Commit Graph
468 Commits
Author SHA1 Message Date
Adam Miller 0e3a2ec80a CrawlJobModel.java
Adding check for application context
2013-03-11 17:07:15 -07:00
Adam Miller b7b7953dc9 EngineResource.java
Pulling individual values for necessary data from crawl jobs instead of building the full CrawlJobModel
CrawlJobModel.java
	Moving some data from methods into fields so they are exposed in the XML representation
EngineModel.java
	Removing full building of CrawlJobModel. Handling instead in EngineResource
Job.ftl
	Updating fields that relied on CrawlJobModel methods which were moved to fields.
2013-03-11 17:07:15 -07:00
Adam Miller 03f4f381af JobResource.java
Removing temporary template loading. Finishing converting XML and HTML generation to use the same data model
CrawlJobModel.java
	Converting to same Map based reports used by XML API
EngineModel.java
	Converting to same Map based reports used by XML API
Engine.ftl
	Creating template for Map based reports
Job.ftl
	Creating template for Map based reports
2013-03-11 17:07:15 -07:00
Adam Miller ef7fd37a0a BeanBrowseResource.java
Removing obsolete method: makePresentableMap()
2013-03-11 17:07:15 -07:00
Adam Miller d26a712211 Fixed bad merge conflict 2013-03-11 17:07:15 -07:00
Adam Miller 60d00e3ef3 Converted data models to be usable for conversion to HTML and XML. Updated Models and freemarker templates to match XML naming 2013-03-11 17:05:18 -07:00
Adam Miller 50e49b6399 Removed commented out code. 2013-03-11 17:05:18 -07:00
Adam Miller 7426c78d73 Removed temporary template loading from file system.
Removed unused method: getAvailableActions
2013-03-11 17:05:18 -07:00
Adam Miller d545899fd2 Removed temporary template loading from file system 2013-03-11 17:05:18 -07:00
Adam Miller 929f3cb489 Scripts and Beans converted to freemarker 2013-03-11 17:05:17 -07:00
Adam Miller 783d48b49c Removing old commented out code 2013-03-11 16:58:28 -07:00
Adam Miller a32f00c2b3 Finishing /engine and /engine/job/{job} 2013-03-11 16:58:28 -07:00
Adam Miller e4ad60dda6 Beginning conversion to freemarker templates. /engine updated 2013-03-11 16:58:28 -07:00
Noah Levitt fb758e53e3 Avoid pulling in hadoop-core.jar and its dependencies; add plugin versions to get rid of warnings 2013-02-22 15:43:15 -08:00
Travis Wellman 56e6a0d504 ScriptResource actually uses the BeanLookupBindings 2013-01-24 10:56:16 -08:00
Travis Wellman 1883d81bc8 bindings on top of beans instead of vice versa 2013-01-23 18:27:40 -08:00
Travis Wellman 91aeb6d0bf disable bean lookup by default
bean lookup may be enabled by setting a variable named beanBindings to
true in the Bindings or in the script.
2013-01-11 12:09:54 -08:00
Travis Wellman 03914a246c BeanLookupBindings for simpler script access to beans
The new class BeanLookupBindings allows scripts to skip getBean calls.
Lines like 'beanname = appCtx.getBean("beanname")' can be left out.
Past scripts remain compatible.
This change effects action directory scripts and rest console scripts.
2013-01-10 18:54:33 -08:00
Noah Levitt 3a65fa25d7 Working on refactoring some stuff into archive-commons 2012-12-18 14:46:52 -08:00
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
2012-09-27 16:56:14 -07:00
Noah Levitt f9703fb5ef * engine/pom.xml
remove incorrect, unnecessary dependency
2012-09-26 14:55:03 -07:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
2012-09-26 14:41:49 -07:00
Noah Levitt e711ff9506 * JobRelatedResource.java
fix code formatting from 01be6d4b
2012-09-04 18:03:38 -07:00
Adam Miller 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans
* JobRelatedResource.java
    Added encoding to URL generation in makePresentableMapFor().
    Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
2012-08-31 15:07:31 -07:00
Adam Miller 52e70a10ef Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans"
* BeanBrowseResource.java
	replaced "script" string in root node with "beans"
2012-08-30 15:35:27 -07:00
Travis Wellman c5d740a6d4 remove some dead or redundant code
* ArchiveUtils
    sb.length() was tested twice and int i was never used
* PrefixFinder
    opCount is never used
* BdbMultipleWorkQueues
    tries is never used
2012-07-11 16:57:13 -07:00
Travis Wellman c1a0525107 suppress unused warnings for serialVersionUid
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
2012-07-10 14:12:33 -07:00
Travis Wellman 6626b44286 generics warnings fixes
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
2012-06-21 18:23:06 -07:00
Noah Levitt 35378df2fa Merge branch 'master' into no-package-scope 2012-06-05 17:58:40 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Noah Levitt e965421281 * WorkQueueFrontier.java
remove unused variable
2012-06-05 16:26:00 -07:00
Travis Wellman 07a2fc65ea use methods to specify which report rather than string constants
* MultiReporter
    deleted. direct method calls are a better way to specify the kind of report desired
    getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
    writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
    MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
    call the appropriate report method instead of passing a string constant
2012-06-01 17:57:25 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt 1015c5fa00 Fix HER-1973 Regression in queue budget overrides
* WorkQueueFrontier.java
    findEligibleURI() - apply sheet overrides before refreshing queue budgets
2012-05-15 17:35:49 -07:00
Noah Levitt 9fffa8fc53 **/pom.xml
bump version to 3.1.2-SNAPSHOT
2012-05-01 18:17:33 -07:00
Noah Levitt 79c40346fc **/pom.xml
* bump version number to 3.1.1 for release
2012-05-01 18:01:06 -07:00
Gordon Mohr e4f12bb20f Merge pull request #6 from ato/codemirror
HER-2001: Use the CodeMirror editor for crawl config and script console
    (Thanks Alex!)
2012-04-25 15:04:10 -07:00
Noah Levitt 42e759c01d List correct defaults for BdbModule cacheSize and cachePercentage with an explanatory comment. 2012-04-25 11:10:56 -07:00
Noah Levitt da28258f90 Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948) 2012-04-19 18:16:53 -07:00
Noah Levitt 4a64c6ba59 Revert "Altered FetchStats to also keep track of the number of URIs that have"
This reverts commit 4c1740ce1e.
2012-04-18 15:53:49 -07:00
Noah Levitt 9551c774af Merge branch 'master' into http-status-deciderule 2012-04-18 15:47:24 -07:00
Alex Osborne f19a72efaf HER-2001: Use the CodeMirror editor for crawl config and script console
This enables syntax-highlighting, line-numbering and auto-indent for the crawl
configuration editor and the script console (for BeanShell, JavaScript and
Groovy).

* BaseResource.java
    add a more generic getStaticRef() in addition to getStylesheetRef()
* EditRepresentation.java
    wrap textarea with CodeMirror in purexml mode.
    disable the "save changes" button until config is edited.
    dynamically resize CodeMirror edit element to fit window height.
* ScriptResource.java
    wrap textarea with CodeMirror.
    switch CodeMirror language modes depending on selected script engine.
* resources/.../restlet/codemirror/
    include just the files from codemirror-2.23.zip necessary for the above
    plus the CodeMirror LICENSE file (MIT)
    plus a README linking to the codemirror website
2012-03-31 18:35:06 +11:00
Noah Levitt 9626b8454a * EditRepresentation.java
make edit page (for cxml mainly) prettier with big textarea (see Kenji's comment on HER-1993 dated 2012-03-27 18:03)
2012-03-30 11:44:28 -07:00
Noah Levitt ea42c0c653 * EngineResource.java
acceptRepresentation() - "I'm sure" parameter is now "im_sure"
2012-03-19 17:30:18 -07:00
Noah Levitt f7505e6673 More HTML cleanup. Html5 doctype declaration, get rid of <br> and
xml-style "/>" tag endings, other cleanup to make it validate.
2012-03-15 18:47:18 -07:00
Noah Levitt 7cde3b4f17 Merge branch 'master' of github.com:internetarchive/heritrix3 into html-cleanup 2012-03-15 15:59:10 -07:00
Kenji Nagahashi 64f24d4c66 HER-1993: In JobResource, write error message instead of throwing RuntimeException when it failed to read
job log or crawl log.
2012-03-09 16:48:11 -08:00
Kenji Nagahashi f05287c8ca restored deleted space after header "Crawl Log" 2012-03-09 16:29:52 -08:00
Kenji Nagahashi adc82c1c6f Second round of HTML cleanup:
- added missing <html> and <body>
- fixed non-working CSS link, added CSS link to all pages
- replaced layout by <br> and &nbsp; with structural tags with CSS
- removed superfluous output, notably in BeanBrowseResource
- <code>-ify variable names in ScriptResource
- added special handling of NaN in doubleToString()
2012-03-03 00:46:56 -08:00
Kenji Nagahashi 46c335c8a3 BeanBrowseResource: fixed malformed HTML. 2012-02-29 13:22:33 -08:00