Commit Graph
465 Commits
Author SHA1 Message Date
Adam Miller ef7fd37a0a BeanBrowseResource.java
Removing obsolete method: makePresentableMap()
2013-03-11 17:07:15 -07:00
Adam Miller d26a712211 Fixed bad merge conflict 2013-03-11 17:07:15 -07:00
Adam Miller 60d00e3ef3 Converted data models to be usable for conversion to HTML and XML. Updated Models and freemarker templates to match XML naming 2013-03-11 17:05:18 -07:00
Adam Miller 50e49b6399 Removed commented out code. 2013-03-11 17:05:18 -07:00
Adam Miller 7426c78d73 Removed temporary template loading from file system.
Removed unused method: getAvailableActions
2013-03-11 17:05:18 -07:00
Adam Miller d545899fd2 Removed temporary template loading from file system 2013-03-11 17:05:18 -07:00
Adam Miller 929f3cb489 Scripts and Beans converted to freemarker 2013-03-11 17:05:17 -07:00
Adam Miller 783d48b49c Removing old commented out code 2013-03-11 16:58:28 -07:00
Adam Miller a32f00c2b3 Finishing /engine and /engine/job/{job} 2013-03-11 16:58:28 -07:00
Adam Miller e4ad60dda6 Beginning conversion to freemarker templates. /engine updated 2013-03-11 16:58:28 -07:00
Noah Levitt fb758e53e3 Avoid pulling in hadoop-core.jar and its dependencies; add plugin versions to get rid of warnings 2013-02-22 15:43:15 -08:00
Travis Wellman 56e6a0d504 ScriptResource actually uses the BeanLookupBindings 2013-01-24 10:56:16 -08:00
Travis Wellman 1883d81bc8 bindings on top of beans instead of vice versa 2013-01-23 18:27:40 -08:00
Travis Wellman 91aeb6d0bf disable bean lookup by default
bean lookup may be enabled by setting a variable named beanBindings to
true in the Bindings or in the script.
2013-01-11 12:09:54 -08:00
Travis Wellman 03914a246c BeanLookupBindings for simpler script access to beans
The new class BeanLookupBindings allows scripts to skip getBean calls.
Lines like 'beanname = appCtx.getBean("beanname")' can be left out.
Past scripts remain compatible.
This change effects action directory scripts and rest console scripts.
2013-01-10 18:54:33 -08:00
Noah Levitt 3a65fa25d7 Working on refactoring some stuff into archive-commons 2012-12-18 14:46:52 -08:00
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
2012-09-27 16:56:14 -07:00
Noah Levitt f9703fb5ef * engine/pom.xml
remove incorrect, unnecessary dependency
2012-09-26 14:55:03 -07:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
2012-09-26 14:41:49 -07:00
Noah Levitt e711ff9506 * JobRelatedResource.java
fix code formatting from 01be6d4b
2012-09-04 18:03:38 -07:00
Adam Miller 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans
* JobRelatedResource.java
    Added encoding to URL generation in makePresentableMapFor().
    Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
2012-08-31 15:07:31 -07:00
Adam Miller 52e70a10ef Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans"
* BeanBrowseResource.java
	replaced "script" string in root node with "beans"
2012-08-30 15:35:27 -07:00
Travis Wellman c5d740a6d4 remove some dead or redundant code
* ArchiveUtils
    sb.length() was tested twice and int i was never used
* PrefixFinder
    opCount is never used
* BdbMultipleWorkQueues
    tries is never used
2012-07-11 16:57:13 -07:00
Travis Wellman c1a0525107 suppress unused warnings for serialVersionUid
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
2012-07-10 14:12:33 -07:00
Travis Wellman 6626b44286 generics warnings fixes
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
2012-06-21 18:23:06 -07:00
Noah Levitt 35378df2fa Merge branch 'master' into no-package-scope 2012-06-05 17:58:40 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Noah Levitt e965421281 * WorkQueueFrontier.java
remove unused variable
2012-06-05 16:26:00 -07:00
Travis Wellman 07a2fc65ea use methods to specify which report rather than string constants
* MultiReporter
    deleted. direct method calls are a better way to specify the kind of report desired
    getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
    writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
    MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
    call the appropriate report method instead of passing a string constant
2012-06-01 17:57:25 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt 1015c5fa00 Fix HER-1973 Regression in queue budget overrides
* WorkQueueFrontier.java
    findEligibleURI() - apply sheet overrides before refreshing queue budgets
2012-05-15 17:35:49 -07:00
Noah Levitt 9fffa8fc53 **/pom.xml
bump version to 3.1.2-SNAPSHOT
2012-05-01 18:17:33 -07:00
Noah Levitt 79c40346fc **/pom.xml
* bump version number to 3.1.1 for release
2012-05-01 18:01:06 -07:00
Gordon Mohr e4f12bb20f Merge pull request #6 from ato/codemirror
HER-2001: Use the CodeMirror editor for crawl config and script console
    (Thanks Alex!)
2012-04-25 15:04:10 -07:00
Noah Levitt 42e759c01d List correct defaults for BdbModule cacheSize and cachePercentage with an explanatory comment. 2012-04-25 11:10:56 -07:00
Noah Levitt da28258f90 Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948) 2012-04-19 18:16:53 -07:00
Noah Levitt 4a64c6ba59 Revert "Altered FetchStats to also keep track of the number of URIs that have"
This reverts commit 4c1740ce1e.
2012-04-18 15:53:49 -07:00
Noah Levitt 9551c774af Merge branch 'master' into http-status-deciderule 2012-04-18 15:47:24 -07:00
Alex Osborne f19a72efaf HER-2001: Use the CodeMirror editor for crawl config and script console
This enables syntax-highlighting, line-numbering and auto-indent for the crawl
configuration editor and the script console (for BeanShell, JavaScript and
Groovy).

* BaseResource.java
    add a more generic getStaticRef() in addition to getStylesheetRef()
* EditRepresentation.java
    wrap textarea with CodeMirror in purexml mode.
    disable the "save changes" button until config is edited.
    dynamically resize CodeMirror edit element to fit window height.
* ScriptResource.java
    wrap textarea with CodeMirror.
    switch CodeMirror language modes depending on selected script engine.
* resources/.../restlet/codemirror/
    include just the files from codemirror-2.23.zip necessary for the above
    plus the CodeMirror LICENSE file (MIT)
    plus a README linking to the codemirror website
2012-03-31 18:35:06 +11:00
Noah Levitt 9626b8454a * EditRepresentation.java
make edit page (for cxml mainly) prettier with big textarea (see Kenji's comment on HER-1993 dated 2012-03-27 18:03)
2012-03-30 11:44:28 -07:00
Noah Levitt ea42c0c653 * EngineResource.java
acceptRepresentation() - "I'm sure" parameter is now "im_sure"
2012-03-19 17:30:18 -07:00
Noah Levitt f7505e6673 More HTML cleanup. Html5 doctype declaration, get rid of <br> and
xml-style "/>" tag endings, other cleanup to make it validate.
2012-03-15 18:47:18 -07:00
Noah Levitt 7cde3b4f17 Merge branch 'master' of github.com:internetarchive/heritrix3 into html-cleanup 2012-03-15 15:59:10 -07:00
Kenji Nagahashi 64f24d4c66 HER-1993: In JobResource, write error message instead of throwing RuntimeException when it failed to read
job log or crawl log.
2012-03-09 16:48:11 -08:00
Kenji Nagahashi f05287c8ca restored deleted space after header "Crawl Log" 2012-03-09 16:29:52 -08:00
Kenji Nagahashi adc82c1c6f Second round of HTML cleanup:
- added missing <html> and <body>
- fixed non-working CSS link, added CSS link to all pages
- replaced layout by <br> and &nbsp; with structural tags with CSS
- removed superfluous output, notably in BeanBrowseResource
- <code>-ify variable names in ScriptResource
- added special handling of NaN in doubleToString()
2012-03-03 00:46:56 -08:00
Kenji Nagahashi 46c335c8a3 BeanBrowseResource: fixed malformed HTML. 2012-02-29 13:22:33 -08:00
Noah Levitt 1d05df7fbe * StatisticsTracker.java
writeReportFile() - reduce log level from warning to info when reusing existing report (normal after crawl is finished)
2012-02-29 11:02:39 -08:00
Noah Levitt 7af7cbf18c Merge branch 'master' of github.com:internetarchive/heritrix3 2012-02-10 15:51:15 -08:00
Noah Levitt db801b3e0f Fix problems handling crawl job paths with characters that require url-encoding, since we sometimes refer to them with file: urls
* PathSharingContext.java, CrawlJob.java
    use java.net.URI to convert paths to and from file: urls
2012-02-10 15:48:09 -08:00