Commit Graph
378 Commits
Author SHA1 Message Date
Adam Miller 929f3cb489 Scripts and Beans converted to freemarker 2013-03-11 17:05:17 -07:00
Adam Miller 783d48b49c Removing old commented out code 2013-03-11 16:58:28 -07:00
Adam Miller a32f00c2b3 Finishing /engine and /engine/job/{job} 2013-03-11 16:58:28 -07:00
Adam Miller e4ad60dda6 Beginning conversion to freemarker templates. /engine updated 2013-03-11 16:58:28 -07:00
Travis Wellman 56e6a0d504 ScriptResource actually uses the BeanLookupBindings 2013-01-24 10:56:16 -08:00
Travis Wellman 1883d81bc8 bindings on top of beans instead of vice versa 2013-01-23 18:27:40 -08:00
Travis Wellman 91aeb6d0bf disable bean lookup by default
bean lookup may be enabled by setting a variable named beanBindings to
true in the Bindings or in the script.
2013-01-11 12:09:54 -08:00
Travis Wellman 03914a246c BeanLookupBindings for simpler script access to beans
The new class BeanLookupBindings allows scripts to skip getBean calls.
Lines like 'beanname = appCtx.getBean("beanname")' can be left out.
Past scripts remain compatible.
This change effects action directory scripts and rest console scripts.
2013-01-10 18:54:33 -08:00
Noah Levitt 3a65fa25d7 Working on refactoring some stuff into archive-commons 2012-12-18 14:46:52 -08:00
Noah Levitt 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics
* CrawledBytesHistotable.java
    accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
    testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
2012-09-27 16:56:14 -07:00
Noah Levitt e711ff9506 * JobRelatedResource.java
fix code formatting from 01be6d4b
2012-09-04 18:03:38 -07:00
Adam Miller 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans
* JobRelatedResource.java
    Added encoding to URL generation in makePresentableMapFor().
    Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
2012-08-31 15:07:31 -07:00
Adam Miller 52e70a10ef Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans"
* BeanBrowseResource.java
	replaced "script" string in root node with "beans"
2012-08-30 15:35:27 -07:00
Travis Wellman c5d740a6d4 remove some dead or redundant code
* ArchiveUtils
    sb.length() was tested twice and int i was never used
* PrefixFinder
    opCount is never used
* BdbMultipleWorkQueues
    tries is never used
2012-07-11 16:57:13 -07:00
Travis Wellman c1a0525107 suppress unused warnings for serialVersionUid
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
2012-07-10 14:12:33 -07:00
Travis Wellman 6626b44286 generics warnings fixes
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
2012-06-21 18:23:06 -07:00
Noah Levitt 35378df2fa Merge branch 'master' into no-package-scope 2012-06-05 17:58:40 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Noah Levitt e965421281 * WorkQueueFrontier.java
remove unused variable
2012-06-05 16:26:00 -07:00
Travis Wellman 07a2fc65ea use methods to specify which report rather than string constants
* MultiReporter
    deleted. direct method calls are a better way to specify the kind of report desired
    getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
    writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
    MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
    call the appropriate report method instead of passing a string constant
2012-06-01 17:57:25 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt 1015c5fa00 Fix HER-1973 Regression in queue budget overrides
* WorkQueueFrontier.java
    findEligibleURI() - apply sheet overrides before refreshing queue budgets
2012-05-15 17:35:49 -07:00
Alex Osborne f19a72efaf HER-2001: Use the CodeMirror editor for crawl config and script console
This enables syntax-highlighting, line-numbering and auto-indent for the crawl
configuration editor and the script console (for BeanShell, JavaScript and
Groovy).

* BaseResource.java
    add a more generic getStaticRef() in addition to getStylesheetRef()
* EditRepresentation.java
    wrap textarea with CodeMirror in purexml mode.
    disable the "save changes" button until config is edited.
    dynamically resize CodeMirror edit element to fit window height.
* ScriptResource.java
    wrap textarea with CodeMirror.
    switch CodeMirror language modes depending on selected script engine.
* resources/.../restlet/codemirror/
    include just the files from codemirror-2.23.zip necessary for the above
    plus the CodeMirror LICENSE file (MIT)
    plus a README linking to the codemirror website
2012-03-31 18:35:06 +11:00
Noah Levitt 9626b8454a * EditRepresentation.java
make edit page (for cxml mainly) prettier with big textarea (see Kenji's comment on HER-1993 dated 2012-03-27 18:03)
2012-03-30 11:44:28 -07:00
Noah Levitt ea42c0c653 * EngineResource.java
acceptRepresentation() - "I'm sure" parameter is now "im_sure"
2012-03-19 17:30:18 -07:00
Noah Levitt f7505e6673 More HTML cleanup. Html5 doctype declaration, get rid of <br> and
xml-style "/>" tag endings, other cleanup to make it validate.
2012-03-15 18:47:18 -07:00
Noah Levitt 7cde3b4f17 Merge branch 'master' of github.com:internetarchive/heritrix3 into html-cleanup 2012-03-15 15:59:10 -07:00
Kenji Nagahashi 64f24d4c66 HER-1993: In JobResource, write error message instead of throwing RuntimeException when it failed to read
job log or crawl log.
2012-03-09 16:48:11 -08:00
Kenji Nagahashi f05287c8ca restored deleted space after header "Crawl Log" 2012-03-09 16:29:52 -08:00
Kenji Nagahashi adc82c1c6f Second round of HTML cleanup:
- added missing <html> and <body>
- fixed non-working CSS link, added CSS link to all pages
- replaced layout by <br> and &nbsp; with structural tags with CSS
- removed superfluous output, notably in BeanBrowseResource
- <code>-ify variable names in ScriptResource
- added special handling of NaN in doubleToString()
2012-03-03 00:46:56 -08:00
Kenji Nagahashi 46c335c8a3 BeanBrowseResource: fixed malformed HTML. 2012-02-29 13:22:33 -08:00
Noah Levitt 1d05df7fbe * StatisticsTracker.java
writeReportFile() - reduce log level from warning to info when reusing existing report (normal after crawl is finished)
2012-02-29 11:02:39 -08:00
Noah Levitt 7af7cbf18c Merge branch 'master' of github.com:internetarchive/heritrix3 2012-02-10 15:51:15 -08:00
Noah Levitt db801b3e0f Fix problems handling crawl job paths with characters that require url-encoding, since we sometimes refer to them with file: urls
* PathSharingContext.java, CrawlJob.java
    use java.net.URI to convert paths to and from file: urls
2012-02-10 15:48:09 -08:00
Noah Levitt d83ba3070b Merge branch 'master' of https://github.com/travisfw/heritrix3 2012-02-09 16:43:19 -08:00
Noah Levitt cfac7ead91 Fix for HER-1985 H3: SurtPrefixDecideRule forgets learned/seed-derived/seed-directive SURT prefixes in checkpoint-resume
* SurtPrefixedDecideRule.java
    make class implement Checkpointable, save surt prefixes to json on checkpoint and load them on recover
* profile-crawler-beans.cxml
    make main SurtPrefixedDecideRule a top-level bean so that it can be checkpointed
* Checkpoint.java, CheckpointService.java
    add some FINE level logging during checkpointing and recovery
2012-02-09 10:05:07 -08:00
Travis Wellman e90fc2965b Sort script engines alphabetically so that they appear consistently.
Note that the name of the engine may be different than the name of the language
it parses. A good example is the Rhino script engine that parses javascript.
2012-01-26 00:37:19 -08:00
Noah Levitt 4e05c89156 Merge branch 'master' of github.com:internetarchive/heritrix3 2012-01-19 13:41:14 -08:00
Noah Levitt 803f9ca979 HER-1984 save script state - implement by adding a map to the application context for arbitrary use, and make sure the app context is available in all scripting environments
* PathSharingContext.java
    new member variable ConcurrentHashMap data and accessor getData()
* ScriptedProcessor.java, ScriptedDecideRule.java
    make appCtx available to scripts; also remove unused member sharedMap
* ActionDirectory.java
    formatting fix
2012-01-19 13:31:57 -08:00
gojomo 20502f54d5 add processErrorOutlinks setting to CandidatesProcessor
default false is old behavior, skip candidates-handling of outlinks from response codes <200 and >=400
if true, these outlinks will be treated the same as others (get scope-tested and enqueued)
2012-01-18 01:23:26 -08:00
gojomo 4829160c84 [HER-1981] H3: overlay setting of alternate queue-precedence (via
BaseQueuePrecedencePolicy.basePrecedence) doesn't stick
(CrawlURI) don't null overlayNames in processingCleanup
(WorkQueueFrontier) move queue precedence recalc to only in handleQueue,
not all (post-wake) reenqueues
2012-01-05 23:01:54 -08:00
Noah Levitt abda8a01fb HER-1963 Unvisited hosts being written twice into H3-generated hosts-report.txt
* HostsReport.java
    write() - remove obsolete code described by this no longer correct comment - "StatisticsTracker doesn't know of zero-completion hosts; so supplement report with those entries from host cache" - StatisticsTracker does know of zero-completion hosts
2011-11-02 13:01:40 -07:00
Noah Levitt 995133dc40 Fix for HER-1962 NPE from missing sheet
* SheetOverlaysManager.java
    getOverlayMap(String) - return null if sheet missing instead of triggering npe
* KeyedProperties.java
    get(String) - check for null return value from getOverlayMap() and log warning
2011-10-27 10:40:30 -07:00
nlevitt 9fb7b0f5ae Fix for HER-1958 race condition on frontier inactive queues
* WorkQueueFrontier.java
    activateInactiveQueue(), deactivateQueue() - synchronize around updates of inactiveQueuesByPrecedence and highestPrecedenceWaiting
2011-10-13 01:47:25 +00:00
nlevitt 189a557af8 Tweak of last checkin
* Heritrix.java
    instanceMain() - move logging of java vendor/version ahead of version check, since the information could be particularly helpful when version check doesn't pass
2011-10-12 18:32:14 +00:00
nlevitt 61240573b3 * Heritrix.java
instanceMain() - log java vendor, version at startup
2011-10-12 18:20:27 +00:00
nlevitt 10b3686ba8 Fix race condition which can happen with multiple concurrent "launch"es - can end up with duplicate ToePools like before r7268, probably other problems
* CrawlJob.java
    launch() - synchronize this method - sort of a blunt fix but probably safest and shouldn't affect performance
2011-10-07 23:18:43 +00:00
nlevitt 19e94f07d1 Fix race condition encountered by Travis and Adam where two ToeThread pools are created, one on "launch" and one on "unpause"
* CrawlController.java
    requestCrawlResume() - do not create ToePool (creation here appeared to be inherited cruft), paused crawl must already have toe pool
2011-10-07 01:54:29 +00:00
nlevitt b4e6f43052 Unset script engine variables after scripts complete, allowing memory to be freed up sooner - see also r7252
* ScriptResource.java, ActionDirectory.java, ScriptedDecideRule.java
2011-10-06 18:01:01 +00:00
nlevitt a76f9818f9 Fix HER-1955 Some annoying interaction between new creation of latest link and disk full java bean
* DiskSpaceMonitor.java
    checkAvailableSpace() - really do ignore non-existent paths, as comment and log message claim is done
2011-09-30 20:07:15 +00:00