- added missing <html> and <body>
- fixed non-working CSS link, added CSS link to all pages
- replaced layout by <br> and with structural tags with CSS
- removed superfluous output, notably in BeanBrowseResource
- <code>-ify variable names in ScriptResource
- added special handling of NaN in doubleToString()
* SurtPrefixedDecideRule.java
make class implement Checkpointable, save surt prefixes to json on checkpoint and load them on recover
* profile-crawler-beans.cxml
make main SurtPrefixedDecideRule a top-level bean so that it can be checkpointed
* Checkpoint.java, CheckpointService.java
add some FINE level logging during checkpointing and recovery
Note that the name of the engine may be different than the name of the language
it parses. A good example is the Rhino script engine that parses javascript.
* PathSharingContext.java
new member variable ConcurrentHashMap data and accessor getData()
* ScriptedProcessor.java, ScriptedDecideRule.java
make appCtx available to scripts; also remove unused member sharedMap
* ActionDirectory.java
formatting fix
default false is old behavior, skip candidates-handling of outlinks from response codes <200 and >=400
if true, these outlinks will be treated the same as others (get scope-tested and enqueued)
* FetchDNS.java
recordDNS() - use local variable for dummy read buffer, instead of member variable that was shared among threads, since RecordingInputStream.read() used the passed buffer as a work area
(using only path)
(RobotsPolicyTest) added for case that failed pre-fix
(RobotsTxtTest) added case similar to problem (but was already OK at
RobotsTxt level)
(RobotsPolicy, *RobotsPolicy) rename method to reflect new behavior
BaseQueuePrecedencePolicy.basePrecedence) doesn't stick
(CrawlURI) don't null overlayNames in processingCleanup
(WorkQueueFrontier) move queue precedence recalc to only in handleQueue,
not all (post-wake) reenqueues
* HostsReport.java
write() - remove obsolete code described by this no longer correct comment - "StatisticsTracker doesn't know of zero-completion hosts; so supplement report with those entries from host cache" - StatisticsTracker does know of zero-completion hosts
* FetchWhois.java
fetch() - record server ip address in CrawlURI attributes list
* CoreAttributeConstants.java
new string constant A_WHOIS_SERVER_IP, key for whois server ip address CrawlURI attribute
* WARCWriterProcessor.java
writeWhoisRecords() - include warc header WARC-IP-Address
* SheetOverlaysManager.java
getOverlayMap(String) - return null if sheet missing instead of triggering npe
* KeyedProperties.java
get(String) - check for null return value from getOverlayMap() and log warning
when starting in background, make better effort to write pid to $HERITRIX_HOME/heritrix.pid on successful start, plus this change:
- echo -n "`date` Starting heritrix"
+ echo -n "`date` Heritrix starting (pid $HERITRIX_PID)"
Travis Wellman