* HardLinker.java
makeHardLink() - wrap unix link() and windows CreateHardLink() and choose depending on platform
* BdbModule.java
doCheckpoint() - use HardLinker.makeHardLink()
* TextSeedModule
refactor announceSeeds to occur in background thread if non-default blockAwaitingSeedLines value is set
signal CountDownLatch on each line, allowing calling thread to proceed at right count
* profile-crawler-beans.cxml
commented-out blockAwaitingSeedLines default settings (-1, meaning wait for all seed lines)
* CrawlController
change pauseAtFinish to runWhileEmpty
add EMPTY state
replace isStateRunning() with isActive() (RUNNING or EMPTY)
trigger finish-test off frontier EMPTY rather than PAUSE
in logging say 'running' rather than 'resumed'
* Frontier
add EMPTY state
* AbstractFrontier
handle EMPTY like RUNNING, transition EMPTY<->RUNNING when appropriate
* StatisticsTracker
handle EMPTY analogous to RUNNING
in logging say 'running' rather than 'resumed'
* CrawlJob, DiskSpaceMonitor
replace isStateRunning() with isActive()
* GenericReplayCharSequence.java
decode() - read another character to check for more content, rather than consult BufferedReader.ready(), since the latter sometimes returns false even when there is more to read
* ReplayCharSequence, GenericReplayCharSequence
use Charset instance rather than String-name
* Recorder
use Charset instance ratehr than String-name
(getContentReplayCharSequence) avoid reusing cached ReplayCharSequence when encoding-in-use has changed since it was created
(getContentReplayPrefixString) allow requested specific charset-interpretation
* ExtractorHTML, ExtractorXML
work with Charset instances
double-check that in-content-declarations are self-consistent before using
recycle matcher instances
change minor charset problems/decisions to a crawl.log annotation
was archived
* CoreAttributeConstants.java
new flag A_HISTORY_GOOD_TO_STORE w/ javadoc
* WriterPoolProcessor.java
shouldWrite() - set A_HISTORY_GOOD_TO_STORE on CrawlURI if we decide to skip writing because of previous fetch with identical digest
* WARCWriterProcessor.java
write() - set A_HISTORY_GOOD_TO_STORE on CrawlURI on successful write to warc
* PersistProcessor.java
shouldStore() - change to return true if A_HISTORY_GOOD_TO_STORE is set on the CrawlURI
redirect url sometimes '0 NOTCRAWLED' in seeds-report.txt even when crawled"
* CandidatesProcessor.java
innerProcess() - set force-fetch on outlinks promoted to seeds
* CrawlURI.java
new field JSONObject extraInfo, methods getExtraInfo() and addExtraInfo()
* WARCWriterProcessor.java
write() - on success, add warcFilename to CrawlURI extraInfo
* CrawlerLoggerModule.java
new config option logExtraInfo, default false
* UriProcessingFormatter.java
new field logExtraInfo
format() - include CrawlURI extraInfo if logExtraInfo is enabled
also include "-" if CrawlURI has no annotations, since this is no longer the last field on the line
* profile-crawler-beans.cxml
<!-- <property name="logExtraInfo" value="false" /> -->
* NonFatalErrorFormatter.java, RuntimeErrorFormatter.java
new constructor to handle logExtraInfo since these subclass UriProcessingFormatter
even when crawled, when original seed also has a regular link to the redirect
url
* CandidatesProcessor.java
innerProcess() - present seed outlinks to the frontier ahead of non-seed outlinks, so that seed version of any duplicated outlink is always the one that's crawled
* FetchHTTP.java
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
by default - this is what my firefox 4.0 for mac sends
* profile-crawler-beans.cxml
commented-out default value
* ExtractorXML.java
if charset not spec'd in http header look for <?xml encoding=""?>
* ExtractorHTML.java
lookForEncodingInContent() -
1. look for <meta http-equiv="content-type"...>
2. if not found then look for <meta charset="">
3. if not found then <?xml encoding=""...?>
* Recorder.java
setCharacterEncoding() - If new encoding is different from old encoding,
close replayCharSequence and set to null, which will trigger recreation on
next retrieval.
* UriUtils.java
NAIVE_LIKELY_URI_PATTERN - revert change and unpublicize
NAIVE_URI_EXCEPTIONS - add some mimetype strings that came up in testing
isLikelyFalsePositive() - unpublicize
* ExtractorXML.java
XML_URI_EXTRACTOR - do not use NAIVE_LIKELY_URI_PATTERN
shouldExtract() - check for mimetype application/vnd.openxmlformats which is not xml
processXml() - use UriUtils.isLikelyUri()
starts with <?xml..., a@href links are treated as embeds
* ExtractorXML.java
shouldExtract() - return true if content starts with "<?xml" only if it
does not also contain "<!doctype html" or "<html" early in the content
* ExtractorXML.java
instead of considering only strings that start with http(s):, consider all
strings that match UriUtils.NAIVE_LIKELY_URI_PATTERN (and, as before,
constitute the entirety of the xml tag content or attribute value)
* UriUtils.java
- NAIVE_LIKELY_URI_PATTERN - add quotes to the excluded characters so it
does not eat the closing quote when matching xml attribute values, and
make visibility public for use in ExtractorXML
- isLikelyUri() - refactor false positive check into new method
isLikelyFalsePositive() so that it can be used in ExtractorXML avoiding
redundant check against NAIVE_LIKELY_URI_PATTERN
(specifically, when they're in the query string)
* UURIFactory.java
fixup() - replace \ with / up to the first ? in the url, instead of
anywhere in the url
and update comments
* UURIFactoryTest.java
backslash test
nlevitt