* CrawlServer, Robotstxt
move autoregister details to Robotstxt, RobotsDirectives
* RobotsDirectives
use ReferenceFieldSerializer so that repeated instances of RobotsDirectives in same context are replaced with backrefs
* RobotstxtTest
(from Kenji) unit test for instance multiplication
* IdenticalDigestDecideRule.java
fix javadoc typo
* CoreAttributeConstants.java
"history-good-to-store" is history
* RecrawlAttributeConstants.java
new "write-tag" constant
* PersistProcessor.java
remove defunct "history-good-to-store" logic; nadd new option onlyStoreIfWriteTagPresent, and update shouldStore() to respect that option
* WriterPoolProcessor.java
remove defunct "history-good-to-store" logic; new method copyForwardWriteTagIfDupe(); override innerRejectProcess() to call copyForwardWriteTagIfDupe()
* ARCWriterProcessor.java, WARCWriterProcessor.java
remove defunct "history-good-to-store" logic; add writeTag to history on write; call copyForwardWriteTagIfDupe() on skipIdenticalDigest
contributed by Adam Wilmer
* FetchHTTP.java
properties, setup for http proxy user/password
* profile-crawler-beans.cxml
commented-out example settings for proxy auth properties
* ActionDirectory.java, CrawlerLoggerModule.java, StatisticsTracker.java, SurtPrefixedDecideRule.java, WriterPoolProcessor.java, profile-crawler-beans.cxml
use camelcase ${launchId}
* CheckpointService.java, JobResource.java
rename getAvailableCheckpointDirectories() to findAvailableCheckpointDirectories() so that it's not treated as a bean property
* WriterPoolProcessor.java, WriterPoolSettingsData.java, WriterPoolMember.java
rename getOutputDirs() to calcOutputDirs() so that it's not treated as a bean property
* ConfigPath.java
let ConfigPathConfigurer to do interpolation of ${launchId}
* ConfigFile.java
let ConfigPathConfigurer do the snapshotting of config file
* CrawlJob.java, PathSharingContext.java
remove launchDir initialization out of CrawlJob into PathSharingContext
* ConfigPathConfigurer.java
- remove special handling of WriterPoolProcessor store paths; instead, look in beans for ConfigPaths within Iterables
- have each ConfigPath hold reference to this ConfigPathConfigurer to use for interpolating ${launchId} and snapshotting config files
* Recorder
object to unsupported content-encodings when first set
* FetchHTTP
note as annotation unsupported content-encodings
* Link
not as annotation when base URI is used for absent via
* ExtractorHTML
downgrade char-sequence reading problem (often a chunking problem) to WARNING from SEVERE
* UriProcessingFormatter, Preformatter, GenerationFileHandler
improve preformat-outside-synchronized optimization so that the LogRecord/CrawlURI doesn't linger until next displaces it
* CrawlURI
(processingCleanup) null more of last-processing-run collected values
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java
change type of storePaths to List<ConfigPath> and handle appropriately
* ConfigPathConfigurer.java
fixupPaths() - old code did not touch WriterPoolProcessor storePaths, since they're deeply nested inside the bean, but they need to be remembered for later interpolation of ${launch-id}, so add special handling
* HardLinker.java
renamed FilesystemLinkMaker.java
* FilesystemLinkMaker.java
add support for symbolic links
* CLibrary.java
new method symlink()
* BdbModule.java
use new class name FilesystemLinkMaker
* CrawlJob.java
at crawl launch, create launch directory launch-{timestamp17}, copy cxml there, symlink "current" to launch dir, inform ConfigPaths
* ConfigPath.java
interpolate ${launch-id} in configured paths
* ConfigFile.java
obtainReader() - snapshot config files to launch dir when they are read
* ActionDirectory.java
default doneDir now ${launch-id}/actions-done
actOn() - symlink from old style done dir action/done to done files
* SurtPrefixedDecideRule.java
default surtsDumpFile now ${launch-id}/surts.dump
pathsFixedUp() - this gets called at build time, but we don't want anything written to disk until launch time, so remove call to dumpSurtPrefixSet() here
* CrawlerLoggerModule.java
default logs dir now ${launch-id}/logs
* StatisticsTracker.java
default reports dir now ${launch-id}/reports
* WriterPoolProcessor.java
default writer base path now ${launch-id}
* profile-crawler-beans.cxml
update to reflect new default paths under launch dirs
* PropertyUtils.java
fix javadoc typo
* pom.xml, .classpath
update references to necessary spring-3.0.5 packages
* SeedModule
merge rather than replace event listeners, so that (now later) autowiring doesn't clobber the self-insertion done by anonymous (non-top-level) SurtPrefixedDecideRule bean
* HeritrixLifecycleProcessor, PathScharingContext
use this new non-default LifecycleProcessor to avoid new Spring3 behavior of auto-start()ing context on refresh(build)
* CrawlController
example of using @Value annotation to set default value: will offer benefits for auto-discovery of defaults for configuration interface, or enforcing maximally-explicit configurations (see [HER-1897])
* profile-crawler-beans.cxml, selftest-crawler-beans.cxml
update templates with spring3 preamble/boilerplate
* TextSeedModule
refactor announceSeeds to occur in background thread if non-default blockAwaitingSeedLines value is set
signal CountDownLatch on each line, allowing calling thread to proceed at right count
* profile-crawler-beans.cxml
commented-out blockAwaitingSeedLines default settings (-1, meaning wait for all seed lines)
* ReplayCharSequence, GenericReplayCharSequence
use Charset instance rather than String-name
* Recorder
use Charset instance ratehr than String-name
(getContentReplayCharSequence) avoid reusing cached ReplayCharSequence when encoding-in-use has changed since it was created
(getContentReplayPrefixString) allow requested specific charset-interpretation
* ExtractorHTML, ExtractorXML
work with Charset instances
double-check that in-content-declarations are self-consistent before using
recycle matcher instances
change minor charset problems/decisions to a crawl.log annotation
was archived
* CoreAttributeConstants.java
new flag A_HISTORY_GOOD_TO_STORE w/ javadoc
* WriterPoolProcessor.java
shouldWrite() - set A_HISTORY_GOOD_TO_STORE on CrawlURI if we decide to skip writing because of previous fetch with identical digest
* WARCWriterProcessor.java
write() - set A_HISTORY_GOOD_TO_STORE on CrawlURI on successful write to warc
* PersistProcessor.java
shouldStore() - change to return true if A_HISTORY_GOOD_TO_STORE is set on the CrawlURI
* CrawlURI.java
new field JSONObject extraInfo, methods getExtraInfo() and addExtraInfo()
* WARCWriterProcessor.java
write() - on success, add warcFilename to CrawlURI extraInfo
* CrawlerLoggerModule.java
new config option logExtraInfo, default false
* UriProcessingFormatter.java
new field logExtraInfo
format() - include CrawlURI extraInfo if logExtraInfo is enabled
also include "-" if CrawlURI has no annotations, since this is no longer the last field on the line
* profile-crawler-beans.cxml
<!-- <property name="logExtraInfo" value="false" /> -->
* NonFatalErrorFormatter.java, RuntimeErrorFormatter.java
new constructor to handle logExtraInfo since these subclass UriProcessingFormatter
* FetchHTTP.java
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
by default - this is what my firefox 4.0 for mac sends
* profile-crawler-beans.cxml
commented-out default value
* ExtractorXML.java
if charset not spec'd in http header look for <?xml encoding=""?>
* ExtractorHTML.java
lookForEncodingInContent() -
1. look for <meta http-equiv="content-type"...>
2. if not found then look for <meta charset="">
3. if not found then <?xml encoding=""...?>
* Recorder.java
setCharacterEncoding() - If new encoding is different from old encoding,
close replayCharSequence and set to null, which will trigger recreation on
next retrieval.
* UriUtils.java
NAIVE_LIKELY_URI_PATTERN - revert change and unpublicize
NAIVE_URI_EXCEPTIONS - add some mimetype strings that came up in testing
isLikelyFalsePositive() - unpublicize
* ExtractorXML.java
XML_URI_EXTRACTOR - do not use NAIVE_LIKELY_URI_PATTERN
shouldExtract() - check for mimetype application/vnd.openxmlformats which is not xml
processXml() - use UriUtils.isLikelyUri()
starts with <?xml..., a@href links are treated as embeds
* ExtractorXML.java
shouldExtract() - return true if content starts with "<?xml" only if it
does not also contain "<!doctype html" or "<html" early in the content
* ExtractorXML.java
instead of considering only strings that start with http(s):, consider all
strings that match UriUtils.NAIVE_LIKELY_URI_PATTERN (and, as before,
constitute the entirety of the xml tag content or attribute value)
* UriUtils.java
- NAIVE_LIKELY_URI_PATTERN - add quotes to the excluded characters so it
does not eat the closing quote when matching xml attribute values, and
make visibility public for use in ExtractorXML
- isLikelyUri() - refactor false positive check into new method
isLikelyFalsePositive() so that it can be used in ExtractorXML avoiding
redundant check against NAIVE_LIKELY_URI_PATTERN
* RobotsDirectives.java
- use plain ConcurrentSkipListSet<String> instead of PrefixSet, to
maintain complete list of prefixes, so we can know the longest match
- allows(String) - return true if longest matching Allow prefix is longer
than or equal to longest matching Disallow prefix
* RobotstxtTest.java
flip expected result of test of generic Allow against specific Disallow
[HER-728] Offer replay stream that has been un-chunked (whether because response was HTTP/1.1 or used chunked in HTTP/1.0 against spec)
[HER-1876] Offer HTTP/1.1 option - for chunked transfer-encoding (but not persistent connections)
* (all)
update to use new (decoded-as-necessary) content streams/CharSequences
[HER-728] Offer replay stream that has been un-chunked (whether because response was HTTP/1.1 or used chunked in HTTP/1.0 against spec)
[HER-1876] Offer HTTP/1.1 option - for chunked transfer-encoding (but not persistent connections)
* FetchHTTP
add 'acceptCompression' and 'useHTTP11' properties, both default false
* Recorder
track whether recorded-input is transfer-encoded (chunked) or content-encoded (gzip etc)
offer alternate replay streams for
(1) raw 'messageBody';
(2) entity (un-chunked if necessary)
(3) content (decompressed if necessary)
always content & GenericReplayCharSequence for CharSequence replays
* GenericReplayCharSequence
always use a stream (rather than random-access buffer)
always decode to prefix buffer first, so short content never touches disk no matter the encoding
* InMemoryReplayCharSequence
deleted; 'Generic' now works similarly for small content and anyway random-access for single-byte-encodings is now rarely possible (given deconding streams)
* RecordingInputStream, RecordingOutputStream
adjust for changed stream names, functionality moved to Recorder
* ReplayCharSequence
use Charset instances rather than names
* ReplayInputStream
add convenience constructor (and tmp-file-destroy) for copying any other inputStream into a seekable ReplayInputStream
Invalid property 'timeoutSeconds' of bean class
[org.archive.modules.fetcher.FetchFTP]: Bean property 'timeoutSeconds' is not
writable or has an invalid setter method. Does the parameter type of the setter
match the return type of the getter?"
* FetchFTP.java
change setTimeoutSeconds(Integer) to setTimeoutSeconds(int)
* FetchFTP.java
setSoTimeoutMs(), getSoTimeoutMs()
(init) - set default value of 20000 ms, same as FetchHTTP
fetch() - set various timeouts on FTPClient - effectively sets connect
timeout and socket timeout on control connection, and socket timeout on
data connection (notably, does not set connect timeout on data connection
w/ current commons-net)
that or preloadSource (a filesystem path), rather than try to handle both
possibilities with one string.
* PersistLoadProcessor.java
* PersistProcessor.java
(regression from h1)
* PersistLoadProcessor.java
change preloadSource from ConfigFile to String, allowing the logic in
PersistProcessor.copyPersistSourceToHistoryMap() to decide how to load it
* URIAuthorityBasedQueueAssignmentPolicy.java
whois urls all go in special "whois" queue
* PreconditionEnforcer.java, CrawlURI.java
move markPrerequisite() method from PreconditionEnforcer to CrawlURI
* profile-crawler-beans.cxml
commented-out whois fetcher clause
* FetchStatusCodes.java
new status codes for whois
* ServerCache.java
getHostFor() - return "whois:" for whois urls, following dns: convention
* UURIFactory.java
ugly hack for whois urls
* SchemeNotInSetDecideRule.java
add whois to list of supported schemes
* WriterPoolProcessor.java
shouldWrite() - we should write successful whois fetches
* WARCWriterProcessor.java
write whois records
* FetchWhois.java
the fetcher
* CookieSpecBase
use Guava InternetDomainName public-suffixes to both (1) prevent storing cookie on public-suffix; (2) prevent looking-up cookies for public-suffixes
* BdbCookieStorage
correct checkpoint/relaunch behavior: don't reuse persisted cookies on normal relaunches; do reuse on resume-from-checkpoint
* WARCWriterProcessor
move checkSize into block protected by catch/finally cleanup of errors
* WARCWriterProcessorTest
test to simulate record-write-failure and verify does not cause pool lockup
* WARCWriterTest, WriterPoolMember
comment and warnings cleanup
* Robotstxt, RobotstxtTest
treat wildcardRules specially, so that it is possible to ask for only explicitly-named matches
* MostFavoredRobotsPolicy
improve comment, change masquerade default to true
* FirstNamedRobotsPolicy, FirstNamedRobotsPolicyTest
policy to consider alternate user-agents in order, using the first that has any explicit (non-wildcard) rules declared (or failing that, the original default user-agent)
* TooManyHopsDecideRule
remove inadvertent reversal of sense of comparison, making default scope reject almost everything (and breaking most self-tests)
gojomo