* CrawlURI.java
getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates
* Link.java
addRelativeToVia() - only log use of base for via the first time, before the annotation has been added
* CoreAttributeConstants.java
new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges
* CrawlURI.java
add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges
* Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java
add httpAuthChallenges argument to populate()
* HttpAuthenticationCredential.java
populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header
* CrawlServer.java
cache httpAuthChallenges
* FetchHTTP.java
getAuthScheme() - cache auth challenges with CrawlURI
configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate()
promoteCredentials() - cache auth challenges with the CrawlServer
* FetchHTTP.java
configureHttp() - enable preemptive authentication parameter
innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want)
* TooManyPathSegmentsDecideRule
Sometimes the path is null, throwing a NPE when the loop in evaluate(CrawlURI)
is entered. In such a case, revert to previous behavior.
* TooManyPathSegmentsDecideRule
evaluate(CrawlURI) previously counted slashes in the full URI.
Now it only counts slashes in the path of the URI, meaning a URI
with slashes in the query string will not be counted as having more
path segments than it has in its path.
* WARCWriterProcessor.java
writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987)
* MultiReporter.java
deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
remove MultiReporter#getReports() and reportTo(String, PrintWriter)
these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.
Now getEngine does what it says it does.
* ResponseContentLengthDecideRule.java
new predicated decide rule that will ACCEPT or REJECT a uri if the content body is within a specified size range
* SurtPrefixedDecideRule.java
make class implement Checkpointable, save surt prefixes to json on checkpoint and load them on recover
* profile-crawler-beans.cxml
make main SurtPrefixedDecideRule a top-level bean so that it can be checkpointed
* Checkpoint.java, CheckpointService.java
add some FINE level logging during checkpointing and recovery
* PathSharingContext.java
new member variable ConcurrentHashMap data and accessor getData()
* ScriptedProcessor.java, ScriptedDecideRule.java
make appCtx available to scripts; also remove unused member sharedMap
* ActionDirectory.java
formatting fix
* FetchDNS.java
recordDNS() - use local variable for dummy read buffer, instead of member variable that was shared among threads, since RecordingInputStream.read() used the passed buffer as a work area
(using only path)
(RobotsPolicyTest) added for case that failed pre-fix
(RobotsTxtTest) added case similar to problem (but was already OK at
RobotsTxt level)
(RobotsPolicy, *RobotsPolicy) rename method to reflect new behavior
BaseQueuePrecedencePolicy.basePrecedence) doesn't stick
(CrawlURI) don't null overlayNames in processingCleanup
(WorkQueueFrontier) move queue precedence recalc to only in handleQueue,
not all (post-wake) reenqueues
an HTTP status between 200 and 299.
Altered the QuotaEnforcer to enforce quotas on the number of URIs that
have an HTTP status between 200 and 299.
Removed confusingly named variables (SUCCESS, SUCCESS_KB, RESPONSES,
RESPONSES_KB) and replaced their references.
Noah Levitt