* CoreAttributeConstants.java
new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges
* CrawlURI.java
add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges
* Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java
add httpAuthChallenges argument to populate()
* HttpAuthenticationCredential.java
populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header
* CrawlServer.java
cache httpAuthChallenges
* FetchHTTP.java
getAuthScheme() - cache auth challenges with CrawlURI
configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate()
promoteCredentials() - cache auth challenges with the CrawlServer
* FetchHTTP.java
configureHttp() - enable preemptive authentication parameter
innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want)
* TooManyPathSegmentsDecideRule
Sometimes the path is null, throwing a NPE when the loop in evaluate(CrawlURI)
is entered. In such a case, revert to previous behavior.
* TooManyPathSegmentsDecideRule
evaluate(CrawlURI) previously counted slashes in the full URI.
Now it only counts slashes in the path of the URI, meaning a URI
with slashes in the query string will not be counted as having more
path segments than it has in its path.
* LaxURI.java
parseUriReference() - fix logic testing if uri string is a path to respect scheme-less uris even if they contain colons
* UURIFactoryTest.java
add test for scheme-less relative uri (with colons)
This reverts commit 6dae8f9a17.
Revert to fix this problem: ... Failed to convert property value of type 'java.lang.String' to required type 'org.archive.spring.ConfigPath' for property 'path'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [java.lang.String] to required type [org.archive.spring.ConfigPath] for property 'path': no matching editors or conversion strategy found
* WARCWriterProcessor.java
writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987)
* UriUtils.java
isLikelyFalsePositive() - do not flag urls starting with ../ as likely false positives
* UriUtilsTest.java
remove test url that is now rejected
* MultiReporter
deleted. direct method calls are a better way to specify the kind of report desired
getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
call the appropriate report method instead of passing a string constant
* MultiReporter.java
deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
remove MultiReporter#getReports() and reportTo(String, PrintWriter)
these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.
Now getEngine does what it says it does.
Noah Levitt