* FetchHTTP.java
configureHttp() - enable preemptive authentication parameter
innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want)
* TooManyPathSegmentsDecideRule
Sometimes the path is null, throwing a NPE when the loop in evaluate(CrawlURI)
is entered. In such a case, revert to previous behavior.
* TooManyPathSegmentsDecideRule
evaluate(CrawlURI) previously counted slashes in the full URI.
Now it only counts slashes in the path of the URI, meaning a URI
with slashes in the query string will not be counted as having more
path segments than it has in its path.
* LaxURI.java
parseUriReference() - fix logic testing if uri string is a path to respect scheme-less uris even if they contain colons
* UURIFactoryTest.java
add test for scheme-less relative uri (with colons)
This reverts commit 6dae8f9a17.
Revert to fix this problem: ... Failed to convert property value of type 'java.lang.String' to required type 'org.archive.spring.ConfigPath' for property 'path'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [java.lang.String] to required type [org.archive.spring.ConfigPath] for property 'path': no matching editors or conversion strategy found
* WARCWriterProcessor.java
writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987)
* UriUtils.java
isLikelyFalsePositive() - do not flag urls starting with ../ as likely false positives
* UriUtilsTest.java
remove test url that is now rejected
* MultiReporter
deleted. direct method calls are a better way to specify the kind of report desired
getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
call the appropriate report method instead of passing a string constant
* MultiReporter.java
deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
remove MultiReporter#getReports() and reportTo(String, PrintWriter)
these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.
Now getEngine does what it says it does.
* ResponseContentLengthDecideRule.java
new predicated decide rule that will ACCEPT or REJECT a uri if the content body is within a specified size range
Kenji Nagahashi