* ConfigPathConfigurer.java
work in two passes: one (BeanPostProcessor) collects all candidates for fixup, two (ApplicationListener) actually performs fixup
suppress fixup of own base path, preventing circulare ref/StackOverflow
* ConfigPath.java
change sense of 'merge' to merge into receiver
* SurtPrefixedDecideRule.java
use ConfigFile, use 'merge' in assignment
* CrawlerLoggerModule.java
use merge in assignment
* CrawlJob.java
(getBeanpathTarget) utility method
* JobResource.java
respect unset ConfigPaths
* ConfigPathEdito.java, ConfigFileEditor.java
better comments, ConfigFile variant
Fix for [HER-1568] ARCReader fails on some newer Alexa ARC files
* ARCConstants.java
added ArcRecordErrors enum for known reading errors
* ARCReaderFactory.java
only check for 'LX' extension in GZIP extra field
* ARCRecord.java
added errors, hasErrors(), and getErrors()
* RecordingOutputStream.java
close ReplayInputStream used to create GenericReplayCharSequence
use 'UTF-8' and not possibly-varying JVm default encoding when none specified
* HopCrossesAssignmentLevelDomainDecideRule.java
port H2 IsCrossTopmostAssignedSurtHopDecideRule to H3 under this new name
* PublicSuffixes.java
update comment, public method for 'Assignment Level Domain'/'Assignment Level SURT' term
* AssignmentLevelSurtQueueAssignmentPolicy
renamed TopmostAssignedSurtQueueAssignmentPolicy
(from nlevitt's H2 commit message:)
* InMemoryReplayCharSequence.java
new, simple ReplayCharSequence that supports any encoding, keeping everything in memory
* GenericReplayCharSequence.java
If the encoding supports random access, memory maps the backing file directly; otherwise decodes to UTF-16 and maps that. Supports the first Integer.MAX_VALUE bytes of the file. Maps up to 64M; moves the map around as necessary for larger files.
* Latin1ByteReplayCharSequence.java
removed, functionality split between InMemoryReplayCharSequence and GenericReplayCharSequence
* ReplayCharSequenceTest.java
xestHugeReplayCharSequence() - uncomment to test a huge replay char sequence
testReplayCharSequenceByteToStringOverflow() - test both UTF-8 and windows-1252
* RecordingOutputStream.java
use the new ReplayCharSequences
* (all, especially CachedBdbMap.java)
pbaclace-contributed patch, ported to H3, plus...
* StatisticsTracker.java
replace LongWrapper with AtomicLong
use non-recursive putIfAbsent-based idiom for get-or-create
* DefaultServerCache.java
clean up host/server get-or-create around non-recursive concurrent idiom
* BdbModule.java
add expectedConcurrency property & je.lock.nLockTables tuning
* TopNSet.java
add synchronization, as calling StatisticsTracker method has lost it
* BeanBrowseResource.java
offers browsing (and limited editting of simple values) of beans in built crawl
* JobRelatedResource.java, ReportGenResource.java
refactor for reuse by other job sub-resources
* JobResource.java
link to job/beans and job/script
* DescriptorUpdater.java
helper for marking-up PropertyDescriptors to aid UI bean access
* CrawlJob.java
don't auto-instantiate context on request
* EngineApplication.java
add uri routes
* FetchHTTP.java, BdbCookieStorage.java, LongWrapper.java WorkQueueFrontier.java, BdbUriUniqFilter.java
improve display when browsed (esp. before launch)
* Sheet.java
make non-abstract; allow for name that might not be beanName
* SheetAssociation.java
define targets as list of names, rather than Sheets themselves
* SheetForDecideRuled.java -> DecideRuledSheetAssociation.java; SheetForSurtPrefixes.java -> SurtPrefixesSheetAssociation.java
make association separate from Sheet -- so same Sheet can be associated more than one place, more than one way
* SheetOverlaysManager.java
match changes to Sheet and SheetAssociations
offer convenience methods for post-construction addition of Sheets, associations
use internal name-to-Sheet mapping, allowing for mid-crawl Sheet additions not in bean factory
* Precedence1SelfTest.java, Precedence4SelfTest.java
update configs for Sheet changes
Fix for [HER-1520] Windows mapped-files cause intermittent ReplayCharSequenceTest.testReplayCharSequenceByteToStringMulti failures, risk other problems
* GenericReplayCharSequence.java
use more diverse filenames, so subsequent decodes don't run into lingering open mapped sections of previous decodes
do delete via new FileUtils.deleteSoonerOrLater utility
* FileUtils.java
(deleteSoonerOrLater) try immediate delete, but if fails, add to a list of files to be retried later
Fix for [HER-1506] ByteReplayCharSequence ignores specified/non-default character encodings, reported by Jean-No?\195?\171l Rivasseau
* RecordingOutputStream.java
use ByteReplayCharSequence only for iso8859-1
* ReplayCharSequenceTest.java
new tests for iso8859-1, windows-1252 and ascii
Fix for [HER-1521] Make the UncompressedARCReader and CompressedARCReader in the ARCReaderFactory as public or protected
* ARCReaderFactory.java, WARCReaderFactory.java
make compressed/uncompressed implementation inner classes public to enable direct reuse
Fix for [HER-1277] speculative javascript link caused 404s for "http%3A" and "www.example.com" like strings
* ExtractorJS.java
perform additional decoding or prepending of "http:" for speculative candidate URIs that appear to need it
* ArchiveUtils.java
add utility set/method with list of all TLDs
Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme
* ExtractorJS.java
(speculativeFixup) maintain origin URI scheme if apparent target host is same
* ExtractorHTMLTest.java, UURIFactoryTest.java
test that relative-URIs with late-position colons aren't interpreted as absolute URIs with long, illegal schemes
* UURIFactory.java
update RFC2396REGEX to require legal scheme
* LaxURI.java
allow illegal candidate scheme to just be interpreted as something else
* UURIFactoryTest.java
test that port-looking-part that's out of integer bounds throws URIException not NumberFormatException
* UURIFactory.java
catch NumberFormatException; allow to surface as URIException
gojomo