Fix for [HER-1568] ARCReader fails on some newer Alexa ARC files
* ARCConstants.java
added ArcRecordErrors enum for known reading errors
* ARCReaderFactory.java
only check for 'LX' extension in GZIP extra field
* ARCRecord.java
added errors, hasErrors(), and getErrors()
* RecordingOutputStream.java
close ReplayInputStream used to create GenericReplayCharSequence
use 'UTF-8' and not possibly-varying JVm default encoding when none specified
* HOWTO-Launch-Heritrix.txt
removed; info mostly about H2 variants
* README.txt
updated wiki link, launch info to match H3
* LICENSE.txt
updated to Apache2
* HopCrossesAssignmentLevelDomainDecideRule.java
port H2 IsCrossTopmostAssignedSurtHopDecideRule to H3 under this new name
* PublicSuffixes.java
update comment, public method for 'Assignment Level Domain'/'Assignment Level SURT' term
* AssignmentLevelSurtQueueAssignmentPolicy
renamed TopmostAssignedSurtQueueAssignmentPolicy
(from nlevitt's H2 commit message:)
* InMemoryReplayCharSequence.java
new, simple ReplayCharSequence that supports any encoding, keeping everything in memory
* GenericReplayCharSequence.java
If the encoding supports random access, memory maps the backing file directly; otherwise decodes to UTF-16 and maps that. Supports the first Integer.MAX_VALUE bytes of the file. Maps up to 64M; moves the map around as necessary for larger files.
* Latin1ByteReplayCharSequence.java
removed, functionality split between InMemoryReplayCharSequence and GenericReplayCharSequence
* ReplayCharSequenceTest.java
xestHugeReplayCharSequence() - uncomment to test a huge replay char sequence
testReplayCharSequenceByteToStringOverflow() - test both UTF-8 and windows-1252
* RecordingOutputStream.java
use the new ReplayCharSequences
* (all, especially CachedBdbMap.java)
pbaclace-contributed patch, ported to H3, plus...
* StatisticsTracker.java
replace LongWrapper with AtomicLong
use non-recursive putIfAbsent-based idiom for get-or-create
* DefaultServerCache.java
clean up host/server get-or-create around non-recursive concurrent idiom
* BdbModule.java
add expectedConcurrency property & je.lock.nLockTables tuning
* TopNSet.java
add synchronization, as calling StatisticsTracker method has lost it
* Heritrix.java
fail-fast with helpful error message if launched with Java < 1.6
* EditRepresentation.java
"save changes" rather than "update content"
* EngineApplication.java
customize restlet error to include back-button and full stack (instead of link to HTTP spec!)
* BeanBrowseResource.java
offers browsing (and limited editting of simple values) of beans in built crawl
* JobRelatedResource.java, ReportGenResource.java
refactor for reuse by other job sub-resources
* JobResource.java
link to job/beans and job/script
* DescriptorUpdater.java
helper for marking-up PropertyDescriptors to aid UI bean access
* CrawlJob.java
don't auto-instantiate context on request
* EngineApplication.java
add uri routes
* FetchHTTP.java, BdbCookieStorage.java, LongWrapper.java WorkQueueFrontier.java, BdbUriUniqFilter.java
improve display when browsed (esp. before launch)
* Sheet.java
make non-abstract; allow for name that might not be beanName
* SheetAssociation.java
define targets as list of names, rather than Sheets themselves
* SheetForDecideRuled.java -> DecideRuledSheetAssociation.java; SheetForSurtPrefixes.java -> SurtPrefixesSheetAssociation.java
make association separate from Sheet -- so same Sheet can be associated more than one place, more than one way
* SheetOverlaysManager.java
match changes to Sheet and SheetAssociations
offer convenience methods for post-construction addition of Sheets, associations
use internal name-to-Sheet mapping, allowing for mid-crawl Sheet additions not in bean factory
* Precedence1SelfTest.java, Precedence4SelfTest.java
update configs for Sheet changes
* BeanShellProcessor.java -> ScriptedProcessor.java; BeanShellDecideRule.java -> ScriptedDecideRule.java
update to use any present JSR-223 scripting engine (requested by free-text name)
update to allow script to come from inline string or file (just as with seeds)
* modules/pom.xml
include groovy and jsr-223-capable JARs
* .classpath
add new JARs to classpath
* BeanShellProcessorTest.java -> ScriptedProcessorTest.java
rename (empty) test class
* DefaultProcessorURI.java
add constructor that supports setting via
* CustomSWFTags.java, CrawlUriSWFAction.java
look for links in a couple more places
* ExtractorSWF.java
ExtractorTagParser.parsePlaceObject2() - read 32 bit clip event
flags when flash version >= 6
* ExtractorSWFTest.java
xestHer1509() - test this fix (rename to enable the test)
* Extractor.java
added instance initializer for DEFAULT_PARAMETERS, and set
@Autowired(required=false) eliminates the need for test
classes to set them explicitly.
* ExtractorCSSTest.java
remove explicit setting of default parameters
* ExtractorHTMLTest.java
remove explicit setting of default parameters
* JerichoExtractorHTMLTest.java
remove explicit setting of default parameters
* ReportGenResource.java
restlet to generate-report-to-file and redirect to view file
* JobResource.java
links to reports from appropriate places for built job
clarify profiles, simplify into-job-dir path-construction
* CrawlJob.java
remove leading '/' from job-relative paths
* EngineApplication.java
install ReportGenResource at appropriate in-job path
* StatisticsTracker.java
fortify for reporting even pre-start/post-stop
add new report types, new groupings: ALL_, LIVE_, END_
dump END_ reports at stop()
dump report by short (class) name
Fix for [HER-1533] robots.txt fetch referer header is via url which is confusing
* FetchHTTP.java
configureMethod() don't set referer header if this is a prerequisite
gojomo