lee/heritrix3
1
0 Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-10-07 04:41:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
108 Commits 41 Branches 34 Tags
c196a1de77a9c42c3600a938eed38ee84dd77663
Commit Graph
9 Commits
Author SHA1 Message Date
gojomo c196a1de77 [HER-1658] CachedBdbMaps not expunging as expected (especially StatisticsTracker.processedSeedsRecords) 
* CachedBdbMap.java
    do expunge on put(), replace()
    add low-memory-sensitive 'canary' to force expunge even if otherwise untriggered
* CachedBdbMapTest.java
    add test of idle expunge in low-memory conditions
* CrawlURI.java
    (getURI) direct access to String URI (don't reuse toString() functionally)
* SeedRecord.java
    support for updating record with later/repeat report
* StatisticsTracker.java
    remove put()s for processedSeedRecords, hostsLastFinished
* BdbFrontier.java
    (getQueueFor) remove put, do in concurrent-compliant manner
 2009-08-04 00:06:22 +00:00
szznax e8a8639277 initial checkin: this class is useful for determining why ArchiveReader fails for problematic W/ARC files 2009-07-17 21:52:21 +00:00
gojomo dd254088e1 [HER-1633] port IsCrossTopmostAssignedSurtHopDecideRule to heritrix3 
* HopCrossesAssignmentLevelDomainDecideRule.java
    port H2 IsCrossTopmostAssignedSurtHopDecideRule to H3 under this new name
* PublicSuffixes.java
    update comment, public method for 'Assignment Level Domain'/'Assignment Level SURT' term
* AssignmentLevelSurtQueueAssignmentPolicy
    renamed TopmostAssignedSurtQueueAssignmentPolicy
 2009-07-08 00:28:56 +00:00
gojomo cb8a374e5d [HER-1636] H3: Failed get of replay char sequence due the OutOfMemoryError: Map failed (port HER-1482) 
(from nlevitt's H2 commit message:)
* InMemoryReplayCharSequence.java
    new, simple ReplayCharSequence that supports any encoding, keeping everything in memory
* GenericReplayCharSequence.java
    If the encoding supports random access, memory maps the backing file directly; otherwise decodes to UTF-16 and maps that. Supports the first Integer.MAX_VALUE bytes of the file. Maps up to 64M; moves the map around as necessary for larger files.
* Latin1ByteReplayCharSequence.java
    removed, functionality split between InMemoryReplayCharSequence and GenericReplayCharSequence
* ReplayCharSequenceTest.java
    xestHugeReplayCharSequence() - uncomment to test a huge replay char sequence
    testReplayCharSequenceByteToStringOverflow() - test both UTF-8 and windows-1252
* RecordingOutputStream.java
    use the new ReplayCharSequences
 2009-07-07 21:53:57 +00:00
gojomo d336ea760f quiet excessive logging 2009-06-05 02:00:07 +00:00
gojomo 9155e5f4f9 copying noah's work to h3 
Fix for [HER-1506] ByteReplayCharSequence ignores specified/non-default character encodings, reported by Jean-No?\195?\171l Rivasseau
* RecordingOutputStream.java
    use ByteReplayCharSequence only for iso8859-1
* ReplayCharSequenceTest.java
    new tests for iso8859-1, windows-1252 and ascii
 2009-05-14 21:19:24 +00:00
gojomo 2dec5dfd1a Fix for [HER-1268] ExtractorHTML not able to extract some links (due to them violating RFC2396) 
* ExtractorHTMLTest.java, UURIFactoryTest.java
    test that relative-URIs with late-position colons aren't interpreted as absolute URIs with long, illegal schemes
* UURIFactory.java
    update RFC2396REGEX to require legal scheme
* LaxURI.java
    allow illegal candidate scheme to just be interpreted as something else
 2009-05-12 22:22:11 +00:00
gojomo 3f7d85d7aa Fix for [HER-1528] NumberFormatException in UURIFactory.checkPort() 
* UURIFactoryTest.java
    test that port-looking-part that's out of integer bounds throws URIException not NumberFormatException
* UURIFactory.java
    catch NumberFormatException; allow to surface as URIException
 2009-05-12 18:21:17 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00