lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-10-04 11:27:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
49 Commits 42 Branches 33 Tags
ca8a02bb1720ea6b164fecdfbf2e2bb2e60b4e0c
Commit Graph
20 Commits
Author SHA1 Message Date
gojomo ca8a02bb17 [HER-1637] upgrade JE jar to 3.3.82 
* commons/pom.xml, .classpath
    update refs to 3.3.82
 2009-05-21 05:46:27 +00:00
szznax 35fbf91626 added joda-time dependency, used in 2009-05-18 17:45:31 +00:00
szznax 3f6d60a054 copy to h3 
Fix for [HER-1588] Arc2Warc transformed WARC has incorrect date in warcinfo record
* Arc2Warc.java
  correct WARC-Date format, and use Joda-time
 2009-05-15 18:19:45 +00:00
gojomo a84e5b5dca copied to h3 from h2 commits 
Misc: maintain object identity of peek()ed objects
* StoredQueueTest.java
    test that consecutive peek()s return exactly identical object
* StoredQueue.java
    cache head object object for peek consistency
 2009-05-15 00:08:29 +00:00
gojomo dc117fb958 * commons/pom.xml 
use latest 3.3.75 bdb je jar
 2009-05-14 22:26:21 +00:00
gojomo a1e9d064da copy to h3 
Fix for [HER-1520] Windows mapped-files cause intermittent ReplayCharSequenceTest.testReplayCharSequenceByteToStringMulti failures, risk other problems
* GenericReplayCharSequence.java
    use more diverse filenames, so subsequent decodes don't run into lingering open mapped sections of previous decodes
    do delete via new FileUtils.deleteSoonerOrLater utility
* FileUtils.java
   (deleteSoonerOrLater) try immediate delete, but if fails, add to a list of files to be retried later
 2009-05-14 21:31:45 +00:00
gojomo 9155e5f4f9 copying noah's work to h3 
Fix for [HER-1506] ByteReplayCharSequence ignores specified/non-default character encodings, reported by Jean-No?\195?\171l Rivasseau
* RecordingOutputStream.java
    use ByteReplayCharSequence only for iso8859-1
* ReplayCharSequenceTest.java
    new tests for iso8859-1, windows-1252 and ascii
 2009-05-14 21:19:24 +00:00
gojomo ad33d302ff copying noah's work to h3 
[HER-1506] correct javadoc on use of these classes
* GenericReplayCharSequence.java
* Latin1ByteReplayCharSequence.java
 2009-05-14 21:12:34 +00:00
gojomo 5baac7cc45 copying noah's work to h3 
[HER-1506] rename classes to reflect true roles
* MultiByteReplayCharSequence.java -> GenericReplayCharSequence.java
* ByteReplayCharSequence.java -> Latin1ByteReplayCharSequence.java
* RecordingOutputStream.java
    update references
 2009-05-14 21:10:44 +00:00
gojomo 37ea37b883 Fix for [HER-1564] Update 'public suffix list' (effective_tld_names.dat) 
* effective_tld_names.dat
    update to latest version from publisuffix.org (Mozilla) -- downloaded 2009-05-14
 2009-05-14 20:56:51 +00:00
gojomo 8c620ae846 Fix for [HER-1632] Update Jericho HTML Parser to 2.6 
* commons/pom.xml
    update requested versions to 2.6.1
 2009-05-14 20:47:29 +00:00
gojomo 9994edb170 HER-565 / improve output if problem recurs 2009-05-14 20:29:01 +00:00
gojomo badbac399f apply to h3 
Fix for [HER-1521] Make the UncompressedARCReader and CompressedARCReader in the ARCReaderFactory as public or protected
* ARCReaderFactory.java, WARCReaderFactory.java
    make compressed/uncompressed implementation inner classes public to enable direct reuse
 2009-05-14 20:24:56 +00:00
gojomo 58486c5f18 (copying from h2 to h3) 
Fix for [HER-1277] speculative javascript link caused 404s for "http%3A" and "www.example.com" like strings
* ExtractorJS.java
    perform additional decoding or prepending of "http:" for speculative candidate URIs that appear to need it
* ArchiveUtils.java
    add utility set/method with list of all TLDs
    
Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme
* ExtractorJS.java
    (speculativeFixup) maintain origin URI scheme if apparent target host is same
 2009-05-13 02:14:15 +00:00
gojomo 2dec5dfd1a Fix for [HER-1268] ExtractorHTML not able to extract some links (due to them violating RFC2396) 
* ExtractorHTMLTest.java, UURIFactoryTest.java
    test that relative-URIs with late-position colons aren't interpreted as absolute URIs with long, illegal schemes
* UURIFactory.java
    update RFC2396REGEX to require legal scheme
* LaxURI.java
    allow illegal candidate scheme to just be interpreted as something else
 2009-05-12 22:22:11 +00:00
gojomo 2a73339211 Forward port for [HER-1525] Bump WARC version to match latest (0.18, June-2008) ISO draft 
* WARCConstants.java, WARCWriter.java, WARCWriterProcessor.java
    remove/update references to pre-0.18 versions or old class names
 2009-05-12 19:51:16 +00:00
gojomo fb97744dfd Fix for [HER-1495] ARCRecord throws null pointer when trying to generate a RecoverableIOException 
* ARCRecord.java
    guard against bull statusBytes in exception message
 2009-05-12 19:43:20 +00:00
gojomo 3f7d85d7aa Fix for [HER-1528] NumberFormatException in UURIFactory.checkPort() 
* UURIFactoryTest.java
    test that port-looking-part that's out of integer bounds throws URIException not NumberFormatException
* UURIFactory.java
    catch NumberFormatException; allow to surface as URIException
 2009-05-12 18:21:17 +00:00
gojomo daf5ed7590 Update pom.xml * 4 
more naming improvements
    ignore test cruft
    update SCM url
 2009-05-12 00:17:01 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00