Commit Graph
6 Commits
Author SHA1 Message Date
gojomo 2dec5dfd1a Fix for [HER-1268] ExtractorHTML not able to extract some links (due to them violating RFC2396)
* ExtractorHTMLTest.java, UURIFactoryTest.java
    test that relative-URIs with late-position colons aren't interpreted as absolute URIs with long, illegal schemes
* UURIFactory.java
    update RFC2396REGEX to require legal scheme
* LaxURI.java
    allow illegal candidate scheme to just be interpreted as something else
2009-05-12 22:22:11 +00:00
gojomo 252cf63863 Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme
* ExtractorHTMLTest.java
    test for maintaining 'https' when speculative, leading-domain URI is on same host
* ExtractorJS.java
    (speculativeFixup) maintain origin URI scheme if apparent target host is same

Fix for [HER-1526] SCRIPT writing script TYPE common trigger of bogus links (eg. 'text/javascript')
Fix for [HER-1611] more javascript false-positives from Google Analytics insert
* ExtractorJS.java
    port of Steve's work in h2: blacklist for URIs, extracted strings to handle common mis-extractions
* ExtractorHTMLTest.java
    port of test for 'text/javascript' false extract
2009-05-12 22:09:22 +00:00
gojomo 2a73339211 Forward port for [HER-1525] Bump WARC version to match latest (0.18, June-2008) ISO draft
* WARCConstants.java, WARCWriter.java, WARCWriterProcessor.java
    remove/update references to pre-0.18 versions or old class names
2009-05-12 19:51:16 +00:00
gojomo 4082195690 Fix for [HER-1529] AggressiveExtractorHTML reports back as ExtractorHTML2 in processor reports
* AggressiveExtractorHTML.java
    correct self-reported name
2009-05-12 18:04:14 +00:00
gojomo daf5ed7590 Update pom.xml * 4
more naming improvements
    ignore test cruft
    update SCM url
2009-05-12 00:17:01 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00