lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-23 07:51:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,776 Commits 40 Branches 31 Tags
bfbd61edd7cd6c1069205f13fe2e16cfeffc648b
Commit Graph
19 Commits
Author SHA1 Message Date
Noah Levitt cbb5e3ab99 Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions. 2016-02-09 15:02:11 -08:00
Hunter Stern 088189ee8b Fix up code based on pull requests comments. Add test for urls with spaces in them. 2016-01-28 15:11:49 -08:00
Hunter Stern 4518bc2da8 Make sure no urls with whitespace and not having good extension slip through 2016-01-20 14:29:42 -08:00
Hunter Stern 12a88d6d23 Allow spaces in urls extracted from JS. 2016-01-20 10:44:20 -08:00
Hunter Stern 23885557bb Allow spaces in JavaScript urls, but only if they have a known good file extension 2016-01-19 18:07:05 -08:00
Noah Levitt 1027f854ec for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js") 2015-04-20 17:08:47 -07:00
Noah Levitt 339e6ec87a for ARI-3723, accept speculative urls with query params with no value 2014-04-04 10:15:07 -07:00
Noah Levitt 705a375daf uses of UriUtils.isLikelyUri() in Extractor{HTML,SWF,XML} with UriUtils.isVeryLikelyUri() to reap the benefits of HER-1523 improvements (should address archive-it issue ARI-3492) 2013-08-09 18:17:11 -07:00
Noah Levitt 3a65fa25d7 Working on refactoring some stuff into archive-commons 2012-12-18 14:46:52 -08:00
Noah Levitt 85118c59c4 Tighten up javascript extraction based on real-world analysis (addresses HER-1523) 
* UriUtils.java
    new method isVeryLikelyUri() with tighter heuristic than isLikelyUri()
* ExtractorJS.java
    use UriUtils.isVeryLikelyUri(), and change order of operations to do fixup before call to isVeryLikelyUri(), since it doesn't expect strings with javascript escaping and stuff
* StringExtractorTestBase.java
    handle test data with expected value null, meaning no outlinks expected
* ExtractorHTMLTest.java
    avoid redundancy by using the extractor created in ContentExtractorTestBase.setUp()
* ExtractorJSTest.java
    some new tests
 2012-12-11 16:27:26 -08:00
Noah Levitt b2a90ddc15 * UriUtils.java 
speculativeFixup() - improve detection of scheme-less intended-absolute-URIs
 2012-12-10 14:10:34 -08:00
Noah Levitt a7a2b3cf7e Make speculative link extractor ignore urls with two consecutive slashes (not preceded by a colon as in http://). We've seen the double slashes lead to crawler traps. 
* UriUtils.java
    isLikelyFalsePositive() - consider likely false positive if it contains '//' (but not '://')
* UriUtilsTest.java
    testDoubleSlashFalsePositives()
 2012-08-24 14:03:45 -07:00
Noah Levitt 99acaa9bfa Fix tests revealed to be failing 
* UriUtils.java
    isLikelyFalsePositive() - do not flag urls starting with ../ as likely false positives
* UriUtilsTest.java
    remove test url that is now rejected
 2012-06-06 09:00:13 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Noah Levitt 9dbe673bc3 * UriUtils.java 
some improvements to likely-uri detection based on crawl log analysis
 2012-01-20 11:23:45 -08:00
nlevitt c5de0048d1 "Fix" for HER-1951 ExtractorJS produces many false positives 
* UriUtils.java
    isLikelyFalsePositive() - check for unusual characters, likely mimetypes, and a couple of other common traps
* UriUtilsTest.java
    add some tests for this code
 2011-09-30 00:48:00 +00:00
nlevitt a2eaa9d6ac HER-1820 followup 
* UriUtils.java
    NAIVE_LIKELY_URI_PATTERN - revert change and unpublicize
    NAIVE_URI_EXCEPTIONS - add some mimetype strings that came up in testing
    isLikelyFalsePositive() - unpublicize
* ExtractorXML.java
    XML_URI_EXTRACTOR - do not use NAIVE_LIKELY_URI_PATTERN
    shouldExtract() - check for mimetype application/vnd.openxmlformats which is not xml
    processXml() - use UriUtils.isLikelyUri()
 2011-04-21 17:18:01 +00:00
nlevitt 70d21ca2da HER-1820 more eager xml link extraction 
* ExtractorXML.java
    instead of considering only strings that start with http(s):, consider all
    strings that match UriUtils.NAIVE_LIKELY_URI_PATTERN (and, as before,
    constitute the entirety of the xml tag content or attribute value)
* UriUtils.java
    - NAIVE_LIKELY_URI_PATTERN - add quotes to the excluded characters so it
      does not eat the closing quote when matching xml attribute values, and
      make visibility public for use in ExtractorXML
    - isLikelyUri() - refactor false positive check into new method
      isLikelyFalsePositive() so that it can be used in ExtractorXML avoiding
      redundant check against NAIVE_LIKELY_URI_PATTERN
 2011-04-20 01:18:00 +00:00
gojomo f933dc922d HER-1763 unify/improve/document likely-URI string tests 
* UriUtils.java
    move likely-URI heuristics here for better consistency, reuse, testing, improvement
* UriUtilsTest.java
    testing for UriUtils
* ExtractorJS.java
    move likely-URI support to UriUtils; use new test
* ExtractoSWF.java
    use new UriUtils likely-URI testing
* ExtractorHTML.java
    move likely-URI testing to UriUtils, use new test
* JerichoExtractorHTML.java
    use new UriUtils likely-URI testing
 2010-04-22 01:08:14 +00:00