Noah Levitt a7a2b3cf7e Make speculative link extractor ignore urls with two consecutive slashes (not preceded by a colon as in http://). We've seen the double slashes lead to crawler traps. ...

* UriUtils.java
    isLikelyFalsePositive() - consider likely false positive if it contains '//' (but not '://')
* UriUtilsTest.java
    testDoubleSlashFalsePositives()

2012-08-24 14:03:45 -07:00

..

.settings

copy 'springy' branch to heritrix3

2009-05-11 22:56:36 +00:00

src

Make speculative link extractor ignore urls with two consecutive slashes (not preceded by a colon as in http://). We've seen the double slashes lead to crawler traps.

2012-08-24 14:03:45 -07:00

.classpath

copy 'springy' branch to heritrix3

2009-05-11 22:56:36 +00:00

.project

copy 'springy' branch to heritrix3

2009-05-11 22:56:36 +00:00

pom.xml

**/pom.xml

2012-05-01 18:17:33 -07:00