lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-26 07:35:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

Bump incoming links to MEDIUM to compete with e.g. runs of redirects.

Browse Source
This commit is contained in:
Andrew Jackson
2016-10-03 16:51:28 +01:00
parent c6c396af8b
commit eaec682dd4
1 changed files with 7 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
contrib/src/main/java/org/archive/crawler/frontier/AMQPUrlReceiver.java
+7 -1
View File
@@ -446,7 +446,13 @@ public class AMQPUrlReceiver
curi.setSchedulingDirective(SchedulingConstants.HIGH);
curi.setPrecedence(1);
} else {
curi.setSchedulingDirective(SchedulingConstants.NORMAL);
/*
* By default, redirects get set to MEDIUM, so to ensure the
* incoming links are caught promptly, even if the crawler hits
* a lot of redirects (which we have observed for e.g. The
* Guardian), we bump them to MEDIUM so these links can compete.
*/
curi.setSchedulingDirective(SchedulingConstants.MEDIUM);
curi.setPrecedence(1);
}