mirror of
https://github.com/internetarchive/heritrix3.git
synced 2026-09-21 21:26:16 +00:00
ExtractorChrome: have frontier consider browser-fetched uris included
Since we now run extractors on subresources there's no reason to schedule and refetch them again. Note that duplicate fetches can still occur if the URI was already scheduled or if the browser itself refetches the resource.
This commit is contained in:
@@ -275,6 +275,12 @@ public class ExtractorChrome extends ContentExtractor {
|
||||
Frontier frontier = controller.getFrontier();
|
||||
curi.getOverlayNames(); // for side-effect of creating the overlayNames list
|
||||
|
||||
// inform the frontier we've already seen this uri so it won't schedule it
|
||||
// we only do this for GETs so a POST doesn't prevent scheduling a GET of the same URI
|
||||
if (request.getMethod().equals("GET")) {
|
||||
frontier.considerIncluded(curi);
|
||||
}
|
||||
|
||||
KeyedProperties.loadOverridesFrom(curi);
|
||||
try {
|
||||
// perform link extraction
|
||||
|
||||
Reference in New Issue
Block a user