ExtractorChrome: have frontier consider browser-fetched uris included

Since we now run extractors on subresources there's no reason to
schedule and refetch them again.

Note that duplicate fetches can still occur if the URI was already
scheduled or if the browser itself refetches the resource.
This commit is contained in:
Alex Osborne
2021-07-24 17:25:21 +09:00
parent bbae794b47
commit 014fb2dcad
@@ -275,6 +275,12 @@ public class ExtractorChrome extends ContentExtractor {
Frontier frontier = controller.getFrontier();
curi.getOverlayNames(); // for side-effect of creating the overlayNames list
// inform the frontier we've already seen this uri so it won't schedule it
// we only do this for GETs so a POST doesn't prevent scheduling a GET of the same URI
if (request.getMethod().equals("GET")) {
frontier.considerIncluded(curi);
}
KeyedProperties.loadOverridesFrom(curi);
try {
// perform link extraction