Alex Osborne
7ff4420801
This adds a link extractor that uses a headless browser via the Chrome Devtools Protocol. It works with Chromium and Google Chrome and while untested likely also other Blink-based browsers such as Brave and Microsoft Edge. It also works with Firefox which implements a subset of the protocol. In addition to ExtractorChrome itself this adds a couple of classes under org.archive.net.chrome for communicating with the browser: * ChromeProcess - manages the lifecycle of the browser child process * ChromeClient - client for the Chrome Devtools Protocol * ChromeWindow - client for controlling a particular browser window Why this extractor when we already have Umbra? The primary goal is to have something that works out of the box without any additional infrastructure beyond having a browser installed. A longer term goal is to be able to take advantage of tighter integration between Heritrix and the browser to do things like recording browser subrequests to WARC files or running ExtractorHTML against the loaded DOM. While this extractor is already usable it is currently quite bare bones and only extracts links from <a> and <area> tags. The intent is to expand it with more functionality over time with features like intercepting subrequests, reusing previously saved resources, taking screenshots or running JavaScript behaviours.
Heritrix
Introduction
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.
Crawl Operators!
Heritrix is designed to respect the robots.txt exclusion directives† and META nofollow tags. Please consider the
load your crawl will place on seed sites and set politeness policies accordingly. Also, always identify your crawl with contact information in the User-Agent so sites that may be adversely affected by your crawl can contact you or adapt their server behavior accordingly.
† The newer wildcard extension to robots.txt is not yet supported.
Getting Started
Developer Documentation
- Developer Manual
- REST API documentation
- JavaDoc: engine, modules, commons, contrib
Latest Releases
Information about releases can be found here.
License
Heritrix is free software; you can redistribute it and/or modify it under the terms of the Apache License, Version 2.0
Some individual source code files are subject to or offered under other licenses. See the included LICENSE.txt file for more information.
Heritrix is distributed with the libraries it depends upon. The libraries can be found under the lib directory in the release distribution, and are used under the terms of their respective licenses, which are included alongside the libraries in the lib directory.