Commit Graph
484 Commits
Author SHA1 Message Date
Alex Osborne 21b808f9c2 MitmProxy: listen on loopback only 2026-11-11 22:02:19 +09:00
Alex Osborne 58258eb640 BiDiJson: map results to null for commands that return Void
Fixes java.lang.UnsupportedOperationException: Unsupported type: class java.lang.Void
2026-07-10 16:12:54 +09:00
Alex Osborne 1392da4fa6 LocalWebDriverBiDi: make close() reliably terminate the browser 2026-07-10 15:43:50 +09:00
Alex Osborne 27d0f05010 LocalWebDriverBiDi: time out and clean up if the browser fails to start 2026-07-10 15:41:28 +09:00
Alex Osborne 99cd6d942d LocalWebDriverBiDi: fail pending commands when the connection is lost 2026-07-10 15:39:34 +09:00
Alex Osborne 4b396426e5 LocalWebDriverBiDi: serialize WebSocket sends
The JDK WebSocket only permits one outstanding sendText() at a time.
With many toe threads issuing BiDi commands concurrently sends can fail with IllegalStateException, so chain them.
2026-07-10 15:36:00 +09:00
Alex Osborne e156997dbd MitmProxy: Fix request recording
This fixes a bug where POST requests would get recorded with a doubled header and a bunch of nulls at the end.

It seems that the onRequestContent() listener doesn't pass us the right buffer when proxying. The buffer seems to contain client-to-server data and the buffer's limit isn't set. So instead this records the request content by overriding newProxyToServerRequestContent.
2026-06-17 16:56:24 +09:00
Alex Osborne 709ac00dd2 Add PaginationBehavior: repeatedly clicks next-page and extracts links 2026-06-11 13:34:55 +09:00
Alex Osborne fc76935fea webdriver: Support Cookie.sameSite value "default" 2026-06-10 18:19:04 +09:00
Alex Osborne 96737799ed Enable XXE protection when parsing crawl job XML
Heritrix's Spring XML based job config format by design allows arbitrary code execution, so this doesn't realistically make things much safer, but I also don't see any reason we need external entity resolution enabled. I suppose there's a chance this could mitigate a generic automated attack that uses XXE but doesn't target Heritrix or Spring XML specifically.

Fixes #711
2026-03-02 10:11:38 +09:00
Leslie Bellony 79396af478 Issue #687 disable strictReplyParsing to accept server partial response 2025-11-18 12:18:34 +01:00
Alex Osborne 2388db4229 Bump commons-lang from 2.6 to 3.18.0 2025-07-21 16:55:31 +09:00
Adam Miller 1ab9b4a206 fix: avoid bdb environment corruption when terminate is called during shutdown 2025-06-11 17:44:52 -07:00
Alex Osborne 61af1d83c4 MitmProxy: Handle URLs generated by browsers but disallowed by java.net.URI 2025-06-11 20:38:54 +09:00
Alex Osborne 8a15156fa1 BrowserProcessor: Catch WebDriver timeouts and exceptions 2025-06-11 17:17:02 +09:00
Alex Osborne c3f9afe64b BrowserProcessor: use FetchHTTP2's configured proxy as an upstream proxy 2025-06-10 09:01:02 +09:00
Alex Osborne 2210cb7262 Browser: Handle navigation abort from downloads starting 2025-06-04 17:44:42 +09:00
Alex Osborne 5131ea4a80 Browser: Disable downloads in Firefox and Chrome
This hopefully will stop us filling up ~/Downloads with random junk.
2025-06-04 17:44:42 +09:00
Alex Osborne 722552feef Add Browser processor using WebDriver BiDi
The Browser processor can load a fetched page in a local web browser, record any requests the browser makes and run behaviors that interact with the page such as scrolling down and extracting links.

This differs from my previous attempt (ExtractorChrome) in a few ways:

- Uses the new WebDriver BiDi standard instead of the Chrome Devtools Protocol. The new protocol is mostly browser-agnostic, more consistent and hopefully more stable.

- Uses a MITM proxy instead of CDP request interception for recording sub-resources. That's partly because BiDi is still missing some key interception APIs. Even so in practice I found the proxy method loads pages faster and more reliably, likely because responses can be streamed incrementally, which helps a lot for large resources or server-sent events.

- Even when HTTP/2 is unavailable, the new FetchHTTP2 module does connection pooling which makes loading browser requests a lot faster. The original FetchHTTP opened a new connection for every request.

- The Browser processor can be configured with a list of behavior beans making it more customizable and extensible.

Obvious areas for future development:

- More Behavior beans: take screenshots, saveg the rendered DOM, run Browsertrix-compatible behavior scripts

- Support for remote WebDrivers (e.g. Selenium Server or cloud services)
2025-06-04 17:44:42 +09:00
Alex Osborne 567e27181d Upgrade webarchive-commons from 1.3.0 to 2.0.1 (removes httpclient 3) 2025-05-21 17:39:21 +09:00
Alex Osborne eb1205f6d5 Upgrade all tests to junit 5 2025-05-09 00:07:18 +09:00
Adam Miller e7fe566919 fix: autoregister Histotable and HighestUriPrecedenceProvider 2025-04-22 17:29:52 -07:00
Adam Miller 75377c5a3b fix: FieldSerializer wasn't passing unit tests. Switch to JavaSerializer 2025-04-21 18:16:47 -07:00
Adam Miller c7784f18e9 feat: Add serializer for Histotable used by HighestUriPrecedenceProvider 2025-04-21 17:20:38 -07:00
Alex Osborne 1a605cf9a6 Remove unused utility class JavaLiterals
Apache Commons StringEscapeUtils which we use for other things provides
this anyway.
2025-04-01 16:11:10 +09:00
Alex Osborne 60f512b4b0 Fix compiler warning: deprecated item is not annotated with @Deprecated 2025-03-31 15:12:21 +09:00
Alex Osborne f6b083ecfb Remove usages of deprecated Integer and Long constructors 2025-03-31 15:04:00 +09:00
Adam Miller b02590cca0 Use AtomicInteger serializer rather than change accessibility 2025-03-14 09:14:26 -07:00
Alex Osborne 4c4510a364 Merge pull request #632 from internetarchive/groovy-config
Add Groovy crawl configs
2024-12-24 12:11:41 +09:00
Alex Osborne d13acddfc4 Remove usages of fastutil
fastutil is our largest dependency, consuming about a third of the
total Heritrix distribution size but we only use a couple of trivial
classes from it.

FPMergeUriUniqFilter (which I'm not sure anyone uses anyway), uses
LongArrayList so this change replaces it with a basic version that does
just enough.

The unsynchronized FastBufferedOutputStream usages are likely
unnecessary these days thanks to the JVM's lock optimisations and for
the one in CrawlerJournal, the GZIPOutputStream is still going to
be synchronizing anyway.
2024-12-20 14:37:20 +09:00
Alex Osborne 19c24b9006 Add Groovy crawl configs
This enables crawl configuration files to use Spring's [Groovy Bean Definition DSL] as an optional alternative to Spring XML. It uses the same bean configuration model but the syntax is more terse and human-readable. No more need for `&` in seed URLs. :-)

```groovy
   checkpointService(CheckpointService) {
        checkpointIntervalMinutes = 15
        checkpointsDir = 'checkpoints'
        forgetAllButLatest = true
   }
```

It also enables some powerful scripting capabilities. For example, defining a custom DecideRule directly in the crawl scope:

```groovy
scope(DecideRuleSequence) {
    rules = [
        new RejectDecideRule(),
        // ACCEPT everything linked from a .pdf file
        new PredicatedDecideRule() {
             boolean evaluate(CrawlURI uri) {
                 return uri.via?.path?.endsWith(".pdf")
             }
        },
        // ...
    ]
}
```

The main downsides are defining nested inner beans can be a bit awkward, some of the errors can be cryptic, and you can't just manipulate the config files with an XML parser.

This commit includes a Groovy version of the default crawl profile for reference, but doesn't expose a way to use it in the UI yet. For now, you need to manually create a `crawler-beans.groovy` file in your job directory.

[Groovy Bean Definition DSL]: https://docs.spring.io/spring-framework/reference/core/beans/basics.html#beans-factory-groovy
2024-12-01 00:14:42 +09:00
Alex Osborne 6de204c53c AutoKryo: Replace SerializingInstantiatorStrategy with StdInstantiatorStrategy
SerializingInstantiatorStrategy when compiled for Java 17 fails when running without `--add-opens java.base/java.io=ALL-UNNAMED`. StdInstantiatorStrategy seems to work OK though.
2024-11-24 19:51:56 +09:00
Alex Osborne dd210c8e70 Replace Spring's removed @Required annotation with our own
Spring 6 removed @Required and they suggest using constructor injection instead. If we switched our beans to that we'd break existing Heritrix crawl configs. So this change implements our own basic version so we still get errors when a @Required property is null.
2024-11-20 16:30:15 +09:00
Alex Osborne 3d1b7e37ca Remove log4j 2024-10-17 19:25:53 +09:00
Alex Osborne e02187ea3b Arc2Warc: Replace joda-time with java.time (JSR-310) 2024-09-13 13:47:00 +09:00
Alex Osborne 74d6b37779 Update to kryo 5.6.0
This eliminates a few more very old dependencies that aren't in Maven Central.

Our direct usage of the unsupported sun.reflect.ReflectionFactory JDK API (which newer compilers complain about) is no longer needed as Kryo now has a SerializingInstantiatorStrategy that does roughly the same thing.
2024-06-18 10:05:56 +09:00
Alex Osborne f4280012ae BdbModule: Fix resource leak on job teardown if sync() throws
This change ensures that close() is always called by moving sync() to a
separate try-block.

Often when a BDB error occurs it causes all subsequent operations
including sync() also throw exceptions. Since we were calling sync()
and close() in the same try-block, sync() throwing would prevent close()
from ever being called and thus leak file handles and BDB worker
threads.
2023-01-23 14:47:07 +09:00
Alex Osborne 71b40cf882 Use Files.createLink() and Files.createSymbolicLink() instead of JNA
We don't need to support Java < 7 anymore. This simplifies the
code and should improve portability to platforms our old version of
JNA doesn't support like Apple Silicon.

Fixes #467
2022-03-02 21:37:27 +09:00
Clemens Klein-Robbenhaar fae2ccb88d Fix name of parameter in setMaxLogFileSize
Fixes #464
2022-02-13 13:01:58 +01:00
Alex Osborne eb67cfb215 Merge pull request #423 from internetarchive/dont-extract-data-uris
Don't extract data URIs
2021-08-03 07:43:17 +09:00
Alex Osborne e33e1ff398 Extractor: ignore data URIs when adding outlinks 2021-07-30 15:57:09 +09:00
Alex Osborne 636948cb09 KeyTool wrapper: fallback to running keytool as a subprocess on JDK 16+
JDK 16 defaults to --illegal-access=deny which means trying to call
KeyTool via reflection now throws IllegalAccessException.

Fixes #417
2021-07-25 11:15:45 +09:00
Adam Miller d2d9d4f157 Adding configurable MaxLogFileSize option to BdbModule 2021-07-19 15:49:56 +09:00
Alex Osborne d643770a7c Put slowest unit tests behind a -DrunSlowTests=true option
Shaves about 2 minutes off the Heritrix build time.
2021-06-22 17:10:50 +09:00
Alex Osborne 5d0a5b2ec4 Remove dependency on mg4j
This eliminates one of the jars that Heritrix depends on that isn't in
Maven Central.

There were only two usages. The first usage was a method in a
CrawlerJournal which as far as I can tell is unused and can be removed.

The second usage was an unnecessary use of MutableString as a line
buffer when saving cookies to a file. Since there doesn't seem any
particular reason buffer per-line here let's just use a BufferedWriter
instead on the whole file which is also slightly simpler and more
efficient.
2021-06-20 23:30:41 +09:00
Andy Jackson 276ced0de3 Allow tuning of BDB-JE evictor and cleaner threads. 2021-05-20 22:22:00 +01:00
Leslie Bellony 63ca9d4c6a Remove static fields and byte loop + rename variable 2020-04-23 17:10:42 +02:00
Leslie Bellony 40725372d6 Add support for the SFTP protocol 2020-04-01 14:47:34 +02:00
Alex Osborne fd514b7bf6 Merge pull request #315 from ukwa/fix-oracle-doc-link
Use the Wayback Machine to repair a link to Oracle docs.
2020-03-05 01:40:39 +09:00
Andrew Jackson b4494a7ffb Use the Wayback Machine to repair a link to Oracle docs. 2020-03-04 15:42:29 +00:00