lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-12 18:47:09 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,639 Commits 40 Branches 31 Tags
3.7.0
Commit Graph

54 Commits

Author SHA1 Message Date
Alex Osborne 4093871de1 Merge pull request #626 from internetarchive/checkpoint-on-shutdown 
Add checkpoint on shutdown and --checkpoint CLI option
 2024-11-28 16:43:45 +09:00
Alex Osborne ec689dbce3 Document checkpointOnShutdown and --checkpoint options in the operating guide 2024-11-24 14:35:52 +09:00
Alex Osborne 585fed8575 Add --checkpoint command-line option 
This makes it possible to select a checkpoint from the command-line when using the --run-job option.
 2024-11-24 13:40:33 +09:00
Alex Osborne 055f4f9f22 Require Java 17 minimum 2024-11-20 15:31:36 +09:00
Alex Osborne 3d1b7e37ca Remove log4j 2024-10-17 19:25:53 +09:00
Alex Osborne 55a179c610 Remove ExtractorChrome 
This never worked that well, is causing random test failures and I don't have any plans to continue developing it.
 2024-08-08 07:21:46 +09:00
Christian Clauss c60af1eba2 Fix typo 2022-10-24 15:06:21 +09:00
Alex Osborne b328ded92e docs: Add 'Checkpointing' section to operating guide 
Fixes #500 #506
 2022-09-30 15:31:21 +09:00
Alex Osborne f736bf2854 docs: Add DecideRuleSequence to bean-reference 2022-09-06 18:02:30 +09:00
Alex Osborne c1ba694add Document robotsTxtOnly robots policy in config guide 2022-07-07 16:18:39 +09:00
Alex Osborne 09f75e389a docs/glossary: Add manifest (M) to discovery path 2022-06-22 15:44:44 +09:00
Alex Osborne a35cd44a73 Docs: Add section to config guide about http and form credentials 
Based on https://github.com/internetarchive/heritrix3/wiki/Credentials
 2022-05-24 12:01:56 +09:00
Clemens Klein-Robbenhaar f5c83ce368 Add support for DNS-over-HTTPS lookups 
- docs: lets see if we can set a link
  to another chapter
 2022-04-09 23:32:55 +02:00
Clemens Klein-Robbenhaar fdde6d87a0 Add support for DNS-over-HTTPS lookups 
- instead of "borrowing" the configured proxy from
  the fetchHttp bean, use proxy values defined via
  global options, to avoid interference with other
  jobs running in parallel (or at least make them
  explicit).
  The "fetchHttp" bean also uses these settings,
  if no bean specific settings are used.
- remove the "enableDnsOverHttpResolves", and rely
  on a non-empty "dnsOverHttpServer" value to signal
  that DoH should be used.
 2022-04-09 23:04:37 +02:00
Clemens Klein-Robbenhaar 9648c98c77 [misc] rst formatting 2022-04-08 12:14:11 +02:00
Clemens Klein-Robbenhaar 2144abb2cb Add support for DNS-over-HTTPS lookups 
- use the "DohResolver" from the dnsjava library
  to make DoH lookups
- to enable and configure it, add two new
  properties
  * "enableDnsOverHttpResolves" (boolean)
  * "dnsOverHttpServer" URL to the DoH Server
- as one use case for DoH is being located
  behind a firewall, also support using a proxy
  to access the DoH server; the proxy from
  the FetchHTTP bean is reused in that case

Fixes #211
 2022-04-08 11:48:52 +02:00
Jamie Hoyle dd01689ace FetchHTTP: SOCKS5 proxy support 
Closes #454
 2022-03-28 10:24:07 +09:00
Oliver Schihin c4edadf1a8 Configuration guide: clarify usage of ContentTypeMatchesRegexDecideRule 2021-12-21 19:10:22 +09:00
Erik Körner 3a5b758d5e Docs (temp) 2021-12-10 16:39:17 +01:00
Erik Körner 17c9694039 Add Dockerfile 2021-12-10 15:35:03 +01:00
Alex Osborne d11d02eeac docs: Add 'Loading Cookies' section to config guide 2021-11-29 12:11:54 +09:00
Alex Osborne 780abb0b1d docs: Update AdoptOpenJDK link as it has been rebranded Adoptium 
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/
 2021-09-11 07:53:18 +09:00
Alex Osborne ea42630608 docs: Add decide rules to bean reference 2021-07-13 21:49:27 +09:00
Alex Osborne 2a027a4064 docs: Fallback to field javadoc comments when setter javadoc unavailable 
This enables us to generate documentation for more bean properties,
although a number of beans have javadoc on initializer code blocks which
makes it hard to access. This affects javadoc and IDE contextual
documentation too so should probably be fixed in the source code itself.
 2021-07-12 19:31:13 +09:00
Alex Osborne c54f18e3ff docs: Add most of the default config beans to the bean reference 
Notably Decide Rules are still to be done.
 2021-07-12 11:39:06 +09:00
Alex Osborne 4dfe48b1a0 docs: Strip @link and @code javadoc directives from bean reference 2021-07-12 09:28:04 +09:00
Alex Osborne 4a5fdcfaf0 docs: Add remaining link extractor to the bean reference 2021-07-12 09:23:08 +09:00
Alex Osborne d10be35f0c docs: Remove duplicate word 'Documentation' from page titles 
Also include '3' to distinguish from the Heritrix 1 manual which is
still prevalent in search results.
 2021-07-12 09:10:57 +09:00
Alex Osborne 63bd15535d docs: Fix links accidentally using Markdown syntax 2021-07-12 09:09:17 +09:00
Alex Osborne b3be9d01fe docs: Start a 'Bean Reference' document generated from the source code 
This is meant to complement the javadoc by providing a reference
more suitable for users trying to configure crawls rather than
developers writing new modules.

The doc generation could still do with some improving and some of the
source javadoc comments need fixing up but this is already useful so I'm
committing what I have so far.
 2021-07-11 15:56:37 +09:00
Alex Osborne 3f20e736d5 docs: Fix incorrectly indented line causing rst warning 2021-07-10 21:11:03 +09:00
Alex Osborne 503e40cfce docs: Add sections on FTP, SFTP and WHOIS to config guide 2021-07-10 21:09:52 +09:00
Alex Osborne c2c2b244de docs: Add a plugin for basic auto-generation of bean examples 2021-07-10 21:09:17 +09:00
Alex Osborne 6fb699c780 docs: s/Most/More/ documentation lives on the wiki 
A large amount of core documentation has now been migrated here. There's
still a lot to go but I don't think 'most' applies anymore.
 2021-07-10 12:42:17 +09:00
Alex Osborne 9b1919dbd3 docs: Remove inaccurate 2018 date from documentation footer 
Large portions of the docs were first published much earlier in other
places and some have been updated since then. Rather than trying to
keep a date range up to date let's just remove the date. We don't use
dates in the source code boilerplate and my understanding is copyright
notices are not mandatory in almost all countries due to the Berne
Convention anyway so it's really just informational.
 2021-07-10 12:37:35 +09:00
Alex Osborne b680bb206b docs: Add subdirectory explanation from wiki install page 2021-07-10 12:13:26 +09:00
Alex Osborne a84e0bb792 docs: Add an operating guide based on the contents of the wiki 2021-07-07 21:30:36 +09:00
Alex Osborne 507d7d91a3 docs: add getting-started.rst, configuring-jobs.rst and glossary.rst 
Compiled from the wiki with some restructuring, reformatting and
updating for the current version of Heritrix.
 2021-07-06 22:44:11 +09:00
Alex Osborne f19dacfe67 Document GET /engine 2020-04-20 18:32:43 +09:00
Alex Osborne f442f76784 Add missing newline 2020-04-20 12:47:17 +09:00
Alex Osborne 33e637b887 Correct indentation to 3-spaces for reST 2020-04-20 12:40:51 +09:00
Alex Osborne 75dc16e8ca Document 'Get Job Status' endpoint 2020-04-20 12:35:49 +09:00
Andy Jackson e08d3ad45d Note feature only applies to forthcoming 3.3 release 2018-11-03 09:30:32 +00:00
Andrew Jackson 3a299d7899 Update API with note about checkpoint launching. 2018-11-01 20:20:17 +09:00
Alex Osborne da9fcf1af6 Fix typos 2018-07-04 22:22:15 +09:00
Alex Osborne 2d48b40ea4 Replace API response section with statuscode tag 2018-07-04 21:46:42 +09:00
Alex Osborne 0cfd62f76e Reformat the rest of the API calls 2018-07-04 21:43:57 +09:00
Alex Osborne 3b3d579cb8 Add actions to API URLs to distinguish them 2018-07-04 21:16:53 +09:00
Alex Osborne 1ff31c135e Add explanatory note about documentation 2018-07-04 19:48:33 +09:00
Alex Osborne 7076dd6388 Try nicer formatting for the first API example 2018-07-04 18:46:15 +09:00