lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-22 14:36:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,471 Commits 40 Branches 31 Tags
d3ca691f8d111d9f9bd1f4ec9eecbbcf02942679
Commit Graph
2471 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alex Osborne d3ca691f8d Update CI tests from JDK 20 to 21 2023-10-11 11:22:17 +09:00
Alex Osborne bd6c6bbff5 Merge pull request #566 from internetarchive/jdk20 
Update groovy to 4.0.15 for JDK 20 support
 2023-10-11 11:09:53 +09:00
Alex Osborne 1a27551e60 Update CI tests from JDK 19 to 20 2023-09-19 11:21:05 +09:00
Alex Osborne 1c07e43ade Update groovy to 4.0.15 for JDK 20 support 2023-09-19 11:21:05 +09:00
Andy Jackson 8563f491a5 Merge pull request #536 from claire9910/oscs_fix_cd5c2voau51t5k2gfi30 
fix(sec): upgrade com.itextpdf:itextpdf to 5.5.12
 2023-03-10 13:12:30 +00:00
Andy Jackson a3952be181 Merge pull request #552 from internetarchive/revert-375-fix-last-modified-millis 
Revert "Use seconds from the FetchCompletedTime for LastModified time"
 2023-03-10 13:05:55 +00:00
Andy Jackson d073b43b48 Revert "Use seconds from the FetchCompletedTime for LastModified time" 2023-03-10 13:05:39 +00:00
Andy Jackson d948364105 Merge pull request #549 from KBNLresearch/master 
Job Status description  now shows that a job is not running anymore
 2023-03-10 12:57:53 +00:00
Andy Jackson 2d0afcdbd7 Merge pull request #375 from phanidv/fix-last-modified-millis 
Use seconds from the FetchCompletedTime for LastModified time
 2023-03-10 11:21:32 +00:00
fedor cdfcce8deb In case of a crashed job, the crawlsummaryreport says: "Finished - Abnormal exit from crawl while the ApplicationContext still reports running==true. So the console shows "Job is ACTIVE - Running", while the process crashed This change reports the exit status in this case. 2023-03-06 15:54:27 +01:00
dependabot[bot] c45d524e74 Bump commons-net from 2.0 to 3.9.0 in /commons 
Bumps commons-net from 2.0 to 3.9.0.

---
updated-dependencies:
- dependency-name: commons-net:commons-net
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-01-23 15:17:30 +09:00
Alex Osborne f4280012ae BdbModule: Fix resource leak on job teardown if sync() throws 
This change ensures that close() is always called by moving sync() to a
separate try-block.

Often when a BDB error occurs it causes all subsequent operations
including sync() also throw exceptions. Since we were calling sync()
and close() in the same try-block, sync() throwing would prevent close()
from ever being called and thus leak file handles and BDB worker
threads.
 2023-01-23 14:47:07 +09:00
Alex Osborne a34d9f3f5e Merge pull request #539 from krakan/dont-warn-for-non-authenticating-calls 
Don't warn for non authenticating calls
 2023-01-23 14:44:16 +09:00
Jonas Linde de5c419d8f Don't log authentication failures when header is missing 
The digest authentication method requires an initial request to get a challenge to use for the subsequent authenmtication digest. This initial request is not an authentication attempt and should not be handled as a failed such.
 2022-11-16 11:24:21 +01:00
Jonas Linde 4bad4713ba Set correct timestamp for lastFailureTime 
The initial sleepMs value will be the negative of the current time minus 6 seconds. Using that value to calculate lastFailureTime will increase that large negative number with 6000 milliseconds for every failure which means that rate limiting won't occur until after millions of failed attempts.
 2022-11-16 11:24:21 +01:00
Alex Osborne b9cb43e910 Update groovy to 3.0.13 for JDK 19 support 2022-11-09 21:11:45 +09:00
Alex Osborne b0fb41e7b5 Update CI tests from JDK 18 to 19 2022-11-09 21:03:30 +09:00
Christian Clauss c60af1eba2 Fix typo 2022-10-24 15:06:21 +09:00
claire9910 7cd4a914c3 update com.itextpdf:itextpdf 5.5.0 to 5.5.12 2022-10-15 22:19:49 +08:00
Alex Osborne b328ded92e docs: Add 'Checkpointing' section to operating guide 
Fixes #500 #506
 2022-09-30 15:31:21 +09:00
Alex Osborne 3c79a0b2b1 Increase DiskSpaceMonitor default pause threshold to 8 GiB 
When there's less than 5 GiB free BDB throws DiskLimitException which
Heritrix will likely be unable to handle gracefully and crawl job will
break in various ways. #499
 2022-09-10 23:17:53 +09:00
Alex Osborne f736bf2854 docs: Add DecideRuleSequence to bean-reference 2022-09-06 18:02:30 +09:00
Andy Jackson c5d9bbd0c0 Fix changelog following tag merge. 2022-07-28 10:58:48 +01:00
Andy Jackson e83f4f9e02 Reset to SNAPSHOT 2022-07-28 10:56:59 +01:00
Andy Jackson 9278651b32 Merge tag '3.4.0-20220727' 
[maven-release-plugin] copy for tag 3.4.0-20220727
 2022-07-28 10:53:55 +01:00
Andy Jackson a00a9618a5 Resync changelog. 2022-07-28 09:06:02 +01:00
Andy Jackson 7c4453cf66 [maven-release-plugin] prepare for next development iteration 2022-07-27 21:41:58 +01:00
Andy Jackson b0505662d7 [maven-release-plugin] prepare release 3.4.0-20220727 2022-07-27 21:41:58 +01:00
Andy Jackson 292d2998cb [maven-release-plugin] prepare release 3.4.0-20220727 3.4.0-20220727 2022-07-27 21:36:19 +01:00
Andy Jackson d17a6a8711 Merge pull request #493 from ukwa/master 
Update changelog as of 2022-07-27
 2022-07-27 21:26:06 +01:00
Andy Jackson b8b6c5e209 Update changelog. 2022-07-27 21:24:33 +01:00
dependabot[bot] 1a91a2c8c1 Bump jsch from 0.1.52 to 0.1.54 in /commons 
Bumps jsch from 0.1.52 to 0.1.54.

---
updated-dependencies:
- dependency-name: com.jcraft:jsch
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-07 23:43:35 +09:00
dependabot[bot] 46ea8e1047 Bump spring-core from 5.3.19 to 5.3.20 in /commons 
Bumps [spring-core](https://github.com/spring-projects/spring-framework) from 5.3.19 to 5.3.20.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.19...v5.3.20)

---
updated-dependencies:
- dependency-name: org.springframework:spring-core
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-07 23:30:29 +09:00
dependabot[bot] 82dee7c89c Bump jsch from 0.1.52 to 0.1.54 in /modules 
Bumps jsch from 0.1.52 to 0.1.54.

---
updated-dependencies:
- dependency-name: com.jcraft:jsch
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-07 23:27:52 +09:00
Alex Osborne c1ba694add Document robotsTxtOnly robots policy in config guide 2022-07-07 16:18:39 +09:00
Alex Osborne 1209b2d42f Add robotsTxtOnly robots policy 
This policy obeys robots.txt but ignores meta nofollow. We're
increasingly frequently encountering sites which fail to be archived
properly due to usage of this meta tag but where the robots.txt file
contains reasonable rules. In many cases it seems to be unintentional,
and we've even had questions from webmasters asking why archiving was
failing when they were explicitly allowing us in robots.txt.
 2022-07-07 16:18:39 +09:00
Alex Osborne 09f75e389a docs/glossary: Add manifest (M) to discovery path 2022-06-22 15:44:44 +09:00
Alex Osborne 1d490c779b Merge pull request #488 from netarchivesuite/crawluri-npe 
Removed a potential NPE in hashCode method to CrawlURI which was fata…
 2022-06-08 13:35:23 +09:00
csrster 702bfa5170 Removed a potential NPE in hashCode method to CrawlURI which was fatal for heritrix Browse Beans functionality. 
(cherry picked from commit 96e47a2f695a3116d89f02e7cb529ca181d21444)
 2022-06-07 17:48:29 +02:00
dependabot[bot] e8be86f7cf Bump gson from 2.8.6 to 2.8.9 in /contrib 
Bumps [gson](https://github.com/google/gson) from 2.8.6 to 2.8.9.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.6...gson-parent-2.8.9)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-05-24 12:32:27 +09:00
Alex Osborne a35cd44a73 Docs: Add section to config guide about http and form credentials 
Based on https://github.com/internetarchive/heritrix3/wiki/Credentials
 2022-05-24 12:01:56 +09:00
Alex Osborne 9644780a54 ExtractorHTML: Add --robots option to main() 
Useful for testing link extraction on pages that use a robots meta tag.
 2022-05-11 16:31:48 +09:00
dependabot[bot] e1e7adf088 Bump spring-core from 5.3.18 to 5.3.19 in /commons 
Bumps [spring-core](https://github.com/spring-projects/spring-framework) from 5.3.18 to 5.3.19.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.18...v5.3.19)

---
updated-dependencies:
- dependency-name: org.springframework:spring-core
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-04-27 15:09:33 +09:00
Alex Osborne 9fe64a6f0c ExtractorChrome: If disabled only connect the browser process on demand 
This avoids creating browser processes for jobs that have it disabled.
If a job has it disabled by default but enables it with a sheet we'll
start it when needed.

We still connect on start when enabled to provide early error feedback.
 2022-04-27 14:17:00 +09:00
Alex Osborne 207adec12c Merge pull request #478 from internetarchive/srcset-fix 2022-04-27 14:11:09 +09:00
Alex Osborne 914756f155 ExtractorHTML: Add a main() method to run the extractor standalone 
This makes troubleshooting link extraction problems much easier.
 2022-04-22 16:44:54 +09:00
Alex Osborne aac74411a1 ExtractorHTML: Fix srcset by normalizing elementContext() to lowercase 
This ensures that when we later compare the context in processEmbed()
we don't need to deal with variants like srcSet or SRCSET. Note that
we're already sometimes lowercasing it later in HTMLLinkContext.get().

Fixes #477.
 2022-04-22 16:44:53 +09:00
Alex Osborne 55078c078d Merge pull request #476 from ClemensRobbenhaar/issue211-dns-over-https 2022-04-10 08:59:59 +09:00
Clemens Klein-Robbenhaar f5c83ce368 Add support for DNS-over-HTTPS lookups 
- docs: lets see if we can set a link
  to another chapter
 2022-04-09 23:32:55 +02:00
Clemens Klein-Robbenhaar ed332bcb9e Add support for DNS-over-HTTPS lookups 
- scrap setter for "enableDnsOverHttpResolve", too
 2022-04-09 23:26:07 +02:00