lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-23 06:56:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,234 Commits 40 Branches 31 Tags
2d64e005cb539464acea0c44b8badb6fa3bb5b90
Commit Graph
2234 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Jackson 2d64e005cb Switch to standard cookie handling. 2021-06-17 14:23:20 +01:00
Andrew Jackson 1d1d2f5abe Avoid two slf4j implementations. 2021-06-17 14:19:12 +01:00
Andrew Jackson 4cd416a5e4 Update complier plugin and use compatible flip(). 2021-06-17 14:01:04 +01:00
Andrew Jackson d892026731 Expire cookies when requested. 2021-06-15 00:31:04 +01:00
Andrew Jackson 81fa60d870 Upgrade HTTPClient to 4.5.x for #245 2021-06-14 23:35:46 +01:00
Andrew Jackson d0221642c3 Update changelog. 2021-06-14 11:04:33 +01:00
Andrew Jackson 09c8d5696b Use HTTPS version for Oracle Maven repo. 2021-06-14 11:03:32 +01:00
Andy Jackson 169b3cc8ab Merge pull request #393 from ukwa/checkpoint-success-stats-392 
Only update last checkpoint stats if the checkpoint completed, for #392.
 2021-05-28 14:20:06 +01:00
Andy Jackson 699f107ccd Merge pull request #394 from ukwa/update-ampq-client 
Update AMPQ client library to address security warning.
 2021-05-28 13:24:23 +01:00
Andy Jackson 921e349939 Update AMPQ client library to address security warning. 2021-05-28 12:47:59 +01:00
Andy Jackson 5caa83d327 Only update last checkpoint stats if the checkpoint completed, for #392. 2021-05-28 12:42:17 +01:00
Andy Jackson 773b4a074d Merge pull request #391 from ukwa/master 
Sync changelog with release.
 2021-05-27 10:27:40 +01:00
Andy Jackson a87937caf9 Sync changelog with release. 2021-05-27 10:26:50 +01:00
Andrew Jackson d3dfb34f83 [maven-release-plugin] prepare for next development iteration 2021-05-27 10:03:10 +01:00
Andrew Jackson 9dcbb240a6 [maven-release-plugin] prepare release 3.4.0-20210527 3.4.0-20210527 2021-05-27 10:03:04 +01:00
Andy Jackson cf0f4ef144 Merge pull request #390 from ukwa/master 
Update changelog.
 2021-05-27 09:46:41 +01:00
Andy Jackson 9df4a0dc05 Update changelog. 2021-05-27 09:45:47 +01:00
Andy Jackson 3b4c442683 Merge pull request #389 from ukwa/update-dependencies-2021-05-26 
Update dependencies 2021 05 26
 2021-05-26 21:56:21 +01:00
Andy Jackson 669cf1ed2b Merge branch 'master' into update-dependencies 2021-05-26 13:40:15 +01:00
Andy Jackson a2c65f4618 Update dependencies for security issues, including changes to tests. 2021-05-26 13:39:30 +01:00
Andy Jackson d175040e33 Merge pull request #386 from ukwa/master 
Bring changlog up to date
 2021-05-20 22:50:43 +01:00
Andrew Jackson 400677a46a Merge branch 'master' of github.com:internetarchive/heritrix3 2021-05-20 22:49:05 +01:00
Andrew Jackson 7c72b8d584 Updated changelog. 2021-05-20 22:47:56 +01:00
Andy Jackson d9fe123b2f Merge pull request #384 from ukwa/configure-bdb-je-cleaner-threads 
Allow tuning of BDB-JE evictor and cleaner threads.
 2021-05-20 22:24:42 +01:00
Andy Jackson 276ced0de3 Allow tuning of BDB-JE evictor and cleaner threads. 2021-05-20 22:22:00 +01:00
Andy Jackson c0dbfb2a0a Merge pull request #383 from ukwa/upgrade-dnsjava 
Update to latest version of dnsjava, for #344
 2021-05-20 21:05:59 +01:00
Andy Jackson d7869de8d9 Merge pull request #262 from kris-sigur/sitemaps 
Support for extracting URLs in sitemaps
 2021-05-20 21:05:13 +01:00
Andy Jackson 396467c535 Merge branch 'master' into sitemaps 2021-05-20 21:03:54 +01:00
Andy Jackson c1bcdd9f9e Merge pull request #341 from internetarchive/noplaylist-ydl 
youtube-dl --no-playlist
 2021-05-20 21:01:39 +01:00
Andy Jackson b2a07b290f Merge pull request #348 from internetarchive/fixes-leaky-file-handles 
Fixes leaky file handles
 2021-05-20 21:00:31 +01:00
Andy Jackson c7b1dfa198 Merge pull request #368 from webdev4422/patch-1 
Fix misspell in comments
 2021-05-20 20:58:41 +01:00
Andy Jackson 045b2516da Update to latest version of dnsjava, for #344 
This commit updates to version 3.3.1 of the dnsjava library.

Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
 2021-05-20 20:44:19 +01:00
Alex Osborne a09479364a Merge pull request #379 from ldko/bean-url-missing-fix 
Avoid error when bean properties have no url available
 2021-05-12 09:31:27 +09:00
Alex Osborne 065e62fc1f Merge pull request #377 from internetarchive/bean-browser-optional-fix 
Handle empty Optionals when browsing beans
 2021-05-12 09:31:04 +09:00
Lauren Ko ac33e2ef55 Avoid error when bean properties have no url available 2021-05-11 16:42:33 -05:00
Alex Osborne 71b3065172 Handle empty Optionals when browsing beans 
addPresentableNestedNames() recursively walks the properties of beans
calling getters. Likely unintentionally this includes calling getClass()
and recursing into the reflection API. When running on JDK 11 the
reflection API has some methods (e.g. in java.lang.module) that return
instances of Optional. With the newer version of Spring's
BeanWrapperImpl encounters an Optional it attempts to unwrap it and
throws IllegalArgumentException if it is empty.

Let's fix this in two ways. Firstly let's avoid walking the reflection
API entirely as its irrelevant for the purposes of bean browsing by
not inspecting the properties of instances of java.lang.Class.

Secondly in case any Heritrix beans start using Optional in future lets
also handle the empty case by also not inspecting, the same as we do for
null.

Fixes #376
Reported-By: Lauren Ko <lauren.ko@unt.edu>
 2021-05-11 21:55:03 +09:00
Kristinn Sigurdsson 52093eb25f Only copy source tag if not null. 2021-04-20 13:18:29 +00:00
webdev4422 5ec7fb87d9 Fix misspell in comments 2021-02-26 08:50:13 +02:00
Alex Osborne 37ce8d6945 Merge pull request #366 from ato/upstream-spring-5.3.3 
Upgrade to Spring 5.3.3
 2021-02-23 15:07:59 +09:00
Alex Osborne 35bb3286f4 Clarify robots protocol support 
Closes #351, #353
 2021-02-15 17:29:49 +09:00
Alex Osborne b10b33821f Fix broken robots.txt link 2021-02-15 17:12:04 +09:00
Alex Osborne 5f52a14afc Upgrade to Spring 5.3.3 
Spring 3.0.5 is 10 years old now and is incompatible with classes using
Java 9+ bytecode and even on Java 8 classes using lambdas (see #337).

We remove spring-asm as it is now part of spring-core module.
 2021-02-15 16:20:00 +09:00
Alex Osborne 5a64141302 Merge pull request #352 from BitBaron/absolute-meta-urls 
Strip quotes from URL value.
 2021-02-15 15:48:50 +09:00
Alex Osborne ba0d51882a Merge pull request #359 from internetarchive/ait-ydl-updates 
ait youtube-dl options
 2021-02-15 15:41:48 +09:00
Barbara Miller 0649751456 --no-cache-dir, --no-playlist 2021-01-04 17:53:46 -08:00
Kristinn Sigurðsson adac067ea7 Update link to robots.txt meta documentation 2020-09-02 12:15:02 +00:00
Neil Minton 2505effadc Strip quotes from URL value. 2020-09-01 10:57:58 -04:00
Adam Miller a9c0c6588b Manage youtube dl temp files which can be closed in the warc writer. 2020-07-31 22:22:25 +00:00
Adam Miller 52a8f345ed Fixing up logging and comments 2020-07-30 23:30:08 +00:00
Adam Miller 00d1c46d60 Ensure Replay Input Stream and File Channels are closed after writing 2020-07-30 23:19:43 +00:00