lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-23 15:06:46 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,219 Commits 40 Branches 31 Tags
cf0f4ef144d7bdb367d7379dc2f4eb8d5c7f88cc
Commit Graph
2219 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andy Jackson cf0f4ef144 Merge pull request #390 from ukwa/master 
Update changelog.
 2021-05-27 09:46:41 +01:00
Andy Jackson 9df4a0dc05 Update changelog. 2021-05-27 09:45:47 +01:00
Andy Jackson 3b4c442683 Merge pull request #389 from ukwa/update-dependencies-2021-05-26 
Update dependencies 2021 05 26
 2021-05-26 21:56:21 +01:00
Andy Jackson 669cf1ed2b Merge branch 'master' into update-dependencies 2021-05-26 13:40:15 +01:00
Andy Jackson a2c65f4618 Update dependencies for security issues, including changes to tests. 2021-05-26 13:39:30 +01:00
Andy Jackson d175040e33 Merge pull request #386 from ukwa/master 
Bring changlog up to date
 2021-05-20 22:50:43 +01:00
Andrew Jackson 400677a46a Merge branch 'master' of github.com:internetarchive/heritrix3 2021-05-20 22:49:05 +01:00
Andrew Jackson 7c72b8d584 Updated changelog. 2021-05-20 22:47:56 +01:00
Andy Jackson d9fe123b2f Merge pull request #384 from ukwa/configure-bdb-je-cleaner-threads 
Allow tuning of BDB-JE evictor and cleaner threads.
 2021-05-20 22:24:42 +01:00
Andy Jackson 276ced0de3 Allow tuning of BDB-JE evictor and cleaner threads. 2021-05-20 22:22:00 +01:00
Andy Jackson c0dbfb2a0a Merge pull request #383 from ukwa/upgrade-dnsjava 
Update to latest version of dnsjava, for #344
 2021-05-20 21:05:59 +01:00
Andy Jackson d7869de8d9 Merge pull request #262 from kris-sigur/sitemaps 
Support for extracting URLs in sitemaps
 2021-05-20 21:05:13 +01:00
Andy Jackson 396467c535 Merge branch 'master' into sitemaps 2021-05-20 21:03:54 +01:00
Andy Jackson c1bcdd9f9e Merge pull request #341 from internetarchive/noplaylist-ydl 
youtube-dl --no-playlist
 2021-05-20 21:01:39 +01:00
Andy Jackson b2a07b290f Merge pull request #348 from internetarchive/fixes-leaky-file-handles 
Fixes leaky file handles
 2021-05-20 21:00:31 +01:00
Andy Jackson c7b1dfa198 Merge pull request #368 from webdev4422/patch-1 
Fix misspell in comments
 2021-05-20 20:58:41 +01:00
Andy Jackson 045b2516da Update to latest version of dnsjava, for #344 
This commit updates to version 3.3.1 of the dnsjava library.

Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
 2021-05-20 20:44:19 +01:00
Alex Osborne a09479364a Merge pull request #379 from ldko/bean-url-missing-fix 
Avoid error when bean properties have no url available
 2021-05-12 09:31:27 +09:00
Alex Osborne 065e62fc1f Merge pull request #377 from internetarchive/bean-browser-optional-fix 
Handle empty Optionals when browsing beans
 2021-05-12 09:31:04 +09:00
Lauren Ko ac33e2ef55 Avoid error when bean properties have no url available 2021-05-11 16:42:33 -05:00
Alex Osborne 71b3065172 Handle empty Optionals when browsing beans 
addPresentableNestedNames() recursively walks the properties of beans
calling getters. Likely unintentionally this includes calling getClass()
and recursing into the reflection API. When running on JDK 11 the
reflection API has some methods (e.g. in java.lang.module) that return
instances of Optional. With the newer version of Spring's
BeanWrapperImpl encounters an Optional it attempts to unwrap it and
throws IllegalArgumentException if it is empty.

Let's fix this in two ways. Firstly let's avoid walking the reflection
API entirely as its irrelevant for the purposes of bean browsing by
not inspecting the properties of instances of java.lang.Class.

Secondly in case any Heritrix beans start using Optional in future lets
also handle the empty case by also not inspecting, the same as we do for
null.

Fixes #376
Reported-By: Lauren Ko <lauren.ko@unt.edu>
 2021-05-11 21:55:03 +09:00
Kristinn Sigurdsson 52093eb25f Only copy source tag if not null. 2021-04-20 13:18:29 +00:00
webdev4422 5ec7fb87d9 Fix misspell in comments 2021-02-26 08:50:13 +02:00
Alex Osborne 37ce8d6945 Merge pull request #366 from ato/upstream-spring-5.3.3 
Upgrade to Spring 5.3.3
 2021-02-23 15:07:59 +09:00
Alex Osborne 35bb3286f4 Clarify robots protocol support 
Closes #351, #353
 2021-02-15 17:29:49 +09:00
Alex Osborne b10b33821f Fix broken robots.txt link 2021-02-15 17:12:04 +09:00
Alex Osborne 5f52a14afc Upgrade to Spring 5.3.3 
Spring 3.0.5 is 10 years old now and is incompatible with classes using
Java 9+ bytecode and even on Java 8 classes using lambdas (see #337).

We remove spring-asm as it is now part of spring-core module.
 2021-02-15 16:20:00 +09:00
Alex Osborne 5a64141302 Merge pull request #352 from BitBaron/absolute-meta-urls 
Strip quotes from URL value.
 2021-02-15 15:48:50 +09:00
Alex Osborne ba0d51882a Merge pull request #359 from internetarchive/ait-ydl-updates 
ait youtube-dl options
 2021-02-15 15:41:48 +09:00
Barbara Miller 0649751456 --no-cache-dir, --no-playlist 2021-01-04 17:53:46 -08:00
Kristinn Sigurðsson adac067ea7 Update link to robots.txt meta documentation 2020-09-02 12:15:02 +00:00
Neil Minton 2505effadc Strip quotes from URL value. 2020-09-01 10:57:58 -04:00
Adam Miller a9c0c6588b Manage youtube dl temp files which can be closed in the warc writer. 2020-07-31 22:22:25 +00:00
Adam Miller 52a8f345ed Fixing up logging and comments 2020-07-30 23:30:08 +00:00
Adam Miller 00d1c46d60 Ensure Replay Input Stream and File Channels are closed after writing 2020-07-30 23:19:43 +00:00
Barbara Miller 1b8e7f73ca youtube-dl --no-playlist 2020-07-29 11:12:33 -07:00
Andrew Jackson 33a3f9a58f Updated changelog based on master. 2020-06-01 14:00:59 +01:00
Alex Osborne 2e6b76d517 Merge pull request #333 from dengliming/patch-1 
Remove deprecated sudo setting.
 2020-06-01 14:05:02 +09:00
Alex Osborne c4644d4e5c Merge pull request #335 from internetarchive/fixes-extractor-multiple-regex-matcher-recycle 
Fixes extractor multiple regex matcher recycle
 2020-06-01 14:03:11 +09:00
Alex Osborne 812e96d2c0 Merge pull request #336 from internetarchive/revert-334-fixes-ftp-response-record-builder 
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
 2020-06-01 13:56:37 +09:00
Alex Osborne 947507b8ec Revert "Warc convention for storing ftp responses has been to use a WARC reso…" 2020-06-01 13:56:12 +09:00
Alex Osborne 4763a8a04b Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder 
Warc convention for storing ftp responses has been to use a WARC reso…
 2020-06-01 13:54:59 +09:00
Adam Miller 60daef90b2 Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle 2020-05-20 16:46:14 +00:00
Adam Miller c0fe2064a1 Merge branch 'fixes-ftp-response-record-builder' 2020-05-20 16:32:36 +00:00
Andrew Jackson 87c349fe13 Re-sync change log. 2020-05-18 15:28:56 +01:00
Andrew Jackson e215b89963 [maven-release-plugin] prepare for next development iteration 2020-05-18 14:08:12 +01:00
Andrew Jackson ef38dc08c3 [maven-release-plugin] prepare release 3.4.0-20200518 3.4.0-20200518 2020-05-18 14:08:04 +01:00
Andrew Jackson 176f796beb Updated changelog. 2020-05-18 14:05:26 +01:00
dengliming b4962d25a8 Remove deprecated sudo setting. 2020-05-12 22:53:40 +08:00
Adam Miller 0db661fb36 Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor 2020-05-05 23:09:53 +00:00