Alex Osborne
065e62fc1f
Merge pull request #377 from internetarchive/bean-browser-optional-fix
...
Handle empty Optionals when browsing beans
2021-05-12 09:31:04 +09:00
Alex Osborne
71b3065172
Handle empty Optionals when browsing beans
...
addPresentableNestedNames() recursively walks the properties of beans
calling getters. Likely unintentionally this includes calling getClass()
and recursing into the reflection API. When running on JDK 11 the
reflection API has some methods (e.g. in java.lang.module) that return
instances of Optional. With the newer version of Spring's
BeanWrapperImpl encounters an Optional it attempts to unwrap it and
throws IllegalArgumentException if it is empty.
Let's fix this in two ways. Firstly let's avoid walking the reflection
API entirely as its irrelevant for the purposes of bean browsing by
not inspecting the properties of instances of java.lang.Class.
Secondly in case any Heritrix beans start using Optional in future lets
also handle the empty case by also not inspecting, the same as we do for
null.
Fixes #376
Reported-By: Lauren Ko <lauren.ko@unt.edu >
2021-05-11 21:55:03 +09:00
Alex Osborne
37ce8d6945
Merge pull request #366 from ato/upstream-spring-5.3.3
...
Upgrade to Spring 5.3.3
2021-02-23 15:07:59 +09:00
Alex Osborne
35bb3286f4
Clarify robots protocol support
...
Closes #351 , #353
2021-02-15 17:29:49 +09:00
Alex Osborne
b10b33821f
Fix broken robots.txt link
2021-02-15 17:12:04 +09:00
Alex Osborne
5f52a14afc
Upgrade to Spring 5.3.3
...
Spring 3.0.5 is 10 years old now and is incompatible with classes using
Java 9+ bytecode and even on Java 8 classes using lambdas (see #337 ).
We remove spring-asm as it is now part of spring-core module.
2021-02-15 16:20:00 +09:00
Alex Osborne
5a64141302
Merge pull request #352 from BitBaron/absolute-meta-urls
...
Strip quotes from URL value.
2021-02-15 15:48:50 +09:00
Alex Osborne
ba0d51882a
Merge pull request #359 from internetarchive/ait-ydl-updates
...
ait youtube-dl options
2021-02-15 15:41:48 +09:00
Barbara Miller
0649751456
--no-cache-dir, --no-playlist
2021-01-04 17:53:46 -08:00
Kristinn Sigurðsson
adac067ea7
Update link to robots.txt meta documentation
2020-09-02 12:15:02 +00:00
Neil Minton
2505effadc
Strip quotes from URL value.
2020-09-01 10:57:58 -04:00
Andrew Jackson
33a3f9a58f
Updated changelog based on master.
2020-06-01 14:00:59 +01:00
Alex Osborne
2e6b76d517
Merge pull request #333 from dengliming/patch-1
...
Remove deprecated sudo setting.
2020-06-01 14:05:02 +09:00
Alex Osborne
c4644d4e5c
Merge pull request #335 from internetarchive/fixes-extractor-multiple-regex-matcher-recycle
...
Fixes extractor multiple regex matcher recycle
2020-06-01 14:03:11 +09:00
Adam Miller
60daef90b2
Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle
2020-05-20 16:46:14 +00:00
Adam Miller
c0fe2064a1
Merge branch 'fixes-ftp-response-record-builder'
2020-05-20 16:32:36 +00:00
Andrew Jackson
87c349fe13
Re-sync change log.
2020-05-18 15:28:56 +01:00
Andrew Jackson
e215b89963
[maven-release-plugin] prepare for next development iteration
2020-05-18 14:08:12 +01:00
Andrew Jackson
ef38dc08c3
[maven-release-plugin] prepare release 3.4.0-20200518
2020-05-18 14:08:04 +01:00
Andrew Jackson
176f796beb
Updated changelog.
2020-05-18 14:05:26 +01:00
dengliming
b4962d25a8
Remove deprecated sudo setting.
2020-05-12 22:53:40 +08:00
Adam Miller
0db661fb36
Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor
2020-05-05 23:09:53 +00:00
Alex Osborne
b84d443f00
Merge pull request #328 from morokosi/fix-matchesregexdeciderule
...
Fix match result is always false in MatchesListRegexDecideRule
2020-05-01 12:56:44 +09:00
morokosi
13ba0b2b67
optimize imports
2020-04-30 22:40:00 +09:00
morokosi
18bee39a4c
fix discarded future value
2020-04-30 22:39:51 +09:00
Alex Osborne
7764f9265f
Merge pull request #320 from bnfleb/sftp
...
Add support for the SFTP protocol
2020-04-24 08:13:39 +09:00
Alex Osborne
8dae196099
Merge pull request #323 from clawia/master
...
Add parsing for HTML tags (data-*)
2020-04-24 07:09:21 +09:00
Leslie Bellony
63ca9d4c6a
Remove static fields and byte loop + rename variable
2020-04-23 17:10:42 +02:00
Leslie Bellony
aad54c3cb3
rename variable + javadoc
2020-04-23 15:43:30 +02:00
Alex Osborne
7a6ea789db
Merge pull request #326 from clawia/crawl-status-and-report
...
Add real crawlStatus in the crawlReport
2020-04-22 23:51:36 +09:00
Clara Wiatrowski
5e948883fa
Add overrides in JerichoExtractorHTMLTest
2020-04-22 15:33:23 +02:00
Clara Wiatrowski
0d1f4ea84f
Add parsing case data-*
2020-04-22 11:48:05 +02:00
Clara Wiatrowski
cc29bb97d1
Revert ExtractorHTML
2020-04-22 11:33:24 +02:00
Clara Wiatrowski
eea3914059
Add missing import
2020-04-21 16:00:21 +02:00
Neil Minton
70a77d931a
Merge pull request #325 from internetarchive/yt-dl-format
...
youtube-dl: request best medium-ish size format
2020-04-20 16:50:24 -04:00
Alex Osborne
f19dacfe67
Document GET /engine
2020-04-20 18:32:43 +09:00
Alex Osborne
f442f76784
Add missing newline
2020-04-20 12:47:17 +09:00
Alex Osborne
33e637b887
Correct indentation to 3-spaces for reST
2020-04-20 12:40:51 +09:00
Alex Osborne
75dc16e8ca
Document 'Get Job Status' endpoint
2020-04-20 12:35:49 +09:00
Clara Wiatrowski
b61f7e574e
Add real crawlStatus in the crawlReport
2020-04-17 14:53:00 +02:00
Clara Wiatrowski
6710c8dba3
Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set)
2020-04-10 15:59:50 +02:00
Barbara Miller
0a2f57fa73
best medium-ish size
2020-04-03 16:18:07 -07:00
Leslie Bellony
40725372d6
Add support for the SFTP protocol
2020-04-01 14:47:34 +02:00
Adam Miller
a384dea2f9
Recycle the Matcher after use.
2020-03-25 22:33:06 +00:00
Andrew Jackson
fc1de30668
Updated changelog post release
2020-03-04 23:21:15 +00:00
Andrew Jackson
ff05de44b6
[maven-release-plugin] prepare for next development iteration
2020-03-04 23:05:41 +00:00
Andrew Jackson
a1bdcb1be7
[maven-release-plugin] prepare release 3.4.0-20200304
2020-03-04 23:05:33 +00:00
Andrew Jackson
384da2e82f
Disable doclint
2020-03-04 23:03:47 +00:00
Andrew Jackson
799c58e291
Merge branch 'master' of github.com:internetarchive/heritrix3
2020-03-04 22:43:24 +00:00
Alex Osborne
fd514b7bf6
Merge pull request #315 from ukwa/fix-oracle-doc-link
...
Use the Wayback Machine to repair a link to Oracle docs.
2020-03-05 01:40:39 +09:00
Alex Osborne