37ce8d6945
Merge pull request #366 from ato/upstream-spring-5.3.3
...
Upgrade to Spring 5.3.3
2021-02-23 15:07:59 +09:00
35bb3286f4
Clarify robots protocol support
...
Closes #351 , #353
2021-02-15 17:29:49 +09:00
b10b33821f
Fix broken robots.txt link
2021-02-15 17:12:04 +09:00
5f52a14afc
Upgrade to Spring 5.3.3
...
Spring 3.0.5 is 10 years old now and is incompatible with classes using
Java 9+ bytecode and even on Java 8 classes using lambdas (see #337 ).
We remove spring-asm as it is now part of spring-core module.
2021-02-15 16:20:00 +09:00
5a64141302
Merge pull request #352 from BitBaron/absolute-meta-urls
...
Strip quotes from URL value.
2021-02-15 15:48:50 +09:00
ba0d51882a
Merge pull request #359 from internetarchive/ait-ydl-updates
...
ait youtube-dl options
2021-02-15 15:41:48 +09:00
0649751456
--no-cache-dir, --no-playlist
2021-01-04 17:53:46 -08:00
adac067ea7
Update link to robots.txt meta documentation
2020-09-02 12:15:02 +00:00
2505effadc
Strip quotes from URL value.
2020-09-01 10:57:58 -04:00
33a3f9a58f
Updated changelog based on master.
2020-06-01 14:00:59 +01:00
2e6b76d517
Merge pull request #333 from dengliming/patch-1
...
Remove deprecated sudo setting.
2020-06-01 14:05:02 +09:00
c4644d4e5c
Merge pull request #335 from internetarchive/fixes-extractor-multiple-regex-matcher-recycle
...
Fixes extractor multiple regex matcher recycle
2020-06-01 14:03:11 +09:00
60daef90b2
Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle
2020-05-20 16:46:14 +00:00
c0fe2064a1
Merge branch 'fixes-ftp-response-record-builder'
2020-05-20 16:32:36 +00:00
87c349fe13
Re-sync change log.
2020-05-18 15:28:56 +01:00
e215b89963
[maven-release-plugin] prepare for next development iteration
2020-05-18 14:08:12 +01:00
ef38dc08c3
[maven-release-plugin] prepare release 3.4.0-20200518
2020-05-18 14:08:04 +01:00
176f796beb
Updated changelog.
2020-05-18 14:05:26 +01:00
b4962d25a8
Remove deprecated sudo setting.
2020-05-12 22:53:40 +08:00
0db661fb36
Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor
2020-05-05 23:09:53 +00:00
b84d443f00
Merge pull request #328 from morokosi/fix-matchesregexdeciderule
...
Fix match result is always false in MatchesListRegexDecideRule
2020-05-01 12:56:44 +09:00
13ba0b2b67
optimize imports
2020-04-30 22:40:00 +09:00
18bee39a4c
fix discarded future value
2020-04-30 22:39:51 +09:00
7764f9265f
Merge pull request #320 from bnfleb/sftp
...
Add support for the SFTP protocol
2020-04-24 08:13:39 +09:00
8dae196099
Merge pull request #323 from clawia/master
...
Add parsing for HTML tags (data-*)
2020-04-24 07:09:21 +09:00
63ca9d4c6a
Remove static fields and byte loop + rename variable
2020-04-23 17:10:42 +02:00
aad54c3cb3
rename variable + javadoc
2020-04-23 15:43:30 +02:00
7a6ea789db
Merge pull request #326 from clawia/crawl-status-and-report
...
Add real crawlStatus in the crawlReport
2020-04-22 23:51:36 +09:00
5e948883fa
Add overrides in JerichoExtractorHTMLTest
2020-04-22 15:33:23 +02:00
0d1f4ea84f
Add parsing case data-*
2020-04-22 11:48:05 +02:00
cc29bb97d1
Revert ExtractorHTML
2020-04-22 11:33:24 +02:00
eea3914059
Add missing import
2020-04-21 16:00:21 +02:00
70a77d931a
Merge pull request #325 from internetarchive/yt-dl-format
...
youtube-dl: request best medium-ish size format
2020-04-20 16:50:24 -04:00
f19dacfe67
Document GET /engine
2020-04-20 18:32:43 +09:00
f442f76784
Add missing newline
2020-04-20 12:47:17 +09:00
33e637b887
Correct indentation to 3-spaces for reST
2020-04-20 12:40:51 +09:00
75dc16e8ca
Document 'Get Job Status' endpoint
2020-04-20 12:35:49 +09:00
b61f7e574e
Add real crawlStatus in the crawlReport
2020-04-17 14:53:00 +02:00
6710c8dba3
Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set)
2020-04-10 15:59:50 +02:00
0a2f57fa73
best medium-ish size
2020-04-03 16:18:07 -07:00
40725372d6
Add support for the SFTP protocol
2020-04-01 14:47:34 +02:00
a384dea2f9
Recycle the Matcher after use.
2020-03-25 22:33:06 +00:00
fc1de30668
Updated changelog post release
2020-03-04 23:21:15 +00:00
ff05de44b6
[maven-release-plugin] prepare for next development iteration
2020-03-04 23:05:41 +00:00
a1bdcb1be7
[maven-release-plugin] prepare release 3.4.0-20200304
2020-03-04 23:05:33 +00:00
384da2e82f
Disable doclint
2020-03-04 23:03:47 +00:00
799c58e291
Merge branch 'master' of github.com:internetarchive/heritrix3
2020-03-04 22:43:24 +00:00
fd514b7bf6
Merge pull request #315 from ukwa/fix-oracle-doc-link
...
Use the Wayback Machine to repair a link to Oracle docs.
2020-03-05 01:40:39 +09:00
9f7887a5fa
Merge pull request #314 from hennekey/parameterize-bloom-filter
...
Utilize the `d` parameter
2020-03-04 16:36:17 +00:00
d5fad01122
Re-sync of changelog prior to release.
2020-03-04 16:21:21 +00:00
Alex Osborne