b84d443f00
Merge pull request #328 from morokosi/fix-matchesregexdeciderule
...
Fix match result is always false in MatchesListRegexDecideRule
2020-05-01 12:56:44 +09:00
13ba0b2b67
optimize imports
2020-04-30 22:40:00 +09:00
18bee39a4c
fix discarded future value
2020-04-30 22:39:51 +09:00
7764f9265f
Merge pull request #320 from bnfleb/sftp
...
Add support for the SFTP protocol
2020-04-24 08:13:39 +09:00
8dae196099
Merge pull request #323 from clawia/master
...
Add parsing for HTML tags (data-*)
2020-04-24 07:09:21 +09:00
63ca9d4c6a
Remove static fields and byte loop + rename variable
2020-04-23 17:10:42 +02:00
aad54c3cb3
rename variable + javadoc
2020-04-23 15:43:30 +02:00
7a6ea789db
Merge pull request #326 from clawia/crawl-status-and-report
...
Add real crawlStatus in the crawlReport
2020-04-22 23:51:36 +09:00
5e948883fa
Add overrides in JerichoExtractorHTMLTest
2020-04-22 15:33:23 +02:00
0d1f4ea84f
Add parsing case data-*
2020-04-22 11:48:05 +02:00
cc29bb97d1
Revert ExtractorHTML
2020-04-22 11:33:24 +02:00
eea3914059
Add missing import
2020-04-21 16:00:21 +02:00
70a77d931a
Merge pull request #325 from internetarchive/yt-dl-format
...
youtube-dl: request best medium-ish size format
2020-04-20 16:50:24 -04:00
f19dacfe67
Document GET /engine
2020-04-20 18:32:43 +09:00
f442f76784
Add missing newline
2020-04-20 12:47:17 +09:00
33e637b887
Correct indentation to 3-spaces for reST
2020-04-20 12:40:51 +09:00
75dc16e8ca
Document 'Get Job Status' endpoint
2020-04-20 12:35:49 +09:00
b61f7e574e
Add real crawlStatus in the crawlReport
2020-04-17 14:53:00 +02:00
6710c8dba3
Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set)
2020-04-10 15:59:50 +02:00
0a2f57fa73
best medium-ish size
2020-04-03 16:18:07 -07:00
40725372d6
Add support for the SFTP protocol
2020-04-01 14:47:34 +02:00
fc1de30668
Updated changelog post release
2020-03-04 23:21:15 +00:00
ff05de44b6
[maven-release-plugin] prepare for next development iteration
2020-03-04 23:05:41 +00:00
a1bdcb1be7
[maven-release-plugin] prepare release 3.4.0-20200304
2020-03-04 23:05:33 +00:00
384da2e82f
Disable doclint
2020-03-04 23:03:47 +00:00
799c58e291
Merge branch 'master' of github.com:internetarchive/heritrix3
2020-03-04 22:43:24 +00:00
fd514b7bf6
Merge pull request #315 from ukwa/fix-oracle-doc-link
...
Use the Wayback Machine to repair a link to Oracle docs.
2020-03-05 01:40:39 +09:00
9f7887a5fa
Merge pull request #314 from hennekey/parameterize-bloom-filter
...
Utilize the `d` parameter
2020-03-04 16:36:17 +00:00
d5fad01122
Re-sync of changelog prior to release.
2020-03-04 16:21:21 +00:00
b4494a7ffb
Use the Wayback Machine to repair a link to Oracle docs.
2020-03-04 15:42:29 +00:00
348b5330bc
Utilize the d parameter
...
There are uses of this class that necessitate a lower false positive
rate than the value that was hard-coded. By making use of the parameter,
as indicated in the JavaDoc, the resulting delegate should meet the
demands of the caller.
2020-03-04 09:37:31 -05:00
e0c82f70e9
Merge pull request #289 from netarchivesuite/bdb-frontier-access
...
Bdb frontier access
2020-03-04 13:41:27 +00:00
7f80b4fd98
Merge pull request #281 from ukwa/upgrade-bdb-je
...
WIP: Upgrade BDB JE to version 7.5.11
2020-03-04 13:36:03 +00:00
977f608c74
Merge pull request #290 from netarchivesuite/crawltrap-regex-timeout
...
Crawltrap regex timeout
2020-03-03 19:27:42 +09:00
e54258756a
Fixed regex timeout handling following suggestion https://github.com/internetarchive/heritrix3/pull/290#discussion_r366711640
2020-03-03 09:02:07 +01:00
60e4053180
Commented this test back in to make Travis happy
2020-03-03 08:52:09 +01:00
66535370ff
Merge pull request #312 from internetarchive/guava-dep
...
Exclude hbase-client's guava 12 transitive dependency
2020-02-28 10:16:55 +09:00
6fcc5e808a
Exclude hbase-client's guava 12 transitive dependency
...
Guava 12 from hbase-client is closer to the root of the dependency tree than guava 17 from webarchive-commons so Maven prefers it. But recent changes to heritrix-commons rely on classes in the newer version of Guava. So let's ensure webarchive-commons wins.
Hopefully this doesn't break the hbase module, I have no way of testing it.
Fixes #311
2020-02-13 16:59:13 +09:00
bc6a15b41d
Use HTTPS to resolve dependencies in Maven Build where possible
2020-02-11 17:48:20 +09:00
015c5e7f80
Merge pull request #308 from ldko/fix-restlet-errors
...
Fix stream closed exception for Paged view
2020-01-31 09:33:26 +09:00
8c1c8009c6
Fix stream closed exception for Paged view
2020-01-30 16:48:38 -06:00
b45f774066
Merge pull request #300 from hennekey/fix-299
...
Use Guice instead of custom implementation
2020-01-30 19:12:05 +09:00
fd9136dbfa
Merge pull request #304 from hennekey/remove-custom-base32
...
Replace custom Base32 encoding
2020-01-30 19:07:09 +09:00
54e05a7864
Correct encoding
...
The previous implementation appears to always have returned upper case,
was able to encode either case, and did not reutrn padding.
2020-01-28 15:12:54 -05:00
3528c990df
Merge pull request #303 from hennekey/refactor-history
...
Replace constant with accessor methods
2020-01-28 14:49:00 +09:00
00fd8c2960
Merge pull request #306 from internetarchive/fix-stream-closed
...
Fix stream closed exception by not closing output stream
2020-01-28 02:17:07 +09:00
a4ce59dbd6
Fix stream closed exception by not closing output stream
...
ServerCall.writeResponseBody() flushes it after we return so it must
remain open.
Fixes #305
2020-01-25 08:25:59 +09:00
b5f95c5e06
Replace custom Base32 encoding
...
Guava is available so a custom implementation is unnecessary.
2020-01-24 12:46:37 -05:00
3003a8e751
Replace constant with accessor methods
...
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
2020-01-24 12:28:11 -05:00
9927e1325f
Merge pull request #296 from hennekey/update-junit
...
Set JUnit version to latest
2020-01-23 16:44:59 +09:00
Alex Osborne