lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-22 13:46:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,136 Commits 38 Branches 33 Tags
6710c8dba38cf8fce6281e36fc6ae0dd1bece5ed
Commit Graph
2136 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Clara Wiatrowski 6710c8dba3 Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set) 2020-04-10 15:59:50 +02:00
Andrew Jackson fc1de30668 Updated changelog post release 2020-03-04 23:21:15 +00:00
Andrew Jackson ff05de44b6 [maven-release-plugin] prepare for next development iteration 2020-03-04 23:05:41 +00:00
Andrew Jackson a1bdcb1be7 [maven-release-plugin] prepare release 3.4.0-20200304 3.4.0-20200304 2020-03-04 23:05:33 +00:00
Andrew Jackson 384da2e82f Disable doclint 2020-03-04 23:03:47 +00:00
Andrew Jackson 799c58e291 Merge branch 'master' of github.com:internetarchive/heritrix3 2020-03-04 22:43:24 +00:00
Alex Osborne fd514b7bf6 Merge pull request #315 from ukwa/fix-oracle-doc-link 
Use the Wayback Machine to repair a link to Oracle docs.
 2020-03-05 01:40:39 +09:00
Andy Jackson 9f7887a5fa Merge pull request #314 from hennekey/parameterize-bloom-filter 
Utilize the `d` parameter
 2020-03-04 16:36:17 +00:00
Andrew Jackson d5fad01122 Re-sync of changelog prior to release. 2020-03-04 16:21:21 +00:00
Andrew Jackson b4494a7ffb Use the Wayback Machine to repair a link to Oracle docs. 2020-03-04 15:42:29 +00:00
Tim Hennekey 348b5330bc Utilize the d parameter 
There are uses of this class that necessitate a lower false positive
rate than the value that was hard-coded. By making use of the parameter,
as indicated in the JavaDoc, the resulting delegate should meet the
demands of the caller.
 2020-03-04 09:37:31 -05:00
Andy Jackson e0c82f70e9 Merge pull request #289 from netarchivesuite/bdb-frontier-access 
Bdb frontier access
 2020-03-04 13:41:27 +00:00
Andy Jackson 7f80b4fd98 Merge pull request #281 from ukwa/upgrade-bdb-je 
WIP: Upgrade BDB JE to version 7.5.11
 2020-03-04 13:36:03 +00:00
Alex Osborne 977f608c74 Merge pull request #290 from netarchivesuite/crawltrap-regex-timeout 
Crawltrap regex timeout
 2020-03-03 19:27:42 +09:00
Colin Rosenthal e54258756a Fixed regex timeout handling following suggestion https://github.com/internetarchive/heritrix3/pull/290#discussion_r366711640 2020-03-03 09:02:07 +01:00
Colin Rosenthal 60e4053180 Commented this test back in to make Travis happy 2020-03-03 08:52:09 +01:00
Alex Osborne 66535370ff Merge pull request #312 from internetarchive/guava-dep 
Exclude hbase-client's guava 12 transitive dependency
 2020-02-28 10:16:55 +09:00
Alex Osborne 6fcc5e808a Exclude hbase-client's guava 12 transitive dependency 
Guava 12 from hbase-client is closer to the root of the dependency tree than guava 17 from webarchive-commons so Maven prefers it. But recent changes to heritrix-commons rely on classes in the newer version of Guava. So let's ensure webarchive-commons wins.

Hopefully this doesn't break the hbase module, I have no way of testing it.

Fixes #311
 2020-02-13 16:59:13 +09:00
Jonathan Leitschuh bc6a15b41d Use HTTPS to resolve dependencies in Maven Build where possible 2020-02-11 17:48:20 +09:00
Alex Osborne 015c5e7f80 Merge pull request #308 from ldko/fix-restlet-errors 
Fix stream closed exception for Paged view
 2020-01-31 09:33:26 +09:00
Lauren Ko 8c1c8009c6 Fix stream closed exception for Paged view 2020-01-30 16:48:38 -06:00
Alex Osborne b45f774066 Merge pull request #300 from hennekey/fix-299 
Use Guice instead of custom implementation
 2020-01-30 19:12:05 +09:00
Alex Osborne fd9136dbfa Merge pull request #304 from hennekey/remove-custom-base32 
Replace custom Base32 encoding
 2020-01-30 19:07:09 +09:00
Tim Hennekey 54e05a7864 Correct encoding 
The previous implementation appears to always have returned upper case,
was able to encode either case, and did not reutrn padding.
 2020-01-28 15:12:54 -05:00
Alex Osborne 3528c990df Merge pull request #303 from hennekey/refactor-history 
Replace constant with accessor methods
 2020-01-28 14:49:00 +09:00
Alex Osborne 00fd8c2960 Merge pull request #306 from internetarchive/fix-stream-closed 
Fix stream closed exception by not closing output stream
 2020-01-28 02:17:07 +09:00
Alex Osborne a4ce59dbd6 Fix stream closed exception by not closing output stream 
ServerCall.writeResponseBody() flushes it after we return so it must
remain open.

Fixes #305
 2020-01-25 08:25:59 +09:00
Tim Hennekey b5f95c5e06 Replace custom Base32 encoding 
Guava is available so a custom implementation is unnecessary.
 2020-01-24 12:46:37 -05:00
Tim Hennekey 3003a8e751 Replace constant with accessor methods 
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
 2020-01-24 12:28:11 -05:00
Alex Osborne 9927e1325f Merge pull request #296 from hennekey/update-junit 
Set JUnit version to latest
 2020-01-23 16:44:59 +09:00
Barbara Miller 8ffcfa9020 Merge pull request #302 from nlevitt/ydl-streaming 
limit ExtractorYoutubeDL heap usage
 2020-01-22 15:08:26 -08:00
Barbara Miller dbade18b58 Merge pull request #301 from nlevitt/fix-logging-config 
fix logging config
 2020-01-22 14:42:22 -08:00
Tim Hennekey 9cb9563da3 Increment the count only when the filter notes it 
Otherwise this is a count of how many times this add method is called,
not how many times an element was noted as being actually added.
 2020-01-22 17:15:24 -05:00
Tim Hennekey 33458f1518 Fix assertions 
By using assertEquals and seting the expected and actual values, the
failure messages become a bit more useful.
 2020-01-22 17:13:58 -05:00
Noah Levitt 6667186826 java 8 compatibility 2020-01-22 14:12:36 -08:00
Noah Levitt 9229a51dcb limit ExtractorYoutubeDL heap usage 
We were seeing OOME due to large youtoube-dl json (for playlists and
such). So instead of storing the json in ram, stream through it, and
stash the contents in an thread-local anonymous tempfile so it can be
written to to warc.
 2020-01-22 13:49:24 -08:00
Noah Levitt 784b3a5dec fix logging config 
by setting system property java.util.logging.config.file, because new
version restlet reconfigures logging after heritrix has already
configured it
 2020-01-22 13:39:24 -08:00
Tim Hennekey 66723aea68 Remove version 
Allow the parent POM to specify the version
 2020-01-21 10:52:15 -05:00
Alex Osborne a8c41aa014 Merge pull request #298 from hennekey/fix-297 
Speed up ObjectIdentityBdbManualCacheTest
 2020-01-16 15:42:06 +09:00
Tim Hennekey 080380b18a Use Guice instead of custom implementation 
This uses the avaialable code in Guice rather than a custom
implementation. It also provides a performance increase (as demonstrated
by the unit tests)
 2020-01-15 18:13:20 -05:00
jkafader 739cf7fdb3 Merge pull request #287 from nlevitt/trough-dedup-fix 
change trough dedup `date` type to varchar.
 2020-01-15 11:21:06 -08:00
Noah Levitt 00a1f2c9b0 Merge pull request #294 from internetarchive/fix-config-post 
Fix 'Method Not Allowed' on POST of config editor form
 2020-01-15 11:01:00 -08:00
Noah Levitt ccbf678f4a Merge pull request #295 from internetarchive/disable-wbm-test 
Disable test that connects to wwwb-dedup.us.archive.org
 2020-01-15 10:56:31 -08:00
Tim Hennekey 0fa87a36e8 Add default constructor to IdentityCacheWrapper 
Kryo is spending a lot of time during serialization handling a
NoSuchMethodException. By adding this default constructor we can skip
doing all that work.
 2020-01-15 13:39:27 -05:00
Tim Hennekey 55308ba90a Move IdentityCacheableWrapper 
This is utilized only in tests so it belongs there
 2020-01-15 13:39:27 -05:00
Tim Hennekey 0fe8463e81 Remove AtomicInteger loop counter 
This test isn't multithreaded so I see no need for this
 2020-01-15 13:39:27 -05:00
Tim Hennekey 611fa325df Remove unused code 
It reduces the clarity of the surrounding code
 2020-01-15 13:39:27 -05:00
Tim Hennekey bb61aa0bcb Fix unchecked warning 2020-01-15 13:39:27 -05:00
Tim Hennekey 4be3e620c3 Set JUnit version to latest 
Updating JUnit to begin updating the unit tests
 2020-01-15 11:08:31 -05:00
Alex Osborne 615578b22d Add IntelliJ IDEA and Heritrix runtime files to .gitignore 
I've seen a few people accidentally commit them.
 2020-01-15 18:19:25 +09:00