lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-23 22:26:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,126 Commits 38 Branches 33 Tags
87566eac61bd77da02b60a7b4fa3efac1f0cf5ed
Commit Graph
2126 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Andy Jackson 87566eac61 Merge pull request #445 from internetarchive/adds-bdbmodule-max-file-size-option 
Adding configurable MaxLogFileSize option to BdbModule
 2023-03-10 11:23:06 +00:00
Adam Miller 953e7c9399 Adding configurable MaxLogFileSize option to BdbModule 2021-07-13 21:29:33 +00:00
jkafader 77e2e9171f Merge pull request #362 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats 
Fixing race condition on totalBytesWritten
 2021-01-14 12:08:10 -08:00
jkafader 2f640fc974 Merge pull request #355 from internetarchive/fixes-prerequisite-status 
Reset CrawlURI status for hasPrerequisite() so that it isn't preserve…
 2021-01-14 12:08:01 -08:00
Adam Miller 1f357c8daf Fixing race condition on totalBytesWritten 2020-11-11 00:02:09 +00:00
Adam Miller e694a4b5c8 Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts 2020-11-07 01:12:56 +00:00
jkafader 288b167399 Merge pull request #354 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats 
Add checkpoint store/load functionality for warc writer stats
 2020-10-15 13:51:09 -07:00
Adam Miller 5504d492a1 Add checkpoint store/load functionality for warc writer stats 2020-10-09 22:14:08 +00:00
jkafader 87ac0dacb2 Merge pull request #349 from BitBaron/absolute-meta-urls 
Better parsing of absolute meta element URLs.
 2020-08-17 12:38:02 -07:00
Neil Minton 941c6c50ad Strip quotes from URL value. 2020-08-14 10:54:18 -04:00
Adam Miller f879226c8e Merge pull request #342 from internetarchive/noplaylist-ydl 
youtube-dl --no-playlist
 2020-08-03 16:40:54 -07:00
jkafader 34bb996e5c Merge pull request #346 from internetarchive/fixes-leaky-file-handles 
Fixes leaky file handles
 2020-08-03 16:22:23 -07:00
Adam Miller a9c0c6588b Manage youtube dl temp files which can be closed in the warc writer. 2020-07-31 22:22:25 +00:00
Adam Miller 52a8f345ed Fixing up logging and comments 2020-07-30 23:30:08 +00:00
Adam Miller 00d1c46d60 Ensure Replay Input Stream and File Channels are closed after writing 2020-07-30 23:19:43 +00:00
Barbara Miller 1b8e7f73ca youtube-dl --no-playlist 2020-07-29 11:12:33 -07:00
Alex Osborne 812e96d2c0 Merge pull request #336 from internetarchive/revert-334-fixes-ftp-response-record-builder 
Revert "Warc convention for storing ftp responses has been to use a WARC reso…"
 2020-06-01 13:56:37 +09:00
Alex Osborne 947507b8ec Revert "Warc convention for storing ftp responses has been to use a WARC reso…" 2020-06-01 13:56:12 +09:00
Alex Osborne 4763a8a04b Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder 
Warc convention for storing ftp responses has been to use a WARC reso…
 2020-06-01 13:54:59 +09:00
Adam Miller 0db661fb36 Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor 2020-05-05 23:09:53 +00:00
jkafader ce409c03c1 Merge pull request #329 from internetarchive/ydl-no-cache 
youtube-dl no cache dir
 2020-04-30 15:03:23 -07:00
jkafader 0a5d46d927 Merge pull request #330 from internetarchive/less_youtubedl 
don't youtubedl receivedFromAMQP
 2020-04-30 14:31:18 -07:00
Barbara Miller aabd16b41e don't youtube-dl receivedFromAMQP 2020-04-30 11:06:13 -07:00
Barbara Miller 633a7cffea no youtube-dl cache dir 2020-04-27 18:15:54 -07:00
Neil Minton 0f8fbaf8f2 Merge pull request #327 from internetarchive/yt-dl-format-medium-ish 
best medium-ish size
 2020-04-20 18:20:00 -04:00
Barbara Miller 4999843ade best medium-ish size 2020-04-20 14:48:51 -07:00
jkafader ea374b3a99 Merge pull request #317 from internetarchive/fixes-extractor-multiple-regex-matcher-recycle 
Recycle the regex Matcher after use.
 2020-04-06 16:04:54 -07:00
Adam Miller a384dea2f9 Recycle the Matcher after use. 2020-03-25 22:33:06 +00:00
Alex Osborne 3528c990df Merge pull request #303 from hennekey/refactor-history 
Replace constant with accessor methods
 2020-01-28 14:49:00 +09:00
Alex Osborne 00fd8c2960 Merge pull request #306 from internetarchive/fix-stream-closed 
Fix stream closed exception by not closing output stream
 2020-01-28 02:17:07 +09:00
Alex Osborne a4ce59dbd6 Fix stream closed exception by not closing output stream 
ServerCall.writeResponseBody() flushes it after we return so it must
remain open.

Fixes #305
 2020-01-25 08:25:59 +09:00
Tim Hennekey 3003a8e751 Replace constant with accessor methods 
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
 2020-01-24 12:28:11 -05:00
Alex Osborne 9927e1325f Merge pull request #296 from hennekey/update-junit 
Set JUnit version to latest
 2020-01-23 16:44:59 +09:00
Barbara Miller 8ffcfa9020 Merge pull request #302 from nlevitt/ydl-streaming 
limit ExtractorYoutubeDL heap usage
 2020-01-22 15:08:26 -08:00
Barbara Miller dbade18b58 Merge pull request #301 from nlevitt/fix-logging-config 
fix logging config
 2020-01-22 14:42:22 -08:00
Noah Levitt 6667186826 java 8 compatibility 2020-01-22 14:12:36 -08:00
Noah Levitt 9229a51dcb limit ExtractorYoutubeDL heap usage 
We were seeing OOME due to large youtoube-dl json (for playlists and
such). So instead of storing the json in ram, stream through it, and
stash the contents in an thread-local anonymous tempfile so it can be
written to to warc.
 2020-01-22 13:49:24 -08:00
Noah Levitt 784b3a5dec fix logging config 
by setting system property java.util.logging.config.file, because new
version restlet reconfigures logging after heritrix has already
configured it
 2020-01-22 13:39:24 -08:00
Tim Hennekey 66723aea68 Remove version 
Allow the parent POM to specify the version
 2020-01-21 10:52:15 -05:00
Alex Osborne a8c41aa014 Merge pull request #298 from hennekey/fix-297 
Speed up ObjectIdentityBdbManualCacheTest
 2020-01-16 15:42:06 +09:00
jkafader 739cf7fdb3 Merge pull request #287 from nlevitt/trough-dedup-fix 
change trough dedup `date` type to varchar.
 2020-01-15 11:21:06 -08:00
Noah Levitt 00a1f2c9b0 Merge pull request #294 from internetarchive/fix-config-post 
Fix 'Method Not Allowed' on POST of config editor form
 2020-01-15 11:01:00 -08:00
Noah Levitt ccbf678f4a Merge pull request #295 from internetarchive/disable-wbm-test 
Disable test that connects to wwwb-dedup.us.archive.org
 2020-01-15 10:56:31 -08:00
Tim Hennekey 0fa87a36e8 Add default constructor to IdentityCacheWrapper 
Kryo is spending a lot of time during serialization handling a
NoSuchMethodException. By adding this default constructor we can skip
doing all that work.
 2020-01-15 13:39:27 -05:00
Tim Hennekey 55308ba90a Move IdentityCacheableWrapper 
This is utilized only in tests so it belongs there
 2020-01-15 13:39:27 -05:00
Tim Hennekey 0fe8463e81 Remove AtomicInteger loop counter 
This test isn't multithreaded so I see no need for this
 2020-01-15 13:39:27 -05:00
Tim Hennekey 611fa325df Remove unused code 
It reduces the clarity of the surrounding code
 2020-01-15 13:39:27 -05:00
Tim Hennekey bb61aa0bcb Fix unchecked warning 2020-01-15 13:39:27 -05:00
Tim Hennekey 4be3e620c3 Set JUnit version to latest 
Updating JUnit to begin updating the unit tests
 2020-01-15 11:08:31 -05:00
Alex Osborne 615578b22d Add IntelliJ IDEA and Heritrix runtime files to .gitignore 
I've seen a few people accidentally commit them.
 2020-01-15 18:19:25 +09:00