lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-24 06:36:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,305 Commits 38 Branches 33 Tags
e33e1ff398f7b89dfce053506069ed2cbabfde8c
Commit Graph
821 Commits
Author SHA1 Message Date
Alex Osborne e33e1ff398 Extractor: ignore data URIs when adding outlinks 2021-07-30 15:57:09 +09:00
Alex Osborne 91ef23eddf ExtractorPDFContext, ExtractorYoutubeDL: use addOutlink() helper method 2021-07-30 14:25:34 +09:00
Alex Osborne 09e3ff3568 Everywhere: Fix dangling JavaDoc comments 
JavaDoc comments need to be directly above a class, method or field
declaration to be used by IDEs and generated documentation. We had some
doc comments that were attached to instance initialization blocks and
thus were being ignored. This invalid doc comment positioning was
presumably an accidental consequence of the conversion of fields to
KeyedProperties.

This change moves most of the dangling doc comments to setter methods
where it can be seen by tools. There were a couple of dangling doc
comments above package statements. These are moved above the class
declaration or removed entirely when empty.
 2021-07-12 21:38:48 +09:00
Adam Miller ca3397e346 Fixing race condition on totalBytesWritten 2021-07-03 12:10:21 +09:00
Adam Miller 111b77b4aa Add checkpoint store/load functionality for warc writer stats 2021-07-03 12:10:21 +09:00
Alex Osborne 200f1cf38f DNS: Use the new protocol-agnostic server IP attribute 
Also fixes an issue where in some cases we would incorrectly output
WARC-IP-Address for DNS records as "/127.0.0.53:53" instead of
"127.0.0.53".
 2021-06-25 16:39:59 +09:00
Alex Osborne c378c46e49 WHOIS: Replace A_WHOIS_SERVER_IP with the generic A_SERVER_IP one 2021-06-25 14:39:26 +09:00
Alex Osborne 6b42c608a5 BaseWARCRecordBuilder: remove now unused getHostAddress(curi) helper 
We now save the server IP at fetch time and retrieve it with
CrawlURI.getServerIP()
 2021-06-25 14:39:26 +09:00
Alex Osborne ef3b428061 FTP: Save server IP at fetch time and use in WARC-IP-Address 2021-06-25 14:39:26 +09:00
Alex Osborne 13c0f65294 FTP: Implement a basic fetch test 2021-06-25 14:09:38 +09:00
Alex Osborne 28ea9c4ba5 HTTP: Populate WARC-IP-Address with server's IP address 
Fixes #396
 2021-06-25 13:34:40 +09:00
Alex Osborne 5a6531b296 CrawlURI: Introduce a protocol-agnostic server-ip attribute 
The vast majority of protocols we use will have a server IP address,
so rather than having a separate one for each protocol let's just use
a single common one.
 2021-06-25 13:24:20 +09:00
Alex Osborne 5d0a5b2ec4 Remove dependency on mg4j 
This eliminates one of the jars that Heritrix depends on that isn't in
Maven Central.

There were only two usages. The first usage was a method in a
CrawlerJournal which as far as I can tell is unused and can be removed.

The second usage was an unnecessary use of MutableString as a line
buffer when saving cookies to a file. Since there doesn't seem any
particular reason buffer per-line here let's just use a BufferedWriter
instead on the whole file which is also slightly simpler and more
efficient.
 2021-06-20 23:30:41 +09:00
Kristinn Sigurdsson 627befe07e Remove unnecessary fiddling with VIA path 2021-06-18 08:42:05 +00:00
Kristinn Sigurdsson 81bef22270 Mark nested sitemap links as sitemaps 2021-06-16 13:07:18 +00:00
Andy Jackson c0dbfb2a0a Merge pull request #383 from ukwa/upgrade-dnsjava 
Update to latest version of dnsjava, for #344
 2021-05-20 21:05:59 +01:00
Andy Jackson 396467c535 Merge branch 'master' into sitemaps 2021-05-20 21:03:54 +01:00
Andy Jackson c1bcdd9f9e Merge pull request #341 from internetarchive/noplaylist-ydl 
youtube-dl --no-playlist
 2021-05-20 21:01:39 +01:00
Andy Jackson b2a07b290f Merge pull request #348 from internetarchive/fixes-leaky-file-handles 
Fixes leaky file handles
 2021-05-20 21:00:31 +01:00
Andy Jackson 045b2516da Update to latest version of dnsjava, for #344 
This commit updates to version 3.3.1 of the dnsjava library.

Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
 2021-05-20 20:44:19 +01:00
Kristinn Sigurdsson 52093eb25f Only copy source tag if not null. 2021-04-20 13:18:29 +00:00
Neil Minton 2505effadc Strip quotes from URL value. 2020-09-01 10:57:58 -04:00
Adam Miller 52a8f345ed Fixing up logging and comments 2020-07-30 23:30:08 +00:00
Adam Miller 00d1c46d60 Ensure Replay Input Stream and File Channels are closed after writing 2020-07-30 23:19:43 +00:00
Alex Osborne 947507b8ec Revert "Warc convention for storing ftp responses has been to use a WARC reso…" 2020-06-01 13:56:12 +09:00
Alex Osborne 4763a8a04b Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder 
Warc convention for storing ftp responses has been to use a WARC reso…
 2020-06-01 13:54:59 +09:00
Adam Miller 60daef90b2 Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle 2020-05-20 16:46:14 +00:00
Adam Miller c0fe2064a1 Merge branch 'fixes-ftp-response-record-builder' 2020-05-20 16:32:36 +00:00
Adam Miller 0db661fb36 Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor 2020-05-05 23:09:53 +00:00
morokosi 13ba0b2b67 optimize imports 2020-04-30 22:40:00 +09:00
morokosi 18bee39a4c fix discarded future value 2020-04-30 22:39:51 +09:00
Alex Osborne 7764f9265f Merge pull request #320 from bnfleb/sftp 
Add support for the SFTP protocol
 2020-04-24 08:13:39 +09:00
Leslie Bellony aad54c3cb3 rename variable + javadoc 2020-04-23 15:43:30 +02:00
Clara Wiatrowski 5e948883fa Add overrides in JerichoExtractorHTMLTest 2020-04-22 15:33:23 +02:00
Clara Wiatrowski 0d1f4ea84f Add parsing case data-* 2020-04-22 11:48:05 +02:00
Clara Wiatrowski cc29bb97d1 Revert ExtractorHTML 2020-04-22 11:33:24 +02:00
Clara Wiatrowski 6710c8dba3 Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set) 2020-04-10 15:59:50 +02:00
Leslie Bellony 40725372d6 Add support for the SFTP protocol 2020-04-01 14:47:34 +02:00
Adam Miller a384dea2f9 Recycle the Matcher after use. 2020-03-25 22:33:06 +00:00
Andy Jackson 7f80b4fd98 Merge pull request #281 from ukwa/upgrade-bdb-je 
WIP: Upgrade BDB JE to version 7.5.11
 2020-03-04 13:36:03 +00:00
Alex Osborne 977f608c74 Merge pull request #290 from netarchivesuite/crawltrap-regex-timeout 
Crawltrap regex timeout
 2020-03-03 19:27:42 +09:00
Colin Rosenthal e54258756a Fixed regex timeout handling following suggestion https://github.com/internetarchive/heritrix3/pull/290#discussion_r366711640 2020-03-03 09:02:07 +01:00
Colin Rosenthal 60e4053180 Commented this test back in to make Travis happy 2020-03-03 08:52:09 +01:00
Alex Osborne 3528c990df Merge pull request #303 from hennekey/refactor-history 
Replace constant with accessor methods
 2020-01-28 14:49:00 +09:00
Tim Hennekey 3003a8e751 Replace constant with accessor methods 
CrawlURI already had the accessor method, and the use of the constant
was a bit inconsistent. This change adds the corresponding mutator
method to make working with the CrawlURI history a bit simpler.
 2020-01-24 12:28:11 -05:00
Alex Osborne 8e875bb17f Merge pull request #288 from netarchivesuite/inline-image-filter 
Attempt to filter out embedded images.
 2020-01-15 12:48:54 +09:00
jkafader 3a8b58998c Merge pull request #285 from nlevitt/warc-writer-chain 
Warc writer chain
 2020-01-13 16:39:42 -08:00
Noah Levitt 31b9960503 basic level of documentation 2019-12-30 16:02:10 -08:00
Colin Rosenthal 204491c357 Attempt to filter out embedded images. 
(cherry picked from commit aa5ff1dbdaefd04652a9c66506d20f1a6ae01dc3)
 2019-11-19 15:04:50 +01:00
Colin Rosenthal 718cc5ae32 Updated manually to new SNAPSHOT version 2019-11-15 14:34:03 +01:00