lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-23 06:56:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,998 Commits 40 Branches 31 Tags
303ffb486e7c331ea4b7eb872c791f800160b2c2
Commit Graph
1998 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 303ffb486e close the rethinkdb connection! 2019-04-10 10:43:22 -07:00
Noah Levitt fc16e4ec3e Merge branch 'master' into trough-dedup 
* master:
  replace System.err.println with logger.info
  Revert "Upgrade httpclient to 4.5.7 and handle cookies more compliantly"
  Removing outdated test.
  Disable questionalbe test.
  Avoid deprecated flag.
  Supply an iterator, for #245
  Updated POM to use latest version.
  Update README.md
  Handle missing closing paren in srcset descriptor
  Teach jericho extractor srcset
  Don't run srcset test against jericho, it doesn't handle it
  Handle commas more compliantly when parsing srcset
  Ensure we start parsing full lines, for #239.
 2019-04-10 10:16:45 -07:00
Noah Levitt 2c6ab5a44a replace System.err.println with logger.info 2019-04-04 17:06:01 -07:00
Noah Levitt 5f5d4615c2 fix some trough dedup bugs 
especially this:

-    writeUrlCache.remove("segmentId");
+    writeUrlCache.remove(segmentId);

and some improvements and tweaks
 2019-04-01 16:14:22 -07:00
Alex Osborne b68731e2b9 Merge pull request #248 from internetarchive/revert-246-upgrade-httpclient 
Revert "Upgrade httpclient to 4.5.7 and handle cookies more compliantly"
 2019-03-28 10:19:22 +09:00
Alex Osborne dd37598470 Revert "Upgrade httpclient to 4.5.7 and handle cookies more compliantly" 2019-03-28 10:14:42 +09:00
Noah Levitt 954043cdf3 Merge pull request #246 from ukwa/upgrade-httpclient 
Upgrade httpclient to 4.5.7 and handle cookies more compliantly
 2019-03-22 10:38:20 -07:00
jkafader 1099c152cf Merge pull request #242 from nlevitt/trough-dedup 
Trough dedup
 2019-03-21 15:49:17 -07:00
Noah Levitt c596f49c8a use constant from rethinkdb lib for default port 2019-03-21 15:34:42 -07:00
Andrew Jackson 03da8c06cf Merge branch 'master' into upgrade-httpclient 2019-03-21 00:06:10 +00:00
Andrew Jackson 044f068d1b Removing outdated test. 2019-03-21 00:06:05 +00:00
Andrew Jackson 629a7adcb6 Disable questionalbe test. 2019-03-20 22:02:58 +00:00
Andrew Jackson 2fbe603878 Avoid deprecated flag. 2019-03-20 22:02:35 +00:00
Andrew Jackson e910bb6a5b Supply an iterator, for #245 2019-03-20 21:18:19 +00:00
Andrew Jackson 0235a0e553 Updated POM to use latest version. 2019-03-20 20:26:51 +00:00
Noah Levitt fd0e86cd64 less alarming logging for normal situation 2019-03-20 11:31:40 -07:00
Adam Miller dff29b1685 Merge pull request #243 from internetarchive/srcset 
Handle commas more compliantly when parsing srcset
 2019-03-19 09:53:23 -07:00
Noah Levitt c4ec9d31fe make TroughCrawlLogFeed use TroughClient, and... 
... configure using rethinkdb url and segment id, instead of write url,
which means it can work if the segment gets reassigned and so forth
***backward incompatible change***
 2019-03-18 22:39:29 -07:00
Alex Osborne fcc3bc2940 Merge pull request #244 from mikeizbicki/patch-1 
Update README.md
 2019-03-19 12:41:25 +09:00
Mike Izbicki 102b5086eb Update README.md 
Fix typo in link
 2019-03-18 20:26:15 -07:00
Alex Osborne 7d91a1d4ee Handle missing closing paren in srcset descriptor 2019-03-16 16:02:48 +09:00
Alex Osborne 2a34fcffd6 Teach jericho extractor srcset 2019-03-16 15:47:50 +09:00
Alex Osborne e1d93e3308 Don't run srcset test against jericho, it doesn't handle it 2019-03-16 12:35:57 +09:00
Alex Osborne 90c52c3a25 Handle commas more compliantly when parsing srcset 
Commas are allowed if they're in the middle of the URL. Consequently:

    srcset="a,b,,c,"   => ["a,b,,c"]
    srcset="a, b,, c," => ["a", "b", "c"]

They occur particularly commonly in data: URLs before the base64 value.

Commas are also allowed in descriptors if they are enclosed by parens:

    srcset="a (b,c),d" => ["a", "d"]

Spec: https://html.spec.whatwg.org/multipage/images.html#parsing-a-srcset-attribute
 2019-03-16 11:53:37 +09:00
Noah Levitt af1a325319 fix bugs around crawl finish 
- make sure we "promote" segments (persist them to hdfs) after we're
  entirely finished writing to theme; in other words, when we reach the
  FINISH state, not while STOPPING
- shut down the promoter thread at crawl finish

plus a couple of tweaks and cleanups
 2019-03-15 14:12:50 -07:00
Noah Levitt 7f7fb3cfe8 allow travis-ci jdk7 failures because contrib... 
... needs java 8 now, and travis-ci builds contrib
 2019-03-14 16:58:22 -07:00
Noah Levitt 82406f4028 promote dirty segments at crawl finish 2019-03-14 15:16:56 -07:00
Noah Levitt 1b89420f99 trough dedup! 2019-03-14 13:47:15 -07:00
Andrew Jackson afa88c025e Ensure we start parsing full lines, for #239. 2019-03-13 21:00:19 +09:00
Noah Levitt c40616e494 Merge pull request #235 from ukwa/retry-failed-dns 
Allow failed lookups to expire, for #234.
 2019-02-25 14:15:38 -08:00
Andrew Jackson 91b18e9bbd Allow failed lookups to expire, for #234. 2019-02-21 23:04:40 +00:00
Andrew Jackson ad9e74d8ad [maven-release-plugin] prepare for next development iteration 2019-02-07 13:52:56 +00:00
Andrew Jackson 83c7044b22 [maven-release-plugin] prepare release 3.4.0-20190207 3.4.0-20190207 2019-02-07 13:52:49 +00:00
Andy Jackson 18d3bb23aa Merge pull request #231 from ukwa/synchronized-enqueuecounts 
Add synchronized statements for internetarchive/heritrix3#221.
 2019-02-07 13:49:50 +00:00
Andy Jackson cb96a44bdc Merge pull request #230 from ukwa/more-robust-fetch-history-processor 
Add checks to guard against server sending 304 in error
 2019-02-07 13:49:23 +00:00
Andrew Jackson 667cf3ac5e As @nlevitt suggestion, a further check. 2019-02-06 21:03:49 +00:00
Andrew Jackson e8e37751bb Merge branch 'master' into more-robust-fetch-history-processor 2019-02-06 20:47:06 +00:00
Andrew Jackson 317b19e7e2 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:33:08 +00:00
Andrew Jackson 9c7d0c299d [maven-release-plugin] prepare release 3.4.0-20190205 3.4.0-20190205 2019-02-05 12:32:57 +00:00
Andrew Jackson ab34a54566 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:29:33 +00:00
Andrew Jackson d1fc5cedb4 [maven-release-plugin] prepare release 3.4.0-20190205-2 2019-02-05 12:29:26 +00:00
Andrew Jackson 096ba4e9c3 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:15:26 +00:00
Andrew Jackson 24ef7eb9d0 Extend to set additional property. 2019-02-05 12:14:17 +00:00
Andrew Jackson 947bb39aff Use argument syntax. 2019-02-05 12:13:20 +00:00
Andrew Jackson 0e1a21c9d8 Skip tests during release process (covered by CI). 2019-02-05 12:11:29 +00:00
Andrew Jackson 765d607e5c Set consistent tag, and include contrib. 2019-02-05 11:57:23 +00:00
Andrew Jackson 24b72e07df Wrong repo spec. 2019-02-05 11:23:18 +00:00
Andrew Jackson fae30ed63c Add build profile for deployment to Maven Central. 2019-02-05 10:32:40 +00:00
Andrew Jackson 1cb399ec18 Avoid headings being treated as lists 2019-02-05 10:32:13 +00:00
Andy Jackson d0a146bc66 Clarification about APIs. 2019-02-05 10:28:18 +00:00