lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-20 22:41:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,144 Commits 40 Branches 31 Tags
3.2.0
Commit Graph
1144 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 484b3574d7 Merge pull request #26 from nlevitt/release-3.2.0 
update readme for 3.2.0 release
3.2.0 		2014-01-10 13:17:46 -08:00
Noah Levitt 0467304f76 update readme for 3.2.0 release 2014-01-10 13:15:48 -08:00
Eldon 752e288ef8 Merge pull request #25 from nlevitt/release-3.2.0 
bump version number to 3.2.0 for release
 2014-01-10 12:47:58 -08:00
Noah Levitt 32c083f720 bump version number to 3.2.0 for release 2014-01-10 12:36:45 -08:00
Adam Miller 9ac0ee057f Merge pull request #24 from nlevitt/url-agnostic-standard 
for url-agnostic dedup, follow "Proposal for Standardizing the Recording...
 2014-01-08 11:38:51 -08:00
Noah Levitt 84e78db40a for url-agnostic dedup, follow "Proposal for Standardizing the Recording of Arbitrary Duplicates in WARC Files" https://docs.google.com/document/d/1QyQBA7Ykgxie75V8Jziz_O7hbhwf7PF6_u9O6w6zgp0 2014-01-07 14:43:26 -08:00
ikreymer 506e68316a Merge pull request #23 from nlevitt/her-1979 
fix HER-1979 so heritrix can run on windows xp
 2013-12-18 01:12:16 -08:00
Noah Levitt 7dd67bd876 fix HER-1979 so heritrix can run on windows xp 2013-12-17 19:25:12 -08:00
Noah Levitt 2388a35a06 useProjectArtifact=false to avoid [WARNING] Cannot include project artifact: ... it doesn't have an associated file or directory 2013-12-15 11:54:36 -08:00
Noah Levitt 0ea10665f1 abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner 2013-12-15 11:53:18 -08:00
Noah Levitt 6f171b575a fix report links 2013-12-11 21:14:04 -08:00
Noah Levitt 3b06c88537 test fix (in ia-web-commons) for HER-2059 2013-12-06 19:22:16 -08:00
Noah Levitt 3c8a0a389f moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons 2013-12-05 17:48:57 -08:00
Noah Levitt 98d0901531 moving org.archive.net.PublicSuffixes to ia-web-commons 2013-12-05 16:13:30 -08:00
Noah Levitt 552794dc89 don't let the exception from commit df2a313797 derail the whole crawl job 2013-11-26 16:08:38 -08:00
Noah Levitt 1163847644 try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted 2013-11-21 18:51:16 -08:00
Noah Levitt 8e1125d062 bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson) 2013-11-20 09:18:31 -08:00
Noah Levitt d93b5449be new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc 2013-11-19 11:05:14 -08:00
Noah Levitt 7089caf3c4 remove unused engine.css 2013-11-15 19:11:19 -08:00
Noah Levitt 119e565acc Adam's responsive UI 2013-11-15 18:59:14 -08:00
Noah Levitt 2b87219496 oops, avoid npe 2013-11-15 17:59:34 -08:00
Noah Levitt df2a313797 throw exception when host cache returns 0.0.0.0 for any host (trying to track down a strange problem) 2013-11-15 17:20:55 -08:00
Ilya Kreymer e2f4b7287f Merge branch 'master' of github.com:internetarchive/heritrix3 2013-11-01 15:44:12 -07:00
Ilya Kreymer 8633ed0ec3 removed ThreadLocalHttpConnectionManager -- it has been moved to ia-web-commons 2013-11-01 15:43:46 -07:00
Noah Levitt b137022b79 Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.) 2013-10-28 17:35:55 -07:00
Noah Levitt 38a12b2008 avoid NPE resuming from checkpoint if "stats" is {} in warcWriter checkpoint json 2013-10-11 14:44:34 -07:00
Noah Levitt a1f84d3cc4 HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility 2013-10-10 17:57:18 -07:00
Noah Levitt 7ad90b3126 Merge branch 'master' of github.com:internetarchive/heritrix3 2013-09-12 12:04:32 -07:00
Noah Levitt 57311db117 use libs version ${project.version} 2013-09-10 15:24:58 -07:00
Noah Levitt 140ba3e89e tweak io block size in FileUtils.appendTo(), referencing performance metrics documented in gnu coreutils (used e.g. for /bin/cat) 2013-09-08 12:41:48 -07:00
Noah Levitt 254811ff7f commented-out default for new setting forgetAllButLatest on CheckpointService 2013-09-08 12:39:54 -07:00
Noah Levitt f096609a50 avoid infinite loop spitting out stack traces on BeansException on startup 2013-09-08 00:11:54 -07:00
Noah Levitt a00141eef7 avoid deadlock in case of BeansException at crawl startup 2013-09-08 00:10:27 -07:00
Noah Levitt 071cbe99cd include urlsWritten in checkpoint, and be backward compatible with old checkpoints 2013-09-07 23:40:17 -07:00
Noah Levitt c49b465d82 custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test 2013-09-07 19:40:11 -07:00
Noah Levitt 52d72e661d part of HER-2048 - WARCWriterProcessor stats resume from checkpoint 2013-09-07 11:35:03 -07:00
Noah Levitt e0f78976d7 part of HER-2048 - restore BdbServerCache from checkpoint 2013-09-07 10:53:22 -07:00
Noah Levitt fd71833050 avoid NPE when attempting to terminate already-finished job 2013-09-07 09:57:33 -07:00
Noah Levitt d01bc4ed55 when resuming from checkpoint, make next checkpoint serial number follow serial number of resumed checkpoint 2013-09-07 09:57:07 -07:00
Noah Levitt d61171f076 CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size 2013-09-07 09:55:58 -07:00
Noah Levitt b670a3c2a7 Merge branch 'master' into always-resumable 2013-09-06 23:32:32 -07:00
Noah Levitt 25d4de186f don't log warning in normal case 2013-09-06 22:04:56 -07:00
Noah Levitt cc0b63066b Merge branch 'master' into always-resumable 2013-09-03 18:38:55 -07:00
Noah Levitt 34cb015ecf adjust PublicSuffixesTest for updated public suffixes list 2013-09-03 18:38:44 -07:00
Noah Levitt 59596c7043 prevent redundant checkpoints when crawl is stopping 2013-09-03 14:24:41 -07:00
Noah Levitt 0ab8eee24e forgetting old checkpoints - roll up previous checkpointed CrawlerJournals (e.g. frontier.recover.gz) into new checkpoint 2013-08-29 10:56:12 -07:00
Noah Levitt 73f6b20e48 Merge branch 'master' into always-resumable 2013-08-28 18:31:42 -07:00
Noah Levitt 981cf5bc9d update public suffixes list 2013-08-28 18:31:28 -07:00
Noah Levitt 1315717002 Merge branch 'master' into always-resumable 2013-08-28 18:14:03 -07:00
Noah Levitt 418c7138a9 handle case where there is no whois server for a domain 2013-08-28 18:13:53 -07:00