lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-20 04:36:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,319 Commits 38 Branches 33 Tags
07293fbfb271b3fd09df2f35c6d76e664695aba2
Commit Graph
1319 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Kenji Nagahashi 07293fbfb2 Merge pull request #28 from nlevitt/hc43 
switch to httpcomponents, get rid of archive-overlay-commons-httpclient
 2014-01-14 14:58:36 -08:00
Noah Levitt 420452195c pull in stock commons-httpclient dependency through ia-web-commons instead of specifying it explicitly 2014-01-14 14:52:24 -08:00
Noah Levitt 5930561042 fix contrib modules that had been using CrawlURI.getHttpMethod() 2014-01-14 14:27:34 -08:00
Noah Levitt c50393fe53 remove some cruft from testing 2014-01-13 21:29:23 -08:00
Noah Levitt 9f88da1735 add javadoc explaining newfound absence of getHttpMethod() 2014-01-13 20:51:08 -08:00
Noah Levitt 2f137dab50 remove obsolete httpcomponents.diff 2014-01-13 12:38:42 -08:00
Noah Levitt a86efb419f bump httpcomponents httpclient dependency to latest 4.3.1 2014-01-12 13:24:48 -08:00
Noah Levitt 16fa2b25f1 selftest-crawler-beans.cxml - replace obsolete BdbCookieStorage with BdbCookieStore 2014-01-12 13:11:25 -08:00
Noah Levitt 41ffae253e mark beginning of content at beginning of test payload, now that Recorder tries to auto-detect 2014-01-12 12:42:46 -08:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43 
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
 2014-01-10 17:52:03 -08:00
Noah Levitt 6bdfec7e7f Merge branch 'master' of github.com:internetarchive/heritrix3 2014-01-10 16:22:51 -08:00
Noah Levitt 749bd6a214 bump version to 3.3.0-SNAPSHOT after 3.2.0 release 2014-01-10 16:22:43 -08:00
Noah Levitt 78a6bea665 Merge pull request #27 from nlevitt/release-3.2.0 
rename dist/README.md to dist/README.txt so that maven bundles it in the...
 2014-01-10 14:16:07 -08:00
Noah Levitt 563c5b034d rename dist/README.md to dist/README.txt so that maven bundles it in the distribution 2014-01-10 14:13:32 -08:00
Noah Levitt 484b3574d7 Merge pull request #26 from nlevitt/release-3.2.0 
update readme for 3.2.0 release
3.2.0 		2014-01-10 13:17:46 -08:00
Noah Levitt 0467304f76 update readme for 3.2.0 release 2014-01-10 13:15:48 -08:00
Eldon 752e288ef8 Merge pull request #25 from nlevitt/release-3.2.0 
bump version number to 3.2.0 for release
 2014-01-10 12:47:58 -08:00
Noah Levitt 32c083f720 bump version number to 3.2.0 for release 2014-01-10 12:36:45 -08:00
Adam Miller 9ac0ee057f Merge pull request #24 from nlevitt/url-agnostic-standard 
for url-agnostic dedup, follow "Proposal for Standardizing the Recording...
 2014-01-08 11:38:51 -08:00
Noah Levitt 84e78db40a for url-agnostic dedup, follow "Proposal for Standardizing the Recording of Arbitrary Duplicates in WARC Files" https://docs.google.com/document/d/1QyQBA7Ykgxie75V8Jziz_O7hbhwf7PF6_u9O6w6zgp0 2014-01-07 14:43:26 -08:00
ikreymer 506e68316a Merge pull request #23 from nlevitt/her-1979 
fix HER-1979 so heritrix can run on windows xp
 2013-12-18 01:12:16 -08:00
Noah Levitt 7dd67bd876 fix HER-1979 so heritrix can run on windows xp 2013-12-17 19:25:12 -08:00
Noah Levitt 2388a35a06 useProjectArtifact=false to avoid [WARNING] Cannot include project artifact: ... it doesn't have an associated file or directory 2013-12-15 11:54:36 -08:00
Noah Levitt 0ea10665f1 abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner 2013-12-15 11:53:18 -08:00
Noah Levitt 6f171b575a fix report links 2013-12-11 21:14:04 -08:00
Noah Levitt 3b06c88537 test fix (in ia-web-commons) for HER-2059 2013-12-06 19:22:16 -08:00
Noah Levitt 3c8a0a389f moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons 2013-12-05 17:48:57 -08:00
Noah Levitt 98d0901531 moving org.archive.net.PublicSuffixes to ia-web-commons 2013-12-05 16:13:30 -08:00
Noah Levitt 552794dc89 don't let the exception from commit df2a313797 derail the whole crawl job 2013-11-26 16:08:38 -08:00
Noah Levitt 1163847644 try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted 2013-11-21 18:51:16 -08:00
Noah Levitt 8e1125d062 bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson) 2013-11-20 09:18:31 -08:00
Noah Levitt d93b5449be new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc 2013-11-19 11:05:14 -08:00
Noah Levitt 7089caf3c4 remove unused engine.css 2013-11-15 19:11:19 -08:00
Noah Levitt 119e565acc Adam's responsive UI 2013-11-15 18:59:14 -08:00
Noah Levitt 2b87219496 oops, avoid npe 2013-11-15 17:59:34 -08:00
Noah Levitt df2a313797 throw exception when host cache returns 0.0.0.0 for any host (trying to track down a strange problem) 2013-11-15 17:20:55 -08:00
Ilya Kreymer e2f4b7287f Merge branch 'master' of github.com:internetarchive/heritrix3 2013-11-01 15:44:12 -07:00
Ilya Kreymer 8633ed0ec3 removed ThreadLocalHttpConnectionManager -- it has been moved to ia-web-commons 2013-11-01 15:43:46 -07:00
Noah Levitt b137022b79 Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.) 2013-10-28 17:35:55 -07:00
Noah Levitt 38a12b2008 avoid NPE resuming from checkpoint if "stats" is {} in warcWriter checkpoint json 2013-10-11 14:44:34 -07:00
Noah Levitt a1f84d3cc4 HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility 2013-10-10 17:57:18 -07:00
Noah Levitt 7ad90b3126 Merge branch 'master' of github.com:internetarchive/heritrix3 2013-09-12 12:04:32 -07:00
Noah Levitt 57311db117 use libs version ${project.version} 2013-09-10 15:24:58 -07:00
Noah Levitt 140ba3e89e tweak io block size in FileUtils.appendTo(), referencing performance metrics documented in gnu coreutils (used e.g. for /bin/cat) 2013-09-08 12:41:48 -07:00
Noah Levitt 254811ff7f commented-out default for new setting forgetAllButLatest on CheckpointService 2013-09-08 12:39:54 -07:00
Noah Levitt f096609a50 avoid infinite loop spitting out stack traces on BeansException on startup 2013-09-08 00:11:54 -07:00
Noah Levitt a00141eef7 avoid deadlock in case of BeansException at crawl startup 2013-09-08 00:10:27 -07:00
Noah Levitt 071cbe99cd include urlsWritten in checkpoint, and be backward compatible with old checkpoints 2013-09-07 23:40:17 -07:00
Noah Levitt c49b465d82 custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test 2013-09-07 19:40:11 -07:00
Noah Levitt 52d72e661d part of HER-2048 - WARCWriterProcessor stats resume from checkpoint 2013-09-07 11:35:03 -07:00