Commit Graph
453 Commits
Author SHA1 Message Date
Noah Levitt 339e6ec87a for ARI-3723, accept speculative urls with query params with no value 2014-04-04 10:15:07 -07:00
Noah Levitt cc838747af pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available 2014-03-31 13:38:54 -07:00
Shriphani Palakodety c81eeced7a added removal listener 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 2f60b5f935 migration to latest guava complete 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 8c9892586a removed guava dependency from heritrix. use the webarchive-commons version 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 1e8e2e2016 prevent intellij from complaining 2014-03-18 06:42:02 -04:00
Shriphani Palakodety b0ad9358e8 bump version in pom.xml 2014-03-18 06:42:02 -04:00
Shriphani Palakodety d0e0ee6bc6 forgot to add expireAfterWrite 2014-03-18 06:42:02 -04:00
Shriphani Palakodety bd1047d2d2 bumped guava version to 11 2014-03-18 06:42:02 -04:00
Noah Levitt e3efa3a5f2 switch dependency from ia-web-commons 1.1.1-SNAPSHOT to webarchive-commons 1.1.1-IA-SNAPSHOT 2014-02-28 12:46:36 -08:00
Noah Levitt cf5e295d42 update httpcomponents, should address NPE we've seen https://issues.apache.org/jira/browse/HTTPCLIENT-1417 2014-02-26 11:35:26 -08:00
Noah Levitt 5e8219b014 pull in json dependency from ia-web-commons 2014-01-31 20:41:05 -08:00
Noah Levitt cba663a107 fix BdbCookieStore so that cookies get replaced correctly! and test it 2014-01-20 20:02:56 -08:00
Noah Levitt 420452195c pull in stock commons-httpclient dependency through ia-web-commons instead of specifying it explicitly 2014-01-14 14:52:24 -08:00
Noah Levitt c50393fe53 remove some cruft from testing 2014-01-13 21:29:23 -08:00
Noah Levitt a86efb419f bump httpcomponents httpclient dependency to latest 4.3.1 2014-01-12 13:24:48 -08:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
2014-01-10 17:52:03 -08:00
Noah Levitt 749bd6a214 bump version to 3.3.0-SNAPSHOT after 3.2.0 release 2014-01-10 16:22:43 -08:00
Noah Levitt 32c083f720 bump version number to 3.2.0 for release 2014-01-10 12:36:45 -08:00
Noah Levitt 7dd67bd876 fix HER-1979 so heritrix can run on windows xp 2013-12-17 19:25:12 -08:00
Noah Levitt 0ea10665f1 abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner 2013-12-15 11:53:18 -08:00
Noah Levitt 3c8a0a389f moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons 2013-12-05 17:48:57 -08:00
Noah Levitt 98d0901531 moving org.archive.net.PublicSuffixes to ia-web-commons 2013-12-05 16:13:30 -08:00
Noah Levitt 8e1125d062 bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson) 2013-11-20 09:18:31 -08:00
Ilya Kreymer e2f4b7287f Merge branch 'master' of github.com:internetarchive/heritrix3 2013-11-01 15:44:12 -07:00
Ilya Kreymer 8633ed0ec3 removed ThreadLocalHttpConnectionManager -- it has been moved to ia-web-commons 2013-11-01 15:43:46 -07:00
Noah Levitt b137022b79 Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.) 2013-10-28 17:35:55 -07:00
Noah Levitt a1f84d3cc4 HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility 2013-10-10 17:57:18 -07:00
Noah Levitt 140ba3e89e tweak io block size in FileUtils.appendTo(), referencing performance metrics documented in gnu coreutils (used e.g. for /bin/cat) 2013-09-08 12:41:48 -07:00
Noah Levitt c49b465d82 custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test 2013-09-07 19:40:11 -07:00
Noah Levitt cc0b63066b Merge branch 'master' into always-resumable 2013-09-03 18:38:55 -07:00
Noah Levitt 34cb015ecf adjust PublicSuffixesTest for updated public suffixes list 2013-09-03 18:38:44 -07:00
Noah Levitt 0ab8eee24e forgetting old checkpoints - roll up previous checkpointed CrawlerJournals (e.g. frontier.recover.gz) into new checkpoint 2013-08-29 10:56:12 -07:00
Noah Levitt 73f6b20e48 Merge branch 'master' into always-resumable 2013-08-28 18:31:42 -07:00
Noah Levitt 981cf5bc9d update public suffixes list 2013-08-28 18:31:28 -07:00
Noah Levitt 43bb42d346 BdbModule#doCheckpoint - support forgetting all but latest checkpoint 2013-08-28 11:22:37 -07:00
Noah Levitt 8e7c26bc8f fix setForgetAllButLatest 2013-08-28 10:58:32 -07:00
Noah Levitt 2fe7d8cd45 some progress on forgetting old checkpoints when checkpointing anew 2013-08-27 18:39:25 -07:00
Noah Levitt a6958951df Reset Recorder state when uri processing is finished (had been seeing cases where a url's entry in crawl.log would sometimes have nonzero size value if the url was erroring out and RecordingOutputStream.open() was never called) 2013-08-12 19:28:28 -07:00
Noah Levitt 317ff7eb65 Merge branch 'master' into hc43 2013-08-09 18:17:24 -07:00
Noah Levitt 705a375daf uses of UriUtils.isLikelyUri() in Extractor{HTML,SWF,XML} with UriUtils.isVeryLikelyUri() to reap the benefits of HER-1523 improvements (should address archive-it issue ARI-3492) 2013-08-09 18:17:11 -07:00
Noah Levitt 7a1f998542 bump httpclient dependency 2013-08-09 17:48:48 -07:00
Noah Levitt 805331fcbe fix off-by-one mistake with integer arithmetic so RecordingInputStreamTest.testReadFullyOrUntil() test passes (how does this pass on master?) 2013-08-09 15:39:42 -07:00
Noah Levitt 750a688670 Merge branch 'master' into hc43
Conflicts:
	commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
	engine/pom.xml
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
2013-08-08 21:14:11 -07:00
Ilya Kreymer b60c43fb81 ARCRecord: Add error offset to position when adjusting for bad header lines 2013-07-18 21:00:01 -07:00
Ilya Kreymer 1b67f5ce6b ArcRecord: Silently skip any header fields before a status line in arc record. This allows the arc reader to be fault-tolerant of 'bad' arcs that have extra headers
inserted before the status line (but are otherwise valid)
2013-07-18 17:08:38 -07:00
Ilya Kreymer 3916175e4d FIX: Add ArchiveRecordHeader.getContentLength() to specifically return the warc/arc record content length,
not including headers
2013-06-21 14:56:31 -07:00
Noah Levitt 03c2075eb1 latest httpclient is 4.3-beta2-SNAPSHOT 2013-05-08 10:21:26 -07:00
Noah Levitt 81d0a23e50 restore build timestamp 2013-04-09 10:42:23 -07:00
Noah Levitt 8a40f9ff13 fix misleading logging 2013-04-01 13:04:33 -07:00