lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-21 22:16:50 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,358 Commits 40 Branches 31 Tags
eccddb3df3ddf09deaff01b44d4e1ecdfd340819
Commit Graph
442 Commits
Author SHA1 Message Date
Noah Levitt 5e8219b014 pull in json dependency from ia-web-commons 2014-01-31 20:41:05 -08:00
Noah Levitt cba663a107 fix BdbCookieStore so that cookies get replaced correctly! and test it 2014-01-20 20:02:56 -08:00
Noah Levitt 420452195c pull in stock commons-httpclient dependency through ia-web-commons instead of specifying it explicitly 2014-01-14 14:52:24 -08:00
Noah Levitt c50393fe53 remove some cruft from testing 2014-01-13 21:29:23 -08:00
Noah Levitt a86efb419f bump httpcomponents httpclient dependency to latest 4.3.1 2014-01-12 13:24:48 -08:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43 
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
 2014-01-10 17:52:03 -08:00
Noah Levitt 749bd6a214 bump version to 3.3.0-SNAPSHOT after 3.2.0 release 2014-01-10 16:22:43 -08:00
Noah Levitt 32c083f720 bump version number to 3.2.0 for release 2014-01-10 12:36:45 -08:00
Noah Levitt 7dd67bd876 fix HER-1979 so heritrix can run on windows xp 2013-12-17 19:25:12 -08:00
Noah Levitt 0ea10665f1 abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner 2013-12-15 11:53:18 -08:00
Noah Levitt 3c8a0a389f moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons 2013-12-05 17:48:57 -08:00
Noah Levitt 98d0901531 moving org.archive.net.PublicSuffixes to ia-web-commons 2013-12-05 16:13:30 -08:00
Noah Levitt 8e1125d062 bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson) 2013-11-20 09:18:31 -08:00
Ilya Kreymer e2f4b7287f Merge branch 'master' of github.com:internetarchive/heritrix3 2013-11-01 15:44:12 -07:00
Ilya Kreymer 8633ed0ec3 removed ThreadLocalHttpConnectionManager -- it has been moved to ia-web-commons 2013-11-01 15:43:46 -07:00
Noah Levitt b137022b79 Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.) 2013-10-28 17:35:55 -07:00
Noah Levitt a1f84d3cc4 HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility 2013-10-10 17:57:18 -07:00
Noah Levitt 140ba3e89e tweak io block size in FileUtils.appendTo(), referencing performance metrics documented in gnu coreutils (used e.g. for /bin/cat) 2013-09-08 12:41:48 -07:00
Noah Levitt c49b465d82 custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test 2013-09-07 19:40:11 -07:00
Noah Levitt cc0b63066b Merge branch 'master' into always-resumable 2013-09-03 18:38:55 -07:00
Noah Levitt 34cb015ecf adjust PublicSuffixesTest for updated public suffixes list 2013-09-03 18:38:44 -07:00
Noah Levitt 0ab8eee24e forgetting old checkpoints - roll up previous checkpointed CrawlerJournals (e.g. frontier.recover.gz) into new checkpoint 2013-08-29 10:56:12 -07:00
Noah Levitt 73f6b20e48 Merge branch 'master' into always-resumable 2013-08-28 18:31:42 -07:00
Noah Levitt 981cf5bc9d update public suffixes list 2013-08-28 18:31:28 -07:00
Noah Levitt 43bb42d346 BdbModule#doCheckpoint - support forgetting all but latest checkpoint 2013-08-28 11:22:37 -07:00
Noah Levitt 8e7c26bc8f fix setForgetAllButLatest 2013-08-28 10:58:32 -07:00
Noah Levitt 2fe7d8cd45 some progress on forgetting old checkpoints when checkpointing anew 2013-08-27 18:39:25 -07:00
Noah Levitt a6958951df Reset Recorder state when uri processing is finished (had been seeing cases where a url's entry in crawl.log would sometimes have nonzero size value if the url was erroring out and RecordingOutputStream.open() was never called) 2013-08-12 19:28:28 -07:00
Noah Levitt 317ff7eb65 Merge branch 'master' into hc43 2013-08-09 18:17:24 -07:00
Noah Levitt 705a375daf uses of UriUtils.isLikelyUri() in Extractor{HTML,SWF,XML} with UriUtils.isVeryLikelyUri() to reap the benefits of HER-1523 improvements (should address archive-it issue ARI-3492) 2013-08-09 18:17:11 -07:00
Noah Levitt 7a1f998542 bump httpclient dependency 2013-08-09 17:48:48 -07:00
Noah Levitt 805331fcbe fix off-by-one mistake with integer arithmetic so RecordingInputStreamTest.testReadFullyOrUntil() test passes (how does this pass on master?) 2013-08-09 15:39:42 -07:00
Noah Levitt 750a688670 Merge branch 'master' into hc43 
Conflicts:
	commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
	engine/pom.xml
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
 2013-08-08 21:14:11 -07:00
Ilya Kreymer b60c43fb81 ARCRecord: Add error offset to position when adjusting for bad header lines 2013-07-18 21:00:01 -07:00
Ilya Kreymer 1b67f5ce6b ArcRecord: Silently skip any header fields before a status line in arc record. This allows the arc reader to be fault-tolerant of 'bad' arcs that have extra headers 
inserted before the status line (but are otherwise valid)
 2013-07-18 17:08:38 -07:00
Ilya Kreymer 3916175e4d FIX: Add ArchiveRecordHeader.getContentLength() to specifically return the warc/arc record content length, 
not including headers
 2013-06-21 14:56:31 -07:00
Noah Levitt 03c2075eb1 latest httpclient is 4.3-beta2-SNAPSHOT 2013-05-08 10:21:26 -07:00
Noah Levitt 81d0a23e50 restore build timestamp 2013-04-09 10:42:23 -07:00
Noah Levitt 8a40f9ff13 fix misleading logging 2013-04-01 13:04:33 -07:00
Ilya Kreymer 45b205061f Fix for testCompressedARCStream - if arc has gzip header but doesn't have IA flags, its still compressed 2013-03-31 18:54:12 -07:00
Noah Levitt 6a0ed067c0 consolidate disparate versions of WARCConstants into one version that lives in ia-web-commons 2013-03-29 17:12:32 -07:00
Ilya Kreymer 2df2427e55 Dependency Fixing: Added LaxHttpParser which can be used explicitly when using the lax header parsing functionality. 
Added HttpParser which just extends LaxHttpParser in the original httpclient package to avoid breaking compatibility with code
which may still reference HttpParser.
Explicit commons code changed to use LaxHttpParser in arc/warc record reading
 2013-03-27 18:29:55 -07:00
Noah Levitt 69eb124b5d add test for handling of bad http headers (to make sure we don't break this when we get rid of archive-overlay) 2013-03-27 14:48:21 -07:00
Noah Levitt 75abd2509f bit of a hack to support shouldFetchBodyRule, given httpcomponents limitation discussed at https://issues.apache.org/jira/browse/HTTPCORE-325 2013-03-20 16:17:34 -07:00
Noah Levitt 956024ec77 update httpclient dependency 2013-03-20 16:11:47 -07:00
Noah Levitt 1993c4388c Merge branch 'master' into hc43 
Conflicts:
	.classpath
 2013-03-20 14:28:13 -07:00
Noah Levitt 8fd361e518 update source/target version to 1.6 since we use 1.6 features, specifically @Override for methods only implementing an interface (not sure why maven wasn't enforcing this); also remove maven-antrun-plugin created timestamp.txt, doesn't seem to be used anywhere 2013-03-20 14:25:22 -07:00
Noah Levitt c509b6e552 remove more eclipse stuff, people like to manage their own 2013-03-20 13:39:28 -07:00
Noah Levitt b4bff3ffb9 new method RecordingInputStream.readToEndOfContent(), because now we can't rely on stopping right after the headers, which is an assumption of readFullyOrUntil() 2013-03-19 18:24:02 -07:00
Noah Levitt 9e054dd1f8 detect http message body begin in RecordingOutputStream, see https://issues.apache.org/jira/browse/HTTPCORE-325 2013-03-19 14:18:11 -07:00