Noah Levitt
2078a9f0e2
remove TmpDirTestCase.java (moving to webarchive-commons)
2015-01-06 16:08:18 -08:00
Noah Levitt
64e3b1b02a
remove RecordingOutputStreamTest.java (moving to webarchive-commons)
2015-01-06 16:01:16 -08:00
Noah Levitt
78e6b8b6ab
replace accidental U+2212 MINUS SIGN with normal ascii 0x2d '-' to avoid compilation error "unmappable character for encoding ASCII"
2014-10-01 21:31:14 -07:00
Noah Levitt
a37a8de4a2
improved implementation not using TreeSet, and some initial work on integration testing of cookie handling
2014-09-28 11:59:14 -07:00
Kenji Nagahashi
29379c7e19
Merge pull request #84 from nlevitt/iipc-webarchive-commons
...
switch to mainline iipc webarchive-commons latest release (1.1.4)
2014-09-11 17:19:08 -07:00
Noah Levitt
ec0839a6fe
remove more unused classes
2014-08-29 15:23:21 -07:00
Noah Levitt
f1b0633a3d
remove unused Transform* classes
2014-08-13 10:53:03 -07:00
Noah Levitt
651d18beb1
replace use of method removed from guava library
2014-08-08 17:38:51 -07:00
Noah Levitt
339e6ec87a
for ARI-3723, accept speculative urls with query params with no value
2014-04-04 10:15:07 -07:00
Noah Levitt
cc838747af
pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available
2014-03-31 13:38:54 -07:00
Shriphani Palakodety
c81eeced7a
added removal listener
2014-03-18 06:42:02 -04:00
Shriphani Palakodety
2f60b5f935
migration to latest guava complete
2014-03-18 06:42:02 -04:00
Shriphani Palakodety
1e8e2e2016
prevent intellij from complaining
2014-03-18 06:42:02 -04:00
Shriphani Palakodety
d0e0ee6bc6
forgot to add expireAfterWrite
2014-03-18 06:42:02 -04:00
Shriphani Palakodety
bd1047d2d2
bumped guava version to 11
2014-03-18 06:42:02 -04:00
Noah Levitt
cba663a107
fix BdbCookieStore so that cookies get replaced correctly! and test it
2014-01-20 20:02:56 -08:00
Noah Levitt
c50393fe53
remove some cruft from testing
2014-01-13 21:29:23 -08:00
Noah Levitt
3a429ffa44
Merge branch 'master' into hc43
...
Conflicts:
commons/pom.xml
commons/src/main/java/org/archive/io/RecordingInputStream.java
commons/src/main/java/org/archive/io/RecordingOutputStream.java
commons/src/main/java/org/archive/io/ReplayInputStream.java
dist/pom.xml
engine/pom.xml
modules/pom.xml
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
pom.xml
2014-01-10 17:52:03 -08:00
Noah Levitt
7dd67bd876
fix HER-1979 so heritrix can run on windows xp
2013-12-17 19:25:12 -08:00
Noah Levitt
3c8a0a389f
moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons
2013-12-05 17:48:57 -08:00
Noah Levitt
98d0901531
moving org.archive.net.PublicSuffixes to ia-web-commons
2013-12-05 16:13:30 -08:00
Ilya Kreymer
e2f4b7287f
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-11-01 15:44:12 -07:00
Ilya Kreymer
8633ed0ec3
removed ThreadLocalHttpConnectionManager -- it has been moved to ia-web-commons
2013-11-01 15:43:46 -07:00
Noah Levitt
b137022b79
Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.)
2013-10-28 17:35:55 -07:00
Noah Levitt
a1f84d3cc4
HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility
2013-10-10 17:57:18 -07:00
Noah Levitt
140ba3e89e
tweak io block size in FileUtils.appendTo(), referencing performance metrics documented in gnu coreutils (used e.g. for /bin/cat)
2013-09-08 12:41:48 -07:00
Noah Levitt
c49b465d82
custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test
2013-09-07 19:40:11 -07:00
Noah Levitt
cc0b63066b
Merge branch 'master' into always-resumable
2013-09-03 18:38:55 -07:00
Noah Levitt
34cb015ecf
adjust PublicSuffixesTest for updated public suffixes list
2013-09-03 18:38:44 -07:00
Noah Levitt
0ab8eee24e
forgetting old checkpoints - roll up previous checkpointed CrawlerJournals (e.g. frontier.recover.gz) into new checkpoint
2013-08-29 10:56:12 -07:00
Noah Levitt
73f6b20e48
Merge branch 'master' into always-resumable
2013-08-28 18:31:42 -07:00
Noah Levitt
981cf5bc9d
update public suffixes list
2013-08-28 18:31:28 -07:00
Noah Levitt
43bb42d346
BdbModule#doCheckpoint - support forgetting all but latest checkpoint
2013-08-28 11:22:37 -07:00
Noah Levitt
8e7c26bc8f
fix setForgetAllButLatest
2013-08-28 10:58:32 -07:00
Noah Levitt
2fe7d8cd45
some progress on forgetting old checkpoints when checkpointing anew
2013-08-27 18:39:25 -07:00
Noah Levitt
a6958951df
Reset Recorder state when uri processing is finished (had been seeing cases where a url's entry in crawl.log would sometimes have nonzero size value if the url was erroring out and RecordingOutputStream.open() was never called)
2013-08-12 19:28:28 -07:00
Noah Levitt
317ff7eb65
Merge branch 'master' into hc43
2013-08-09 18:17:24 -07:00
Noah Levitt
705a375daf
uses of UriUtils.isLikelyUri() in Extractor{HTML,SWF,XML} with UriUtils.isVeryLikelyUri() to reap the benefits of HER-1523 improvements (should address archive-it issue ARI-3492)
2013-08-09 18:17:11 -07:00
Noah Levitt
805331fcbe
fix off-by-one mistake with integer arithmetic so RecordingInputStreamTest.testReadFullyOrUntil() test passes (how does this pass on master?)
2013-08-09 15:39:42 -07:00
Noah Levitt
750a688670
Merge branch 'master' into hc43
...
Conflicts:
commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
engine/pom.xml
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
2013-08-08 21:14:11 -07:00
Ilya Kreymer
b60c43fb81
ARCRecord: Add error offset to position when adjusting for bad header lines
2013-07-18 21:00:01 -07:00
Ilya Kreymer
1b67f5ce6b
ArcRecord: Silently skip any header fields before a status line in arc record. This allows the arc reader to be fault-tolerant of 'bad' arcs that have extra headers
...
inserted before the status line (but are otherwise valid)
2013-07-18 17:08:38 -07:00
Ilya Kreymer
3916175e4d
FIX: Add ArchiveRecordHeader.getContentLength() to specifically return the warc/arc record content length,
...
not including headers
2013-06-21 14:56:31 -07:00
Noah Levitt
81d0a23e50
restore build timestamp
2013-04-09 10:42:23 -07:00
Noah Levitt
8a40f9ff13
fix misleading logging
2013-04-01 13:04:33 -07:00
Ilya Kreymer
45b205061f
Fix for testCompressedARCStream - if arc has gzip header but doesn't have IA flags, its still compressed
2013-03-31 18:54:12 -07:00
Noah Levitt
6a0ed067c0
consolidate disparate versions of WARCConstants into one version that lives in ia-web-commons
2013-03-29 17:12:32 -07:00
Ilya Kreymer
2df2427e55
Dependency Fixing: Added LaxHttpParser which can be used explicitly when using the lax header parsing functionality.
...
Added HttpParser which just extends LaxHttpParser in the original httpclient package to avoid breaking compatibility with code
which may still reference HttpParser.
Explicit commons code changed to use LaxHttpParser in arc/warc record reading
2013-03-27 18:29:55 -07:00
Noah Levitt
69eb124b5d
add test for handling of bad http headers (to make sure we don't break this when we get rid of archive-overlay)
2013-03-27 14:48:21 -07:00
Noah Levitt
75abd2509f
bit of a hack to support shouldFetchBodyRule, given httpcomponents limitation discussed at https://issues.apache.org/jira/browse/HTTPCORE-325
2013-03-20 16:17:34 -07:00
Noah Levitt