Noah Levitt
|
339e6ec87a
|
for ARI-3723, accept speculative urls with query params with no value
|
2014-04-04 10:15:07 -07:00 |
|
Noah Levitt
|
cc838747af
|
pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available
|
2014-03-31 13:38:54 -07:00 |
|
Shriphani Palakodety
|
c81eeced7a
|
added removal listener
|
2014-03-18 06:42:02 -04:00 |
|
Shriphani Palakodety
|
2f60b5f935
|
migration to latest guava complete
|
2014-03-18 06:42:02 -04:00 |
|
Shriphani Palakodety
|
8c9892586a
|
removed guava dependency from heritrix. use the webarchive-commons version
|
2014-03-18 06:42:02 -04:00 |
|
Shriphani Palakodety
|
1e8e2e2016
|
prevent intellij from complaining
|
2014-03-18 06:42:02 -04:00 |
|
Shriphani Palakodety
|
b0ad9358e8
|
bump version in pom.xml
|
2014-03-18 06:42:02 -04:00 |
|
Shriphani Palakodety
|
d0e0ee6bc6
|
forgot to add expireAfterWrite
|
2014-03-18 06:42:02 -04:00 |
|
Shriphani Palakodety
|
bd1047d2d2
|
bumped guava version to 11
|
2014-03-18 06:42:02 -04:00 |
|
Noah Levitt
|
e3efa3a5f2
|
switch dependency from ia-web-commons 1.1.1-SNAPSHOT to webarchive-commons 1.1.1-IA-SNAPSHOT
|
2014-02-28 12:46:36 -08:00 |
|
Noah Levitt
|
cf5e295d42
|
update httpcomponents, should address NPE we've seen https://issues.apache.org/jira/browse/HTTPCLIENT-1417
|
2014-02-26 11:35:26 -08:00 |
|
Noah Levitt
|
5e8219b014
|
pull in json dependency from ia-web-commons
|
2014-01-31 20:41:05 -08:00 |
|
Noah Levitt
|
cba663a107
|
fix BdbCookieStore so that cookies get replaced correctly! and test it
|
2014-01-20 20:02:56 -08:00 |
|
Noah Levitt
|
420452195c
|
pull in stock commons-httpclient dependency through ia-web-commons instead of specifying it explicitly
|
2014-01-14 14:52:24 -08:00 |
|
Noah Levitt
|
c50393fe53
|
remove some cruft from testing
|
2014-01-13 21:29:23 -08:00 |
|
Noah Levitt
|
a86efb419f
|
bump httpcomponents httpclient dependency to latest 4.3.1
|
2014-01-12 13:24:48 -08:00 |
|
Noah Levitt
|
3a429ffa44
|
Merge branch 'master' into hc43
Conflicts:
commons/pom.xml
commons/src/main/java/org/archive/io/RecordingInputStream.java
commons/src/main/java/org/archive/io/RecordingOutputStream.java
commons/src/main/java/org/archive/io/ReplayInputStream.java
dist/pom.xml
engine/pom.xml
modules/pom.xml
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
pom.xml
|
2014-01-10 17:52:03 -08:00 |
|
Noah Levitt
|
749bd6a214
|
bump version to 3.3.0-SNAPSHOT after 3.2.0 release
|
2014-01-10 16:22:43 -08:00 |
|
Noah Levitt
|
32c083f720
|
bump version number to 3.2.0 for release
|
2014-01-10 12:36:45 -08:00 |
|
Noah Levitt
|
7dd67bd876
|
fix HER-1979 so heritrix can run on windows xp
|
2013-12-17 19:25:12 -08:00 |
|
Noah Levitt
|
0ea10665f1
|
abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner
|
2013-12-15 11:53:18 -08:00 |
|
Noah Levitt
|
3c8a0a389f
|
moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons
|
2013-12-05 17:48:57 -08:00 |
|
Noah Levitt
|
98d0901531
|
moving org.archive.net.PublicSuffixes to ia-web-commons
|
2013-12-05 16:13:30 -08:00 |
|
Noah Levitt
|
8e1125d062
|
bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson)
|
2013-11-20 09:18:31 -08:00 |
|
Ilya Kreymer
|
e2f4b7287f
|
Merge branch 'master' of github.com:internetarchive/heritrix3
|
2013-11-01 15:44:12 -07:00 |
|
Ilya Kreymer
|
8633ed0ec3
|
removed ThreadLocalHttpConnectionManager -- it has been moved to ia-web-commons
|
2013-11-01 15:43:46 -07:00 |
|
Noah Levitt
|
b137022b79
|
Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.)
|
2013-10-28 17:35:55 -07:00 |
|
Noah Levitt
|
a1f84d3cc4
|
HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility
|
2013-10-10 17:57:18 -07:00 |
|
Noah Levitt
|
140ba3e89e
|
tweak io block size in FileUtils.appendTo(), referencing performance metrics documented in gnu coreutils (used e.g. for /bin/cat)
|
2013-09-08 12:41:48 -07:00 |
|
Noah Levitt
|
c49b465d82
|
custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test
|
2013-09-07 19:40:11 -07:00 |
|
Noah Levitt
|
cc0b63066b
|
Merge branch 'master' into always-resumable
|
2013-09-03 18:38:55 -07:00 |
|
Noah Levitt
|
34cb015ecf
|
adjust PublicSuffixesTest for updated public suffixes list
|
2013-09-03 18:38:44 -07:00 |
|
Noah Levitt
|
0ab8eee24e
|
forgetting old checkpoints - roll up previous checkpointed CrawlerJournals (e.g. frontier.recover.gz) into new checkpoint
|
2013-08-29 10:56:12 -07:00 |
|
Noah Levitt
|
73f6b20e48
|
Merge branch 'master' into always-resumable
|
2013-08-28 18:31:42 -07:00 |
|
Noah Levitt
|
981cf5bc9d
|
update public suffixes list
|
2013-08-28 18:31:28 -07:00 |
|
Noah Levitt
|
43bb42d346
|
BdbModule#doCheckpoint - support forgetting all but latest checkpoint
|
2013-08-28 11:22:37 -07:00 |
|
Noah Levitt
|
8e7c26bc8f
|
fix setForgetAllButLatest
|
2013-08-28 10:58:32 -07:00 |
|
Noah Levitt
|
2fe7d8cd45
|
some progress on forgetting old checkpoints when checkpointing anew
|
2013-08-27 18:39:25 -07:00 |
|
Noah Levitt
|
a6958951df
|
Reset Recorder state when uri processing is finished (had been seeing cases where a url's entry in crawl.log would sometimes have nonzero size value if the url was erroring out and RecordingOutputStream.open() was never called)
|
2013-08-12 19:28:28 -07:00 |
|
Noah Levitt
|
317ff7eb65
|
Merge branch 'master' into hc43
|
2013-08-09 18:17:24 -07:00 |
|
Noah Levitt
|
705a375daf
|
uses of UriUtils.isLikelyUri() in Extractor{HTML,SWF,XML} with UriUtils.isVeryLikelyUri() to reap the benefits of HER-1523 improvements (should address archive-it issue ARI-3492)
|
2013-08-09 18:17:11 -07:00 |
|
Noah Levitt
|
7a1f998542
|
bump httpclient dependency
|
2013-08-09 17:48:48 -07:00 |
|
Noah Levitt
|
805331fcbe
|
fix off-by-one mistake with integer arithmetic so RecordingInputStreamTest.testReadFullyOrUntil() test passes (how does this pass on master?)
|
2013-08-09 15:39:42 -07:00 |
|
Noah Levitt
|
750a688670
|
Merge branch 'master' into hc43
Conflicts:
commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
engine/pom.xml
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
|
2013-08-08 21:14:11 -07:00 |
|
Ilya Kreymer
|
b60c43fb81
|
ARCRecord: Add error offset to position when adjusting for bad header lines
|
2013-07-18 21:00:01 -07:00 |
|
Ilya Kreymer
|
1b67f5ce6b
|
ArcRecord: Silently skip any header fields before a status line in arc record. This allows the arc reader to be fault-tolerant of 'bad' arcs that have extra headers
inserted before the status line (but are otherwise valid)
|
2013-07-18 17:08:38 -07:00 |
|
Ilya Kreymer
|
3916175e4d
|
FIX: Add ArchiveRecordHeader.getContentLength() to specifically return the warc/arc record content length,
not including headers
|
2013-06-21 14:56:31 -07:00 |
|
Noah Levitt
|
03c2075eb1
|
latest httpclient is 4.3-beta2-SNAPSHOT
|
2013-05-08 10:21:26 -07:00 |
|
Noah Levitt
|
81d0a23e50
|
restore build timestamp
|
2013-04-09 10:42:23 -07:00 |
|
Noah Levitt
|
8a40f9ff13
|
fix misleading logging
|
2013-04-01 13:04:33 -07:00 |
|