Noah Levitt
b7afc627ed
fix typo
2015-09-24 15:31:12 -07:00
Noah Levitt
9aabcb3272
add license header to new class
2015-09-23 12:07:38 -07:00
Noah Levitt
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
Noah Levitt
1027f854ec
for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js")
2015-04-20 17:08:47 -07:00
Noah Levitt
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
Noah Levitt
1bd8b141aa
bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39
2015-02-11 18:53:34 -08:00
Noah Levitt
2078a9f0e2
remove TmpDirTestCase.java (moving to webarchive-commons)
2015-01-06 16:08:18 -08:00
Noah Levitt
64e3b1b02a
remove RecordingOutputStreamTest.java (moving to webarchive-commons)
2015-01-06 16:01:16 -08:00
Noah Levitt
78e6b8b6ab
replace accidental U+2212 MINUS SIGN with normal ascii 0x2d '-' to avoid compilation error "unmappable character for encoding ASCII"
2014-10-01 21:31:14 -07:00
Noah Levitt
e5e5cd87e4
configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency
2014-10-01 13:38:39 -07:00
Noah Levitt
a37a8de4a2
improved implementation not using TreeSet, and some initial work on integration testing of cookie handling
2014-09-28 11:59:14 -07:00
Kenji Nagahashi
29379c7e19
Merge pull request #84 from nlevitt/iipc-webarchive-commons
...
switch to mainline iipc webarchive-commons latest release (1.1.4)
2014-09-11 17:19:08 -07:00
Noah Levitt
1d0f60a2be
webarchive-commons 1.1.4 has been released, bump dependency
2014-09-10 12:31:01 -07:00
Noah Levitt
935b8f9658
remove unused dependency jets3t (thanks kenji)
2014-09-02 11:00:34 -07:00
Noah Levitt
ec0839a6fe
remove more unused classes
2014-08-29 15:23:21 -07:00
Kenji Nagahashi
bb511796ec
Update POMs for newly imported Wayback-dedup module.
...
lock httpclient version to 4.3.3 stated by
heritrix-commons. have to exclude hadoop’s
dependency on jets3t 0.9.0 as it pulls in httpcore 4.1.2.
add easy mock 3.1 to heritrix-contrib.
2014-08-25 16:44:48 -07:00
Noah Levitt
f1b0633a3d
remove unused Transform* classes
2014-08-13 10:53:03 -07:00
Noah Levitt
651d18beb1
replace use of method removed from guava library
2014-08-08 17:38:51 -07:00
Noah Levitt
21f17a0d9f
switch to mainline iipc webarchive-commons latest release
2014-08-08 17:26:21 -07:00
Noah Levitt
339e6ec87a
for ARI-3723, accept speculative urls with query params with no value
2014-04-04 10:15:07 -07:00
Noah Levitt
cc838747af
pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available
2014-03-31 13:38:54 -07:00
Shriphani Palakodety
c81eeced7a
added removal listener
2014-03-18 06:42:02 -04:00
Shriphani Palakodety
2f60b5f935
migration to latest guava complete
2014-03-18 06:42:02 -04:00
Shriphani Palakodety
8c9892586a
removed guava dependency from heritrix. use the webarchive-commons version
2014-03-18 06:42:02 -04:00
Shriphani Palakodety
1e8e2e2016
prevent intellij from complaining
2014-03-18 06:42:02 -04:00
Shriphani Palakodety
b0ad9358e8
bump version in pom.xml
2014-03-18 06:42:02 -04:00
Shriphani Palakodety
d0e0ee6bc6
forgot to add expireAfterWrite
2014-03-18 06:42:02 -04:00
Shriphani Palakodety
bd1047d2d2
bumped guava version to 11
2014-03-18 06:42:02 -04:00
Noah Levitt
e3efa3a5f2
switch dependency from ia-web-commons 1.1.1-SNAPSHOT to webarchive-commons 1.1.1-IA-SNAPSHOT
2014-02-28 12:46:36 -08:00
Noah Levitt
cf5e295d42
update httpcomponents, should address NPE we've seen https://issues.apache.org/jira/browse/HTTPCLIENT-1417
2014-02-26 11:35:26 -08:00
Noah Levitt
5e8219b014
pull in json dependency from ia-web-commons
2014-01-31 20:41:05 -08:00
Noah Levitt
cba663a107
fix BdbCookieStore so that cookies get replaced correctly! and test it
2014-01-20 20:02:56 -08:00
Noah Levitt
420452195c
pull in stock commons-httpclient dependency through ia-web-commons instead of specifying it explicitly
2014-01-14 14:52:24 -08:00
Noah Levitt
c50393fe53
remove some cruft from testing
2014-01-13 21:29:23 -08:00
Noah Levitt
a86efb419f
bump httpcomponents httpclient dependency to latest 4.3.1
2014-01-12 13:24:48 -08:00
Noah Levitt
3a429ffa44
Merge branch 'master' into hc43
...
Conflicts:
commons/pom.xml
commons/src/main/java/org/archive/io/RecordingInputStream.java
commons/src/main/java/org/archive/io/RecordingOutputStream.java
commons/src/main/java/org/archive/io/ReplayInputStream.java
dist/pom.xml
engine/pom.xml
modules/pom.xml
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
pom.xml
2014-01-10 17:52:03 -08:00
Noah Levitt
749bd6a214
bump version to 3.3.0-SNAPSHOT after 3.2.0 release
2014-01-10 16:22:43 -08:00
Noah Levitt
32c083f720
bump version number to 3.2.0 for release
2014-01-10 12:36:45 -08:00
Noah Levitt
7dd67bd876
fix HER-1979 so heritrix can run on windows xp
2013-12-17 19:25:12 -08:00
Noah Levitt
0ea10665f1
abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner
2013-12-15 11:53:18 -08:00
Noah Levitt
3c8a0a389f
moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons
2013-12-05 17:48:57 -08:00
Noah Levitt
98d0901531
moving org.archive.net.PublicSuffixes to ia-web-commons
2013-12-05 16:13:30 -08:00
Noah Levitt
8e1125d062
bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson)
2013-11-20 09:18:31 -08:00
Ilya Kreymer
e2f4b7287f
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-11-01 15:44:12 -07:00
Ilya Kreymer
8633ed0ec3
removed ThreadLocalHttpConnectionManager -- it has been moved to ia-web-commons
2013-11-01 15:43:46 -07:00
Noah Levitt
b137022b79
Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.)
2013-10-28 17:35:55 -07:00
Noah Levitt
a1f84d3cc4
HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility
2013-10-10 17:57:18 -07:00
Noah Levitt
140ba3e89e
tweak io block size in FileUtils.appendTo(), referencing performance metrics documented in gnu coreutils (used e.g. for /bin/cat)
2013-09-08 12:41:48 -07:00
Noah Levitt
c49b465d82
custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test
2013-09-07 19:40:11 -07:00
Noah Levitt
cc0b63066b
Merge branch 'master' into always-resumable
2013-09-03 18:38:55 -07:00
Noah Levitt