lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-21 13:16:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,921 Commits 38 Branches 33 Tags
fca796265d58afa628ae80e404a7abca09cef4e7
Commit Graph
485 Commits
Author SHA1 Message Date
Andrew Jackson d85d2a3fe8 Move OneLineSimpleLayout into commons so all can use it. 2017-10-12 22:10:58 +01:00
Noah Levitt ae8a0ef95f update webarchive-commons dependency to incorporate fix to warc size calculation, and uncomment test that exercises the fix 2017-10-11 14:54:37 -07:00
Noah Levitt 3892c17b56 ughhh appease java 8 javadoc rules 2017-06-15 17:29:22 -07:00
Mohamed Elsayed d13bfb6a37 Remove unit tests which moved over from Heritrix3 to webarchive-commons 2017-03-09 10:26:32 -08:00
Noah Levitt 36ea1a9fb2 try to fix remaining serialization tests in oraclejdk8 by using ConcurrentSkipListMap instead of ConcurrentHashMap 2016-06-07 17:55:45 -05:00
Noah Levitt 99d87e8f23 hopefully fix serialization tests in oraclejdk8 by using TreeSet instead of HashSet in KeyedProperties.java 2016-06-07 14:53:11 -05:00
Noah Levitt cbb5e3ab99 Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions. 2016-02-09 15:02:11 -08:00
Hunter Stern 088189ee8b Fix up code based on pull requests comments. Add test for urls with spaces in them. 2016-01-28 15:11:49 -08:00
Hunter Stern 4518bc2da8 Make sure no urls with whitespace and not having good extension slip through 2016-01-20 14:29:42 -08:00
Hunter Stern 12a88d6d23 Allow spaces in urls extracted from JS. 2016-01-20 10:44:20 -08:00
Hunter Stern 23885557bb Allow spaces in JavaScript urls, but only if they have a known good file extension 2016-01-19 18:07:05 -08:00
Noah Levitt 3b12a751e1 to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly 2016-01-14 12:17:19 -08:00
Hunter Stern 34cffd512a Add ability to submit forms with enctype="multipart/form-data" 2016-01-14 12:05:38 -08:00
Noah Levitt b7afc627ed fix typo 2015-09-24 15:31:12 -07:00
Noah Levitt 9aabcb3272 add license header to new class 2015-09-23 12:07:38 -07:00
Noah Levitt 12dde2b693 load keytool utility main class dynamically, trying both the old and new (java 8) class names 2015-09-23 12:06:11 -07:00
Noah Levitt 1027f854ec for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js") 2015-04-20 17:08:47 -07:00
Noah Levitt 8009275075 make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/ 2015-02-23 11:36:29 -08:00
Noah Levitt 1bd8b141aa bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39 2015-02-11 18:53:34 -08:00
Noah Levitt 2078a9f0e2 remove TmpDirTestCase.java (moving to webarchive-commons) 2015-01-06 16:08:18 -08:00
Noah Levitt 64e3b1b02a remove RecordingOutputStreamTest.java (moving to webarchive-commons) 2015-01-06 16:01:16 -08:00
Noah Levitt 78e6b8b6ab replace accidental U+2212 MINUS SIGN with normal ascii 0x2d '-' to avoid compilation error "unmappable character for encoding ASCII" 2014-10-01 21:31:14 -07:00
Noah Levitt e5e5cd87e4 configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency 2014-10-01 13:38:39 -07:00
Noah Levitt a37a8de4a2 improved implementation not using TreeSet, and some initial work on integration testing of cookie handling 2014-09-28 11:59:14 -07:00
Kenji Nagahashi 29379c7e19 Merge pull request #84 from nlevitt/iipc-webarchive-commons 
switch to mainline iipc webarchive-commons latest release (1.1.4)
 2014-09-11 17:19:08 -07:00
Noah Levitt 1d0f60a2be webarchive-commons 1.1.4 has been released, bump dependency 2014-09-10 12:31:01 -07:00
Noah Levitt 935b8f9658 remove unused dependency jets3t (thanks kenji) 2014-09-02 11:00:34 -07:00
Noah Levitt ec0839a6fe remove more unused classes 2014-08-29 15:23:21 -07:00
Kenji Nagahashi bb511796ec Update POMs for newly imported Wayback-dedup module. 
lock httpclient version to 4.3.3 stated by
	heritrix-commons. have to exclude hadoop’s
	dependency on jets3t 0.9.0 as it pulls in		httpcore 4.1.2.
	add easy mock 3.1 to heritrix-contrib.
 2014-08-25 16:44:48 -07:00
Noah Levitt f1b0633a3d remove unused Transform* classes 2014-08-13 10:53:03 -07:00
Noah Levitt 651d18beb1 replace use of method removed from guava library 2014-08-08 17:38:51 -07:00
Noah Levitt 21f17a0d9f switch to mainline iipc webarchive-commons latest release 2014-08-08 17:26:21 -07:00
Noah Levitt 339e6ec87a for ARI-3723, accept speculative urls with query params with no value 2014-04-04 10:15:07 -07:00
Noah Levitt cc838747af pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available 2014-03-31 13:38:54 -07:00
Shriphani Palakodety c81eeced7a added removal listener 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 2f60b5f935 migration to latest guava complete 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 8c9892586a removed guava dependency from heritrix. use the webarchive-commons version 2014-03-18 06:42:02 -04:00
Shriphani Palakodety 1e8e2e2016 prevent intellij from complaining 2014-03-18 06:42:02 -04:00
Shriphani Palakodety b0ad9358e8 bump version in pom.xml 2014-03-18 06:42:02 -04:00
Shriphani Palakodety d0e0ee6bc6 forgot to add expireAfterWrite 2014-03-18 06:42:02 -04:00
Shriphani Palakodety bd1047d2d2 bumped guava version to 11 2014-03-18 06:42:02 -04:00
Noah Levitt e3efa3a5f2 switch dependency from ia-web-commons 1.1.1-SNAPSHOT to webarchive-commons 1.1.1-IA-SNAPSHOT 2014-02-28 12:46:36 -08:00
Noah Levitt cf5e295d42 update httpcomponents, should address NPE we've seen https://issues.apache.org/jira/browse/HTTPCLIENT-1417 2014-02-26 11:35:26 -08:00
Noah Levitt 5e8219b014 pull in json dependency from ia-web-commons 2014-01-31 20:41:05 -08:00
Noah Levitt cba663a107 fix BdbCookieStore so that cookies get replaced correctly! and test it 2014-01-20 20:02:56 -08:00
Noah Levitt 420452195c pull in stock commons-httpclient dependency through ia-web-commons instead of specifying it explicitly 2014-01-14 14:52:24 -08:00
Noah Levitt c50393fe53 remove some cruft from testing 2014-01-13 21:29:23 -08:00
Noah Levitt a86efb419f bump httpcomponents httpclient dependency to latest 4.3.1 2014-01-12 13:24:48 -08:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43 
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
 2014-01-10 17:52:03 -08:00
Noah Levitt 749bd6a214 bump version to 3.3.0-SNAPSHOT after 3.2.0 release 2014-01-10 16:22:43 -08:00