lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-22 05:35:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,345 Commits 38 Branches 33 Tags
6b19daf1d28a8f4848bfb537b2cb45bb36dc7ea7
Commit Graph
78 Commits
Author SHA1 Message Date
Andrew Jackson ddb695abad [maven-release-plugin] prepare for next development iteration 2021-08-03 09:56:26 +01:00
Andrew Jackson ba44cce7f2 [maven-release-plugin] prepare release 3.4.0-20210803 2021-08-03 09:56:20 +01:00
Andy Jackson 4d5b2dc3fe Merge pull request #397 from ukwa/upgrade-httpclient-4-5 
Upgrade httpclient to 4.5
 2021-08-02 09:19:30 +01:00
Alex Osborne f4bd5d1015 Upgrade Groovy to latest stable version (3.0.8) for JDK 16 compatibility 
Fixes #419
 2021-07-25 14:11:13 +09:00
Alex Osborne 13c0f65294 FTP: Implement a basic fetch test 2021-06-25 14:09:38 +09:00
Andrew Jackson 0e93e22ba9 [maven-release-plugin] prepare for next development iteration 2021-06-21 15:42:52 +01:00
Andrew Jackson 846f2bc2cc [maven-release-plugin] prepare release 3.4.0-20210621 2021-06-21 15:42:46 +01:00
Andrew Jackson a5beb7d6e9 [maven-release-plugin] prepare for next development iteration 2021-06-18 09:41:30 +01:00
Andrew Jackson 30598f8402 [maven-release-plugin] prepare release 3.4.0-20210618 2021-06-18 09:41:24 +01:00
Andrew Jackson 3423cab6a6 [maven-release-plugin] prepare for next development iteration 2021-06-17 14:39:00 +01:00
Andrew Jackson 7368ded9c2 [maven-release-plugin] prepare release 3.4.0-20210617 2021-06-17 14:38:54 +01:00
Andrew Jackson 1d1d2f5abe Avoid two slf4j implementations. 2021-06-17 14:19:12 +01:00
Andrew Jackson d3dfb34f83 [maven-release-plugin] prepare for next development iteration 2021-05-27 10:03:10 +01:00
Andrew Jackson 9dcbb240a6 [maven-release-plugin] prepare release 3.4.0-20210527 2021-05-27 10:03:04 +01:00
Andy Jackson a2c65f4618 Update dependencies for security issues, including changes to tests. 2021-05-26 13:39:30 +01:00
Andy Jackson 396467c535 Merge branch 'master' into sitemaps 2021-05-20 21:03:54 +01:00
Andrew Jackson e215b89963 [maven-release-plugin] prepare for next development iteration 2020-05-18 14:08:12 +01:00
Andrew Jackson ef38dc08c3 [maven-release-plugin] prepare release 3.4.0-20200518 2020-05-18 14:08:04 +01:00
Leslie Bellony 40725372d6 Add support for the SFTP protocol 2020-04-01 14:47:34 +02:00
Andrew Jackson ff05de44b6 [maven-release-plugin] prepare for next development iteration 2020-03-04 23:05:41 +00:00
Andrew Jackson a1bdcb1be7 [maven-release-plugin] prepare release 3.4.0-20200304 2020-03-04 23:05:33 +00:00
Noah Levitt cefad96036 Merge pull request #276 from internetarchive/jdk11-upgrade-jetty 
JDK11 support: upgrade to Jetty 9.4.19, Restlet 2.4.0 and drop JDK 7 support
 2019-08-07 10:48:51 -07:00
Alex Osborne 9d1341c731 JDK11 support: upgrade jetty to 9.4.19 in modules tests 
Fixes #268 ssl handshake_failure. Support for jdk11 was added in jetty
9.4.14 but we may as well bump to the latest stable version.

In Jetty 9 it appears the request is logged after the response is sent.
Thus it was racing with the assertions that check the client IP. So to
fix this rather stashing the 'lastRequest' we just make the server
echo the client's IP in a response header.

Some other minor tweaks were needed due to changes in Jetty behaviour:

- We stop checking the length of the raw response. It doesn't tell us
  anything and easily varies.

- Jetty now generates Set-Cookie with a space after the ;

- Jetty now lowercases the word "basic" in WWW-Authenticate header

- testLaxUrlEncoding(): Jetty now rejects bad paths with a 400 error so
  we disable the response checks. The actual request line is
  still checked which is the important thing.

Note: This patch does not affect the version of jetty used by the
Heritrix admin console. That will be tackled separately.
 2019-08-05 14:35:09 +09:00
Alex Osborne d3d19e2248 Prevent newer surefire from invoking FetchHTTPTests directly 
It seems older versions only matched *Test.java but the new version
is now incorrectly including it.
 2019-08-02 17:57:09 +09:00
Alex Osborne 412885fd7d JDK11 support: upgrade maven-surefire-plugin to 2.22.2 
Under JDK11 our old version of it throws ClassNotFoundExceptions when
tests load some builtin classes like javax.transaction.xa.Xid.

Fixes #266
 2019-08-02 16:43:20 +09:00
kristinn 8ddaf41db2 Add crawler-commons dependency. 
Bump commons-io dependency version to match crawler-commons.
 2019-07-05 11:27:15 +00:00
Andrew Jackson b3961a2f96 [maven-release-plugin] prepare for next development iteration 2019-04-18 15:36:28 +01:00
Andrew Jackson c7c6141ee1 [maven-release-plugin] prepare release 3.4.0-20190418 2019-04-18 15:36:20 +01:00
Andrew Jackson ad9e74d8ad [maven-release-plugin] prepare for next development iteration 2019-02-07 13:52:56 +00:00
Andrew Jackson 83c7044b22 [maven-release-plugin] prepare release 3.4.0-20190207 2019-02-07 13:52:49 +00:00
Andrew Jackson 317b19e7e2 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:33:08 +00:00
Andrew Jackson 9c7d0c299d [maven-release-plugin] prepare release 3.4.0-20190205 2019-02-05 12:32:57 +00:00
Andrew Jackson ab34a54566 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:29:33 +00:00
Andrew Jackson d1fc5cedb4 [maven-release-plugin] prepare release 3.4.0-20190205-2 2019-02-05 12:29:26 +00:00
Andrew Jackson 096ba4e9c3 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:15:26 +00:00
Noah Levitt e5e5cd87e4 configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency 2014-10-01 13:38:39 -07:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43 
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
 2014-01-10 17:52:03 -08:00
Noah Levitt 749bd6a214 bump version to 3.3.0-SNAPSHOT after 3.2.0 release 2014-01-10 16:22:43 -08:00
Noah Levitt 32c083f720 bump version number to 3.2.0 for release 2014-01-10 12:36:45 -08:00
Noah Levitt 0ea10665f1 abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner 2013-12-15 11:53:18 -08:00
Noah Levitt 8e1125d062 bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson) 2013-11-20 09:18:31 -08:00
Noah Levitt 1993c4388c Merge branch 'master' into hc43 
Conflicts:
	.classpath
 2013-03-20 14:28:13 -07:00
Noah Levitt 8fd361e518 update source/target version to 1.6 since we use 1.6 features, specifically @Override for methods only implementing an interface (not sure why maven wasn't enforcing this); also remove maven-antrun-plugin created timestamp.txt, doesn't seem to be used anywhere 2013-03-20 14:25:22 -07:00
Noah Levitt 12da824316 Merge branch 'master' into hc43 2013-03-18 23:32:51 -07:00
Noah Levitt fb758e53e3 Avoid pulling in hadoop-core.jar and its dependencies; add plugin versions to get rid of warnings 2013-02-22 15:43:15 -08:00
Noah Levitt a20824aea7 Merge branch 'master' into new-fetchhttp-only 
Conflicts:
	.classpath
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
	modules/src/test/java/org/archive/modules/fetcher/FetchHTTPTest.java
 2012-11-30 17:04:28 -08:00
Noah Levitt b9cbbbd248 Bunch of unit tests for FetchHTTP 
* FetchHTTPTest.java
    a bunch of new tests
* FetchHTTP.java
    make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides)
* modules/pom.xml, .classpath
    dependencies for testing
* engine/pom.xml
    update jetty to 6.1.26 since we require that for the new tests
 2012-09-26 14:41:49 -07:00
Noah Levitt 77559bf81c **/pom.xml 
bump version number because of api change
 2012-09-13 17:47:26 -07:00
Noah Levitt 3f2295353a test dependencies belong in engine/ now too 2012-08-30 12:16:25 -07:00
Noah Levitt f502292eef commons-codec was already pulled in from elsewhere 2012-08-29 19:06:37 -07:00