lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-22 23:41:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,051 Commits 40 Branches 31 Tags
adds-subqueue-support-forced-queue-assignment
Commit Graph
501 Commits
Author SHA1 Message Date
Noah LevittandGitHub cefad96036 Merge pull request #276 from internetarchive/jdk11-upgrade-jetty 
JDK11 support: upgrade to Jetty 9.4.19, Restlet 2.4.0 and drop JDK 7 support
 2019-08-07 10:48:51 -07:00
Noah LevittandGitHub a1f2c9513a Merge pull request #273 from internetarchive/jdk11-remove-old-bdbcache 
JDK11 support: remove unused class ObjectIdentityBdbCache and tests
 2019-08-07 10:48:40 -07:00
Alex Osborne 0782fb0481 JDK11: upgrade engine to jetty 9.4.19 and restlet 2.4.0 
Jetty 9.4.12+ is required for TLS to work correctly under JDK11
(due to SSL handshake failures). In order to upgrade
Jetty we also need to upgrade Restlet.

There is one intentional change in behaviour to simplify
upgrading. We remove a workaround for an old [webkit bug]
where the browser claimed to prefer application/xml. The
bug was fixed in 2011.

[webkit bug]: https://bugs.webkit.org/show_bug.cgi?id=27267

Summary of Jetty API changes:
- package names changed such as org.mortbay -> org.eclipse
- SocketConnector and SslSocketConnector merged to ServerConnector
- HashUserRealm split into UserStore and HashLoginService
- SecurityHandler -> ConstraintSecurityHandler

Summary of Restlet API changes:
- some classes have moved package (Request, Response, Router etc)
- ServerResource replaces Resource
- represent(), acceptRepresentation() renamed to get(), post()
- constructors were replaced by an init() method
- setModifiable() was removed
- getRequest().getEntityAsForm() -> new Form(entity)
- Guard -> ChallengeAuthenticator

Fixes #275
 2019-08-06 15:16:37 +09:00
Alex Osborne a5c2b46208 JDK11 support: remove unused class ObjectIdentityBdbCache and tests 
In 2011 ObjectIdentityBdbCache was replaced by
ObjectIdentityBdbManualCache in order to "remove dependence on
GC/finalization/PhantomReference magic". It hasn't been used since.
As of JDK11 the ObjectIdentityBdbCache unit tests are failing so let's
remove it.
 2019-08-02 17:10:13 +09:00
Alex Osborne 412885fd7d JDK11 support: upgrade maven-surefire-plugin to 2.22.2 
Under JDK11 our old version of it throws ClassNotFoundExceptions when
tests load some builtin classes like javax.transaction.xa.Xid.

Fixes #266
 2019-08-02 16:43:20 +09:00
Noah Levitt 1bd8b713c6 quiet org.mortbay.log (jetty?) logging 
There is already a clause for this in logging.properties, but it's using
log4j. It was dumping stack traces every time the client was dubious of
heritrix's self-signed certificate.

Why do we have so many identical log4j.xml's? 🤷‍♂️
 2019-04-30 16:17:35 -07:00
Andrew Jackson b3961a2f96 [maven-release-plugin] prepare for next development iteration 2019-04-18 15:36:28 +01:00
Andrew Jackson c7c6141ee1 [maven-release-plugin] prepare release 3.4.0-20190418 2019-04-18 15:36:20 +01:00
Andrew Jackson ad9e74d8ad [maven-release-plugin] prepare for next development iteration 2019-02-07 13:52:56 +00:00
Andrew Jackson 83c7044b22 [maven-release-plugin] prepare release 3.4.0-20190207 2019-02-07 13:52:49 +00:00
Andrew Jackson 317b19e7e2 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:33:08 +00:00
Andrew Jackson 9c7d0c299d [maven-release-plugin] prepare release 3.4.0-20190205 2019-02-05 12:32:57 +00:00
Andrew Jackson ab34a54566 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:29:33 +00:00
Andrew Jackson d1fc5cedb4 [maven-release-plugin] prepare release 3.4.0-20190205-2 2019-02-05 12:29:26 +00:00
Andrew Jackson 096ba4e9c3 [maven-release-plugin] prepare for next development iteration 2019-02-05 12:15:26 +00:00
Edward BettsandAlex Osborne aa705bef2e Correct spelling mistakes. 2018-11-04 19:49:37 +09:00
Andrew Jackson d85d2a3fe8 Move OneLineSimpleLayout into commons so all can use it. 2017-10-12 22:10:58 +01:00
Noah Levitt ae8a0ef95f update webarchive-commons dependency to incorporate fix to warc size calculation, and uncomment test that exercises the fix 2017-10-11 14:54:37 -07:00
Noah Levitt 3892c17b56 ughhh appease java 8 javadoc rules 2017-06-15 17:29:22 -07:00
Mohamed ElsayedandNoah Levitt d13bfb6a37 Remove unit tests which moved over from Heritrix3 to webarchive-commons 2017-03-09 10:26:32 -08:00
Noah Levitt 36ea1a9fb2 try to fix remaining serialization tests in oraclejdk8 by using ConcurrentSkipListMap instead of ConcurrentHashMap 2016-06-07 17:55:45 -05:00
Noah Levitt 99d87e8f23 hopefully fix serialization tests in oraclejdk8 by using TreeSet instead of HashSet in KeyedProperties.java 2016-06-07 14:53:11 -05:00
Noah Levitt cbb5e3ab99 Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions. 2016-02-09 15:02:11 -08:00
Hunter Stern 088189ee8b Fix up code based on pull requests comments. Add test for urls with spaces in them. 2016-01-28 15:11:49 -08:00
Hunter Stern 4518bc2da8 Make sure no urls with whitespace and not having good extension slip through 2016-01-20 14:29:42 -08:00
Hunter Stern 12a88d6d23 Allow spaces in urls extracted from JS. 2016-01-20 10:44:20 -08:00
Hunter Stern 23885557bb Allow spaces in JavaScript urls, but only if they have a known good file extension 2016-01-19 18:07:05 -08:00
Noah Levitt 3b12a751e1 to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly 2016-01-14 12:17:19 -08:00
Hunter SternandNoah Levitt 34cffd512a Add ability to submit forms with enctype="multipart/form-data" 2016-01-14 12:05:38 -08:00
Noah Levitt b7afc627ed fix typo 2015-09-24 15:31:12 -07:00
Noah Levitt 9aabcb3272 add license header to new class 2015-09-23 12:07:38 -07:00
Noah Levitt 12dde2b693 load keytool utility main class dynamically, trying both the old and new (java 8) class names 2015-09-23 12:06:11 -07:00
Noah Levitt 1027f854ec for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js") 2015-04-20 17:08:47 -07:00
Noah Levitt 8009275075 make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/ 2015-02-23 11:36:29 -08:00
Noah Levitt 1bd8b141aa bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39 2015-02-11 18:53:34 -08:00
Noah Levitt 2078a9f0e2 remove TmpDirTestCase.java (moving to webarchive-commons) 2015-01-06 16:08:18 -08:00
Noah Levitt 64e3b1b02a remove RecordingOutputStreamTest.java (moving to webarchive-commons) 2015-01-06 16:01:16 -08:00
Noah Levitt 78e6b8b6ab replace accidental U+2212 MINUS SIGN with normal ascii 0x2d '-' to avoid compilation error "unmappable character for encoding ASCII" 2014-10-01 21:31:14 -07:00
Noah Levitt e5e5cd87e4 configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency 2014-10-01 13:38:39 -07:00
Noah Levitt a37a8de4a2 improved implementation not using TreeSet, and some initial work on integration testing of cookie handling 2014-09-28 11:59:14 -07:00
Kenji Nagahashi 29379c7e19 Merge pull request #84 from nlevitt/iipc-webarchive-commons 
switch to mainline iipc webarchive-commons latest release (1.1.4)
 2014-09-11 17:19:08 -07:00
Noah Levitt 1d0f60a2be webarchive-commons 1.1.4 has been released, bump dependency 2014-09-10 12:31:01 -07:00
Noah Levitt 935b8f9658 remove unused dependency jets3t (thanks kenji) 2014-09-02 11:00:34 -07:00
Noah Levitt ec0839a6fe remove more unused classes 2014-08-29 15:23:21 -07:00
Kenji Nagahashi bb511796ec Update POMs for newly imported Wayback-dedup module. 
lock httpclient version to 4.3.3 stated by
	heritrix-commons. have to exclude hadoop’s
	dependency on jets3t 0.9.0 as it pulls in		httpcore 4.1.2.
	add easy mock 3.1 to heritrix-contrib.
 2014-08-25 16:44:48 -07:00
Noah Levitt f1b0633a3d remove unused Transform* classes 2014-08-13 10:53:03 -07:00
Noah Levitt 651d18beb1 replace use of method removed from guava library 2014-08-08 17:38:51 -07:00
Noah Levitt 21f17a0d9f switch to mainline iipc webarchive-commons latest release 2014-08-08 17:26:21 -07:00
Noah Levitt 339e6ec87a for ARI-3723, accept speculative urls with query params with no value 2014-04-04 10:15:07 -07:00
Noah Levitt cc838747af pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available 2014-03-31 13:38:54 -07:00