lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-10-02 02:17:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,410 Commits 41 Branches 33 Tags
ed332bcb9e84d65bee5ce71a4f278fe459b60cc1
Commit Graph
559 Commits
Author SHA1 Message Date
Alex Osborne 69f231b8ab Merge pull request #468 from internetarchive/remove-jna 2022-03-03 17:16:34 +09:00
Alex Osborne 98f42de863 Merge pull request #460 from internetarchive/dependabot/maven/commons/org.springframework-spring-core-5.3.14 2022-03-03 17:14:46 +09:00
Alex Osborne 71b40cf882 Use Files.createLink() and Files.createSymbolicLink() instead of JNA 
We don't need to support Java < 7 anymore. This simplifies the
code and should improve portability to platforms our old version of
JNA doesn't support like Apple Silicon.

Fixes #467
 2022-03-02 21:37:27 +09:00
Clemens Klein-Robbenhaar fae2ccb88d Fix name of parameter in setMaxLogFileSize 
Fixes #464
 2022-02-13 13:01:58 +01:00
dependabot[bot] ab27a48e26 Bump spring-core from 5.3.3 to 5.3.14 in /commons 
Bumps [spring-core](https://github.com/spring-projects/spring-framework) from 5.3.3 to 5.3.14.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](https://github.com/spring-projects/spring-framework/compare/v5.3.3...v5.3.14)

---
updated-dependencies:
- dependency-name: org.springframework:spring-core
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-27 03:03:08 +00:00
Andrew Jackson 024a7772b5 [maven-release-plugin] prepare for next development iteration 2021-09-23 22:08:33 +01:00
Andrew Jackson fedff1e2c7 [maven-release-plugin] prepare release 3.4.0-20210923 2021-09-23 22:08:30 +01:00
Andrew Jackson ddb695abad [maven-release-plugin] prepare for next development iteration 2021-08-03 09:56:26 +01:00
Andrew Jackson ba44cce7f2 [maven-release-plugin] prepare release 3.4.0-20210803 2021-08-03 09:56:20 +01:00
Alex Osborne eb67cfb215 Merge pull request #423 from internetarchive/dont-extract-data-uris 
Don't extract data URIs
 2021-08-03 07:43:17 +09:00
Alex Osborne e33e1ff398 Extractor: ignore data URIs when adding outlinks 2021-07-30 15:57:09 +09:00
Alex Osborne 636948cb09 KeyTool wrapper: fallback to running keytool as a subprocess on JDK 16+ 
JDK 16 defaults to --illegal-access=deny which means trying to call
KeyTool via reflection now throws IllegalAccessException.

Fixes #417
 2021-07-25 11:15:45 +09:00
Adam Miller d2d9d4f157 Adding configurable MaxLogFileSize option to BdbModule 2021-07-19 15:49:56 +09:00
Alex Osborne 6835d0940e Merge pull request #407 from internetarchive/fix-maven-warnings 
Fix a couple of boring maven warnings
 2021-06-25 11:52:53 +09:00
Alex Osborne 039634cee0 Update relocated dependencies 
Fixes the Maven warnings:

    [WARNING] The artifact poi:poi:jar:2.5.1 has been relocated to
        poi:poi:jar:2.5.1-final-20040804
    [WARNING] The artifact itext:itext:jar:1.3 has been relocated to
        com.lowagie:itext:jar:1.3
 2021-06-22 20:52:55 +09:00
Alex Osborne d643770a7c Put slowest unit tests behind a -DrunSlowTests=true option 
Shaves about 2 minutes off the Heritrix build time.
 2021-06-22 17:10:50 +09:00
Andrew Jackson 0e93e22ba9 [maven-release-plugin] prepare for next development iteration 2021-06-21 15:42:52 +01:00
Andrew Jackson 846f2bc2cc [maven-release-plugin] prepare release 3.4.0-20210621 2021-06-21 15:42:46 +01:00
Alex Osborne 5d0a5b2ec4 Remove dependency on mg4j 
This eliminates one of the jars that Heritrix depends on that isn't in
Maven Central.

There were only two usages. The first usage was a method in a
CrawlerJournal which as far as I can tell is unused and can be removed.

The second usage was an unnecessary use of MutableString as a line
buffer when saving cookies to a file. Since there doesn't seem any
particular reason buffer per-line here let's just use a BufferedWriter
instead on the whole file which is also slightly simpler and more
efficient.
 2021-06-20 23:30:41 +09:00
Andrew Jackson a5beb7d6e9 [maven-release-plugin] prepare for next development iteration 2021-06-18 09:41:30 +01:00
Andrew Jackson 30598f8402 [maven-release-plugin] prepare release 3.4.0-20210618 2021-06-18 09:41:24 +01:00
Andrew Jackson 3423cab6a6 [maven-release-plugin] prepare for next development iteration 2021-06-17 14:39:00 +01:00
Andrew Jackson 7368ded9c2 [maven-release-plugin] prepare release 3.4.0-20210617 2021-06-17 14:38:54 +01:00
Andrew Jackson d3dfb34f83 [maven-release-plugin] prepare for next development iteration 2021-05-27 10:03:10 +01:00
Andrew Jackson 9dcbb240a6 [maven-release-plugin] prepare release 3.4.0-20210527 2021-05-27 10:03:04 +01:00
Andy Jackson 669cf1ed2b Merge branch 'master' into update-dependencies 2021-05-26 13:40:15 +01:00
Andy Jackson a2c65f4618 Update dependencies for security issues, including changes to tests. 2021-05-26 13:39:30 +01:00
Andy Jackson 276ced0de3 Allow tuning of BDB-JE evictor and cleaner threads. 2021-05-20 22:22:00 +01:00
Andy Jackson c0dbfb2a0a Merge pull request #383 from ukwa/upgrade-dnsjava 
Update to latest version of dnsjava, for #344
 2021-05-20 21:05:59 +01:00
Andy Jackson 396467c535 Merge branch 'master' into sitemaps 2021-05-20 21:03:54 +01:00
Andy Jackson 045b2516da Update to latest version of dnsjava, for #344 
This commit updates to version 3.3.1 of the dnsjava library.

Incidentally, the restlet.org certificate is currently invalid, so this
patch switched to maven.restlet.com.
 2021-05-20 20:44:19 +01:00
Alex Osborne 5f52a14afc Upgrade to Spring 5.3.3 
Spring 3.0.5 is 10 years old now and is incompatible with classes using
Java 9+ bytecode and even on Java 8 classes using lambdas (see #337).

We remove spring-asm as it is now part of spring-core module.
 2021-02-15 16:20:00 +09:00
Andrew Jackson e215b89963 [maven-release-plugin] prepare for next development iteration 2020-05-18 14:08:12 +01:00
Andrew Jackson ef38dc08c3 [maven-release-plugin] prepare release 3.4.0-20200518 2020-05-18 14:08:04 +01:00
Leslie Bellony 63ca9d4c6a Remove static fields and byte loop + rename variable 2020-04-23 17:10:42 +02:00
Leslie Bellony 40725372d6 Add support for the SFTP protocol 2020-04-01 14:47:34 +02:00
Andrew Jackson ff05de44b6 [maven-release-plugin] prepare for next development iteration 2020-03-04 23:05:41 +00:00
Andrew Jackson a1bdcb1be7 [maven-release-plugin] prepare release 3.4.0-20200304 2020-03-04 23:05:33 +00:00
Alex Osborne fd514b7bf6 Merge pull request #315 from ukwa/fix-oracle-doc-link 
Use the Wayback Machine to repair a link to Oracle docs.
 2020-03-05 01:40:39 +09:00
Andrew Jackson b4494a7ffb Use the Wayback Machine to repair a link to Oracle docs. 2020-03-04 15:42:29 +00:00
Tim Hennekey 348b5330bc Utilize the d parameter 
There are uses of this class that necessitate a lower false positive
rate than the value that was hard-coded. By making use of the parameter,
as indicated in the JavaDoc, the resulting delegate should meet the
demands of the caller.
 2020-03-04 09:37:31 -05:00
Andy Jackson 7f80b4fd98 Merge pull request #281 from ukwa/upgrade-bdb-je 
WIP: Upgrade BDB JE to version 7.5.11
 2020-03-04 13:36:03 +00:00
Jonathan Leitschuh bc6a15b41d Use HTTPS to resolve dependencies in Maven Build where possible 2020-02-11 17:48:20 +09:00
Alex Osborne b45f774066 Merge pull request #300 from hennekey/fix-299 
Use Guice instead of custom implementation
 2020-01-30 19:12:05 +09:00
Alex Osborne fd9136dbfa Merge pull request #304 from hennekey/remove-custom-base32 
Replace custom Base32 encoding
 2020-01-30 19:07:09 +09:00
Tim Hennekey 54e05a7864 Correct encoding 
The previous implementation appears to always have returned upper case,
was able to encode either case, and did not reutrn padding.
 2020-01-28 15:12:54 -05:00
Tim Hennekey b5f95c5e06 Replace custom Base32 encoding 
Guava is available so a custom implementation is unnecessary.
 2020-01-24 12:46:37 -05:00
Alex Osborne 9927e1325f Merge pull request #296 from hennekey/update-junit 
Set JUnit version to latest
 2020-01-23 16:44:59 +09:00
Tim Hennekey 9cb9563da3 Increment the count only when the filter notes it 
Otherwise this is a count of how many times this add method is called,
not how many times an element was noted as being actually added.
 2020-01-22 17:15:24 -05:00
Tim Hennekey 66723aea68 Remove version 
Allow the parent POM to specify the version
 2020-01-21 10:52:15 -05:00