Alex Osborne
6f32d018be
Merge pull request #405 from internetarchive/faster-tests
...
Speed up test suite
2021-06-25 11:52:40 +09:00
Alex Osborne
04f958e987
Merge pull request #404 from internetarchive/github-ci
...
Switch from Travis CI to Github Actions
2021-06-25 11:52:01 +09:00
Alex Osborne
c0cb68f754
Speed up the unit tests by changing some 1s polling sleeps to 250ms
...
Lots of 1 second sleeps add up fast. On my PC this reduces the runtime
of `mvn clean test` from about 3m 40s to about 1m 50s.
Ideally we'd probably re-architect some this to use a notification
mechanism instead of polling but that's easier said than done and this
is a pretty big improvement by itself.
2021-06-22 17:10:50 +09:00
Alex Osborne
16a49d3527
Remove arbitrary 1.5 second sleep() when launching jobs
...
I think the sleep is supposed to make launch() not return until the job
has actually been launched but it doesn't work as launch()
and getCrawlController() are both synchronized therefore the
launcher thread can't actually call startContext() until launch()
returns after sleeping.
So let's replace the sleep call with join and unsynchronize launch()
so it doesn't deadlock. All the relevant methods it calls seem to be
synchronized so I think it's no worse to not synchronize it itself.
2021-06-22 17:10:50 +09:00
Alex Osborne
d643770a7c
Put slowest unit tests behind a -DrunSlowTests=true option
...
Shaves about 2 minutes off the Heritrix build time.
2021-06-22 17:10:50 +09:00
Alex Osborne
175f2938d1
Workaround Maven connection timeout in Github Action
2021-06-22 13:13:18 +09:00
Alex Osborne
a5c610f488
Switch from Travis CI to Github Actions
...
The CI builds are no longer working as travis-ci.org has been
discontinued so switch to using Github Actions instead.
We need to use a custom settings.xml for now as Maven now blocks
non-https repositories by default and builds.archive.org isn't
available via https.
2021-06-22 12:16:08 +09:00
Andrew Jackson
c9369da6b8
Updated changelog.
2021-06-21 22:09:51 +01:00
Andrew Jackson
0e93e22ba9
[maven-release-plugin] prepare for next development iteration
2021-06-21 15:42:52 +01:00
Andrew Jackson
846f2bc2cc
[maven-release-plugin] prepare release 3.4.0-20210621
2021-06-21 15:42:46 +01:00
Andrew Jackson
33a01a3bb6
Merge branch 'master' of github.com:internetarchive/heritrix3
2021-06-21 14:28:38 +01:00
Andrew Jackson
6d0ae8f916
Remove unnecessary Maven setup that breaks cross-JDK builds.
2021-06-21 14:18:52 +01:00
Andrew Jackson
c75e710183
Update Changlog.
2021-06-21 14:18:06 +01:00
Andy Jackson
31e54ccdb0
Merge pull request #402 from internetarchive/remove-mg4j
...
Remove dependency on mg4j
2021-06-20 23:08:24 +01:00
Alex Osborne
b917a32eab
Merge pull request #401 from kris-sigur/shutdown
...
Graceful UI shutdown
2021-06-20 23:47:44 +09:00
Alex Osborne
5d0a5b2ec4
Remove dependency on mg4j
...
This eliminates one of the jars that Heritrix depends on that isn't in
Maven Central.
There were only two usages. The first usage was a method in a
CrawlerJournal which as far as I can tell is unused and can be removed.
The second usage was an unnecessary use of MutableString as a line
buffer when saving cookies to a file. Since there doesn't seem any
particular reason buffer per-line here let's just use a BufferedWriter
instead on the whole file which is also slightly simpler and more
efficient.
2021-06-20 23:30:41 +09:00
Kristinn Sigurdsson
37913cbb2e
Graceful UI shutdown
2021-06-18 10:56:20 +00:00
Andrew Jackson
a34e40a9f8
Update changelog.
2021-06-18 10:25:54 +01:00
Andy Jackson
50de3eea08
Merge pull request #400 from kris-sigur/robots-sitemap
...
Remove unnecessary fiddling with VIA path in ExtractorRobotsTxt
2021-06-18 10:14:11 +01:00
Andrew Jackson
07ada5f838
Update changelog.
2021-06-18 09:57:51 +01:00
Kristinn Sigurdsson
627befe07e
Remove unnecessary fiddling with VIA path
2021-06-18 08:42:05 +00:00
Andrew Jackson
a5beb7d6e9
[maven-release-plugin] prepare for next development iteration
2021-06-18 09:41:30 +01:00
Andrew Jackson
30598f8402
[maven-release-plugin] prepare release 3.4.0-20210618
2021-06-18 09:41:24 +01:00
Andrew Jackson
22b07e151a
Merge branch 'master' of github.com:internetarchive/heritrix3
2021-06-18 09:37:37 +01:00
Andrew Jackson
768c6ef458
Update changelog.
2021-06-18 09:37:13 +01:00
Andy Jackson
b60ae999cc
Merge pull request #399 from ukwa/master
...
Switch to properties that enforce Java 8 compatibility.
2021-06-17 21:23:39 +01:00
Andrew Jackson
2007ddb005
Switch to properties that enforce Java 8 compatibility.
2021-06-17 21:22:04 +01:00
Andrew Jackson
18e8779398
Updated changelog.
2021-06-17 16:42:23 +01:00
Andrew Jackson
3423cab6a6
[maven-release-plugin] prepare for next development iteration
2021-06-17 14:39:00 +01:00
Andrew Jackson
7368ded9c2
[maven-release-plugin] prepare release 3.4.0-20210617
2021-06-17 14:38:54 +01:00
Andrew Jackson
a5164dbcea
Update changelog.
2021-06-17 14:34:22 +01:00
Andy Jackson
4b82d6060c
Merge pull request #398 from kris-sigur/sitemap-tweak
...
Annotate nested sitemap links as sitemaps
2021-06-17 10:10:58 +01:00
Kristinn Sigurdsson
81bef22270
Mark nested sitemap links as sitemaps
2021-06-16 13:07:18 +00:00
Andrew Jackson
d0221642c3
Update changelog.
2021-06-14 11:04:33 +01:00
Andrew Jackson
09c8d5696b
Use HTTPS version for Oracle Maven repo.
2021-06-14 11:03:32 +01:00
Andy Jackson
169b3cc8ab
Merge pull request #393 from ukwa/checkpoint-success-stats-392
...
Only update last checkpoint stats if the checkpoint completed, for #392 .
2021-05-28 14:20:06 +01:00
Andy Jackson
699f107ccd
Merge pull request #394 from ukwa/update-ampq-client
...
Update AMPQ client library to address security warning.
2021-05-28 13:24:23 +01:00
Andy Jackson
921e349939
Update AMPQ client library to address security warning.
2021-05-28 12:47:59 +01:00
Andy Jackson
5caa83d327
Only update last checkpoint stats if the checkpoint completed, for #392 .
2021-05-28 12:42:17 +01:00
Andy Jackson
773b4a074d
Merge pull request #391 from ukwa/master
...
Sync changelog with release.
2021-05-27 10:27:40 +01:00
Andy Jackson
a87937caf9
Sync changelog with release.
2021-05-27 10:26:50 +01:00
Andrew Jackson
d3dfb34f83
[maven-release-plugin] prepare for next development iteration
2021-05-27 10:03:10 +01:00
Andrew Jackson
9dcbb240a6
[maven-release-plugin] prepare release 3.4.0-20210527
2021-05-27 10:03:04 +01:00
Andy Jackson
cf0f4ef144
Merge pull request #390 from ukwa/master
...
Update changelog.
2021-05-27 09:46:41 +01:00
Andy Jackson
9df4a0dc05
Update changelog.
2021-05-27 09:45:47 +01:00
Andy Jackson
3b4c442683
Merge pull request #389 from ukwa/update-dependencies-2021-05-26
...
Update dependencies 2021 05 26
2021-05-26 21:56:21 +01:00
Andy Jackson
669cf1ed2b
Merge branch 'master' into update-dependencies
2021-05-26 13:40:15 +01:00
Andy Jackson
a2c65f4618
Update dependencies for security issues, including changes to tests.
2021-05-26 13:39:30 +01:00
Andy Jackson
d175040e33
Merge pull request #386 from ukwa/master
...
Bring changlog up to date
2021-05-20 22:50:43 +01:00
Andrew Jackson
400677a46a
Merge branch 'master' of github.com:internetarchive/heritrix3
2021-05-20 22:49:05 +01:00
Alex Osborne