25a7aa9c8d
Merge pull request #141 from vonrosen/ARI-4659
...
Disable SNI for a request if that request failed due to an SNI error …
2016-01-15 11:13:42 -08:00
368d586eb1
fix NullPointerException when using old HtmlFormCredential login
2016-01-14 16:00:08 -08:00
3b12a751e1
to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly
2016-01-14 12:17:19 -08:00
309847cc40
Be sure to return at least one of the candidates
2016-01-14 12:05:38 -08:00
704ec62487
Remove unneeded constant
2016-01-14 12:05:38 -08:00
a2a39fe052
Make sure to check form encoding type when addign form to curi data
2016-01-14 12:05:38 -08:00
93bca3d55c
More changes for https://webarchive.jira.com/browse/ARI-4656
2016-01-14 12:05:38 -08:00
4997014b34
More changes
2016-01-14 12:05:38 -08:00
ad329ceea8
Save enctype of form in submission curi
2016-01-14 12:05:38 -08:00
34cffd512a
Add ability to submit forms with enctype="multipart/form-data"
2016-01-14 12:05:38 -08:00
afa4f6535b
If multiple username inputs in login form, favor the one with the word 'login' in it.
2016-01-14 12:05:38 -08:00
eeeea6f0c4
Merge pull request #122 from nlevitt/form-login-checked-boxes
...
only submit checkbox and radio button form fields if they are on by d…
2016-01-14 08:00:49 -08:00
9e4c7f0024
Merge branch 'master' into novel-quotas
...
* master:
license header
check that sourceTag of CrawlURI actually matches configured sourceTag
remove already-outdated stuff from javadoc
handle multiple clauses for same user agent in robots.txt
Hook in submitted seeds properly.
avoid spurious logging
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
2016-01-13 13:44:54 -08:00
22336fb71b
fix stats in unusual case of "failed" fetch with response > 0 (only case where this can happen currently is if basic auth is configured for a url, but fails and url returns "401 Unauthorized")
2016-01-12 17:34:27 -08:00
73be17af06
Disable SNI for a request if that request failed due to an SNI error 'javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name'
2016-01-12 16:58:48 -08:00
3dbf754d56
Merge pull request #137 from nlevitt/seed-limits
...
SourceSeedDecideRule, SeedLimitsEnforcer
2016-01-12 16:39:17 -08:00
99c5af0cf2
comment out currently failing test (needs webarchive-commons pull request merged)
2015-12-14 10:46:50 -08:00
f1173789f7
fix test now that ContentDigestHistoryTest.shouldProcess() is smarter
2015-12-11 16:13:17 -08:00
38931f152a
handle multiple clauses for same user agent in robots.txt
2015-12-11 15:42:16 -08:00
dfc1dd940e
tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests
2015-12-11 12:49:02 -08:00
c20a06ec27
Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch
...
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-11 11:35:11 -08:00
294cfa3524
limited test of warc writer stats
2015-12-11 11:22:03 -08:00
9ce12d2dbe
add warc stats to CrawlURI.getData() dynamic attributes list after writing records; new method WARCWriterProcessor.getStats() to expose cumulative stats
2015-12-11 11:21:05 -08:00
35a7954882
refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code
2015-12-11 11:05:44 -08:00
566c1a600d
remove warning which can be misleading if crawl is configured to write only certain urls to warcs
2015-12-11 11:01:26 -08:00
5559a5353d
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
2015-12-03 17:43:26 -08:00
17f58f4bf8
Remove uneeded logger
2015-11-06 10:43:58 -08:00
580391e0a7
Move most of addCookie functionality into AbstractCookieStore
2015-11-06 10:40:43 -08:00
3142bbd485
Revise based on feedback in pull request from nlevitt
2015-11-02 13:07:11 -08:00
44ac1030f2
Make sure to synchronize in addCookie now that domain count limits are imposed and modify unit test to also take into account domain limits
2015-10-16 17:12:42 -07:00
d9684e913b
Restrict number of cookies per domain to 50
2015-10-16 14:05:56 -07:00
5ec6c738a6
Revert "Possible fix that probably isn't needed"
...
This reverts commit c6f019a095
2015-10-13 17:21:28 -07:00
c6f019a095
Possible fix that probably isn't needed
2015-10-13 14:18:06 -07:00
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
f097939a1c
Don't call setRealm so other realms that can be used.
2015-08-14 13:01:54 -07:00
4e22f1f809
Allow realm to be set by server for basic auth.
2015-07-07 13:06:29 -07:00
3c4d09f6c0
only submit checkbox and radio button form fields if they are on by default ("checked" attribute is present)
2015-06-15 17:27:59 -07:00
1027f854ec
for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js")
2015-04-20 17:08:47 -07:00
458cbb51b0
Fix for HER-2082
...
https://webarchive.jira.com/browse/HER-2082
WARCWriterProcessor.java
Add last hop character and link context to outlinks in metadata records
2015-03-26 16:19:09 -07:00
f5107f60b8
Add annotation for server-not-modified duplicates.
2015-03-20 13:46:20 +00:00
dff219e884
Harmonizing whitespace use
2015-03-13 14:32:23 +00:00
843f6468c4
Use digest from previous capture when encountering 304s.
...
The previous capture may also have been a 304 but there must be an
original 200 capture at the start. This assumption may fail when using
an index generated by a version of Heritrix without this fix. But even
then you should be no worse off than before.
2015-03-13 14:30:39 +00:00
3ffe5b8b4f
Stop automatically writing WARC-Payload-Digest for revisit records.
...
Refer that functionality to the relevant revisit profile classes.
2015-03-11 13:52:31 +00:00
a0c3fcdae0
avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory
2015-02-23 14:09:25 -08:00
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
3609abe38c
Merge pull request #108 from nlevitt/aitfive-203
...
kafka crawl log feed
2014-11-14 10:29:41 -08:00
d96b1f8eca
Merge pull request #100 from vonrosen/ari-4061
...
Allow discovery of urls in content attribute of meta tags.
2014-11-08 00:05:00 -08:00
7312ad597e
so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ)
2014-11-07 13:21:42 -08:00
9a41b31264
Handle case where form does not have an action defined.
2014-11-04 11:22:38 -08:00
Noah Levitt