Hunter
bfbd61edd7
Merge pull request #145 from vonrosen/ARI-4713
...
Make some urls with whitespace acceptable to JavaScript extractor.
2016-02-11 08:27:52 -08:00
Noah Levitt
cbb5e3ab99
Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions.
2016-02-09 15:02:11 -08:00
Hunter Stern
30ce2f09a0
Take care of possible NullPointerException
2016-02-09 08:22:39 -08:00
Hunter Stern
e7c46753ce
Fix formatting
2016-01-29 18:26:18 -08:00
Hunter Stern
e83ff04e86
More fixes for https://webarchive.jira.com/browse/ARI-4656
2016-01-29 17:25:06 -08:00
Hunter Stern
088189ee8b
Fix up code based on pull requests comments. Add test for urls with spaces in them.
2016-01-28 15:11:49 -08:00
Hunter Stern
12a88d6d23
Allow spaces in urls extracted from JS.
2016-01-20 10:44:20 -08:00
Hunter Stern
23885557bb
Allow spaces in JavaScript urls, but only if they have a known good file extension
2016-01-19 18:07:05 -08:00
Hunter
34c90966a8
Merge pull request #143 from nlevitt/form-login-extra-inputs
...
handle login forms with <input type="text"> fields in addition to use…
2016-01-19 08:37:05 -08:00
Noah Levitt
25a7aa9c8d
Merge pull request #141 from vonrosen/ARI-4659
...
Disable SNI for a request if that request failed due to an SNI error …
2016-01-15 11:13:42 -08:00
Noah Levitt
e92e5f2bb3
Merge branch 'master' into form-login-extra-inputs
...
* also fix new FormLoginProcessorTest.testFormLoginExtraInputs() in context of form-login-multipart changes
* master:
fix NullPointerException when using old HtmlFormCredential login
to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly
Be sure to return at least one of the candidates
Remove unneeded constant
Make sure to check form encoding type when addign form to curi data
More changes for https://webarchive.jira.com/browse/ARI-4656
More changes
Save enctype of form in submission curi
Add ability to submit forms with enctype="multipart/form-data"
If multiple username inputs in login form, favor the one with the word 'login' in it.
2016-01-15 09:39:24 -08:00
Noah Levitt
c1abbfa5bd
handle login forms with <input type="text"> fields in addition to username (for this to work there must be exactly one input field whose name contains the string "username")
2016-01-14 18:02:54 -08:00
Noah Levitt
368d586eb1
fix NullPointerException when using old HtmlFormCredential login
2016-01-14 16:00:08 -08:00
Noah Levitt
3b12a751e1
to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly
2016-01-14 12:17:19 -08:00
Hunter Stern and Noah Levitt
309847cc40
Be sure to return at least one of the candidates
2016-01-14 12:05:38 -08:00
Hunter Stern and Noah Levitt
704ec62487
Remove unneeded constant
2016-01-14 12:05:38 -08:00
Hunter Stern and Noah Levitt
a2a39fe052
Make sure to check form encoding type when addign form to curi data
2016-01-14 12:05:38 -08:00
Hunter Stern and Noah Levitt
93bca3d55c
More changes for https://webarchive.jira.com/browse/ARI-4656
2016-01-14 12:05:38 -08:00
Hunter Stern and Noah Levitt
4997014b34
More changes
2016-01-14 12:05:38 -08:00
Hunter Stern and Noah Levitt
ad329ceea8
Save enctype of form in submission curi
2016-01-14 12:05:38 -08:00
Hunter Stern and Noah Levitt
34cffd512a
Add ability to submit forms with enctype="multipart/form-data"
2016-01-14 12:05:38 -08:00
Hunter Stern and Noah Levitt
afa4f6535b
If multiple username inputs in login form, favor the one with the word 'login' in it.
2016-01-14 12:05:38 -08:00
Hunter
eeeea6f0c4
Merge pull request #122 from nlevitt/form-login-checked-boxes
...
only submit checkbox and radio button form fields if they are on by d…
2016-01-14 08:00:49 -08:00
Noah Levitt
9e4c7f0024
Merge branch 'master' into novel-quotas
...
* master:
license header
check that sourceTag of CrawlURI actually matches configured sourceTag
remove already-outdated stuff from javadoc
handle multiple clauses for same user agent in robots.txt
Hook in submitted seeds properly.
avoid spurious logging
try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability
2016-01-13 13:44:54 -08:00
Noah Levitt
22336fb71b
fix stats in unusual case of "failed" fetch with response > 0 (only case where this can happen currently is if basic auth is configured for a url, but fails and url returns "401 Unauthorized")
2016-01-12 17:34:27 -08:00
Hunter Stern
73be17af06
Disable SNI for a request if that request failed due to an SNI error 'javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name'
2016-01-12 16:58:48 -08:00
Adam Miller
3dbf754d56
Merge pull request #137 from nlevitt/seed-limits
...
SourceSeedDecideRule, SeedLimitsEnforcer
2016-01-12 16:39:17 -08:00
Noah Levitt
99c5af0cf2
comment out currently failing test (needs webarchive-commons pull request merged)
2015-12-14 10:46:50 -08:00
Noah Levitt
f1173789f7
fix test now that ContentDigestHistoryTest.shouldProcess() is smarter
2015-12-11 16:13:17 -08:00
Noah Levitt
38931f152a
handle multiple clauses for same user agent in robots.txt
2015-12-11 15:42:16 -08:00
Noah Levitt
dfc1dd940e
tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests
2015-12-11 12:49:02 -08:00
Noah Levitt
c20a06ec27
Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch
...
* seed-limits:
change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable
fix checkpointing problems with new statsBySource
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality
2015-12-11 11:35:11 -08:00
Noah Levitt
294cfa3524
limited test of warc writer stats
2015-12-11 11:22:03 -08:00
Noah Levitt
9ce12d2dbe
add warc stats to CrawlURI.getData() dynamic attributes list after writing records; new method WARCWriterProcessor.getStats() to expose cumulative stats
2015-12-11 11:21:05 -08:00
Noah Levitt
35a7954882
refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code
2015-12-11 11:05:44 -08:00
Noah Levitt
566c1a600d
remove warning which can be misleading if crawl is configured to write only certain urls to warcs
2015-12-11 11:01:26 -08:00
Noah Levitt
5559a5353d
SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds
2015-12-03 17:43:26 -08:00
Hunter Stern
17f58f4bf8
Remove uneeded logger
2015-11-06 10:43:58 -08:00
Hunter Stern
580391e0a7
Move most of addCookie functionality into AbstractCookieStore
2015-11-06 10:40:43 -08:00
Hunter Stern
3142bbd485
Revise based on feedback in pull request from nlevitt
2015-11-02 13:07:11 -08:00
Hunter Stern
44ac1030f2
Make sure to synchronize in addCookie now that domain count limits are imposed and modify unit test to also take into account domain limits
2015-10-16 17:12:42 -07:00
Hunter Stern
d9684e913b
Restrict number of cookies per domain to 50
2015-10-16 14:05:56 -07:00
Hunter Stern
5ec6c738a6
Revert "Possible fix that probably isn't needed"
...
This reverts commit c6f019a095
2015-10-13 17:21:28 -07:00
Hunter Stern
c6f019a095
Possible fix that probably isn't needed
2015-10-13 14:18:06 -07:00
Hunter
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
Noah Levitt
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
Hunter Stern
f097939a1c
Don't call setRealm so other realms that can be used.
2015-08-14 13:01:54 -07:00
Hunter Stern
4e22f1f809
Allow realm to be set by server for basic auth.
2015-07-07 13:06:29 -07:00
Noah Levitt
3c4d09f6c0
only submit checkbox and radio button form fields if they are on by default ("checked" attribute is present)
2015-06-15 17:27:59 -07:00
Noah Levitt
1027f854ec
for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js")
2015-04-20 17:08:47 -07:00
Hunter