Noah Levitt
38931f152a
handle multiple clauses for same user agent in robots.txt
2015-12-11 15:42:16 -08:00
Hunter Stern
17f58f4bf8
Remove uneeded logger
2015-11-06 10:43:58 -08:00
Hunter Stern
580391e0a7
Move most of addCookie functionality into AbstractCookieStore
2015-11-06 10:40:43 -08:00
Hunter Stern
3142bbd485
Revise based on feedback in pull request from nlevitt
2015-11-02 13:07:11 -08:00
Hunter Stern
44ac1030f2
Make sure to synchronize in addCookie now that domain count limits are imposed and modify unit test to also take into account domain limits
2015-10-16 17:12:42 -07:00
Hunter Stern
d9684e913b
Restrict number of cookies per domain to 50
2015-10-16 14:05:56 -07:00
Hunter Stern
5ec6c738a6
Revert "Possible fix that probably isn't needed"
...
This reverts commit c6f019a095
2015-10-13 17:21:28 -07:00
Hunter Stern
c6f019a095
Possible fix that probably isn't needed
2015-10-13 14:18:06 -07:00
Hunter
79ebb0e03c
Merge pull request #129 from nlevitt/java8keytool
...
load keytool utility main class dynamically, trying both the old and …
2015-10-06 16:09:43 -07:00
Noah Levitt
12dde2b693
load keytool utility main class dynamically, trying both the old and new (java 8) class names
2015-09-23 12:06:11 -07:00
Hunter Stern
f097939a1c
Don't call setRealm so other realms that can be used.
2015-08-14 13:01:54 -07:00
Hunter Stern
4e22f1f809
Allow realm to be set by server for basic auth.
2015-07-07 13:06:29 -07:00
Noah Levitt
1027f854ec
for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js")
2015-04-20 17:08:47 -07:00
Adam Miller
458cbb51b0
Fix for HER-2082
...
https://webarchive.jira.com/browse/HER-2082
WARCWriterProcessor.java
Add last hop character and link context to outlinks in metadata records
2015-03-26 16:19:09 -07:00
Kristinn Sigurðsson
f5107f60b8
Add annotation for server-not-modified duplicates.
2015-03-20 13:46:20 +00:00
Kristinn Sigurðsson
dff219e884
Harmonizing whitespace use
2015-03-13 14:32:23 +00:00
Kristinn Sigurðsson
843f6468c4
Use digest from previous capture when encountering 304s.
...
The previous capture may also have been a 304 but there must be an
original 200 capture at the start. This assumption may fail when using
an index generated by a version of Heritrix without this fix. But even
then you should be no worse off than before.
2015-03-13 14:30:39 +00:00
Kristinn Sigurðsson
3ffe5b8b4f
Stop automatically writing WARC-Payload-Digest for revisit records.
...
Refer that functionality to the relevant revisit profile classes.
2015-03-11 13:52:31 +00:00
Noah Levitt
a0c3fcdae0
avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory
2015-02-23 14:09:25 -08:00
Noah Levitt
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
vonrosen
3609abe38c
Merge pull request #108 from nlevitt/aitfive-203
...
kafka crawl log feed
2014-11-14 10:29:41 -08:00
Noah Levitt
d96b1f8eca
Merge pull request #100 from vonrosen/ari-4061
...
Allow discovery of urls in content attribute of meta tags.
2014-11-08 00:05:00 -08:00
Noah Levitt
7312ad597e
so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ)
2014-11-07 13:21:42 -08:00
Hunter Stern
9a41b31264
Handle case where form does not have an action defined.
2014-11-04 11:22:38 -08:00
Hunter Stern
78e98365d6
Remove unused variable.
2014-10-16 17:51:08 +00:00
vonrosen
5909042b84
Merge pull request #99 from nlevitt/aitfive-39
...
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
2014-10-16 17:24:37 +00:00
Noah Levitt
a27dbd182d
change DecideRuleSequence extraInfo field name 'source' to 'seed' (for reason external to heritrix)
2014-10-14 15:40:07 -07:00
Hunter Stern
3a82b37e52
Ensure Jerico extractor html doesn't call new metadata test method.
2014-10-10 23:33:22 +00:00
Hunter Stern
3347de3d39
Make meta tag content url extraction rules more generate so as to apply to all meta tags 'content' attributes, not just specific og:video.
2014-10-09 23:42:24 +00:00
Hunter Stern
ff5c59c555
Allow extraction of content url in opengraph video meta tag - og:video
2014-10-09 17:57:05 +00:00
Kristinn Sigurðsson
0b16ff0fad
Fix for HER-2074
...
https://webarchive.jira.com/browse/HER-2074
2014-10-03 11:30:38 +00:00
Noah Levitt
5cb5f05437
add field "extra info" to DecideRuleSequence log, which is a json object with entries for host, via, source and hop path
2014-10-02 02:34:32 -07:00
Noah Levitt
e5e5cd87e4
configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency
2014-10-01 13:38:39 -07:00
Noah Levitt
060054e6b3
avoid java 7 api
2014-10-01 13:38:14 -07:00
Noah Levitt
6faa1b8c1c
fix misleading variable name
2014-09-30 11:43:09 -07:00
Noah Levitt
8d98882eb7
an optimization suggested by Gordon: organize cookies by host (domain), rather than top private domain, and supply a composite list of cookies from the host and each parent domain to the http client library
2014-09-29 18:32:31 -07:00
Noah Levitt
5fd120dffd
some javadocs and other comments, some minor refactoring, change delimiter to ';' for the sortable key
2014-09-29 17:51:24 -07:00
Noah Levitt
87b408cc23
rename limited cookie store facade class
2014-09-28 14:35:22 -07:00
Noah Levitt
54eb1c950f
rudimentary test of cookie at url with ip address as host
2014-09-28 14:30:01 -07:00
Noah Levitt
5c21ad1762
AbstractCookieStore.getCookieFor(CrawlURI) computes top private domain, so FetchHTTP doesn't have to know anything about that
2014-09-28 14:29:04 -07:00
Noah Levitt
4aa85af1b0
a few tests exercising some basic domain/subdomain cookie behaviors
2014-09-28 14:15:27 -07:00
Noah Levitt
a37a8de4a2
improved implementation not using TreeSet, and some initial work on integration testing of cookie handling
2014-09-28 11:59:14 -07:00
Noah Levitt
6588641017
restore saving and loading, checkpoints, testing
2014-09-27 13:55:07 -07:00
Noah Levitt
111294e887
oops, commit classes for initial bdb implementation
2014-09-27 12:22:50 -07:00
Noah Levitt
47b7caad29
initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains
2014-09-26 20:56:45 -07:00
Noah Levitt
886a0f2136
(forgot to add SimpleHostCookieSetStore) initial proof of concept fix for HER-2070 "cookie monster" bug
2014-09-26 20:05:37 -07:00
Noah Levitt
f769e0636d
initial proof of concept fix for HER-2070 "cookie monster" bug
2014-09-26 20:01:52 -07:00
Kenji Nagahashi
29379c7e19
Merge pull request #84 from nlevitt/iipc-webarchive-commons
...
switch to mainline iipc webarchive-commons latest release (1.1.4)
2014-09-11 17:19:08 -07:00
Kenji Nagahashi
6ada6801de
Merge pull request #94 from nlevitt/ari-3724
...
do not require "+" (plus sign) before @OPERATOR_CONTACT_URL@ in user-age...
2014-09-11 17:17:55 -07:00
Noah Levitt
e2064237cf
Merge pull request #87 from vonrosen/ari-3977
...
Allow xml extractor to handle urls in CDATA.
2014-09-10 12:42:34 -07:00
Noah Levitt