 Noah Levitt
|
32c083f720
|
bump version number to 3.2.0 for release
|
2014-01-10 12:36:45 -08:00 |
|
|
Noah Levitt
|
84e78db40a
|
for url-agnostic dedup, follow "Proposal for Standardizing the Recording of Arbitrary Duplicates in WARC Files" https://docs.google.com/document/d/1QyQBA7Ykgxie75V8Jziz_O7hbhwf7PF6_u9O6w6zgp0
|
2014-01-07 14:43:26 -08:00 |
|
|
Noah Levitt
|
0ea10665f1
|
abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner
|
2013-12-15 11:53:18 -08:00 |
|
|
Noah Levitt
|
3b06c88537
|
test fix (in ia-web-commons) for HER-2059
|
2013-12-06 19:22:16 -08:00 |
|
|
Noah Levitt
|
8e1125d062
|
bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson)
|
2013-11-20 09:18:31 -08:00 |
|
|
Noah Levitt
|
d93b5449be
|
new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc
|
2013-11-19 11:05:14 -08:00 |
|
|
Noah Levitt
|
2b87219496
|
oops, avoid npe
|
2013-11-15 17:59:34 -08:00 |
|
|
Noah Levitt
|
df2a313797
|
throw exception when host cache returns 0.0.0.0 for any host (trying to track down a strange problem)
|
2013-11-15 17:20:55 -08:00 |
|
|
Noah Levitt
|
b137022b79
|
Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.)
|
2013-10-28 17:35:55 -07:00 |
|
|
Noah Levitt
|
38a12b2008
|
avoid NPE resuming from checkpoint if "stats" is {} in warcWriter checkpoint json
|
2013-10-11 14:44:34 -07:00 |
|
|
Noah Levitt
|
a1f84d3cc4
|
HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility
|
2013-10-10 17:57:18 -07:00 |
|
|
Noah Levitt
|
071cbe99cd
|
include urlsWritten in checkpoint, and be backward compatible with old checkpoints
|
2013-09-07 23:40:17 -07:00 |
|
|
Noah Levitt
|
c49b465d82
|
custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test
|
2013-09-07 19:40:11 -07:00 |
|
|
Noah Levitt
|
52d72e661d
|
part of HER-2048 - WARCWriterProcessor stats resume from checkpoint
|
2013-09-07 11:35:03 -07:00 |
|
|
Noah Levitt
|
e0f78976d7
|
part of HER-2048 - restore BdbServerCache from checkpoint
|
2013-09-07 10:53:22 -07:00 |
|
|
Noah Levitt
|
25d4de186f
|
don't log warning in normal case
|
2013-09-06 22:04:56 -07:00 |
|
|
Noah Levitt
|
418c7138a9
|
handle case where there is no whois server for a domain
|
2013-08-28 18:13:53 -07:00 |
|
|
Noah Levitt
|
4abfcb01b0
|
HER-1895 support HTTP Refresh header
|
2013-08-26 18:33:02 -07:00 |
|
|
Noah Levitt
|
68adaa70fd
|
need to close the recorder here so it can calculate the content length properly
|
2013-08-09 19:14:58 -07:00 |
|
|
Noah Levitt
|
705a375daf
|
uses of UriUtils.isLikelyUri() in Extractor{HTML,SWF,XML} with UriUtils.isVeryLikelyUri() to reap the benefits of HER-1523 improvements (should address archive-it issue ARI-3492)
|
2013-08-09 18:17:11 -07:00 |
|
|
Noah Levitt
|
8c64cba9ae
|
TransclusionDecideRule.java - consider form submission "S" hop as navlink like "L", not transcluded
|
2013-08-05 16:05:02 -07:00 |
|
|
Noah Levitt
|
f4f3b9be38
|
add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs
|
2013-07-26 17:50:36 -07:00 |
|
|
Noah Levitt
|
72094213a4
|
MirrorWriterProcessor.java - change "path" setting to a ConfigPath and set default value "${launchId}/mirror"
|
2013-07-22 17:54:06 -07:00 |
|
|
Adam Miller
|
b85828135b
|
Convert MatchesStatusCodeDecideRule and NotMatchesStatusCodeDecideRule to use CrawlURI.getFetchStatus() to avoid null status codes and broaden use.
Added test cases for MatchesStatusCodeDecideRule and NotMatchesStatusCodeDecideRule
Fix comments for youtube extractor.
|
2013-06-21 12:03:41 -07:00 |
|
|
Noah Levitt
|
f2984881f6
|
address HER-2042 - new annotation duplicate:uriAgnosticDigest for all url agnostic duplicates, whether written to warcs or not; stats consult this annotation so that duplicate/novel numbers are correct
|
2013-06-07 17:17:24 -07:00 |
|
|
Noah Levitt
|
ffd248f780
|
HER-2040 ExtractorJS shouldExtract when content-type is application/json
|
2013-06-03 11:23:16 -07:00 |
|
|
Noah Levitt
|
d4c5bd6e98
|
avoid npe by setting default <input> type "text" when unspecified
|
2013-05-14 18:17:09 -07:00 |
|
|
Noah Levitt
|
d972731958
|
ExtractorHTMLForms.java - handle a corner case extracting attribute values
|
2013-05-14 16:23:41 -07:00 |
|
|
Noah Levitt
|
7bd010e3c7
|
HTMLForm.java - default html <input> type is text, so consider <input> with no type as candidateUsernameInput
|
2013-05-13 22:53:37 -07:00 |
|
|
Noah Levitt
|
bea5c6f267
|
followup on HER-2031 ExtractorHTMLForms.java - improve regexes looking for form and input attributes, mainly to avoid maxing cpu for long periods of time for certain input
|
2013-05-13 22:37:02 -07:00 |
|
|
Noah Levitt
|
ec824e9724
|
avoid need for getRawData() method by introducing Link.hasData()
|
2013-05-13 17:43:45 -07:00 |
|
|
Kristinn Sigurðsson
|
7c5b2ceb00
|
Added a data map to Link. Becomes the CrawlURI's data map when Link is
promoted to CrawlURI
|
2013-05-13 15:18:02 +00:00 |
|
|
Noah Levitt
|
d5693563f8
|
fix bug pointed out in https://webarchive.jira.com/browse/ARI-3176?focusedCommentId=35314 - url-agnostic dedupe not reported in host report (and other reports)
|
2013-04-29 18:11:24 -07:00 |
|
|
Noah Levitt
|
6a0ed067c0
|
consolidate disparate versions of WARCConstants into one version that lives in ia-web-commons
|
2013-03-29 17:12:32 -07:00 |
|
|
Kenji Nagahashi
|
a4a3ce60cd
|
fix failing ContentDigestHistoryTest
|
2013-03-26 15:06:31 -07:00 |
|
|
Noah Levitt
|
5d8fe0443d
|
Merge branch 'master' into HER-2031
|
2013-03-23 19:16:05 -07:00 |
|
|
Noah Levitt
|
8fd361e518
|
update source/target version to 1.6 since we use 1.6 features, specifically @Override for methods only implementing an interface (not sure why maven wasn't enforcing this); also remove maven-antrun-plugin created timestamp.txt, doesn't seem to be used anywhere
|
2013-03-20 14:25:22 -07:00 |
|
|
Noah Levitt
|
c509b6e552
|
remove more eclipse stuff, people like to manage their own
|
2013-03-20 13:39:28 -07:00 |
|
|
Noah Levitt
|
b0c6619f80
|
Merge branch 'master' of github.com:internetarchive/heritrix3
|
2013-03-18 20:02:03 -07:00 |
|
|
Noah Levitt
|
0c9e2a1f4d
|
ContentDigestHistory{Loader,Storer} - ignore uris with empty content body (deduping that is counterproductive)
|
2013-03-18 20:01:52 -07:00 |
|
|
Noah Levitt
|
ff36546eda
|
fix serverless whois url example in comment
|
2013-03-18 14:03:39 -07:00 |
|
|
gojomo
|
c5c54ac138
|
comment-out debug output
|
2013-03-12 11:42:47 -07:00 |
|
|
gojomo
|
7c8295ee82
|
checkpoint support
|
2013-03-12 11:42:29 -07:00 |
|
|
gojomo
|
a844878e7d
|
Merge branch 'simple_logins' into HER-2031
|
2013-03-12 01:16:31 -07:00 |
|
|
gojomo
|
3f8590fe44
|
when found in CrawlURI, add WARC-Simple-Form-Province-Status header
|
2013-03-11 20:57:41 -07:00 |
|
|
gojomo
|
bf9cab8eb2
|
when POST/submit-data requested, alter request method accordingly
|
2013-03-11 20:57:09 -07:00 |
|
|
gojomo
|
bbac7d7320
|
per (overlay) settings, spawn high-priority POST when appropriate; annotate/add-headers to describe
|
2013-03-11 20:56:31 -07:00 |
|
|
gojomo
|
ed639c8342
|
extract HTMLForm INPUT details
|
2013-03-11 20:55:20 -07:00 |
|
|
gojomo
|
94a61b4fb7
|
retain offsets of discovered FORMs to aid later processing
|
2013-03-11 20:53:41 -07:00 |
|
|
gojomo
|
6b0abae8ee
|
new constants, hop-type; refactoring to help support form-submissions
|
2013-03-11 20:52:59 -07:00 |
|