vonrosen
ba48f96d93
Merge pull request #78 from nlevitt/extract-html-no-mimetype
...
treat content as html and extract links if it looks like html, even if m...
2014-07-22 15:41:44 -07:00
Noah Levitt
4b75dfa16f
treat content as html and extract links if it looks like html, even if mime type belies that
2014-07-17 21:09:51 -07:00
Noah Levitt
cb6ac8b8b5
restore writeRevisitFor* settings, but issue warning in setter and ignore the value
2014-07-14 10:53:37 -07:00
Noah Levitt
da30439d61
oops, fix 304 tests
2014-07-11 12:47:15 -07:00
Noah Levitt
77492802b3
test 304 not modified revist profile in FetchHTTPTest
2014-07-10 21:30:54 -07:00
Noah Levitt
2292ea9287
remove deprecated unused stuff
2014-07-10 21:29:47 -07:00
Noah Levitt
810ad91b9d
set revisit profile on 304 not modified
2014-07-10 21:28:52 -07:00
Noah Levitt
905a98c0c2
use RevisitProfile for tallying stats
2014-07-10 21:27:50 -07:00
Noah Levitt
d9fd66878b
Merge branch 'master' into revisit-redux
...
Conflicts:
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
2014-07-10 15:01:43 -07:00
Noah Levitt
ee52f48be9
remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper
2014-07-09 10:14:24 -07:00
Noah Levitt
bed1e58adf
copy over and adjust equals(), hashCode(), compareTo() from old Link class to CrawlURI, so that tests pass
2014-07-07 16:59:18 -07:00
Noah Levitt
bfac3c8bbc
ensure non-null pathFromSeed to fix some unit tests
2014-07-07 12:25:27 -07:00
Noah Levitt
3c2e1315bb
other changes required for merge from master into HER-2039
2014-07-07 11:48:36 -07:00
Noah Levitt
f23a59a918
Merge branch 'master' into HER-2039
...
Conflicts:
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
modules/src/main/java/org/archive/modules/extractor/Link.java
modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
2014-07-07 11:45:27 -07:00
Noah Levitt
f2055ee2ae
in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data from Link
2014-07-02 13:17:42 -07:00
Hunter Stern
5eade58de4
Fix for https://webarchive.jira.com/browse/ARI-3943
2014-06-27 16:20:42 -07:00
Kristinn Sigurðsson
ece0149339
Fix for HER-2068
2014-06-04 14:59:13 +00:00
Noah Levitt
67a0c59758
add A_ANNOTATIONS to persistentKeys so that CrawlURI doesn't lose its annotations when deferred
2014-05-22 16:48:01 -07:00
Noah Levitt
aff5b9ac4e
for ARI-3712, add extracted links relative to both via and base, and annotate with "extractorSWFRelToVia", "extractorSWFRelToBase", or "extractorSWFRelToBoth" if resulting link is the same whether relative to base or via
2014-05-15 13:31:29 -07:00
Noah Levitt
519717fd7b
make addRelativeToBase() and addRelativeToVia() return the link added, or null; copy getAnnotations() from CrawlURI
2014-05-15 13:27:50 -07:00
Adam Miller
ae878d29cf
Code cleanup
2014-05-01 15:22:16 -07:00
Adam Miller
f2bebd41b0
Cleanup / Optimization
2014-05-01 12:48:23 -07:00
Adam Miller
753404f662
Adding ViaSurtPrefixedDecideRule. Returns true if a surtform 'via' matches any surt in a specified list
2014-05-01 12:43:26 -07:00
Noah Levitt
e73810d0a6
fix unit test now that we accept speculative urls with query params with no value
2014-04-10 17:25:27 -07:00
Noah Levitt
36a207ce07
fix FormLoginProcessor checkpointing
2014-04-02 17:33:09 -07:00
Noah Levitt
878fd9f7a9
oops, update test to expect post data as url-encoded query string
2014-04-01 15:18:17 -07:00
Noah Levitt
552200b2fa
fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass
2014-03-31 20:38:17 -07:00
Noah Levitt
a82886b747
restore support for sending http post data, add unit test
2014-03-31 18:09:23 -07:00
Noah Levitt
23e8d41651
avoid NPE; root cause was assumption that LoadingCache.asMap().get() would call CacheLoader.load() but it doesnt, see https://code.google.com/p/guava-libraries/wiki/MapMakerMigration
2014-03-31 13:40:29 -07:00
Noah Levitt
26a577ff6e
basic unit tests for FormLoginProcessor
2014-03-31 13:38:39 -07:00
Noah Levitt
0d7f633fdb
pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available
2014-03-31 13:36:44 -07:00
Shriphani Palakodety
84f29e4e10
build with the right key and value types
2014-03-18 06:42:03 -04:00
Shriphani Palakodety
2f60b5f935
migration to latest guava complete
2014-03-18 06:42:02 -04:00
Noah Levitt
6a92e88b1c
Merge pull request #40 from internetarchive/her-2064
...
For https://webarchive.jira.com/browse/HER-2064
2014-02-28 16:48:42 -08:00
Noah Levitt
cf775bec94
remove extraneous code related to logging removed in last commit
2014-02-21 15:46:52 -08:00
Noah Levitt
fd806a514d
remove extra logging from performance test
2014-02-21 13:28:13 -08:00
Noah Levitt
39f06a1f1c
fix slowness caused by BdbCookieStore by maintaining cookie list in memory parallel to the one stored on disk; add tests for consistency and performance under load
2014-02-21 13:26:58 -08:00
Hunter Stern
d6e61b83da
For https://webarchive.jira.com/browse/HER-2064
2014-02-13 16:59:05 -08:00
Hunter Stern
06ba50c410
Reverting previous commit: https://github.com/internetarchive/heritrix3/commit/164d0fbdc5fa32cd90edd9f1b4091ed876552232
2014-02-13 16:50:21 -08:00
Hunter Stern
164d0fbdc5
For https://webarchive.jira.com/browse/HER-2064
2014-02-13 16:42:58 -08:00
Noah Levitt
af44522fdf
FetchHTTPRequest - include CrawlURI.getData().get("customHttpRequestHeaders"), if any, in http request
2014-02-03 18:41:23 -08:00
Noah Levitt
edd62f3de0
fix HER-2063 - omit port in Host request header when it is default for the scheme
2014-02-03 17:18:40 -08:00
Kristinn Sigurðsson
cc18521759
Merge remote-tracking branch 'internetarchive/master' into revisit-redux
2014-01-30 15:06:01 +00:00
Kristinn Sigurðsson
701457ed4b
ContentDigestHistoryLoader now sets the revisit profile on the CrawlURI
...
after determining that it is a revisit.
2014-01-30 15:03:48 +00:00
Kristinn Sigurðsson
2905cc556c
Revisit headers now come correctly formated from the revisit profile.
2014-01-30 15:02:40 +00:00
Kristinn Sigurðsson
685f5d5d6a
Store refersToDate in String format (as intended to write in WARC).
...
Retain long setter for convenience.
2014-01-30 15:01:41 +00:00
Noah Levitt
31aea014d9
also handle case where Charset.forName() throws IllegalCharsetNameException, as opposed to UnsupportedCharsetException
2014-01-28 16:36:51 -08:00
Noah Levitt
161b359979
Avoid the exception below by handling bad charsets in FetchHTTP. Restore annotation "unsatisfiableCharsetInHeader:...". Includes unit tests for good and bad charsets.
...
SEVERE thread-5973 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.nio.charset.UnsupportedCharsetException: binary occured when trying to process 'http://example.com/ ' at step ABOUT_TO_BEGIN_PROCESSOR in
java.nio.charset.UnsupportedCharsetException: binary
at java.nio.charset.Charset.forName(Charset.java:543)
at org.apache.http.entity.ContentType.<init>(ContentType.java:111)
at org.apache.http.entity.ContentType.create(ContentType.java:210)
at org.apache.http.entity.ContentType.get(ContentType.java:258)
at org.apache.http.entity.ContentType.getOrDefault(ContentType.java:277)
at org.archive.modules.fetcher.FetchHTTP.setCharacterEncoding(FetchHTTP.java:580)
at org.archive.modules.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:693)
at org.archive.modules.Processor.innerProcessResult(Processor.java:175)
at org.archive.modules.Processor.process(Processor.java:142)
at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)
at org.archive.crawler.framework.ToeThread.run(ToeThread.java:148)
2014-01-28 16:19:44 -08:00
Noah Levitt
b803ea0ec0
whoops! send escaped path+query on http request line; had been sending raw spaces (includes unit test)
2014-01-27 16:57:37 -08:00
Noah Levitt
26a5dfebdc
fix NullPointerException in case of 401 with no auth challenge (includes unit test)
2014-01-27 11:47:37 -08:00