lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-09-24 22:55:45 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,156 Commits 38 Branches 33 Tags
b84d443f00a75d68fa022c61ba5b8a106bd6344a
Commit Graph
839 Commits
Author SHA1 Message Date
Noah Levitt 51a1cad300 new class DecideRuleSequenceWithAMQPFeed publishes scoping decisions to amqp 2014-08-29 17:08:32 -07:00
Adam Miller bf248f407b FetchHTTP.java 
Handle response header with a null or empty Content-Length
 2014-08-27 17:14:05 -07:00
Noah Levitt 04921073ca fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering() 2014-08-19 17:14:32 -07:00
Noah Levitt 9a0c0656b2 move HostsReport.fixup(String) to CrawlHost.fixUpName() so it can be called elsewhere 2014-08-12 16:45:53 -07:00
Noah Levitt 65cf7942c2 replace use of another method removed in updated guava library 2014-08-11 11:23:59 -07:00
Hunter Stern 7f1700400b Allow xml extractor to handle urls in CDATA. 2014-08-01 17:37:31 -07:00
vonrosen 4298c3de69 Merge pull request #83 from nlevitt/revisit-redux 
oops! count novel urls/bytes for hosts report, etc
 2014-07-25 16:08:33 -07:00
Noah Levitt e5ecde6c2f oops! count novel urls/bytes for hosts report, etc 2014-07-25 15:57:53 -07:00
Noah Levitt e7b034c60e Merge pull request #82 from kris-sigur/HER-2071 
Fix for HER-2071
 2014-07-25 12:13:02 -07:00
Kristinn Sigurðsson 8edebc0943 Fix for HER-2071 2014-07-25 13:32:08 +00:00
vonrosen ba48f96d93 Merge pull request #78 from nlevitt/extract-html-no-mimetype 
treat content as html and extract links if it looks like html, even if m...
 2014-07-22 15:41:44 -07:00
Noah Levitt 410e1709b7 ExtractorHTML when a/@href links include the attribute data-remote="true", include that info the viaContext; and in FetchHTTPRequest, when a CrawlURI has data-remote="true", add the http header "X-Requested-With: XMLHttpRequest"; data-remote is a jquery thing that means the url can be requested via ajax in a browser... some urls work wrong without the X-Requested-With header... this is not a great place for a non-generic thing like this, but where else could it go... 2014-07-18 19:48:15 -07:00
Noah Levitt 4b75dfa16f treat content as html and extract links if it looks like html, even if mime type belies that 2014-07-17 21:09:51 -07:00
Noah Levitt cb6ac8b8b5 restore writeRevisitFor* settings, but issue warning in setter and ignore the value 2014-07-14 10:53:37 -07:00
Noah Levitt da30439d61 oops, fix 304 tests 2014-07-11 12:47:15 -07:00
Noah Levitt 77492802b3 test 304 not modified revist profile in FetchHTTPTest 2014-07-10 21:30:54 -07:00
Noah Levitt 2292ea9287 remove deprecated unused stuff 2014-07-10 21:29:47 -07:00
Noah Levitt 810ad91b9d set revisit profile on 304 not modified 2014-07-10 21:28:52 -07:00
Noah Levitt 905a98c0c2 use RevisitProfile for tallying stats 2014-07-10 21:27:50 -07:00
Noah Levitt d9fd66878b Merge branch 'master' into revisit-redux 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-10 15:01:43 -07:00
Noah Levitt ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper 2014-07-09 10:14:24 -07:00
Noah Levitt bed1e58adf copy over and adjust equals(), hashCode(), compareTo() from old Link class to CrawlURI, so that tests pass 2014-07-07 16:59:18 -07:00
Noah Levitt bfac3c8bbc ensure non-null pathFromSeed to fix some unit tests 2014-07-07 12:25:27 -07:00
Noah Levitt 3c2e1315bb other changes required for merge from master into HER-2039 2014-07-07 11:48:36 -07:00
Noah Levitt f23a59a918 Merge branch 'master' into HER-2039 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
	modules/src/main/java/org/archive/modules/extractor/Link.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-07 11:45:27 -07:00
Noah Levitt f2055ee2ae in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data from Link 2014-07-02 13:17:42 -07:00
Hunter Stern 5eade58de4 Fix for https://webarchive.jira.com/browse/ARI-3943 2014-06-27 16:20:42 -07:00
Kristinn Sigurðsson ece0149339 Fix for HER-2068 2014-06-04 14:59:13 +00:00
Noah Levitt 67a0c59758 add A_ANNOTATIONS to persistentKeys so that CrawlURI doesn't lose its annotations when deferred 2014-05-22 16:48:01 -07:00
Noah Levitt aff5b9ac4e for ARI-3712, add extracted links relative to both via and base, and annotate with "extractorSWFRelToVia", "extractorSWFRelToBase", or "extractorSWFRelToBoth" if resulting link is the same whether relative to base or via 2014-05-15 13:31:29 -07:00
Noah Levitt 519717fd7b make addRelativeToBase() and addRelativeToVia() return the link added, or null; copy getAnnotations() from CrawlURI 2014-05-15 13:27:50 -07:00
Adam Miller ae878d29cf Code cleanup 2014-05-01 15:22:16 -07:00
Adam Miller f2bebd41b0 Cleanup / Optimization 2014-05-01 12:48:23 -07:00
Adam Miller 753404f662 Adding ViaSurtPrefixedDecideRule. Returns true if a surtform 'via' matches any surt in a specified list 2014-05-01 12:43:26 -07:00
Noah Levitt e73810d0a6 fix unit test now that we accept speculative urls with query params with no value 2014-04-10 17:25:27 -07:00
Noah Levitt 36a207ce07 fix FormLoginProcessor checkpointing 2014-04-02 17:33:09 -07:00
Noah Levitt 878fd9f7a9 oops, update test to expect post data as url-encoded query string 2014-04-01 15:18:17 -07:00
Noah Levitt 552200b2fa fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass 2014-03-31 20:38:17 -07:00
Noah Levitt a82886b747 restore support for sending http post data, add unit test 2014-03-31 18:09:23 -07:00
Noah Levitt 23e8d41651 avoid NPE; root cause was assumption that LoadingCache.asMap().get() would call CacheLoader.load() but it doesnt, see https://code.google.com/p/guava-libraries/wiki/MapMakerMigration 2014-03-31 13:40:29 -07:00
Noah Levitt 26a577ff6e basic unit tests for FormLoginProcessor 2014-03-31 13:38:39 -07:00
Noah Levitt 0d7f633fdb pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available 2014-03-31 13:36:44 -07:00
Shriphani Palakodety 84f29e4e10 build with the right key and value types 2014-03-18 06:42:03 -04:00
Shriphani Palakodety 2f60b5f935 migration to latest guava complete 2014-03-18 06:42:02 -04:00
Noah Levitt 6a92e88b1c Merge pull request #40 from internetarchive/her-2064 
For https://webarchive.jira.com/browse/HER-2064
 2014-02-28 16:48:42 -08:00
Noah Levitt cf775bec94 remove extraneous code related to logging removed in last commit 2014-02-21 15:46:52 -08:00
Noah Levitt fd806a514d remove extra logging from performance test 2014-02-21 13:28:13 -08:00
Noah Levitt 39f06a1f1c fix slowness caused by BdbCookieStore by maintaining cookie list in memory parallel to the one stored on disk; add tests for consistency and performance under load 2014-02-21 13:26:58 -08:00
Hunter Stern d6e61b83da For https://webarchive.jira.com/browse/HER-2064 2014-02-13 16:59:05 -08:00
Hunter Stern 06ba50c410 Reverting previous commit: https://github.com/internetarchive/heritrix3/commit/164d0fbdc5fa32cd90edd9f1b4091ed876552232 2014-02-13 16:50:21 -08:00