lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-23 15:06:46 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,520 Commits 40 Branches 31 Tags
40eae4cbceaa17fd0f4ebc5872cfaafca545eab7
Commit Graph
1520 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 40eae4cbce Merge branch 'master' into hbase-cdh5 
Conflicts:
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-23 13:33:58 -07:00
vonrosen ba48f96d93 Merge pull request #78 from nlevitt/extract-html-no-mimetype 
treat content as html and extract links if it looks like html, even if m...
 2014-07-22 15:41:44 -07:00
Kristinn Sigurðsson d0ebd40578 Merge pull request #79 from nlevitt/revisit-redux 
Revisit redux
 2014-07-21 08:40:30 +00:00
Noah Levitt 4b75dfa16f treat content as html and extract links if it looks like html, even if mime type belies that 2014-07-17 21:09:51 -07:00
Noah Levitt cb6ac8b8b5 restore writeRevisitFor* settings, but issue warning in setter and ignore the value 2014-07-14 10:53:37 -07:00
Noah Levitt da30439d61 oops, fix 304 tests 2014-07-11 12:47:15 -07:00
Noah Levitt 77492802b3 test 304 not modified revist profile in FetchHTTPTest 2014-07-10 21:30:54 -07:00
Noah Levitt 2292ea9287 remove deprecated unused stuff 2014-07-10 21:29:47 -07:00
Noah Levitt 810ad91b9d set revisit profile on 304 not modified 2014-07-10 21:28:52 -07:00
Noah Levitt 905a98c0c2 use RevisitProfile for tallying stats 2014-07-10 21:27:50 -07:00
Noah Levitt d9fd66878b Merge branch 'master' into revisit-redux 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-10 15:01:43 -07:00
Kristinn Sigurðsson f101bba509 Merge pull request #76 from nlevitt/HER-2039 
HER-2039 remove class Link, use CrawlURI

It should be noted that this is an API change that will affect any custom link extractors that people may have built.
 2014-07-10 09:41:00 +00:00
Noah Levitt ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper 2014-07-09 10:14:24 -07:00
Noah Levitt 7bd9f3acaf switch usage of Link to CrawlURI in heritrix-contrib 2014-07-08 11:56:56 -07:00
Noah Levitt 6e9a0a4d05 only write WARC-Refers-To header when value is not null 2014-07-07 19:21:07 -07:00
Noah Levitt 2854798964 new parameter keySuffix-- If not null, keySuffix is appended to the lookup key when loading and storing digest history. Thus the key looks like {digest}{keySuffix}, e.g. "sha1:22SFHXERHNFOEY6WK7YOUN4PFIPZSB4D-1193". The purpose is to support multiple namespaces in a single hbase table, to avoid proliferation of small tables. The reason we use a suffix instead of a prefix is to leave open the possibility of deduplication across these different namespaces at some point in the future. 2014-07-07 19:05:39 -07:00
Noah Levitt bed1e58adf copy over and adjust equals(), hashCode(), compareTo() from old Link class to CrawlURI, so that tests pass 2014-07-07 16:59:18 -07:00
Noah Levitt bfac3c8bbc ensure non-null pathFromSeed to fix some unit tests 2014-07-07 12:25:27 -07:00
Noah Levitt 3c2e1315bb other changes required for merge from master into HER-2039 2014-07-07 11:48:36 -07:00
Noah Levitt f23a59a918 Merge branch 'master' into HER-2039 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
	modules/src/main/java/org/archive/modules/extractor/Link.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-07 11:45:27 -07:00
Adam Miller c48de14663 Merge pull request #75 from nlevitt/uri-inherited-data-fix 
in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data ...
 2014-07-02 15:45:56 -07:00
Noah Levitt f2055ee2ae in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data from Link 2014-07-02 13:17:42 -07:00
Noah Levitt 47e4e98608 Merge pull request #74 from vonrosen/ari-3943 
Fix for https://webarchive.jira.com/browse/ARI-3943
 2014-06-27 17:00:40 -07:00
Hunter Stern 5eade58de4 Fix for https://webarchive.jira.com/browse/ARI-3943 2014-06-27 16:20:42 -07:00
Noah Levitt 3e35c967ab upgrade hbase to cdh5 2014-06-16 18:33:25 -07:00
Noah Levitt 8c7b0fed92 Merge pull request #73 from kris-sigur/HER-2068 
Treat codebase as link hops, not embeds
 2014-06-04 13:03:04 -07:00
Kristinn Sigurðsson ece0149339 Fix for HER-2068 2014-06-04 14:59:13 +00:00
Roger G. Coram fc43a29459 Merge pull request #68 from nlevitt/handle-apply-overlay-exc 
handle exception determining whether to apply overlay
 2014-05-28 09:59:54 +01:00
vonrosen 12186af93d Merge pull request #72 from nlevitt/persist-annotations 
add A_ANNOTATIONS to persistentKeys so that CrawlURI doesn't lose its an...
 2014-05-22 16:49:37 -07:00
Noah Levitt 67a0c59758 add A_ANNOTATIONS to persistentKeys so that CrawlURI doesn't lose its annotations when deferred 2014-05-22 16:48:01 -07:00
vonrosen f2bc21435f Merge pull request #71 from nlevitt/checkpoint-no-ui-hang 
avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not...
 2014-05-22 16:44:22 -07:00
Noah Levitt 39814d1efd avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress 2014-05-22 16:40:44 -07:00
vonrosen 1c0d6c60fb Merge pull request #70 from nlevitt/ari-3712 
for ARI-3712, add extracted links relative to both via and base, and annotate with "extractorSWFRelToVia", "extractorSWFRelToBase", or "extractorSWFRelToBoth" if resulting link is the same whether relative to base or via
 2014-05-22 10:58:21 -07:00
Noah Levitt 26222ef660 Merge pull request #69 from vonrosen/ari-3853 
For https://webarchive.jira.com/browse/ARI-3865
 2014-05-15 17:38:41 -07:00
Hunter Stern d570498fd9 New Unit tests added now. 2014-05-15 17:37:26 -07:00
Hunter Stern 80880b4409 Unit test and other changes. 2014-05-15 17:34:43 -07:00
Noah Levitt aff5b9ac4e for ARI-3712, add extracted links relative to both via and base, and annotate with "extractorSWFRelToVia", "extractorSWFRelToBase", or "extractorSWFRelToBoth" if resulting link is the same whether relative to base or via 2014-05-15 13:31:29 -07:00
Noah Levitt 519717fd7b make addRelativeToBase() and addRelativeToVia() return the link added, or null; copy getAnnotations() from CrawlURI 2014-05-15 13:27:50 -07:00
Hunter Stern f24cfff6af New youtube channel video extractor for https://webarchive.jira.com/browse/ARI-3865 2014-05-13 17:41:42 -07:00
Noah Levitt 963e04e50c handle exception determining whether to apply overlay 2014-05-13 15:47:21 -07:00
Adam Miller 3e213c82bc Merge pull request #67 from nlevitt/umbra 
don't log severe with stack trace on normal amqp shutdown
 2014-05-13 14:44:09 -07:00
Noah Levitt 08a4aafa23 don't log severe with stack trace on normal amqp shutdown 2014-05-13 14:39:35 -07:00
Noah Levitt 38cd14ecac Merge pull request #63 from adam-miller/ExtractorPDFContent 
Contrib - ExtractorPDFContent
 2014-05-13 14:09:20 -07:00
Adam Miller aaecd6d78c Merge branch 'master' of git://github.com/internetarchive/heritrix3 into ExtractorPDFContent 2014-05-13 14:04:29 -07:00
Adam Miller fb42c07c92 bugfix, new test case. 2014-05-13 14:03:03 -07:00
Adam Miller ab7a60f9e3 Merge pull request #66 from nlevitt/cosmetics 
oops, make "exit java process" button work again
 2014-05-13 13:31:11 -07:00
Adam Miller 85e7a5ee78 Bugfix & new test cases. 2014-05-12 16:52:32 -07:00
Noah Levitt ba68649ba0 Merge remote-tracking branch 'origin/master' into cosmetics 2014-05-07 10:02:06 -07:00
Noah Levitt b7f8a0669f Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics 2014-05-07 10:02:00 -07:00
Noah Levitt 33e782e2fd oops, make "exit java process" button work again 2014-05-07 10:01:47 -07:00