lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-22 22:46:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,496 Commits 40 Branches 31 Tags
4b75dfa16f78cd3d9e5db48936ff9388ca5ab2aa
Commit Graph
1496 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt 4b75dfa16f treat content as html and extract links if it looks like html, even if mime type belies that 2014-07-17 21:09:51 -07:00
Kristinn Sigurðsson f101bba509 Merge pull request #76 from nlevitt/HER-2039 
HER-2039 remove class Link, use CrawlURI

It should be noted that this is an API change that will affect any custom link extractors that people may have built.
 2014-07-10 09:41:00 +00:00
Noah Levitt ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper 2014-07-09 10:14:24 -07:00
Noah Levitt 7bd9f3acaf switch usage of Link to CrawlURI in heritrix-contrib 2014-07-08 11:56:56 -07:00
Noah Levitt bed1e58adf copy over and adjust equals(), hashCode(), compareTo() from old Link class to CrawlURI, so that tests pass 2014-07-07 16:59:18 -07:00
Noah Levitt bfac3c8bbc ensure non-null pathFromSeed to fix some unit tests 2014-07-07 12:25:27 -07:00
Noah Levitt 3c2e1315bb other changes required for merge from master into HER-2039 2014-07-07 11:48:36 -07:00
Noah Levitt f23a59a918 Merge branch 'master' into HER-2039 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
	modules/src/main/java/org/archive/modules/extractor/Link.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-07 11:45:27 -07:00
Adam Miller c48de14663 Merge pull request #75 from nlevitt/uri-inherited-data-fix 
in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data ...
 2014-07-02 15:45:56 -07:00
Noah Levitt f2055ee2ae in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data from Link 2014-07-02 13:17:42 -07:00
Noah Levitt 47e4e98608 Merge pull request #74 from vonrosen/ari-3943 
Fix for https://webarchive.jira.com/browse/ARI-3943
 2014-06-27 17:00:40 -07:00
Hunter Stern 5eade58de4 Fix for https://webarchive.jira.com/browse/ARI-3943 2014-06-27 16:20:42 -07:00
Noah Levitt 8c7b0fed92 Merge pull request #73 from kris-sigur/HER-2068 
Treat codebase as link hops, not embeds
 2014-06-04 13:03:04 -07:00
Kristinn Sigurðsson ece0149339 Fix for HER-2068 2014-06-04 14:59:13 +00:00
Roger G. Coram fc43a29459 Merge pull request #68 from nlevitt/handle-apply-overlay-exc 
handle exception determining whether to apply overlay
 2014-05-28 09:59:54 +01:00
vonrosen 12186af93d Merge pull request #72 from nlevitt/persist-annotations 
add A_ANNOTATIONS to persistentKeys so that CrawlURI doesn't lose its an...
 2014-05-22 16:49:37 -07:00
Noah Levitt 67a0c59758 add A_ANNOTATIONS to persistentKeys so that CrawlURI doesn't lose its annotations when deferred 2014-05-22 16:48:01 -07:00
vonrosen f2bc21435f Merge pull request #71 from nlevitt/checkpoint-no-ui-hang 
avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not...
 2014-05-22 16:44:22 -07:00
Noah Levitt 39814d1efd avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress 2014-05-22 16:40:44 -07:00
vonrosen 1c0d6c60fb Merge pull request #70 from nlevitt/ari-3712 
for ARI-3712, add extracted links relative to both via and base, and annotate with "extractorSWFRelToVia", "extractorSWFRelToBase", or "extractorSWFRelToBoth" if resulting link is the same whether relative to base or via
 2014-05-22 10:58:21 -07:00
Noah Levitt 26222ef660 Merge pull request #69 from vonrosen/ari-3853 
For https://webarchive.jira.com/browse/ARI-3865
 2014-05-15 17:38:41 -07:00
Hunter Stern d570498fd9 New Unit tests added now. 2014-05-15 17:37:26 -07:00
Hunter Stern 80880b4409 Unit test and other changes. 2014-05-15 17:34:43 -07:00
Noah Levitt aff5b9ac4e for ARI-3712, add extracted links relative to both via and base, and annotate with "extractorSWFRelToVia", "extractorSWFRelToBase", or "extractorSWFRelToBoth" if resulting link is the same whether relative to base or via 2014-05-15 13:31:29 -07:00
Noah Levitt 519717fd7b make addRelativeToBase() and addRelativeToVia() return the link added, or null; copy getAnnotations() from CrawlURI 2014-05-15 13:27:50 -07:00
Hunter Stern f24cfff6af New youtube channel video extractor for https://webarchive.jira.com/browse/ARI-3865 2014-05-13 17:41:42 -07:00
Noah Levitt 963e04e50c handle exception determining whether to apply overlay 2014-05-13 15:47:21 -07:00
Adam Miller 3e213c82bc Merge pull request #67 from nlevitt/umbra 
don't log severe with stack trace on normal amqp shutdown
 2014-05-13 14:44:09 -07:00
Noah Levitt 08a4aafa23 don't log severe with stack trace on normal amqp shutdown 2014-05-13 14:39:35 -07:00
Noah Levitt 38cd14ecac Merge pull request #63 from adam-miller/ExtractorPDFContent 
Contrib - ExtractorPDFContent
 2014-05-13 14:09:20 -07:00
Adam Miller aaecd6d78c Merge branch 'master' of git://github.com/internetarchive/heritrix3 into ExtractorPDFContent 2014-05-13 14:04:29 -07:00
Adam Miller fb42c07c92 bugfix, new test case. 2014-05-13 14:03:03 -07:00
Adam Miller ab7a60f9e3 Merge pull request #66 from nlevitt/cosmetics 
oops, make "exit java process" button work again
 2014-05-13 13:31:11 -07:00
Adam Miller 85e7a5ee78 Bugfix & new test cases. 2014-05-12 16:52:32 -07:00
Noah Levitt ba68649ba0 Merge remote-tracking branch 'origin/master' into cosmetics 2014-05-07 10:02:06 -07:00
Noah Levitt b7f8a0669f Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics 2014-05-07 10:02:00 -07:00
Noah Levitt 33e782e2fd oops, make "exit java process" button work again 2014-05-07 10:01:47 -07:00
Adam Miller eb32d7a8b8 Merge pull request #65 from nlevitt/umbra 
shut down the starter-restarter thread at crawl finish!!
 2014-05-02 15:47:44 -07:00
Noah Levitt 949c9f086c shut down the starter-restarter thread at crawl finish!! 2014-05-02 15:42:29 -07:00
Noah Levitt dd510cbf3b Merge pull request #64 from adam-miller/ViaSurtPrefixedDecideRule 
Via surt prefixed decide rule
 2014-05-01 18:14:25 -07:00
Adam Miller ae878d29cf Code cleanup 2014-05-01 15:22:16 -07:00
Adam Miller f2bebd41b0 Cleanup / Optimization 2014-05-01 12:48:23 -07:00
Adam Miller 753404f662 Adding ViaSurtPrefixedDecideRule. Returns true if a surtform 'via' matches any surt in a specified list 2014-05-01 12:43:26 -07:00
Adam Miller 2b1c330be6 Further cleanup 2014-04-30 12:18:14 -07:00
Adam Miller e6d646d4b4 Cleanup / optimization 2014-04-30 12:14:32 -07:00
Adam Miller 34a9172e69 Organize Imports 2014-04-29 17:25:43 -07:00
Adam Miller 5181c1ef64 Cleaning up test cases and removing unused code 2014-04-29 17:11:18 -07:00
Adam Miller 33277445f6 Merge branch 'master' of git://github.com/internetarchive/heritrix3 into ExtractorPDFContent 2014-04-29 17:06:55 -07:00
Adam Miller 49069f2dd1 Adding comment for library dependencies 2014-04-29 17:06:11 -07:00
Adam Miller 7e8a66e7d8 New Test cases, fixing bugs 2014-04-29 10:50:04 -07:00