lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-17 04:57:11 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,678 Commits 40 Branches 31 Tags
12dde2b69361f79eaa3ea53193d2014f31fd730a
Commit Graph

467 Commits

Author SHA1 Message Date
Noah Levitt 12dde2b693 load keytool utility main class dynamically, trying both the old and new (java 8) class names 2015-09-23 12:06:11 -07:00
Kristinn Sigurðsson 4ea7efdfc8 Set character set to UTF-8 when passing through files. 2015-02-10 16:07:37 +00:00
vonrosen 3609abe38c Merge pull request #108 from nlevitt/aitfive-203 
kafka crawl log feed
 2014-11-14 10:29:41 -08:00
Noah Levitt 7312ad597e so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ) 2014-11-07 13:21:42 -08:00
Adam Miller fb23b40e74 CandidatesProcessor.java 
Keep CrawlURI outlinks intact for use in SeedRecord reporting.
 2014-10-22 13:18:25 -07:00
vonrosen 5909042b84 Merge pull request #99 from nlevitt/aitfive-39 
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
 2014-10-16 17:24:37 +00:00
Kenji Nagahashi 5821fe7b1e FIX corner-case of bean browser failing due to an exception from hashCode(). 2014-09-11 11:22:01 -07:00
Kenji Nagahashi 2a38fa0b12 Don’t let exception from property getter fail entire bean-browser. 2014-08-25 13:36:25 -07:00
Noah Levitt 591f0f9f0f expose forAllPendingDo() as public so it can be used elsewhere 2014-08-13 13:02:28 -07:00
Noah Levitt 9a0c0656b2 move HostsReport.fixup(String) to CrawlHost.fixUpName() so it can be called elsewhere 2014-08-12 16:45:53 -07:00
Noah Levitt 51081a652c make AbstractFrontier.needReenqueuing() public so it can be called elsewhere 2014-08-12 16:20:25 -07:00
Noah Levitt 47a62f7c6f remove a couple of crufty lines of code 2014-08-12 16:13:59 -07:00
Noah Levitt 08e24bbdff update guess at log line length based on real world data 2014-08-12 16:13:11 -07:00
Noah Levitt 905a98c0c2 use RevisitProfile for tallying stats 2014-07-10 21:27:50 -07:00
Noah Levitt ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper 2014-07-09 10:14:24 -07:00
Noah Levitt f23a59a918 Merge branch 'master' into HER-2039 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
	modules/src/main/java/org/archive/modules/extractor/Link.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-07 11:45:27 -07:00
Roger G. Coram fc43a29459 Merge pull request #68 from nlevitt/handle-apply-overlay-exc 
handle exception determining whether to apply overlay
 2014-05-28 09:59:54 +01:00
Noah Levitt 39814d1efd avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress 2014-05-22 16:40:44 -07:00
Noah Levitt 963e04e50c handle exception determining whether to apply overlay 2014-05-13 15:47:21 -07:00
Noah Levitt ba68649ba0 Merge remote-tracking branch 'origin/master' into cosmetics 2014-05-07 10:02:06 -07:00
Noah Levitt 33e782e2fd oops, make "exit java process" button work again 2014-05-07 10:01:47 -07:00
Noah Levitt d777c198bd HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji) 2014-04-24 17:23:01 -07:00
Shriphani Palakodety 2f60b5f935 migration to latest guava complete 2014-03-18 06:42:02 -04:00
Noah Levitt ea19dacbd2 handle main job log handler like current launch job log handler, close on teardown 2014-02-26 17:19:50 -08:00
Martins Balodis 88dc01305f fix job.log file handler was left open when jobdir is removed 2014-02-26 21:15:38 +02:00
Kristinn Sigurðsson df72bc8afc Minor tidying up 2014-01-15 12:51:35 +00:00
Kristinn Sigurðsson bc77e82da5 Fix for WorkQueueFrontier.deleteURIs mishandles deletions from retired 
queues (HER-2062)
 2014-01-15 12:49:41 +00:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43 
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
 2014-01-10 17:52:03 -08:00
Noah Levitt 552794dc89 don't let the exception from commit df2a313797 derail the whole crawl job 2013-11-26 16:08:38 -08:00
Noah Levitt 1163847644 try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted 2013-11-21 18:51:16 -08:00
Noah Levitt d93b5449be new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc 2013-11-19 11:05:14 -08:00
Noah Levitt 7089caf3c4 remove unused engine.css 2013-11-15 19:11:19 -08:00
Noah Levitt f096609a50 avoid infinite loop spitting out stack traces on BeansException on startup 2013-09-08 00:11:54 -07:00
Noah Levitt a00141eef7 avoid deadlock in case of BeansException at crawl startup 2013-09-08 00:10:27 -07:00
Noah Levitt fd71833050 avoid NPE when attempting to terminate already-finished job 2013-09-07 09:57:33 -07:00
Noah Levitt d01bc4ed55 when resuming from checkpoint, make next checkpoint serial number follow serial number of resumed checkpoint 2013-09-07 09:57:07 -07:00
Noah Levitt d61171f076 CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size 2013-09-07 09:55:58 -07:00
Noah Levitt 59596c7043 prevent redundant checkpoints when crawl is stopping 2013-09-03 14:24:41 -07:00
Noah Levitt 2fe7d8cd45 some progress on forgetting old checkpoints when checkpointing anew 2013-08-27 18:39:25 -07:00
Noah Levitt 77a12e63b8 make special "host" "whois:" appear correctly in hosts report 2013-08-27 17:39:07 -07:00
Noah Levitt d1bb1a4637 further improve performance of BdbUriUniqFilter.forgetAllSchemeAuthorityMatching() by skipping conversion of key data to long and comparing bytes directly 2013-08-21 11:44:46 -07:00
Noah Levitt a06c141740 rename BdbUriUniqFilter.forgetSchemeHost() to forgetAllSchemeAuthorityMatching(), improve performance 2013-08-21 11:31:05 -07:00
Noah Levitt 750a688670 Merge branch 'master' into hc43 
Conflicts:
	commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
	engine/pom.xml
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
 2013-08-08 21:14:11 -07:00
Noah Levitt f4f3b9be38 add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs 2013-07-26 17:50:36 -07:00
Noah Levitt b9726c8d4f new method BdbUriUniqFilter.forgetSchemeHost(String schemeHost), and unit test 2013-07-25 18:51:26 -07:00
Noah Levitt b6b3ae6970 make log level check for debug/performance logging consistent to avoid unnecessary operations 2013-07-25 17:28:16 -07:00
Kenji Nagahashi b88242ee16 bean browser: handle array of non-Object element type properly. 
(just avoiding throwing exception - further change would be necessary
for better rendering.)
 2013-06-16 08:21:53 -07:00
Kenji Nagahashi 9ef2fb0f7f Merge branch 'topic/scripting-xml-fix' 2013-06-10 15:36:13 -07:00
Kenji Nagahashi 358afd4486 revised scripting console xml response fix based on comments. 
removed scripting console actions from ScriptModel into ScriptingConsole
class (resurrection of ScriptExec).
added basic ScriptingConsoleTest.
 2013-06-10 14:17:27 -07:00
Noah Levitt f2984881f6 address HER-2042 - new annotation duplicate:uriAgnosticDigest for all url agnostic duplicates, whether written to warcs or not; stats consult this annotation so that duplicate/novel numbers are correct 2013-06-07 17:17:24 -07:00