lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-31 03:41:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,515 Commits 40 Branches 31 Tags
8edebc09439254c8eaee0396d166ce6562a10f4e
Commit Graph
585 Commits
Author SHA1 Message Date
Noah Levitt 905a98c0c2 use RevisitProfile for tallying stats 2014-07-10 21:27:50 -07:00
Noah Levitt ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper 2014-07-09 10:14:24 -07:00
Noah Levitt f23a59a918 Merge branch 'master' into HER-2039 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
	modules/src/main/java/org/archive/modules/extractor/Link.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-07 11:45:27 -07:00
Roger G. Coram fc43a29459 Merge pull request #68 from nlevitt/handle-apply-overlay-exc 
handle exception determining whether to apply overlay
 2014-05-28 09:59:54 +01:00
Noah Levitt 39814d1efd avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress 2014-05-22 16:40:44 -07:00
Noah Levitt 963e04e50c handle exception determining whether to apply overlay 2014-05-13 15:47:21 -07:00
Noah Levitt ba68649ba0 Merge remote-tracking branch 'origin/master' into cosmetics 2014-05-07 10:02:06 -07:00
Noah Levitt b7f8a0669f Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics 2014-05-07 10:02:00 -07:00
Noah Levitt 33e782e2fd oops, make "exit java process" button work again 2014-05-07 10:01:47 -07:00
Noah Levitt d777c198bd HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji) 2014-04-24 17:23:01 -07:00
Noah Levitt 0cd1043cb1 change memory bar color, use consistent non-capitalization on buttons 2014-04-21 11:53:40 -07:00
Noah Levitt 1b72ce5dd8 remove extra </div> 2014-04-21 11:53:05 -07:00
Noah Levitt cfb9474f7a cosmetic changes 2014-04-21 09:06:15 -07:00
Noah Levitt 552200b2fa fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass 2014-03-31 20:38:17 -07:00
Noah Levitt 20a5f31e19 ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet) 2014-03-31 18:57:33 -07:00
Shriphani Palakodety 2f60b5f935 migration to latest guava complete 2014-03-18 06:42:02 -04:00
Noah Levitt ea19dacbd2 handle main job log handler like current launch job log handler, close on teardown 2014-02-26 17:19:50 -08:00
Martins Balodis 88dc01305f fix job.log file handler was left open when jobdir is removed 2014-02-26 21:15:38 +02:00
Kristinn Sigurðsson df72bc8afc Minor tidying up 2014-01-15 12:51:35 +00:00
Kristinn Sigurðsson bc77e82da5 Fix for WorkQueueFrontier.deleteURIs mishandles deletions from retired 
queues (HER-2062)
 2014-01-15 12:49:41 +00:00
Noah Levitt 16fa2b25f1 selftest-crawler-beans.cxml - replace obsolete BdbCookieStorage with BdbCookieStore 2014-01-12 13:11:25 -08:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43 
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
 2014-01-10 17:52:03 -08:00
Noah Levitt 749bd6a214 bump version to 3.3.0-SNAPSHOT after 3.2.0 release 2014-01-10 16:22:43 -08:00
Noah Levitt 32c083f720 bump version number to 3.2.0 for release 2014-01-10 12:36:45 -08:00
Noah Levitt 0ea10665f1 abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner 2013-12-15 11:53:18 -08:00
Noah Levitt 6f171b575a fix report links 2013-12-11 21:14:04 -08:00
Noah Levitt 552794dc89 don't let the exception from commit df2a313797 derail the whole crawl job 2013-11-26 16:08:38 -08:00
Noah Levitt 1163847644 try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted 2013-11-21 18:51:16 -08:00
Noah Levitt 8e1125d062 bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson) 2013-11-20 09:18:31 -08:00
Noah Levitt d93b5449be new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc 2013-11-19 11:05:14 -08:00
Noah Levitt 7089caf3c4 remove unused engine.css 2013-11-15 19:11:19 -08:00
Noah Levitt 119e565acc Adam's responsive UI 2013-11-15 18:59:14 -08:00
Noah Levitt a1f84d3cc4 HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility 2013-10-10 17:57:18 -07:00
Noah Levitt 254811ff7f commented-out default for new setting forgetAllButLatest on CheckpointService 2013-09-08 12:39:54 -07:00
Noah Levitt f096609a50 avoid infinite loop spitting out stack traces on BeansException on startup 2013-09-08 00:11:54 -07:00
Noah Levitt a00141eef7 avoid deadlock in case of BeansException at crawl startup 2013-09-08 00:10:27 -07:00
Noah Levitt fd71833050 avoid NPE when attempting to terminate already-finished job 2013-09-07 09:57:33 -07:00
Noah Levitt d01bc4ed55 when resuming from checkpoint, make next checkpoint serial number follow serial number of resumed checkpoint 2013-09-07 09:57:07 -07:00
Noah Levitt d61171f076 CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size 2013-09-07 09:55:58 -07:00
Noah Levitt 59596c7043 prevent redundant checkpoints when crawl is stopping 2013-09-03 14:24:41 -07:00
Noah Levitt 2fe7d8cd45 some progress on forgetting old checkpoints when checkpointing anew 2013-08-27 18:39:25 -07:00
Noah Levitt 77a12e63b8 make special "host" "whois:" appear correctly in hosts report 2013-08-27 17:39:07 -07:00
Noah Levitt d1bb1a4637 further improve performance of BdbUriUniqFilter.forgetAllSchemeAuthorityMatching() by skipping conversion of key data to long and comparing bytes directly 2013-08-21 11:44:46 -07:00
Noah Levitt a06c141740 rename BdbUriUniqFilter.forgetSchemeHost() to forgetAllSchemeAuthorityMatching(), improve performance 2013-08-21 11:31:05 -07:00
Noah Levitt 750a688670 Merge branch 'master' into hc43 
Conflicts:
	commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
	engine/pom.xml
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
	modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
 2013-08-08 21:14:11 -07:00
Noah Levitt f4f3b9be38 add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs 2013-07-26 17:50:36 -07:00
Noah Levitt b9726c8d4f new method BdbUriUniqFilter.forgetSchemeHost(String schemeHost), and unit test 2013-07-25 18:51:26 -07:00
Noah Levitt b6b3ae6970 make log level check for debug/performance logging consistent to avoid unnecessary operations 2013-07-25 17:28:16 -07:00
Noah Levitt 452304ef12 profile-crawler-beans.cxml - include commented out default value for WARCWriterProcessor "template" setting - see http://tech.groups.yahoo.com/group/archive-crawler/message/8182 2013-07-18 18:38:33 -07:00
Kenji Nagahashi b88242ee16 bean browser: handle array of non-Object element type properly. 
(just avoiding throwing exception - further change would be necessary
for better rendering.)
 2013-06-16 08:21:53 -07:00