lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-07-28 18:31:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,705 Commits 40 Branches 31 Tags
ce31cada5f97288ea3ffa6fe4f85fee1981cbc1a
Commit Graph
610 Commits
Author SHA1 Message Date
Noah Levitt 7929e16d6a fix checkpointing problems with new statsBySource 2015-12-04 21:33:40 -08:00
Noah Levitt ce4a865764 add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality 2015-12-03 16:13:34 -08:00
Hunter 79ebb0e03c Merge pull request #129 from nlevitt/java8keytool 
load keytool utility main class dynamically, trying both the old and …
 2015-10-06 16:09:43 -07:00
Noah Levitt 11d4bab1ac catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening 2015-10-05 11:46:35 -07:00
Noah Levitt 12dde2b693 load keytool utility main class dynamically, trying both the old and new (java 8) class names 2015-09-23 12:06:11 -07:00
Noah Levitt 8009275075 make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/ 2015-02-23 11:36:29 -08:00
Kristinn Sigurðsson 4ea7efdfc8 Set character set to UTF-8 when passing through files. 2015-02-10 16:07:37 +00:00
vonrosen 3609abe38c Merge pull request #108 from nlevitt/aitfive-203 
kafka crawl log feed
 2014-11-14 10:29:41 -08:00
Noah Levitt 7312ad597e so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ) 2014-11-07 13:21:42 -08:00
Adam Miller fb23b40e74 CandidatesProcessor.java 
Keep CrawlURI outlinks intact for use in SeedRecord reporting.
 2014-10-22 13:18:25 -07:00
vonrosen 5909042b84 Merge pull request #99 from nlevitt/aitfive-39 
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
 2014-10-16 17:24:37 +00:00
Noah Levitt 5f7dc49599 Merge pull request #95 from kngenie/patch-beanbrowser-failure 
FIX corner-case of bean browser failing due to an exception from hashCode()
 2014-10-03 19:21:27 -07:00
Noah Levitt e5e5cd87e4 configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency 2014-10-01 13:38:39 -07:00
Noah Levitt 610a1e1e97 cookie store is called BdbCookieStore again 2014-09-28 14:32:34 -07:00
Noah Levitt a37a8de4a2 improved implementation not using TreeSet, and some initial work on integration testing of cookie handling 2014-09-28 11:59:14 -07:00
Noah Levitt 47b7caad29 initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains 2014-09-26 20:56:45 -07:00
Noah Levitt f769e0636d initial proof of concept fix for HER-2070 "cookie monster" bug 2014-09-26 20:01:52 -07:00
Kenji Nagahashi 5821fe7b1e FIX corner-case of bean browser failing due to an exception from hashCode(). 2014-09-11 11:22:01 -07:00
Kenji Nagahashi 2a38fa0b12 Don’t let exception from property getter fail entire bean-browser. 2014-08-25 13:36:25 -07:00
Noah Levitt 04921073ca fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering() 2014-08-19 17:14:32 -07:00
Noah Levitt 591f0f9f0f expose forAllPendingDo() as public so it can be used elsewhere 2014-08-13 13:02:28 -07:00
Noah Levitt 9a0c0656b2 move HostsReport.fixup(String) to CrawlHost.fixUpName() so it can be called elsewhere 2014-08-12 16:45:53 -07:00
Noah Levitt 51081a652c make AbstractFrontier.needReenqueuing() public so it can be called elsewhere 2014-08-12 16:20:25 -07:00
Noah Levitt 47a62f7c6f remove a couple of crufty lines of code 2014-08-12 16:13:59 -07:00
Noah Levitt 08e24bbdff update guess at log line length based on real world data 2014-08-12 16:13:11 -07:00
Noah Levitt 905a98c0c2 use RevisitProfile for tallying stats 2014-07-10 21:27:50 -07:00
Noah Levitt ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper 2014-07-09 10:14:24 -07:00
Noah Levitt f23a59a918 Merge branch 'master' into HER-2039 
Conflicts:
	modules/src/main/java/org/archive/modules/CrawlURI.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
	modules/src/main/java/org/archive/modules/extractor/Link.java
	modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
 2014-07-07 11:45:27 -07:00
Roger G. Coram fc43a29459 Merge pull request #68 from nlevitt/handle-apply-overlay-exc 
handle exception determining whether to apply overlay
 2014-05-28 09:59:54 +01:00
Noah Levitt 39814d1efd avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress 2014-05-22 16:40:44 -07:00
Noah Levitt 963e04e50c handle exception determining whether to apply overlay 2014-05-13 15:47:21 -07:00
Noah Levitt ba68649ba0 Merge remote-tracking branch 'origin/master' into cosmetics 2014-05-07 10:02:06 -07:00
Noah Levitt b7f8a0669f Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics 2014-05-07 10:02:00 -07:00
Noah Levitt 33e782e2fd oops, make "exit java process" button work again 2014-05-07 10:01:47 -07:00
Noah Levitt d777c198bd HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji) 2014-04-24 17:23:01 -07:00
Noah Levitt 0cd1043cb1 change memory bar color, use consistent non-capitalization on buttons 2014-04-21 11:53:40 -07:00
Noah Levitt 1b72ce5dd8 remove extra </div> 2014-04-21 11:53:05 -07:00
Noah Levitt cfb9474f7a cosmetic changes 2014-04-21 09:06:15 -07:00
Noah Levitt 552200b2fa fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass 2014-03-31 20:38:17 -07:00
Noah Levitt 20a5f31e19 ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet) 2014-03-31 18:57:33 -07:00
Shriphani Palakodety 2f60b5f935 migration to latest guava complete 2014-03-18 06:42:02 -04:00
Noah Levitt ea19dacbd2 handle main job log handler like current launch job log handler, close on teardown 2014-02-26 17:19:50 -08:00
Martins Balodis 88dc01305f fix job.log file handler was left open when jobdir is removed 2014-02-26 21:15:38 +02:00
Kristinn Sigurðsson df72bc8afc Minor tidying up 2014-01-15 12:51:35 +00:00
Kristinn Sigurðsson bc77e82da5 Fix for WorkQueueFrontier.deleteURIs mishandles deletions from retired 
queues (HER-2062)
 2014-01-15 12:49:41 +00:00
Noah Levitt 16fa2b25f1 selftest-crawler-beans.cxml - replace obsolete BdbCookieStorage with BdbCookieStore 2014-01-12 13:11:25 -08:00
Noah Levitt 3a429ffa44 Merge branch 'master' into hc43 
Conflicts:
	commons/pom.xml
	commons/src/main/java/org/archive/io/RecordingInputStream.java
	commons/src/main/java/org/archive/io/RecordingOutputStream.java
	commons/src/main/java/org/archive/io/ReplayInputStream.java
	dist/pom.xml
	engine/pom.xml
	modules/pom.xml
	modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
	modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
	pom.xml
 2014-01-10 17:52:03 -08:00
Noah Levitt 749bd6a214 bump version to 3.3.0-SNAPSHOT after 3.2.0 release 2014-01-10 16:22:43 -08:00
Noah Levitt 32c083f720 bump version number to 3.2.0 for release 2014-01-10 12:36:45 -08:00
Noah Levitt 0ea10665f1 abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner 2013-12-15 11:53:18 -08:00