Kristinn Sigurðsson
3495851029
JavaDoc typo
2015-07-13 15:10:44 +00:00
Kristinn Sigurðsson
b15fbbd13e
Enable suppression of empty hosts in report.
...
Until now hosts that have never been crawled are included in the hosts
report. This can be a very large number of hosts and is infrequently of
interest. This makes it possible to exclude the empty hosts from the
report. Default behavior remains to include them.
The option is highlighted by including it in the relevant, commented-out
section of the default profile.
2015-07-13 15:08:03 +00:00
Kristinn Sigurðsson
0cf009eb4c
Use default value in sample profile.
2015-07-13 14:58:23 +00:00
Kristinn Sigurðsson
96b927d9fa
Make sorting of host report configurable.
...
Can now set a maximum size for sorted reports. Longer reports are
unsorted. Default behavior is still to sort all reports.
Default profile updated to reflect this new option in a commented-out
section.
2015-07-13 14:55:26 +00:00
Kristinn Sigurðsson
bf638939b0
Remove unused "top" hosts tracking.
...
This used to be used for an overview report that was available in the H1
UI. Had become vestigial code that served no purpose.
2015-07-13 14:32:34 +00:00
Noah Levitt
8009275075
make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/
2015-02-23 11:36:29 -08:00
Kristinn Sigurðsson
4ea7efdfc8
Set character set to UTF-8 when passing through files.
2015-02-10 16:07:37 +00:00
vonrosen
3609abe38c
Merge pull request #108 from nlevitt/aitfive-203
...
kafka crawl log feed
2014-11-14 10:29:41 -08:00
Noah Levitt
7312ad597e
so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ)
2014-11-07 13:21:42 -08:00
Adam Miller
fb23b40e74
CandidatesProcessor.java
...
Keep CrawlURI outlinks intact for use in SeedRecord reporting.
2014-10-22 13:18:25 -07:00
vonrosen
5909042b84
Merge pull request #99 from nlevitt/aitfive-39
...
AMQPCrawlLogFeed, DecideRuleSequenceWithAMQPFeed, DecideRuleSequence.logExtraInfo
2014-10-16 17:24:37 +00:00
Noah Levitt
5f7dc49599
Merge pull request #95 from kngenie/patch-beanbrowser-failure
...
FIX corner-case of bean browser failing due to an exception from hashCode()
2014-10-03 19:21:27 -07:00
Noah Levitt
e5e5cd87e4
configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency
2014-10-01 13:38:39 -07:00
Noah Levitt
610a1e1e97
cookie store is called BdbCookieStore again
2014-09-28 14:32:34 -07:00
Noah Levitt
a37a8de4a2
improved implementation not using TreeSet, and some initial work on integration testing of cookie handling
2014-09-28 11:59:14 -07:00
Noah Levitt
47b7caad29
initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains
2014-09-26 20:56:45 -07:00
Noah Levitt
f769e0636d
initial proof of concept fix for HER-2070 "cookie monster" bug
2014-09-26 20:01:52 -07:00
Kenji Nagahashi
5821fe7b1e
FIX corner-case of bean browser failing due to an exception from hashCode().
2014-09-11 11:22:01 -07:00
Kenji Nagahashi
2a38fa0b12
Don’t let exception from property getter fail entire bean-browser.
2014-08-25 13:36:25 -07:00
Noah Levitt
04921073ca
fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering()
2014-08-19 17:14:32 -07:00
Noah Levitt
591f0f9f0f
expose forAllPendingDo() as public so it can be used elsewhere
2014-08-13 13:02:28 -07:00
Noah Levitt
9a0c0656b2
move HostsReport.fixup(String) to CrawlHost.fixUpName() so it can be called elsewhere
2014-08-12 16:45:53 -07:00
Noah Levitt
51081a652c
make AbstractFrontier.needReenqueuing() public so it can be called elsewhere
2014-08-12 16:20:25 -07:00
Noah Levitt
47a62f7c6f
remove a couple of crufty lines of code
2014-08-12 16:13:59 -07:00
Noah Levitt
08e24bbdff
update guess at log line length based on real world data
2014-08-12 16:13:11 -07:00
Noah Levitt
905a98c0c2
use RevisitProfile for tallying stats
2014-07-10 21:27:50 -07:00
Noah Levitt
ee52f48be9
remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper
2014-07-09 10:14:24 -07:00
Noah Levitt
f23a59a918
Merge branch 'master' into HER-2039
...
Conflicts:
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
modules/src/main/java/org/archive/modules/extractor/Link.java
modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
2014-07-07 11:45:27 -07:00
Roger G. Coram
fc43a29459
Merge pull request #68 from nlevitt/handle-apply-overlay-exc
...
handle exception determining whether to apply overlay
2014-05-28 09:59:54 +01:00
Noah Levitt
39814d1efd
avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress
2014-05-22 16:40:44 -07:00
Noah Levitt
963e04e50c
handle exception determining whether to apply overlay
2014-05-13 15:47:21 -07:00
Noah Levitt
ba68649ba0
Merge remote-tracking branch 'origin/master' into cosmetics
2014-05-07 10:02:06 -07:00
Noah Levitt
b7f8a0669f
Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics
2014-05-07 10:02:00 -07:00
Noah Levitt
33e782e2fd
oops, make "exit java process" button work again
2014-05-07 10:01:47 -07:00
Noah Levitt
d777c198bd
HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji)
2014-04-24 17:23:01 -07:00
Noah Levitt
0cd1043cb1
change memory bar color, use consistent non-capitalization on buttons
2014-04-21 11:53:40 -07:00
Noah Levitt
1b72ce5dd8
remove extra </div>
2014-04-21 11:53:05 -07:00
Noah Levitt
cfb9474f7a
cosmetic changes
2014-04-21 09:06:15 -07:00
Noah Levitt
552200b2fa
fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass
2014-03-31 20:38:17 -07:00
Noah Levitt
20a5f31e19
ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet)
2014-03-31 18:57:33 -07:00
Shriphani Palakodety
2f60b5f935
migration to latest guava complete
2014-03-18 06:42:02 -04:00
Noah Levitt
ea19dacbd2
handle main job log handler like current launch job log handler, close on teardown
2014-02-26 17:19:50 -08:00
Martins Balodis
88dc01305f
fix job.log file handler was left open when jobdir is removed
2014-02-26 21:15:38 +02:00
Kristinn Sigurðsson
df72bc8afc
Minor tidying up
2014-01-15 12:51:35 +00:00
Kristinn Sigurðsson
bc77e82da5
Fix for WorkQueueFrontier.deleteURIs mishandles deletions from retired
...
queues (HER-2062)
2014-01-15 12:49:41 +00:00
Noah Levitt
16fa2b25f1
selftest-crawler-beans.cxml - replace obsolete BdbCookieStorage with BdbCookieStore
2014-01-12 13:11:25 -08:00
Noah Levitt
3a429ffa44
Merge branch 'master' into hc43
...
Conflicts:
commons/pom.xml
commons/src/main/java/org/archive/io/RecordingInputStream.java
commons/src/main/java/org/archive/io/RecordingOutputStream.java
commons/src/main/java/org/archive/io/ReplayInputStream.java
dist/pom.xml
engine/pom.xml
modules/pom.xml
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
pom.xml
2014-01-10 17:52:03 -08:00
Noah Levitt
749bd6a214
bump version to 3.3.0-SNAPSHOT after 3.2.0 release
2014-01-10 16:22:43 -08:00
Noah Levitt
32c083f720
bump version number to 3.2.0 for release
2014-01-10 12:36:45 -08:00
Noah Levitt
0ea10665f1
abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner
2013-12-15 11:53:18 -08:00
Kristinn Sigurðsson