Noah Levitt
905a98c0c2
use RevisitProfile for tallying stats
2014-07-10 21:27:50 -07:00
Noah Levitt
ee52f48be9
remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper
2014-07-09 10:14:24 -07:00
Noah Levitt
f23a59a918
Merge branch 'master' into HER-2039
...
Conflicts:
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/extractor/ExtractorSWF.java
modules/src/main/java/org/archive/modules/extractor/Link.java
modules/src/main/java/org/archive/modules/writer/WARCWriterProcessor.java
2014-07-07 11:45:27 -07:00
Roger G. Coram
fc43a29459
Merge pull request #68 from nlevitt/handle-apply-overlay-exc
...
handle exception determining whether to apply overlay
2014-05-28 09:59:54 +01:00
Noah Levitt
39814d1efd
avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress
2014-05-22 16:40:44 -07:00
Noah Levitt
963e04e50c
handle exception determining whether to apply overlay
2014-05-13 15:47:21 -07:00
Noah Levitt
ba68649ba0
Merge remote-tracking branch 'origin/master' into cosmetics
2014-05-07 10:02:06 -07:00
Noah Levitt
b7f8a0669f
Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics
2014-05-07 10:02:00 -07:00
Noah Levitt
33e782e2fd
oops, make "exit java process" button work again
2014-05-07 10:01:47 -07:00
Noah Levitt
d777c198bd
HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji)
2014-04-24 17:23:01 -07:00
Noah Levitt
0cd1043cb1
change memory bar color, use consistent non-capitalization on buttons
2014-04-21 11:53:40 -07:00
Noah Levitt
1b72ce5dd8
remove extra </div>
2014-04-21 11:53:05 -07:00
Noah Levitt
cfb9474f7a
cosmetic changes
2014-04-21 09:06:15 -07:00
Noah Levitt
20a5f31e19
ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet)
2014-03-31 18:57:33 -07:00
Shriphani Palakodety
2f60b5f935
migration to latest guava complete
2014-03-18 06:42:02 -04:00
Noah Levitt
ea19dacbd2
handle main job log handler like current launch job log handler, close on teardown
2014-02-26 17:19:50 -08:00
Martins Balodis
88dc01305f
fix job.log file handler was left open when jobdir is removed
2014-02-26 21:15:38 +02:00
Kristinn Sigurðsson
df72bc8afc
Minor tidying up
2014-01-15 12:51:35 +00:00
Kristinn Sigurðsson
bc77e82da5
Fix for WorkQueueFrontier.deleteURIs mishandles deletions from retired
...
queues (HER-2062)
2014-01-15 12:49:41 +00:00
Noah Levitt
3a429ffa44
Merge branch 'master' into hc43
...
Conflicts:
commons/pom.xml
commons/src/main/java/org/archive/io/RecordingInputStream.java
commons/src/main/java/org/archive/io/RecordingOutputStream.java
commons/src/main/java/org/archive/io/ReplayInputStream.java
dist/pom.xml
engine/pom.xml
modules/pom.xml
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
pom.xml
2014-01-10 17:52:03 -08:00
Noah Levitt
6f171b575a
fix report links
2013-12-11 21:14:04 -08:00
Noah Levitt
552794dc89
don't let the exception from commit df2a313797 derail the whole crawl job
2013-11-26 16:08:38 -08:00
Noah Levitt
1163847644
try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted
2013-11-21 18:51:16 -08:00
Noah Levitt
d93b5449be
new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc
2013-11-19 11:05:14 -08:00
Noah Levitt
7089caf3c4
remove unused engine.css
2013-11-15 19:11:19 -08:00
Noah Levitt
119e565acc
Adam's responsive UI
2013-11-15 18:59:14 -08:00
Noah Levitt
a1f84d3cc4
HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility
2013-10-10 17:57:18 -07:00
Noah Levitt
254811ff7f
commented-out default for new setting forgetAllButLatest on CheckpointService
2013-09-08 12:39:54 -07:00
Noah Levitt
f096609a50
avoid infinite loop spitting out stack traces on BeansException on startup
2013-09-08 00:11:54 -07:00
Noah Levitt
a00141eef7
avoid deadlock in case of BeansException at crawl startup
2013-09-08 00:10:27 -07:00
Noah Levitt
fd71833050
avoid NPE when attempting to terminate already-finished job
2013-09-07 09:57:33 -07:00
Noah Levitt
d01bc4ed55
when resuming from checkpoint, make next checkpoint serial number follow serial number of resumed checkpoint
2013-09-07 09:57:07 -07:00
Noah Levitt
d61171f076
CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size
2013-09-07 09:55:58 -07:00
Noah Levitt
59596c7043
prevent redundant checkpoints when crawl is stopping
2013-09-03 14:24:41 -07:00
Noah Levitt
2fe7d8cd45
some progress on forgetting old checkpoints when checkpointing anew
2013-08-27 18:39:25 -07:00
Noah Levitt
77a12e63b8
make special "host" "whois:" appear correctly in hosts report
2013-08-27 17:39:07 -07:00
Noah Levitt
d1bb1a4637
further improve performance of BdbUriUniqFilter.forgetAllSchemeAuthorityMatching() by skipping conversion of key data to long and comparing bytes directly
2013-08-21 11:44:46 -07:00
Noah Levitt
a06c141740
rename BdbUriUniqFilter.forgetSchemeHost() to forgetAllSchemeAuthorityMatching(), improve performance
2013-08-21 11:31:05 -07:00
Noah Levitt
750a688670
Merge branch 'master' into hc43
...
Conflicts:
commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
engine/pom.xml
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
2013-08-08 21:14:11 -07:00
Noah Levitt
f4f3b9be38
add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs
2013-07-26 17:50:36 -07:00
Noah Levitt
b9726c8d4f
new method BdbUriUniqFilter.forgetSchemeHost(String schemeHost), and unit test
2013-07-25 18:51:26 -07:00
Noah Levitt
b6b3ae6970
make log level check for debug/performance logging consistent to avoid unnecessary operations
2013-07-25 17:28:16 -07:00
Noah Levitt
452304ef12
profile-crawler-beans.cxml - include commented out default value for WARCWriterProcessor "template" setting - see http://tech.groups.yahoo.com/group/archive-crawler/message/8182
2013-07-18 18:38:33 -07:00
Kenji Nagahashi
b88242ee16
bean browser: handle array of non-Object element type properly.
...
(just avoiding throwing exception - further change would be necessary
for better rendering.)
2013-06-16 08:21:53 -07:00
Kenji Nagahashi
9ef2fb0f7f
Merge branch 'topic/scripting-xml-fix'
2013-06-10 15:36:13 -07:00
Kenji Nagahashi
358afd4486
revised scripting console xml response fix based on comments.
...
removed scripting console actions from ScriptModel into ScriptingConsole
class (resurrection of ScriptExec).
added basic ScriptingConsoleTest.
2013-06-10 14:17:27 -07:00
Noah Levitt
f2984881f6
address HER-2042 - new annotation duplicate:uriAgnosticDigest for all url agnostic duplicates, whether written to warcs or not; stats consult this annotation so that duplicate/novel numbers are correct
2013-06-07 17:17:24 -07:00
Kenji Nagahashi
3a54600a79
ADD unit test for XmlMarshaller
2013-05-29 12:03:49 -07:00
Kenji Nagahashi
c53b821c85
FIX: 73f58dc56a broke XML variant of scripting console response, and will
...
produce incompatible structure (extra nesting with scriptExec element).
moved ScriptExecution code to ScriptModel, which is now a bean, not Map.
XmlMarshaller is extended to support simple JavaBeans to XML marshalling,
borrowing annotation classes from javax.xml.
2013-05-28 13:37:04 -07:00
Kenji Nagahashi
7f2aacba2c
BeanLookupBindings: check appCtx != null explicitly, rather than using
...
assert.
2013-05-28 13:20:23 -07:00