Noah Levitt
963e04e50c
handle exception determining whether to apply overlay
2014-05-13 15:47:21 -07:00
Noah Levitt
ba68649ba0
Merge remote-tracking branch 'origin/master' into cosmetics
2014-05-07 10:02:06 -07:00
Noah Levitt
b7f8a0669f
Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics
2014-05-07 10:02:00 -07:00
Noah Levitt
33e782e2fd
oops, make "exit java process" button work again
2014-05-07 10:01:47 -07:00
Noah Levitt
d777c198bd
HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji)
2014-04-24 17:23:01 -07:00
Noah Levitt
0cd1043cb1
change memory bar color, use consistent non-capitalization on buttons
2014-04-21 11:53:40 -07:00
Noah Levitt
1b72ce5dd8
remove extra </div>
2014-04-21 11:53:05 -07:00
Noah Levitt
cfb9474f7a
cosmetic changes
2014-04-21 09:06:15 -07:00
Noah Levitt
20a5f31e19
ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet)
2014-03-31 18:57:33 -07:00
Shriphani Palakodety
2f60b5f935
migration to latest guava complete
2014-03-18 06:42:02 -04:00
Noah Levitt
ea19dacbd2
handle main job log handler like current launch job log handler, close on teardown
2014-02-26 17:19:50 -08:00
Martins Balodis
88dc01305f
fix job.log file handler was left open when jobdir is removed
2014-02-26 21:15:38 +02:00
Kristinn Sigurðsson
df72bc8afc
Minor tidying up
2014-01-15 12:51:35 +00:00
Kristinn Sigurðsson
bc77e82da5
Fix for WorkQueueFrontier.deleteURIs mishandles deletions from retired
...
queues (HER-2062)
2014-01-15 12:49:41 +00:00
Noah Levitt
3a429ffa44
Merge branch 'master' into hc43
...
Conflicts:
commons/pom.xml
commons/src/main/java/org/archive/io/RecordingInputStream.java
commons/src/main/java/org/archive/io/RecordingOutputStream.java
commons/src/main/java/org/archive/io/ReplayInputStream.java
dist/pom.xml
engine/pom.xml
modules/pom.xml
modules/src/main/java/org/archive/modules/extractor/ExtractorHTTP.java
modules/src/main/java/org/archive/modules/recrawl/FetchHistoryProcessor.java
pom.xml
2014-01-10 17:52:03 -08:00
Noah Levitt
6f171b575a
fix report links
2013-12-11 21:14:04 -08:00
Noah Levitt
552794dc89
don't let the exception from commit df2a313797 derail the whole crawl job
2013-11-26 16:08:38 -08:00
Noah Levitt
1163847644
try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted
2013-11-21 18:51:16 -08:00
Noah Levitt
d93b5449be
new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc
2013-11-19 11:05:14 -08:00
Noah Levitt
7089caf3c4
remove unused engine.css
2013-11-15 19:11:19 -08:00
Noah Levitt
119e565acc
Adam's responsive UI
2013-11-15 18:59:14 -08:00
Noah Levitt
a1f84d3cc4
HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility
2013-10-10 17:57:18 -07:00
Noah Levitt
254811ff7f
commented-out default for new setting forgetAllButLatest on CheckpointService
2013-09-08 12:39:54 -07:00
Noah Levitt
f096609a50
avoid infinite loop spitting out stack traces on BeansException on startup
2013-09-08 00:11:54 -07:00
Noah Levitt
a00141eef7
avoid deadlock in case of BeansException at crawl startup
2013-09-08 00:10:27 -07:00
Noah Levitt
fd71833050
avoid NPE when attempting to terminate already-finished job
2013-09-07 09:57:33 -07:00
Noah Levitt
d01bc4ed55
when resuming from checkpoint, make next checkpoint serial number follow serial number of resumed checkpoint
2013-09-07 09:57:07 -07:00
Noah Levitt
d61171f076
CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size
2013-09-07 09:55:58 -07:00
Noah Levitt
59596c7043
prevent redundant checkpoints when crawl is stopping
2013-09-03 14:24:41 -07:00
Noah Levitt
2fe7d8cd45
some progress on forgetting old checkpoints when checkpointing anew
2013-08-27 18:39:25 -07:00
Noah Levitt
77a12e63b8
make special "host" "whois:" appear correctly in hosts report
2013-08-27 17:39:07 -07:00
Noah Levitt
d1bb1a4637
further improve performance of BdbUriUniqFilter.forgetAllSchemeAuthorityMatching() by skipping conversion of key data to long and comparing bytes directly
2013-08-21 11:44:46 -07:00
Noah Levitt
a06c141740
rename BdbUriUniqFilter.forgetSchemeHost() to forgetAllSchemeAuthorityMatching(), improve performance
2013-08-21 11:31:05 -07:00
Noah Levitt
750a688670
Merge branch 'master' into hc43
...
Conflicts:
commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
engine/pom.xml
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
2013-08-08 21:14:11 -07:00
Noah Levitt
f4f3b9be38
add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs
2013-07-26 17:50:36 -07:00
Noah Levitt
b9726c8d4f
new method BdbUriUniqFilter.forgetSchemeHost(String schemeHost), and unit test
2013-07-25 18:51:26 -07:00
Noah Levitt
b6b3ae6970
make log level check for debug/performance logging consistent to avoid unnecessary operations
2013-07-25 17:28:16 -07:00
Noah Levitt
452304ef12
profile-crawler-beans.cxml - include commented out default value for WARCWriterProcessor "template" setting - see http://tech.groups.yahoo.com/group/archive-crawler/message/8182
2013-07-18 18:38:33 -07:00
Kenji Nagahashi
b88242ee16
bean browser: handle array of non-Object element type properly.
...
(just avoiding throwing exception - further change would be necessary
for better rendering.)
2013-06-16 08:21:53 -07:00
Kenji Nagahashi
9ef2fb0f7f
Merge branch 'topic/scripting-xml-fix'
2013-06-10 15:36:13 -07:00
Kenji Nagahashi
358afd4486
revised scripting console xml response fix based on comments.
...
removed scripting console actions from ScriptModel into ScriptingConsole
class (resurrection of ScriptExec).
added basic ScriptingConsoleTest.
2013-06-10 14:17:27 -07:00
Noah Levitt
f2984881f6
address HER-2042 - new annotation duplicate:uriAgnosticDigest for all url agnostic duplicates, whether written to warcs or not; stats consult this annotation so that duplicate/novel numbers are correct
2013-06-07 17:17:24 -07:00
Kenji Nagahashi
3a54600a79
ADD unit test for XmlMarshaller
2013-05-29 12:03:49 -07:00
Kenji Nagahashi
c53b821c85
FIX: 73f58dc56a broke XML variant of scripting console response, and will
...
produce incompatible structure (extra nesting with scriptExec element).
moved ScriptExecution code to ScriptModel, which is now a bean, not Map.
XmlMarshaller is extended to support simple JavaBeans to XML marshalling,
borrowing annotation classes from javax.xml.
2013-05-28 13:37:04 -07:00
Kenji Nagahashi
7f2aacba2c
BeanLookupBindings: check appCtx != null explicitly, rather than using
...
assert.
2013-05-28 13:20:23 -07:00
Noah Levitt
73888819dc
fix bug where scripting console no longer retains your script when you hit execute
2013-05-15 14:23:53 -07:00
Noah Levitt
5b4d8c7f0d
use intended File.separator instead of File.pathSeparator
2013-05-13 22:06:09 -07:00
Kenji Nagahashi
3841d096d5
removed commented-out old code.
2013-05-11 15:44:25 -07:00
Kenji Nagahashi
2825cb736a
script conole: say "line" when linesExecuted==1.
...
generate global variables guide from model.availableGlobalVariables.
2013-05-11 15:44:25 -07:00
Kenji Nagahashi
73f58dc56a
scripting console:
...
fix bug: exception from script execution is not displayed.
remove extra space around exception and rawOutput.
involves a bit more than bug fix. introduced ScriptExecution class for
encapsulating script, execution, and result.
2013-05-11 15:44:25 -07:00
Noah Levitt