32c083f720
bump version number to 3.2.0 for release
2014-01-10 12:36:45 -08:00
0ea10665f1
abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner
2013-12-15 11:53:18 -08:00
6f171b575a
fix report links
2013-12-11 21:14:04 -08:00
552794dc89
don't let the exception from commit df2a313797 derail the whole crawl job
2013-11-26 16:08:38 -08:00
1163847644
try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted
2013-11-21 18:51:16 -08:00
8e1125d062
bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson)
2013-11-20 09:18:31 -08:00
d93b5449be
new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc
2013-11-19 11:05:14 -08:00
7089caf3c4
remove unused engine.css
2013-11-15 19:11:19 -08:00
119e565acc
Adam's responsive UI
2013-11-15 18:59:14 -08:00
a1f84d3cc4
HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility
2013-10-10 17:57:18 -07:00
254811ff7f
commented-out default for new setting forgetAllButLatest on CheckpointService
2013-09-08 12:39:54 -07:00
f096609a50
avoid infinite loop spitting out stack traces on BeansException on startup
2013-09-08 00:11:54 -07:00
a00141eef7
avoid deadlock in case of BeansException at crawl startup
2013-09-08 00:10:27 -07:00
fd71833050
avoid NPE when attempting to terminate already-finished job
2013-09-07 09:57:33 -07:00
d01bc4ed55
when resuming from checkpoint, make next checkpoint serial number follow serial number of resumed checkpoint
2013-09-07 09:57:07 -07:00
d61171f076
CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size
2013-09-07 09:55:58 -07:00
59596c7043
prevent redundant checkpoints when crawl is stopping
2013-09-03 14:24:41 -07:00
2fe7d8cd45
some progress on forgetting old checkpoints when checkpointing anew
2013-08-27 18:39:25 -07:00
77a12e63b8
make special "host" "whois:" appear correctly in hosts report
2013-08-27 17:39:07 -07:00
d1bb1a4637
further improve performance of BdbUriUniqFilter.forgetAllSchemeAuthorityMatching() by skipping conversion of key data to long and comparing bytes directly
2013-08-21 11:44:46 -07:00
a06c141740
rename BdbUriUniqFilter.forgetSchemeHost() to forgetAllSchemeAuthorityMatching(), improve performance
2013-08-21 11:31:05 -07:00
f4f3b9be38
add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs
2013-07-26 17:50:36 -07:00
b9726c8d4f
new method BdbUriUniqFilter.forgetSchemeHost(String schemeHost), and unit test
2013-07-25 18:51:26 -07:00
b6b3ae6970
make log level check for debug/performance logging consistent to avoid unnecessary operations
2013-07-25 17:28:16 -07:00
452304ef12
profile-crawler-beans.cxml - include commented out default value for WARCWriterProcessor "template" setting - see http://tech.groups.yahoo.com/group/archive-crawler/message/8182
2013-07-18 18:38:33 -07:00
b88242ee16
bean browser: handle array of non-Object element type properly.
...
(just avoiding throwing exception - further change would be necessary
for better rendering.)
2013-06-16 08:21:53 -07:00
9ef2fb0f7f
Merge branch 'topic/scripting-xml-fix'
2013-06-10 15:36:13 -07:00
358afd4486
revised scripting console xml response fix based on comments.
...
removed scripting console actions from ScriptModel into ScriptingConsole
class (resurrection of ScriptExec).
added basic ScriptingConsoleTest.
2013-06-10 14:17:27 -07:00
f2984881f6
address HER-2042 - new annotation duplicate:uriAgnosticDigest for all url agnostic duplicates, whether written to warcs or not; stats consult this annotation so that duplicate/novel numbers are correct
2013-06-07 17:17:24 -07:00
3a54600a79
ADD unit test for XmlMarshaller
2013-05-29 12:03:49 -07:00
c53b821c85
FIX: 73f58dc56a broke XML variant of scripting console response, and will
...
produce incompatible structure (extra nesting with scriptExec element).
moved ScriptExecution code to ScriptModel, which is now a bean, not Map.
XmlMarshaller is extended to support simple JavaBeans to XML marshalling,
borrowing annotation classes from javax.xml.
2013-05-28 13:37:04 -07:00
7f2aacba2c
BeanLookupBindings: check appCtx != null explicitly, rather than using
...
assert.
2013-05-28 13:20:23 -07:00
73888819dc
fix bug where scripting console no longer retains your script when you hit execute
2013-05-15 14:23:53 -07:00
5b4d8c7f0d
use intended File.separator instead of File.pathSeparator
2013-05-13 22:06:09 -07:00
3841d096d5
removed commented-out old code.
2013-05-11 15:44:25 -07:00
2825cb736a
script conole: say "line" when linesExecuted==1.
...
generate global variables guide from model.availableGlobalVariables.
2013-05-11 15:44:25 -07:00
73f58dc56a
scripting console:
...
fix bug: exception from script execution is not displayed.
remove extra space around exception and rawOutput.
involves a bit more than bug fix. introduced ScriptExecution class for
encapsulating script, execution, and result.
2013-05-11 15:44:25 -07:00
e531de0ed3
avoid polluting logs with long stacktraces when a request comes in for a report from an unbuilt job (a fairly common case in archive-it)
2013-04-29 19:02:22 -07:00
dafd95eef7
fix for HER-2037 final crawl reports written after crawl is "FINISHED" - call dumpReports() in StatisticsTracker.stop() which (since the HER-1954 changes) runs at the right time, after all url processing has finished but before the FINISHED state is officially reached
2013-04-22 21:23:43 -07:00
1713093699
remove stray space
2013-04-22 21:18:35 -07:00
3d90506b99
use LinkedHashMap to ensure consistent ordering of xml api
2013-04-09 10:34:02 -07:00
08e18fd692
avoid creating unnecessary AlerthThreadGroups
2013-04-05 09:35:23 -07:00
475d07e8ca
attempt to address HER-2036 occasional strange memory leak after crawl finishes - have ToePool discard its reference to CrawlController at crawl finish
2013-04-05 09:34:58 -07:00
d07bf80e02
add link to engine page on job page
2013-04-03 23:50:23 -07:00
5a7cd8a6a8
disable codemirror search which add-on which seems to fight with browser search all the time
2013-04-03 17:16:04 -07:00
fd668e82ba
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-04-03 16:39:45 -07:00
9f33b3b667
button on engine page to ask java to run garbage collector
2013-04-03 16:39:32 -07:00
c98c1d0e29
changed CrawlerLoggerModule.setupLogs() from private to protected so that sub-class can modify logging configuration.
2013-04-02 17:55:03 -07:00
4dd46e6af8
quick fix to ClassCastException in ToePool.shortReportLineTo().
...
fixed a typo in Job.ftl.
2013-04-01 23:45:22 -07:00
1ac1dd9bab
fix typo which removed call to candidate.setFullVia(), should fix failing Precedence3SelfTest
2013-03-29 13:17:58 -07:00
Noah Levitt