Noah Levitt
a00141eef7
avoid deadlock in case of BeansException at crawl startup
2013-09-08 00:10:27 -07:00
Noah Levitt
fd71833050
avoid NPE when attempting to terminate already-finished job
2013-09-07 09:57:33 -07:00
Noah Levitt
d01bc4ed55
when resuming from checkpoint, make next checkpoint serial number follow serial number of resumed checkpoint
2013-09-07 09:57:07 -07:00
Noah Levitt
d61171f076
CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size
2013-09-07 09:55:58 -07:00
Noah Levitt
59596c7043
prevent redundant checkpoints when crawl is stopping
2013-09-03 14:24:41 -07:00
Noah Levitt
2fe7d8cd45
some progress on forgetting old checkpoints when checkpointing anew
2013-08-27 18:39:25 -07:00
Noah Levitt
77a12e63b8
make special "host" "whois:" appear correctly in hosts report
2013-08-27 17:39:07 -07:00
Noah Levitt
d1bb1a4637
further improve performance of BdbUriUniqFilter.forgetAllSchemeAuthorityMatching() by skipping conversion of key data to long and comparing bytes directly
2013-08-21 11:44:46 -07:00
Noah Levitt
a06c141740
rename BdbUriUniqFilter.forgetSchemeHost() to forgetAllSchemeAuthorityMatching(), improve performance
2013-08-21 11:31:05 -07:00
Noah Levitt
750a688670
Merge branch 'master' into hc43
...
Conflicts:
commons/src/main/java/org/apache/commons/httpclient/HttpParser.java
engine/pom.xml
modules/src/main/java/org/archive/modules/CrawlURI.java
modules/src/main/java/org/archive/modules/deciderules/MatchesStatusCodeDecideRule.java
modules/src/main/java/org/archive/modules/deciderules/NotMatchesStatusCodeDecideRule.java
modules/src/main/java/org/archive/modules/fetcher/FetchHTTP.java
2013-08-08 21:14:11 -07:00
Noah Levitt
f4f3b9be38
add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs
2013-07-26 17:50:36 -07:00
Noah Levitt
b9726c8d4f
new method BdbUriUniqFilter.forgetSchemeHost(String schemeHost), and unit test
2013-07-25 18:51:26 -07:00
Noah Levitt
b6b3ae6970
make log level check for debug/performance logging consistent to avoid unnecessary operations
2013-07-25 17:28:16 -07:00
Noah Levitt
452304ef12
profile-crawler-beans.cxml - include commented out default value for WARCWriterProcessor "template" setting - see http://tech.groups.yahoo.com/group/archive-crawler/message/8182
2013-07-18 18:38:33 -07:00
Kenji Nagahashi
b88242ee16
bean browser: handle array of non-Object element type properly.
...
(just avoiding throwing exception - further change would be necessary
for better rendering.)
2013-06-16 08:21:53 -07:00
Kenji Nagahashi
9ef2fb0f7f
Merge branch 'topic/scripting-xml-fix'
2013-06-10 15:36:13 -07:00
Kenji Nagahashi
358afd4486
revised scripting console xml response fix based on comments.
...
removed scripting console actions from ScriptModel into ScriptingConsole
class (resurrection of ScriptExec).
added basic ScriptingConsoleTest.
2013-06-10 14:17:27 -07:00
Noah Levitt
f2984881f6
address HER-2042 - new annotation duplicate:uriAgnosticDigest for all url agnostic duplicates, whether written to warcs or not; stats consult this annotation so that duplicate/novel numbers are correct
2013-06-07 17:17:24 -07:00
Kenji Nagahashi
3a54600a79
ADD unit test for XmlMarshaller
2013-05-29 12:03:49 -07:00
Kenji Nagahashi
c53b821c85
FIX: 73f58dc56a broke XML variant of scripting console response, and will
...
produce incompatible structure (extra nesting with scriptExec element).
moved ScriptExecution code to ScriptModel, which is now a bean, not Map.
XmlMarshaller is extended to support simple JavaBeans to XML marshalling,
borrowing annotation classes from javax.xml.
2013-05-28 13:37:04 -07:00
Kenji Nagahashi
7f2aacba2c
BeanLookupBindings: check appCtx != null explicitly, rather than using
...
assert.
2013-05-28 13:20:23 -07:00
Noah Levitt
73888819dc
fix bug where scripting console no longer retains your script when you hit execute
2013-05-15 14:23:53 -07:00
Kristinn Sigurðsson
858523db94
HER-2039: Initial work. Eliminated Link class and dealt with resultant
...
errors. Not tested any further than that.
2013-05-15 13:36:01 +00:00
Noah Levitt
5b4d8c7f0d
use intended File.separator instead of File.pathSeparator
2013-05-13 22:06:09 -07:00
Kenji Nagahashi
3841d096d5
removed commented-out old code.
2013-05-11 15:44:25 -07:00
Kenji Nagahashi
2825cb736a
script conole: say "line" when linesExecuted==1.
...
generate global variables guide from model.availableGlobalVariables.
2013-05-11 15:44:25 -07:00
Kenji Nagahashi
73f58dc56a
scripting console:
...
fix bug: exception from script execution is not displayed.
remove extra space around exception and rawOutput.
involves a bit more than bug fix. introduced ScriptExecution class for
encapsulating script, execution, and result.
2013-05-11 15:44:25 -07:00
Noah Levitt
e531de0ed3
avoid polluting logs with long stacktraces when a request comes in for a report from an unbuilt job (a fairly common case in archive-it)
2013-04-29 19:02:22 -07:00
Noah Levitt
dafd95eef7
fix for HER-2037 final crawl reports written after crawl is "FINISHED" - call dumpReports() in StatisticsTracker.stop() which (since the HER-1954 changes) runs at the right time, after all url processing has finished but before the FINISHED state is officially reached
2013-04-22 21:23:43 -07:00
Noah Levitt
1713093699
remove stray space
2013-04-22 21:18:35 -07:00
Noah Levitt
3d90506b99
use LinkedHashMap to ensure consistent ordering of xml api
2013-04-09 10:34:02 -07:00
Noah Levitt
08e18fd692
avoid creating unnecessary AlerthThreadGroups
2013-04-05 09:35:23 -07:00
Noah Levitt
475d07e8ca
attempt to address HER-2036 occasional strange memory leak after crawl finishes - have ToePool discard its reference to CrawlController at crawl finish
2013-04-05 09:34:58 -07:00
Noah Levitt
d07bf80e02
add link to engine page on job page
2013-04-03 23:50:23 -07:00
Noah Levitt
5a7cd8a6a8
disable codemirror search which add-on which seems to fight with browser search all the time
2013-04-03 17:16:04 -07:00
Noah Levitt
fd668e82ba
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-04-03 16:39:45 -07:00
Noah Levitt
9f33b3b667
button on engine page to ask java to run garbage collector
2013-04-03 16:39:32 -07:00
Kenji Nagahashi
c98c1d0e29
changed CrawlerLoggerModule.setupLogs() from private to protected so that sub-class can modify logging configuration.
2013-04-02 17:55:03 -07:00
Kenji Nagahashi
4dd46e6af8
quick fix to ClassCastException in ToePool.shortReportLineTo().
...
fixed a typo in Job.ftl.
2013-04-01 23:45:22 -07:00
Noah Levitt
1ac1dd9bab
fix typo which removed call to candidate.setFullVia(), should fix failing Precedence3SelfTest
2013-03-29 13:17:58 -07:00
Noah Levitt
dcbbec4bbb
fix for HER-2035 xml representation of job url can be invalid
2013-03-29 12:16:07 -07:00
Noah Levitt
f988a26e10
move some code into EngineModel constructor and restore reverse chronological ordering of crawl jobs
2013-03-28 19:09:14 -07:00
Noah Levitt
6a2b4a4a76
Merge branch 'master' into freemarker
...
Conflicts:
.classpath
2013-03-28 18:01:16 -07:00
Noah Levitt
5d8fe0443d
Merge branch 'master' into HER-2031
2013-03-23 19:16:05 -07:00
Noah Levitt
1993c4388c
Merge branch 'master' into hc43
...
Conflicts:
.classpath
2013-03-20 14:28:13 -07:00
Noah Levitt
8fd361e518
update source/target version to 1.6 since we use 1.6 features, specifically @Override for methods only implementing an interface (not sure why maven wasn't enforcing this); also remove maven-antrun-plugin created timestamp.txt, doesn't seem to be used anywhere
2013-03-20 14:25:22 -07:00
Noah Levitt
c509b6e552
remove more eclipse stuff, people like to manage their own
2013-03-20 13:39:28 -07:00
Noah Levitt
12da824316
Merge branch 'master' into hc43
2013-03-18 23:32:51 -07:00
Adam Miller
2f4d437734
Beans.ftl
...
Fixed a bug that showed objects with no properties as null instead of displaying the class name
Job.ftl
Fixed a bug that prevented the configuration referenced paths from displaying
Script.ftl
Fixed a bug that rendered extra whitespace before and after the script block
2013-03-15 16:06:06 -07:00
gojomo
a844878e7d
Merge branch 'simple_logins' into HER-2031
2013-03-12 01:16:31 -07:00
Noah Levitt