lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-08 15:51:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,084 Commits 40 Branches 31 Tags
705a375daf4840635449d7061d12e20489b4787f
Commit Graph
468 Commits
Author SHA1 Message Date
Noah Levitt f4f3b9be38 add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs 2013-07-26 17:50:36 -07:00
Noah Levitt b9726c8d4f new method BdbUriUniqFilter.forgetSchemeHost(String schemeHost), and unit test 2013-07-25 18:51:26 -07:00
Noah Levitt b6b3ae6970 make log level check for debug/performance logging consistent to avoid unnecessary operations 2013-07-25 17:28:16 -07:00
Noah Levitt 452304ef12 profile-crawler-beans.cxml - include commented out default value for WARCWriterProcessor "template" setting - see http://tech.groups.yahoo.com/group/archive-crawler/message/8182 2013-07-18 18:38:33 -07:00
Kenji Nagahashi b88242ee16 bean browser: handle array of non-Object element type properly. 
(just avoiding throwing exception - further change would be necessary
for better rendering.)
 2013-06-16 08:21:53 -07:00
Kenji Nagahashi 9ef2fb0f7f Merge branch 'topic/scripting-xml-fix' 2013-06-10 15:36:13 -07:00
Kenji Nagahashi 358afd4486 revised scripting console xml response fix based on comments. 
removed scripting console actions from ScriptModel into ScriptingConsole
class (resurrection of ScriptExec).
added basic ScriptingConsoleTest.
 2013-06-10 14:17:27 -07:00
Noah Levitt f2984881f6 address HER-2042 - new annotation duplicate:uriAgnosticDigest for all url agnostic duplicates, whether written to warcs or not; stats consult this annotation so that duplicate/novel numbers are correct 2013-06-07 17:17:24 -07:00
Kenji Nagahashi 3a54600a79 ADD unit test for XmlMarshaller 2013-05-29 12:03:49 -07:00
Kenji Nagahashi c53b821c85 FIX: 73f58dc56a broke XML variant of scripting console response, and will 
produce incompatible structure (extra nesting with scriptExec element).
  moved ScriptExecution code to ScriptModel, which is now a bean, not Map.
  XmlMarshaller is extended to support simple JavaBeans to XML marshalling,
  borrowing annotation classes from javax.xml.
 2013-05-28 13:37:04 -07:00
Kenji Nagahashi 7f2aacba2c BeanLookupBindings: check appCtx != null explicitly, rather than using 
assert.
 2013-05-28 13:20:23 -07:00
Noah Levitt 73888819dc fix bug where scripting console no longer retains your script when you hit execute 2013-05-15 14:23:53 -07:00
Noah Levitt 5b4d8c7f0d use intended File.separator instead of File.pathSeparator 2013-05-13 22:06:09 -07:00
Kenji Nagahashi 3841d096d5 removed commented-out old code. 2013-05-11 15:44:25 -07:00
Kenji Nagahashi 2825cb736a script conole: say "line" when linesExecuted==1. 
generate global variables guide from model.availableGlobalVariables.
 2013-05-11 15:44:25 -07:00
Kenji Nagahashi 73f58dc56a scripting console: 
fix bug: exception from script execution is not displayed.
remove extra space around exception and rawOutput.
involves a bit more than bug fix. introduced ScriptExecution class for
encapsulating script, execution, and result.
 2013-05-11 15:44:25 -07:00
Noah Levitt e531de0ed3 avoid polluting logs with long stacktraces when a request comes in for a report from an unbuilt job (a fairly common case in archive-it) 2013-04-29 19:02:22 -07:00
Noah Levitt dafd95eef7 fix for HER-2037 final crawl reports written after crawl is "FINISHED" - call dumpReports() in StatisticsTracker.stop() which (since the HER-1954 changes) runs at the right time, after all url processing has finished but before the FINISHED state is officially reached 2013-04-22 21:23:43 -07:00
Noah Levitt 1713093699 remove stray space 2013-04-22 21:18:35 -07:00
Noah Levitt 3d90506b99 use LinkedHashMap to ensure consistent ordering of xml api 2013-04-09 10:34:02 -07:00
Noah Levitt 08e18fd692 avoid creating unnecessary AlerthThreadGroups 2013-04-05 09:35:23 -07:00
Noah Levitt 475d07e8ca attempt to address HER-2036 occasional strange memory leak after crawl finishes - have ToePool discard its reference to CrawlController at crawl finish 2013-04-05 09:34:58 -07:00
Noah Levitt d07bf80e02 add link to engine page on job page 2013-04-03 23:50:23 -07:00
Noah Levitt 5a7cd8a6a8 disable codemirror search which add-on which seems to fight with browser search all the time 2013-04-03 17:16:04 -07:00
Noah Levitt fd668e82ba Merge branch 'master' of github.com:internetarchive/heritrix3 2013-04-03 16:39:45 -07:00
Noah Levitt 9f33b3b667 button on engine page to ask java to run garbage collector 2013-04-03 16:39:32 -07:00
Kenji Nagahashi c98c1d0e29 changed CrawlerLoggerModule.setupLogs() from private to protected so that sub-class can modify logging configuration. 2013-04-02 17:55:03 -07:00
Kenji Nagahashi 4dd46e6af8 quick fix to ClassCastException in ToePool.shortReportLineTo(). 
fixed a typo in Job.ftl.
 2013-04-01 23:45:22 -07:00
Noah Levitt 1ac1dd9bab fix typo which removed call to candidate.setFullVia(), should fix failing Precedence3SelfTest 2013-03-29 13:17:58 -07:00
Noah Levitt dcbbec4bbb fix for HER-2035 xml representation of job url can be invalid 2013-03-29 12:16:07 -07:00
Noah Levitt f988a26e10 move some code into EngineModel constructor and restore reverse chronological ordering of crawl jobs 2013-03-28 19:09:14 -07:00
Noah Levitt 6a2b4a4a76 Merge branch 'master' into freemarker 
Conflicts:
	.classpath
 2013-03-28 18:01:16 -07:00
Adam Miller 2f4d437734 Beans.ftl 
Fixed a bug that showed objects with no properties as null instead of displaying the class name
Job.ftl
	Fixed a bug that prevented the configuration referenced paths from displaying
Script.ftl
	Fixed a bug that rendered extra whitespace before and after the script block
 2013-03-15 16:06:06 -07:00
gojomo a844878e7d Merge branch 'simple_logins' into HER-2031 2013-03-12 01:16:31 -07:00
gojomo eb0760a37e refactor for runCandidatesChain; also run outCandidates 2013-03-11 20:50:18 -07:00
Adam Miller 0e3a2ec80a CrawlJobModel.java 
Adding check for application context
 2013-03-11 17:07:15 -07:00
Adam Miller b7b7953dc9 EngineResource.java 
Pulling individual values for necessary data from crawl jobs instead of building the full CrawlJobModel
CrawlJobModel.java
	Moving some data from methods into fields so they are exposed in the XML representation
EngineModel.java
	Removing full building of CrawlJobModel. Handling instead in EngineResource
Job.ftl
	Updating fields that relied on CrawlJobModel methods which were moved to fields.
 2013-03-11 17:07:15 -07:00
Adam Miller 03f4f381af JobResource.java 
Removing temporary template loading. Finishing converting XML and HTML generation to use the same data model
CrawlJobModel.java
	Converting to same Map based reports used by XML API
EngineModel.java
	Converting to same Map based reports used by XML API
Engine.ftl
	Creating template for Map based reports
Job.ftl
	Creating template for Map based reports
 2013-03-11 17:07:15 -07:00
Adam Miller ef7fd37a0a BeanBrowseResource.java 
Removing obsolete method: makePresentableMap()
 2013-03-11 17:07:15 -07:00
Adam Miller d26a712211 Fixed bad merge conflict 2013-03-11 17:07:15 -07:00
Adam Miller 60d00e3ef3 Converted data models to be usable for conversion to HTML and XML. Updated Models and freemarker templates to match XML naming 2013-03-11 17:05:18 -07:00
Adam Miller 50e49b6399 Removed commented out code. 2013-03-11 17:05:18 -07:00
Adam Miller 7426c78d73 Removed temporary template loading from file system. 
Removed unused method: getAvailableActions
 2013-03-11 17:05:18 -07:00
Adam Miller d545899fd2 Removed temporary template loading from file system 2013-03-11 17:05:18 -07:00
Adam Miller 929f3cb489 Scripts and Beans converted to freemarker 2013-03-11 17:05:17 -07:00
Adam Miller 783d48b49c Removing old commented out code 2013-03-11 16:58:28 -07:00
Adam Miller a32f00c2b3 Finishing /engine and /engine/job/{job} 2013-03-11 16:58:28 -07:00
Adam Miller e4ad60dda6 Beginning conversion to freemarker templates. /engine updated 2013-03-11 16:58:28 -07:00
Travis Wellman 56e6a0d504 ScriptResource actually uses the BeanLookupBindings 2013-01-24 10:56:16 -08:00
Travis Wellman 1883d81bc8 bindings on top of beans instead of vice versa 2013-01-23 18:27:40 -08:00