358afd4486
revised scripting console xml response fix based on comments.
...
removed scripting console actions from ScriptModel into ScriptingConsole
class (resurrection of ScriptExec).
added basic ScriptingConsoleTest.
2013-06-10 14:17:27 -07:00
c53b821c85
FIX: 73f58dc56a broke XML variant of scripting console response, and will
...
produce incompatible structure (extra nesting with scriptExec element).
moved ScriptExecution code to ScriptModel, which is now a bean, not Map.
XmlMarshaller is extended to support simple JavaBeans to XML marshalling,
borrowing annotation classes from javax.xml.
2013-05-28 13:37:04 -07:00
7f2aacba2c
BeanLookupBindings: check appCtx != null explicitly, rather than using
...
assert.
2013-05-28 13:20:23 -07:00
73888819dc
fix bug where scripting console no longer retains your script when you hit execute
2013-05-15 14:23:53 -07:00
5b4d8c7f0d
use intended File.separator instead of File.pathSeparator
2013-05-13 22:06:09 -07:00
3841d096d5
removed commented-out old code.
2013-05-11 15:44:25 -07:00
73f58dc56a
scripting console:
...
fix bug: exception from script execution is not displayed.
remove extra space around exception and rawOutput.
involves a bit more than bug fix. introduced ScriptExecution class for
encapsulating script, execution, and result.
2013-05-11 15:44:25 -07:00
e531de0ed3
avoid polluting logs with long stacktraces when a request comes in for a report from an unbuilt job (a fairly common case in archive-it)
2013-04-29 19:02:22 -07:00
dafd95eef7
fix for HER-2037 final crawl reports written after crawl is "FINISHED" - call dumpReports() in StatisticsTracker.stop() which (since the HER-1954 changes) runs at the right time, after all url processing has finished but before the FINISHED state is officially reached
2013-04-22 21:23:43 -07:00
3d90506b99
use LinkedHashMap to ensure consistent ordering of xml api
2013-04-09 10:34:02 -07:00
08e18fd692
avoid creating unnecessary AlerthThreadGroups
2013-04-05 09:35:23 -07:00
475d07e8ca
attempt to address HER-2036 occasional strange memory leak after crawl finishes - have ToePool discard its reference to CrawlController at crawl finish
2013-04-05 09:34:58 -07:00
5a7cd8a6a8
disable codemirror search which add-on which seems to fight with browser search all the time
2013-04-03 17:16:04 -07:00
fd668e82ba
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-04-03 16:39:45 -07:00
9f33b3b667
button on engine page to ask java to run garbage collector
2013-04-03 16:39:32 -07:00
c98c1d0e29
changed CrawlerLoggerModule.setupLogs() from private to protected so that sub-class can modify logging configuration.
2013-04-02 17:55:03 -07:00
4dd46e6af8
quick fix to ClassCastException in ToePool.shortReportLineTo().
...
fixed a typo in Job.ftl.
2013-04-01 23:45:22 -07:00
1ac1dd9bab
fix typo which removed call to candidate.setFullVia(), should fix failing Precedence3SelfTest
2013-03-29 13:17:58 -07:00
dcbbec4bbb
fix for HER-2035 xml representation of job url can be invalid
2013-03-29 12:16:07 -07:00
f988a26e10
move some code into EngineModel constructor and restore reverse chronological ordering of crawl jobs
2013-03-28 19:09:14 -07:00
6a2b4a4a76
Merge branch 'master' into freemarker
...
Conflicts:
.classpath
2013-03-28 18:01:16 -07:00
a844878e7d
Merge branch 'simple_logins' into HER-2031
2013-03-12 01:16:31 -07:00
eb0760a37e
refactor for runCandidatesChain; also run outCandidates
2013-03-11 20:50:18 -07:00
0e3a2ec80a
CrawlJobModel.java
...
Adding check for application context
2013-03-11 17:07:15 -07:00
b7b7953dc9
EngineResource.java
...
Pulling individual values for necessary data from crawl jobs instead of building the full CrawlJobModel
CrawlJobModel.java
Moving some data from methods into fields so they are exposed in the XML representation
EngineModel.java
Removing full building of CrawlJobModel. Handling instead in EngineResource
Job.ftl
Updating fields that relied on CrawlJobModel methods which were moved to fields.
2013-03-11 17:07:15 -07:00
03f4f381af
JobResource.java
...
Removing temporary template loading. Finishing converting XML and HTML generation to use the same data model
CrawlJobModel.java
Converting to same Map based reports used by XML API
EngineModel.java
Converting to same Map based reports used by XML API
Engine.ftl
Creating template for Map based reports
Job.ftl
Creating template for Map based reports
2013-03-11 17:07:15 -07:00
ef7fd37a0a
BeanBrowseResource.java
...
Removing obsolete method: makePresentableMap()
2013-03-11 17:07:15 -07:00
d26a712211
Fixed bad merge conflict
2013-03-11 17:07:15 -07:00
60d00e3ef3
Converted data models to be usable for conversion to HTML and XML. Updated Models and freemarker templates to match XML naming
2013-03-11 17:05:18 -07:00
50e49b6399
Removed commented out code.
2013-03-11 17:05:18 -07:00
7426c78d73
Removed temporary template loading from file system.
...
Removed unused method: getAvailableActions
2013-03-11 17:05:18 -07:00
d545899fd2
Removed temporary template loading from file system
2013-03-11 17:05:18 -07:00
929f3cb489
Scripts and Beans converted to freemarker
2013-03-11 17:05:17 -07:00
783d48b49c
Removing old commented out code
2013-03-11 16:58:28 -07:00
a32f00c2b3
Finishing /engine and /engine/job/{job}
2013-03-11 16:58:28 -07:00
e4ad60dda6
Beginning conversion to freemarker templates. /engine updated
2013-03-11 16:58:28 -07:00
56e6a0d504
ScriptResource actually uses the BeanLookupBindings
2013-01-24 10:56:16 -08:00
1883d81bc8
bindings on top of beans instead of vice versa
2013-01-23 18:27:40 -08:00
91aeb6d0bf
disable bean lookup by default
...
bean lookup may be enabled by setting a variable named beanBindings to
true in the Bindings or in the script.
2013-01-11 12:09:54 -08:00
03914a246c
BeanLookupBindings for simpler script access to beans
...
The new class BeanLookupBindings allows scripts to skip getBean calls.
Lines like 'beanname = appCtx.getBean("beanname")' can be left out.
Past scripts remain compatible.
This change effects action directory scripts and rest console scripts.
2013-01-10 18:54:33 -08:00
3a65fa25d7
Working on refactoring some stuff into archive-commons
2012-12-18 14:46:52 -08:00
4470ba512e
Count url-agnostic content digest duplicates as duplicates in statistics
...
* CrawledBytesHistotable.java
accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
2012-09-27 16:56:14 -07:00
e711ff9506
* JobRelatedResource.java
...
fix code formatting from 01be6d4b
2012-09-04 18:03:38 -07:00
01be6d4b18
Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans
...
* JobRelatedResource.java
Added encoding to URL generation in makePresentableMapFor().
Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
2012-08-31 15:07:31 -07:00
52e70a10ef
Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans"
...
* BeanBrowseResource.java
replaced "script" string in root node with "beans"
2012-08-30 15:35:27 -07:00
c5d740a6d4
remove some dead or redundant code
...
* ArchiveUtils
sb.length() was tested twice and int i was never used
* PrefixFinder
opCount is never used
* BdbMultipleWorkQueues
tries is never used
2012-07-11 16:57:13 -07:00
c1a0525107
suppress unused warnings for serialVersionUid
...
Toward the goal of having Eclipse warnings be useful to look at, suppressing
unused warnings for serialVersionUid will significantly reduce clutter.
2012-07-10 14:12:33 -07:00
6626b44286
generics warnings fixes
...
This commit does not change any functionality, just
alters generics to reduce warnings in Eclipse.
2012-06-21 18:23:06 -07:00
35378df2fa
Merge branch 'master' into no-package-scope
2012-06-05 17:58:40 -07:00
3dce1e909d
Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.)
2012-06-05 17:55:41 -07:00
Kenji Nagahashi