Noah Levitt
60d93cbb5b
Remove obsolete <scm> section that pointed to old svn repository
2013-02-05 18:37:56 -08:00
Noah Levitt
878e3b1afe
Remove spurious "throws" declaration
2013-02-05 18:37:22 -08:00
Noah Levitt
bda50a56b1
Merge pull request #19 from travisfw/bean-bindings
...
BeanLookupBindings for simpler script access to beans
2013-01-24 13:59:29 -08:00
Noah Levitt
bf82ea3524
Rename to README.md so github treats it as markdown
2013-01-24 13:57:02 -08:00
Noah Levitt
c26e1c9732
Rename to README.md so github treats it as markdown
2013-01-24 13:56:20 -08:00
Noah Levitt
a99b2a26b4
Convert readme to markdown
2013-01-24 13:54:37 -08:00
Travis Wellman
56e6a0d504
ScriptResource actually uses the BeanLookupBindings
2013-01-24 10:56:16 -08:00
Travis Wellman
1883d81bc8
bindings on top of beans instead of vice versa
2013-01-23 18:27:40 -08:00
Travis Wellman
91aeb6d0bf
disable bean lookup by default
...
bean lookup may be enabled by setting a variable named beanBindings to
true in the Bindings or in the script.
2013-01-11 12:09:54 -08:00
Travis Wellman
03914a246c
BeanLookupBindings for simpler script access to beans
...
The new class BeanLookupBindings allows scripts to skip getBean calls.
Lines like 'beanname = appCtx.getBean("beanname")' can be left out.
Past scripts remain compatible.
This change effects action directory scripts and rest console scripts.
2013-01-10 18:54:33 -08:00
Noah Levitt
d27f67897e
Fix for HER-2029 Checkpoints recover error on windows - replace CLibrary.INSTANCE.link() with FilesystemLinkMaker.makeHardLink() in doRecover() - contributed by Andrés Aguilar
2013-01-09 18:28:05 -08:00
Noah Levitt
5af4ff657a
Avoid pulling in [non-archive-overlay] commons-httpclient from archive-commons dependency
2013-01-04 18:48:05 -08:00
Noah Levitt
c904cef241
Quiet down SSL warnings from restlet server by default
2013-01-04 10:56:20 -08:00
Noah Levitt
3a6ade1e09
More stuff moved to archive-commons
2013-01-03 17:44:30 -08:00
Noah Levitt
9d5986c078
Moving GZIPMembersInputStream from heritrix to archive-commons
2013-01-03 17:16:50 -08:00
Noah Levitt
543e8820ef
Merge branch 'master' into archive-commons-refactor
2012-12-20 02:44:57 -08:00
Noah Levitt
0de9869c0f
Tweaks to new UURIFactory
2012-12-20 02:42:16 -08:00
Noah Levitt
9168813ed7
Fix bugs in refactoring of UURI/UsableURI
2012-12-20 02:32:56 -08:00
Noah Levitt
3a65fa25d7
Working on refactoring some stuff into archive-commons
2012-12-18 14:46:52 -08:00
Noah Levitt
0eac126f7d
Special WARC-Profile for uri-agnostic revisit records. See HER-2022, http://tech.groups.yahoo.com/group/archive-crawler/message/7806
2012-12-13 14:11:14 -08:00
Noah Levitt
5586d5452d
* JerichoExtractorHTMLTest.java
...
avoid redundancy by using extractor built in setUp()
makeExtractor() - call extractor.setExtractorJS(new ExtractorJS()) since we got rid of the static ExtractorJS stuff
testConditionalComment1() - override to skip the test since it fails with JerichoExtractorHTML
2012-12-11 17:00:23 -08:00
Noah Levitt
85118c59c4
Tighten up javascript extraction based on real-world analysis (addresses HER-1523)
...
* UriUtils.java
new method isVeryLikelyUri() with tighter heuristic than isLikelyUri()
* ExtractorJS.java
use UriUtils.isVeryLikelyUri(), and change order of operations to do fixup before call to isVeryLikelyUri(), since it doesn't expect strings with javascript escaping and stuff
* StringExtractorTestBase.java
handle test data with expected value null, meaning no outlinks expected
* ExtractorHTMLTest.java
avoid redundancy by using the extractor created in ContentExtractorTestBase.setUp()
* ExtractorJSTest.java
some new tests
2012-12-11 16:27:26 -08:00
Noah Levitt
b1cd3d30a0
* ExtractorHTMLTest.java
...
makeExtractor() - call setExtractorJS(new ExtractorJS()) so testSpeculativeLinkExtraction() passes
2012-12-10 17:31:55 -08:00
Noah Levitt
256b16249a
* ExtractorJS.java
...
refactor so considerStrings() is not static, allowing it to be overridden in subclasses
* ExtractorHTML.java, ExtractorJS.java
add @Autowired parameter extractorJS used to process inline javascript, instead of call to static ExtractorJS.considerStrings()
2012-12-10 16:51:42 -08:00
Noah Levitt
b2a90ddc15
* UriUtils.java
...
speculativeFixup() - improve detection of scheme-less intended-absolute-URIs
2012-12-10 14:10:34 -08:00
Noah Levitt
d3dc153c5f
* ModuleTestBase.java
...
runTest() - convenience logging of test failures
2012-12-10 13:56:45 -08:00
Noah Levitt
e44cf9ca9b
A little cleanup of FetchHTTPTest refactoring
2012-12-04 18:43:15 -08:00
Noah Levitt
5354960f00
Refactor FetchHTTPTest to use a TestSuite so that starting and shutting down the test http servers happen only once at start and finish respectively
2012-12-04 17:25:09 -08:00
Noah Levitt
2c0dbed399
.classpath
...
include junit as a regular dependency not managed by eclipse, so source jar can be attached
2012-12-04 17:20:39 -08:00
Noah Levitt
e30cb83540
* FetchHTTPTest.java
...
testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix lets pass through to mimic browser behavior.
2012-12-04 16:25:09 -08:00
Noah Levitt
b981fd7423
* ArchiveReader.java
...
add "implements Closeable" since it already has the close() method
2012-12-03 17:09:22 -08:00
Noah Levitt
92b302f164
* ExtractorMultipleRegexTest.java
...
avoid "constant string too long" compile error
2012-11-16 19:08:18 -08:00
Noah Levitt
94df568b4f
* ExtractorMultipleRegex.java
...
remove temporary performance testing code
2012-11-16 18:34:28 -08:00
Noah Levitt
aa98ec3327
* ExtractorMultipleRegex.java
...
keep cache of groovy Template objects, since they are expensive to create
2012-11-16 18:30:41 -08:00
Noah Levitt
eeee01a52d
* ExtractorMultipleRegex.java
...
javadocs
2012-11-16 16:42:43 -08:00
Noah Levitt
89dfef468c
* ExtractorMultipleRegexTest.java
...
test passes now; extractor gets what it can, which is most of the scroll down urls
2012-11-15 19:18:59 -08:00
Noah Levitt
dfd7a8a7d3
* ExtractorMultipleRegexTest.java
...
turns out that __adt parameter can be found near the json blob - most, but not all, expected links are now found
2012-11-15 18:58:11 -08:00
Noah Levitt
22ea489c57
* ExtractorMultipleRegex.java
...
remove the "fooIndex" thing from available bindings, since it's kinda hacky and turned out not to be needed for our use case
2012-11-15 18:56:54 -08:00
Noah Levitt
4af58f38e5
* CrawlURI.java
...
outLinks, outCandidates - use LinkedHashSet to ensure predictable order (any reason not to do that?)
2012-11-15 18:55:37 -08:00
Noah Levitt
75280a13a8
* ExtractorMultipleRegexTest.java
...
working on making test work - first two scroll-down urls are extracted successfully, others fail
2012-11-15 17:27:41 -08:00
Noah Levitt
13e1f9d64d
* ExtractorMultipleRegex.java
...
fix omission from last commit, uriRegex
2012-11-15 14:01:01 -08:00
Noah Levitt
6ec4b5a973
* ExtractorMultipleRegex.java
...
more refactoring to avoid redundant operations
2012-11-15 13:55:11 -08:00
Noah Levitt
43dbcbfe81
* ExtractorMultipleRegex.java
...
refactor for readability
2012-11-15 13:35:16 -08:00
Noah Levitt
3f44a4bc66
Merge branch 'master' into MoreImpliedExtractor
2012-11-15 12:35:17 -08:00
Noah Levitt
8395e372e4
* ScriptedProcessor.java
...
getEngine() - fix logic to respect isolateThreads setting
2012-11-15 12:33:48 -08:00
Noah Levitt
ba5bca807e
* ExtractorMultipleRegex.java
...
use groovy templating facility
2012-11-15 12:32:42 -08:00
Noah Levitt
6e2d2129fe
* ExtractorMultipleRegexTest.java
...
use real use case actual content, actual urls from facebook for testing
* ExtractorMultipleRegex.java
some comments, little twiddles
2012-11-15 11:16:57 -08:00
Noah Levitt
1e2d86dc63
* ContentExtractorTestBase.java
...
createRecorder(String,String) - new method for creating test Recorder and specifying charset
createRecorder(String) - deprecate (use default charset as before)
2012-11-15 11:14:25 -08:00
Noah Levitt
1d994493e2
more progress on ExtractorMultipleRegex
2012-11-14 19:58:43 -08:00
Travis Wellman
1998337645
tmp commit
2012-11-14 17:58:44 -08:00
Noah Levitt