Noah Levitt
75bb19c4ea
switch .classpath from archive-commons to ia-web-commons
2013-03-19 11:25:13 -07:00
Noah Levitt
9fa198944a
depend on new "ia-web-commons" instead of "archive-commons"
2013-03-18 23:32:05 -07:00
Noah Levitt
b0c6619f80
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-03-18 20:02:03 -07:00
Noah Levitt
0c9e2a1f4d
ContentDigestHistory{Loader,Storer} - ignore uris with empty content body (deduping that is counterproductive)
2013-03-18 20:01:52 -07:00
Noah Levitt
ff36546eda
fix serverless whois url example in comment
2013-03-18 14:03:39 -07:00
Noah Levitt
8cbb84b8df
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-02-22 15:43:30 -08:00
Noah Levitt
fb758e53e3
Avoid pulling in hadoop-core.jar and its dependencies; add plugin versions to get rid of warnings
2013-02-22 15:43:15 -08:00
Noah Levitt
0841840baf
Fix for HER-2032 - test stream for compression; add tests with and without compression
2013-02-05 18:45:14 -08:00
Noah Levitt
60d93cbb5b
Remove obsolete <scm> section that pointed to old svn repository
2013-02-05 18:37:56 -08:00
Noah Levitt
878e3b1afe
Remove spurious "throws" declaration
2013-02-05 18:37:22 -08:00
Noah Levitt
bda50a56b1
Merge pull request #19 from travisfw/bean-bindings
...
BeanLookupBindings for simpler script access to beans
2013-01-24 13:59:29 -08:00
Noah Levitt
bf82ea3524
Rename to README.md so github treats it as markdown
2013-01-24 13:57:02 -08:00
Noah Levitt
c26e1c9732
Rename to README.md so github treats it as markdown
2013-01-24 13:56:20 -08:00
Noah Levitt
a99b2a26b4
Convert readme to markdown
2013-01-24 13:54:37 -08:00
Travis Wellman
56e6a0d504
ScriptResource actually uses the BeanLookupBindings
2013-01-24 10:56:16 -08:00
Travis Wellman
1883d81bc8
bindings on top of beans instead of vice versa
2013-01-23 18:27:40 -08:00
Travis Wellman
91aeb6d0bf
disable bean lookup by default
...
bean lookup may be enabled by setting a variable named beanBindings to
true in the Bindings or in the script.
2013-01-11 12:09:54 -08:00
Travis Wellman
03914a246c
BeanLookupBindings for simpler script access to beans
...
The new class BeanLookupBindings allows scripts to skip getBean calls.
Lines like 'beanname = appCtx.getBean("beanname")' can be left out.
Past scripts remain compatible.
This change effects action directory scripts and rest console scripts.
2013-01-10 18:54:33 -08:00
Noah Levitt
d27f67897e
Fix for HER-2029 Checkpoints recover error on windows - replace CLibrary.INSTANCE.link() with FilesystemLinkMaker.makeHardLink() in doRecover() - contributed by Andrés Aguilar
2013-01-09 18:28:05 -08:00
Noah Levitt
5af4ff657a
Avoid pulling in [non-archive-overlay] commons-httpclient from archive-commons dependency
2013-01-04 18:48:05 -08:00
Noah Levitt
c904cef241
Quiet down SSL warnings from restlet server by default
2013-01-04 10:56:20 -08:00
Noah Levitt
3a6ade1e09
More stuff moved to archive-commons
2013-01-03 17:44:30 -08:00
Noah Levitt
9d5986c078
Moving GZIPMembersInputStream from heritrix to archive-commons
2013-01-03 17:16:50 -08:00
Noah Levitt
543e8820ef
Merge branch 'master' into archive-commons-refactor
2012-12-20 02:44:57 -08:00
Noah Levitt
0de9869c0f
Tweaks to new UURIFactory
2012-12-20 02:42:16 -08:00
Noah Levitt
9168813ed7
Fix bugs in refactoring of UURI/UsableURI
2012-12-20 02:32:56 -08:00
Noah Levitt
3a65fa25d7
Working on refactoring some stuff into archive-commons
2012-12-18 14:46:52 -08:00
Noah Levitt
0eac126f7d
Special WARC-Profile for uri-agnostic revisit records. See HER-2022, http://tech.groups.yahoo.com/group/archive-crawler/message/7806
2012-12-13 14:11:14 -08:00
Noah Levitt
5586d5452d
* JerichoExtractorHTMLTest.java
...
avoid redundancy by using extractor built in setUp()
makeExtractor() - call extractor.setExtractorJS(new ExtractorJS()) since we got rid of the static ExtractorJS stuff
testConditionalComment1() - override to skip the test since it fails with JerichoExtractorHTML
2012-12-11 17:00:23 -08:00
Noah Levitt
85118c59c4
Tighten up javascript extraction based on real-world analysis (addresses HER-1523)
...
* UriUtils.java
new method isVeryLikelyUri() with tighter heuristic than isLikelyUri()
* ExtractorJS.java
use UriUtils.isVeryLikelyUri(), and change order of operations to do fixup before call to isVeryLikelyUri(), since it doesn't expect strings with javascript escaping and stuff
* StringExtractorTestBase.java
handle test data with expected value null, meaning no outlinks expected
* ExtractorHTMLTest.java
avoid redundancy by using the extractor created in ContentExtractorTestBase.setUp()
* ExtractorJSTest.java
some new tests
2012-12-11 16:27:26 -08:00
Noah Levitt
b1cd3d30a0
* ExtractorHTMLTest.java
...
makeExtractor() - call setExtractorJS(new ExtractorJS()) so testSpeculativeLinkExtraction() passes
2012-12-10 17:31:55 -08:00
Noah Levitt
256b16249a
* ExtractorJS.java
...
refactor so considerStrings() is not static, allowing it to be overridden in subclasses
* ExtractorHTML.java, ExtractorJS.java
add @Autowired parameter extractorJS used to process inline javascript, instead of call to static ExtractorJS.considerStrings()
2012-12-10 16:51:42 -08:00
Noah Levitt
b2a90ddc15
* UriUtils.java
...
speculativeFixup() - improve detection of scheme-less intended-absolute-URIs
2012-12-10 14:10:34 -08:00
Noah Levitt
d3dc153c5f
* ModuleTestBase.java
...
runTest() - convenience logging of test failures
2012-12-10 13:56:45 -08:00
Noah Levitt
e44cf9ca9b
A little cleanup of FetchHTTPTest refactoring
2012-12-04 18:43:15 -08:00
Noah Levitt
5354960f00
Refactor FetchHTTPTest to use a TestSuite so that starting and shutting down the test http servers happen only once at start and finish respectively
2012-12-04 17:25:09 -08:00
Noah Levitt
2c0dbed399
.classpath
...
include junit as a regular dependency not managed by eclipse, so source jar can be attached
2012-12-04 17:20:39 -08:00
Noah Levitt
e30cb83540
* FetchHTTPTest.java
...
testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix lets pass through to mimic browser behavior.
2012-12-04 16:25:09 -08:00
Noah Levitt
b981fd7423
* ArchiveReader.java
...
add "implements Closeable" since it already has the close() method
2012-12-03 17:09:22 -08:00
Noah Levitt
92b302f164
* ExtractorMultipleRegexTest.java
...
avoid "constant string too long" compile error
2012-11-16 19:08:18 -08:00
Noah Levitt
94df568b4f
* ExtractorMultipleRegex.java
...
remove temporary performance testing code
2012-11-16 18:34:28 -08:00
Noah Levitt
aa98ec3327
* ExtractorMultipleRegex.java
...
keep cache of groovy Template objects, since they are expensive to create
2012-11-16 18:30:41 -08:00
Noah Levitt
eeee01a52d
* ExtractorMultipleRegex.java
...
javadocs
2012-11-16 16:42:43 -08:00
Noah Levitt
89dfef468c
* ExtractorMultipleRegexTest.java
...
test passes now; extractor gets what it can, which is most of the scroll down urls
2012-11-15 19:18:59 -08:00
Noah Levitt
dfd7a8a7d3
* ExtractorMultipleRegexTest.java
...
turns out that __adt parameter can be found near the json blob - most, but not all, expected links are now found
2012-11-15 18:58:11 -08:00
Noah Levitt
22ea489c57
* ExtractorMultipleRegex.java
...
remove the "fooIndex" thing from available bindings, since it's kinda hacky and turned out not to be needed for our use case
2012-11-15 18:56:54 -08:00
Noah Levitt
4af58f38e5
* CrawlURI.java
...
outLinks, outCandidates - use LinkedHashSet to ensure predictable order (any reason not to do that?)
2012-11-15 18:55:37 -08:00
Noah Levitt
75280a13a8
* ExtractorMultipleRegexTest.java
...
working on making test work - first two scroll-down urls are extracted successfully, others fail
2012-11-15 17:27:41 -08:00
Noah Levitt
13e1f9d64d
* ExtractorMultipleRegex.java
...
fix omission from last commit, uriRegex
2012-11-15 14:01:01 -08:00
Noah Levitt
6ec4b5a973
* ExtractorMultipleRegex.java
...
more refactoring to avoid redundant operations
2012-11-15 13:55:11 -08:00
Noah Levitt