Noah Levitt
bf82ea3524
Rename to README.md so github treats it as markdown
2013-01-24 13:57:02 -08:00
Noah Levitt
c26e1c9732
Rename to README.md so github treats it as markdown
2013-01-24 13:56:20 -08:00
Noah Levitt
a99b2a26b4
Convert readme to markdown
2013-01-24 13:54:37 -08:00
Noah Levitt
d27f67897e
Fix for HER-2029 Checkpoints recover error on windows - replace CLibrary.INSTANCE.link() with FilesystemLinkMaker.makeHardLink() in doRecover() - contributed by Andrés Aguilar
2013-01-09 18:28:05 -08:00
Noah Levitt
5af4ff657a
Avoid pulling in [non-archive-overlay] commons-httpclient from archive-commons dependency
2013-01-04 18:48:05 -08:00
Noah Levitt
c904cef241
Quiet down SSL warnings from restlet server by default
2013-01-04 10:56:20 -08:00
Noah Levitt
3a6ade1e09
More stuff moved to archive-commons
2013-01-03 17:44:30 -08:00
Noah Levitt
9d5986c078
Moving GZIPMembersInputStream from heritrix to archive-commons
2013-01-03 17:16:50 -08:00
Noah Levitt
543e8820ef
Merge branch 'master' into archive-commons-refactor
2012-12-20 02:44:57 -08:00
Noah Levitt
0de9869c0f
Tweaks to new UURIFactory
2012-12-20 02:42:16 -08:00
Noah Levitt
9168813ed7
Fix bugs in refactoring of UURI/UsableURI
2012-12-20 02:32:56 -08:00
Noah Levitt
3a65fa25d7
Working on refactoring some stuff into archive-commons
2012-12-18 14:46:52 -08:00
Noah Levitt
0eac126f7d
Special WARC-Profile for uri-agnostic revisit records. See HER-2022, http://tech.groups.yahoo.com/group/archive-crawler/message/7806
2012-12-13 14:11:14 -08:00
Noah Levitt
5586d5452d
* JerichoExtractorHTMLTest.java
...
avoid redundancy by using extractor built in setUp()
makeExtractor() - call extractor.setExtractorJS(new ExtractorJS()) since we got rid of the static ExtractorJS stuff
testConditionalComment1() - override to skip the test since it fails with JerichoExtractorHTML
2012-12-11 17:00:23 -08:00
Noah Levitt
85118c59c4
Tighten up javascript extraction based on real-world analysis (addresses HER-1523)
...
* UriUtils.java
new method isVeryLikelyUri() with tighter heuristic than isLikelyUri()
* ExtractorJS.java
use UriUtils.isVeryLikelyUri(), and change order of operations to do fixup before call to isVeryLikelyUri(), since it doesn't expect strings with javascript escaping and stuff
* StringExtractorTestBase.java
handle test data with expected value null, meaning no outlinks expected
* ExtractorHTMLTest.java
avoid redundancy by using the extractor created in ContentExtractorTestBase.setUp()
* ExtractorJSTest.java
some new tests
2012-12-11 16:27:26 -08:00
Noah Levitt
b1cd3d30a0
* ExtractorHTMLTest.java
...
makeExtractor() - call setExtractorJS(new ExtractorJS()) so testSpeculativeLinkExtraction() passes
2012-12-10 17:31:55 -08:00
Noah Levitt
256b16249a
* ExtractorJS.java
...
refactor so considerStrings() is not static, allowing it to be overridden in subclasses
* ExtractorHTML.java, ExtractorJS.java
add @Autowired parameter extractorJS used to process inline javascript, instead of call to static ExtractorJS.considerStrings()
2012-12-10 16:51:42 -08:00
Noah Levitt
b2a90ddc15
* UriUtils.java
...
speculativeFixup() - improve detection of scheme-less intended-absolute-URIs
2012-12-10 14:10:34 -08:00
Noah Levitt
d3dc153c5f
* ModuleTestBase.java
...
runTest() - convenience logging of test failures
2012-12-10 13:56:45 -08:00
Noah Levitt
e44cf9ca9b
A little cleanup of FetchHTTPTest refactoring
2012-12-04 18:43:15 -08:00
Noah Levitt
5354960f00
Refactor FetchHTTPTest to use a TestSuite so that starting and shutting down the test http servers happen only once at start and finish respectively
2012-12-04 17:25:09 -08:00
Noah Levitt
2c0dbed399
.classpath
...
include junit as a regular dependency not managed by eclipse, so source jar can be attached
2012-12-04 17:20:39 -08:00
Noah Levitt
e30cb83540
* FetchHTTPTest.java
...
testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix lets pass through to mimic browser behavior.
2012-12-04 16:25:09 -08:00
Noah Levitt
b981fd7423
* ArchiveReader.java
...
add "implements Closeable" since it already has the close() method
2012-12-03 17:09:22 -08:00
Noah Levitt
92b302f164
* ExtractorMultipleRegexTest.java
...
avoid "constant string too long" compile error
2012-11-16 19:08:18 -08:00
Noah Levitt
94df568b4f
* ExtractorMultipleRegex.java
...
remove temporary performance testing code
2012-11-16 18:34:28 -08:00
Noah Levitt
aa98ec3327
* ExtractorMultipleRegex.java
...
keep cache of groovy Template objects, since they are expensive to create
2012-11-16 18:30:41 -08:00
Noah Levitt
eeee01a52d
* ExtractorMultipleRegex.java
...
javadocs
2012-11-16 16:42:43 -08:00
Noah Levitt
89dfef468c
* ExtractorMultipleRegexTest.java
...
test passes now; extractor gets what it can, which is most of the scroll down urls
2012-11-15 19:18:59 -08:00
Noah Levitt
dfd7a8a7d3
* ExtractorMultipleRegexTest.java
...
turns out that __adt parameter can be found near the json blob - most, but not all, expected links are now found
2012-11-15 18:58:11 -08:00
Noah Levitt
22ea489c57
* ExtractorMultipleRegex.java
...
remove the "fooIndex" thing from available bindings, since it's kinda hacky and turned out not to be needed for our use case
2012-11-15 18:56:54 -08:00
Noah Levitt
4af58f38e5
* CrawlURI.java
...
outLinks, outCandidates - use LinkedHashSet to ensure predictable order (any reason not to do that?)
2012-11-15 18:55:37 -08:00
Noah Levitt
75280a13a8
* ExtractorMultipleRegexTest.java
...
working on making test work - first two scroll-down urls are extracted successfully, others fail
2012-11-15 17:27:41 -08:00
Noah Levitt
13e1f9d64d
* ExtractorMultipleRegex.java
...
fix omission from last commit, uriRegex
2012-11-15 14:01:01 -08:00
Noah Levitt
6ec4b5a973
* ExtractorMultipleRegex.java
...
more refactoring to avoid redundant operations
2012-11-15 13:55:11 -08:00
Noah Levitt
43dbcbfe81
* ExtractorMultipleRegex.java
...
refactor for readability
2012-11-15 13:35:16 -08:00
Noah Levitt
3f44a4bc66
Merge branch 'master' into MoreImpliedExtractor
2012-11-15 12:35:17 -08:00
Noah Levitt
8395e372e4
* ScriptedProcessor.java
...
getEngine() - fix logic to respect isolateThreads setting
2012-11-15 12:33:48 -08:00
Noah Levitt
ba5bca807e
* ExtractorMultipleRegex.java
...
use groovy templating facility
2012-11-15 12:32:42 -08:00
Noah Levitt
6e2d2129fe
* ExtractorMultipleRegexTest.java
...
use real use case actual content, actual urls from facebook for testing
* ExtractorMultipleRegex.java
some comments, little twiddles
2012-11-15 11:16:57 -08:00
Noah Levitt
1e2d86dc63
* ContentExtractorTestBase.java
...
createRecorder(String,String) - new method for creating test Recorder and specifying charset
createRecorder(String) - deprecate (use default charset as before)
2012-11-15 11:14:25 -08:00
Noah Levitt
1d994493e2
more progress on ExtractorMultipleRegex
2012-11-14 19:58:43 -08:00
Travis Wellman
1998337645
tmp commit
2012-11-14 17:58:44 -08:00
gojomo
ed65493db0
alternate testing of multiple local binds more likely to work more places
2012-10-01 16:38:17 -07:00
gojomo
b4baea0a13
make maven requested version match eclipse project classpath
2012-10-01 16:37:31 -07:00
gojomo
37e6ab802a
Merge branch 'master' of github.com:internetarchive/heritrix3
2012-10-01 13:41:12 -07:00
Noah Levitt
4470ba512e
Count url-agnostic content digest duplicates as duplicates in statistics
...
* CrawledBytesHistotable.java
accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates
* ContentDigestHistoryTest.java
testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation
2012-09-27 16:56:14 -07:00
Noah Levitt
a1d135edbc
* WARCWriterProcessor.java
...
include warcFileOffset in CrawlURI extraInfo (optionally logged in crawl log if logExtraInfo is enabled)
2012-09-27 16:51:50 -07:00
Noah Levitt
e2a1e3bb53
* ContentDigestHistoryTest.java
...
testWarcDedupe() - open the written with with WARCReader, check that each record is as expected
2012-09-27 14:34:20 -07:00
Noah Levitt
a1a5fc7379
* WARCWriterProcessor.java
...
writeRevisitUriAgnosticDigest() - put angle brackets around WARC-Refers-To uri value, since warc record ids appear everywhere with those brackets
2012-09-27 14:20:00 -07:00
Noah Levitt