Commit Graph
846 Commits
Author SHA1 Message Date
Noah Levitt 99acaa9bfa Fix tests revealed to be failing
* UriUtils.java
    isLikelyFalsePositive() - do not flag urls starting with ../ as likely false positives
* UriUtilsTest.java
    remove test url that is now rejected
2012-06-06 09:00:13 -07:00
Noah Levitt 342ea33be5 Move test classes into test area of source tree
* LongToIntConsistentHashTest.java, UriUtilsTest.java
2012-06-05 18:03:33 -07:00
Noah Levitt 35378df2fa Merge branch 'master' into no-package-scope 2012-06-05 17:58:40 -07:00
Noah Levitt 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) 2012-06-05 17:55:41 -07:00
Noah Levitt e965421281 * WorkQueueFrontier.java
remove unused variable
2012-06-05 16:26:00 -07:00
Travis Wellman ee1e25ca69 Merge pull request #11 from travisfw/new-reports
deleting MultiReporter interface
2012-06-05 16:24:43 -07:00
Travis Wellman 07a2fc65ea use methods to specify which report rather than string constants
* MultiReporter
    deleted. direct method calls are a better way to specify the kind of report desired
    getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
    writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
    MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
    call the appropriate report method instead of passing a string constant
2012-06-01 17:57:25 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt 3bebf10674 * ScriptedDecideRule.java
getEngine() - tiny cosmetic fixups
2012-05-31 16:29:33 -07:00
Travis Wellman 392a4b3fd0 correct synchronization and threadlocal logic
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.

Now getEngine does what it says it does.
2012-05-31 16:09:42 -07:00
Noah Levitt 1015c5fa00 Fix HER-1973 Regression in queue budget overrides
* WorkQueueFrontier.java
    findEligibleURI() - apply sheet overrides before refreshing queue budgets
2012-05-15 17:35:49 -07:00
Noah Levitt 809981ef02 * PersistProcessor.java
populatePersistEnvFromLog() - handle exception deserializing line
2012-05-10 19:00:27 -07:00
Noah Levitt 9fffa8fc53 **/pom.xml
bump version to 3.1.2-SNAPSHOT
2012-05-01 18:17:33 -07:00
Noah Levitt 43d57dcaf4 * README.txt
update for 3.1.1
3.1.1
2012-05-01 18:05:38 -07:00
Noah Levitt 79c40346fc **/pom.xml
* bump version number to 3.1.1 for release
2012-05-01 18:01:06 -07:00
Noah Levitt a7c63b8d00 * FetchHTTP.java
remove redundant nonsensical comment
2012-05-01 17:53:52 -07:00
Gordon Mohr e4f12bb20f Merge pull request #6 from ato/codemirror
HER-2001: Use the CodeMirror editor for crawl config and script console
    (Thanks Alex!)
2012-04-25 15:04:10 -07:00
Noah Levitt 42e759c01d List correct defaults for BdbModule cacheSize and cachePercentage with an explanatory comment. 2012-04-25 11:10:56 -07:00
Noah Levitt a2880c4306 HER-1990 large robots.txt problematic
* Robotstxt.java
   enforce max size of robots.txt
2012-04-20 13:57:53 -07:00
Noah Levitt 52d5307a9b Merge branch 'her-1948' 2012-04-20 12:33:19 -07:00
Noah Levitt 8d0a1678de * effective_tld_names.dat
update to latest from publicsuffix.org
2012-04-20 12:25:57 -07:00
Noah Levitt 3fed078817 Merge pull request #9 from kngenie/publicsuffixes2
Publicsuffixes2
2012-04-20 12:19:47 -07:00
Noah Levitt da28258f90 Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948) 2012-04-19 18:16:53 -07:00
Noah Levitt 4a64c6ba59 Revert "Altered FetchStats to also keep track of the number of URIs that have"
This reverts commit 4c1740ce1e.
2012-04-18 15:53:49 -07:00
Noah Levitt 9551c774af Merge branch 'master' into http-status-deciderule 2012-04-18 15:47:24 -07:00
Noah Levitt 8e853c5f8d HER-2005 existing content length decide rule is not flexible enough
* ResponseContentLengthDecideRule.java
    new predicated decide rule that will ACCEPT or REJECT a uri if the content body is within a specified size range
2012-04-18 13:15:15 -07:00
Noah Levitt 0a4d56b638 Fix for HER-2004 SEVERE error in FetchWhois on some government domains
* FetchWhois.java
    addWhoisLinks() - handle IllegalStateException from InternetDomainName.topPrivateDomain()
2012-04-18 10:37:24 -07:00
Kenji Nagahashi 27d621f21a renamed PublicSuffixes2 to PublicSuffixes.
deleted debug println.
2012-04-18 10:14:51 -07:00
Kenji Nagahashi a0d52e6c17 PublicSuffixes2Test still had tests for PublicSuffix. fixed. 2012-04-18 10:14:51 -07:00
Kenji Nagahashi b2d62a81ff trial rewrite of PublicSuffixes for small memory footprint (no significant performance gain) 2012-04-18 10:14:50 -07:00
Noah Levitt 31df3a8f03 Merge branch 'master' into speculative-links 2012-04-13 18:16:08 -07:00
Travis Wellman 52eef84658 note about preselector.recheckScope for use of IpAddressSetDecideRule 2012-04-12 17:34:21 -07:00
Travis Wellman 54b1e245eb gitignore for maven build artifacts 2012-04-12 17:32:57 -07:00
Travis Wellman 563da14330 Merge pull request #7 from travisfw/IpAddressSetDecideRule
Ip address set decide rule
2012-04-12 16:52:00 -07:00
Travis Wellman 57d2f42954 formatting 2012-04-12 16:50:22 -07:00
Travis Wellman 64e9cc367b remove println, make class public 2012-04-12 16:46:16 -07:00
Travis Wellman 2a3945ff79 Merge branch 'master' of github.com:internetarchive/heritrix3 into IpAddressSetDecideRule 2012-04-02 18:17:44 -07:00
Travis Wellman f9f9e62f4c Reject or accept matching a set of IP addresses. 2012-04-02 18:16:23 -07:00
Alex Osborne f19a72efaf HER-2001: Use the CodeMirror editor for crawl config and script console
This enables syntax-highlighting, line-numbering and auto-indent for the crawl
configuration editor and the script console (for BeanShell, JavaScript and
Groovy).

* BaseResource.java
    add a more generic getStaticRef() in addition to getStylesheetRef()
* EditRepresentation.java
    wrap textarea with CodeMirror in purexml mode.
    disable the "save changes" button until config is edited.
    dynamically resize CodeMirror edit element to fit window height.
* ScriptResource.java
    wrap textarea with CodeMirror.
    switch CodeMirror language modes depending on selected script engine.
* resources/.../restlet/codemirror/
    include just the files from codemirror-2.23.zip necessary for the above
    plus the CodeMirror LICENSE file (MIT)
    plus a README linking to the codemirror website
2012-03-31 18:35:06 +11:00
Noah Levitt 9626b8454a * EditRepresentation.java
make edit page (for cxml mainly) prettier with big textarea (see Kenji's comment on HER-1993 dated 2012-03-27 18:03)
2012-03-30 11:44:28 -07:00
Noah Levitt 335b32aecc * ExtractorHTML.java
some fixup on RELEVANT_TAG_EXTRACTOR javadoc comment
2012-03-20 15:14:14 -07:00
Noah Levitt 4eab6461ed Merge remote-tracking branch 'adam-miller/master' 2012-03-20 15:06:58 -07:00
Adam Miller a108795d50 HER-1998 - Adding comments, adjusting test case to prevent subclasses from failing new tests 2012-03-19 18:19:00 -07:00
Noah Levitt ea42c0c653 * EngineResource.java
acceptRepresentation() - "I'm sure" parameter is now "im_sure"
2012-03-19 17:30:18 -07:00
Adam Miller 1a333fbd35 HER-1998 - ExtractorHTML modified to parse html inside conditional comments. Still ignores normal comments 2012-03-16 14:36:15 -07:00
Noah Levitt f7505e6673 More HTML cleanup. Html5 doctype declaration, get rid of <br> and
xml-style "/>" tag endings, other cleanup to make it validate.
2012-03-15 18:47:18 -07:00
Noah Levitt 52703c6eb4 * ArchiveUtils.java
doubleToString() - return "NaN" for NaN (inf symbol didn't quite make sense to me)
2012-03-15 18:46:01 -07:00
Noah Levitt 7cde3b4f17 Merge branch 'master' of github.com:internetarchive/heritrix3 into html-cleanup 2012-03-15 15:59:10 -07:00
Kenji Nagahashi 64f24d4c66 HER-1993: In JobResource, write error message instead of throwing RuntimeException when it failed to read
job log or crawl log.
2012-03-09 16:48:11 -08:00
Kenji Nagahashi f05287c8ca restored deleted space after header "Crawl Log" 2012-03-09 16:29:52 -08:00