Commit Graph
840 Commits
Author SHA1 Message Date
Travis Wellman 07a2fc65ea use methods to specify which report rather than string constants
* MultiReporter
    deleted. direct method calls are a better way to specify the kind of report desired
    getReports was never used to choose from the array of strings. no use case for MultiReporter.
* ArchiveUtils
    writeReportToString was not used, and it used the now deleted MultiReporter#reportTo
* DevUtils, Frontier, ToePool, AbstractFrontier, WorkQueueFrontier
    MultiReporter -> Reporter and make methods that implemented named multireport public
* FrontierNonemptyReport, FrontierSummaryReport
    call the appropriate report method instead of passing a string constant
2012-06-01 17:57:25 -07:00
Travis Wellman 493e2423f6 begin removing MultiReporter
* MultiReporter.java
    deprecated
* ToePool, ToeThread, WorkQueue, WorkQueueFrontier, PrecedenceProvider, CrawlURI, FetchStats.java
    remove MultiReporter#getReports() and reportTo(String, PrintWriter)
    these are not used as MultiReporter specifies, and MultiReporter is going away anyhow
2012-06-01 17:14:41 -07:00
Noah Levitt 3bebf10674 * ScriptedDecideRule.java
getEngine() - tiny cosmetic fixups
2012-05-31 16:29:33 -07:00
Travis Wellman 392a4b3fd0 correct synchronization and threadlocal logic
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.

Now getEngine does what it says it does.
2012-05-31 16:09:42 -07:00
Noah Levitt 1015c5fa00 Fix HER-1973 Regression in queue budget overrides
* WorkQueueFrontier.java
    findEligibleURI() - apply sheet overrides before refreshing queue budgets
2012-05-15 17:35:49 -07:00
Noah Levitt 809981ef02 * PersistProcessor.java
populatePersistEnvFromLog() - handle exception deserializing line
2012-05-10 19:00:27 -07:00
Noah Levitt 9fffa8fc53 **/pom.xml
bump version to 3.1.2-SNAPSHOT
2012-05-01 18:17:33 -07:00
Noah Levitt 43d57dcaf4 * README.txt
update for 3.1.1
3.1.1
2012-05-01 18:05:38 -07:00
Noah Levitt 79c40346fc **/pom.xml
* bump version number to 3.1.1 for release
2012-05-01 18:01:06 -07:00
Noah Levitt a7c63b8d00 * FetchHTTP.java
remove redundant nonsensical comment
2012-05-01 17:53:52 -07:00
Gordon Mohr e4f12bb20f Merge pull request #6 from ato/codemirror
HER-2001: Use the CodeMirror editor for crawl config and script console
    (Thanks Alex!)
2012-04-25 15:04:10 -07:00
Noah Levitt 42e759c01d List correct defaults for BdbModule cacheSize and cachePercentage with an explanatory comment. 2012-04-25 11:10:56 -07:00
Noah Levitt a2880c4306 HER-1990 large robots.txt problematic
* Robotstxt.java
   enforce max size of robots.txt
2012-04-20 13:57:53 -07:00
Noah Levitt 52d5307a9b Merge branch 'her-1948' 2012-04-20 12:33:19 -07:00
Noah Levitt 8d0a1678de * effective_tld_names.dat
update to latest from publicsuffix.org
2012-04-20 12:25:57 -07:00
Noah Levitt 3fed078817 Merge pull request #9 from kngenie/publicsuffixes2
Publicsuffixes2
2012-04-20 12:19:47 -07:00
Noah Levitt da28258f90 Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948) 2012-04-19 18:16:53 -07:00
Noah Levitt 4a64c6ba59 Revert "Altered FetchStats to also keep track of the number of URIs that have"
This reverts commit 4c1740ce1e.
2012-04-18 15:53:49 -07:00
Noah Levitt 9551c774af Merge branch 'master' into http-status-deciderule 2012-04-18 15:47:24 -07:00
Noah Levitt 8e853c5f8d HER-2005 existing content length decide rule is not flexible enough
* ResponseContentLengthDecideRule.java
    new predicated decide rule that will ACCEPT or REJECT a uri if the content body is within a specified size range
2012-04-18 13:15:15 -07:00
Noah Levitt 0a4d56b638 Fix for HER-2004 SEVERE error in FetchWhois on some government domains
* FetchWhois.java
    addWhoisLinks() - handle IllegalStateException from InternetDomainName.topPrivateDomain()
2012-04-18 10:37:24 -07:00
Kenji Nagahashi 27d621f21a renamed PublicSuffixes2 to PublicSuffixes.
deleted debug println.
2012-04-18 10:14:51 -07:00
Kenji Nagahashi a0d52e6c17 PublicSuffixes2Test still had tests for PublicSuffix. fixed. 2012-04-18 10:14:51 -07:00
Kenji Nagahashi b2d62a81ff trial rewrite of PublicSuffixes for small memory footprint (no significant performance gain) 2012-04-18 10:14:50 -07:00
Noah Levitt 31df3a8f03 Merge branch 'master' into speculative-links 2012-04-13 18:16:08 -07:00
Travis Wellman 52eef84658 note about preselector.recheckScope for use of IpAddressSetDecideRule 2012-04-12 17:34:21 -07:00
Travis Wellman 54b1e245eb gitignore for maven build artifacts 2012-04-12 17:32:57 -07:00
Travis Wellman 563da14330 Merge pull request #7 from travisfw/IpAddressSetDecideRule
Ip address set decide rule
2012-04-12 16:52:00 -07:00
Travis Wellman 57d2f42954 formatting 2012-04-12 16:50:22 -07:00
Travis Wellman 64e9cc367b remove println, make class public 2012-04-12 16:46:16 -07:00
Travis Wellman 2a3945ff79 Merge branch 'master' of github.com:internetarchive/heritrix3 into IpAddressSetDecideRule 2012-04-02 18:17:44 -07:00
Travis Wellman f9f9e62f4c Reject or accept matching a set of IP addresses. 2012-04-02 18:16:23 -07:00
Alex Osborne f19a72efaf HER-2001: Use the CodeMirror editor for crawl config and script console
This enables syntax-highlighting, line-numbering and auto-indent for the crawl
configuration editor and the script console (for BeanShell, JavaScript and
Groovy).

* BaseResource.java
    add a more generic getStaticRef() in addition to getStylesheetRef()
* EditRepresentation.java
    wrap textarea with CodeMirror in purexml mode.
    disable the "save changes" button until config is edited.
    dynamically resize CodeMirror edit element to fit window height.
* ScriptResource.java
    wrap textarea with CodeMirror.
    switch CodeMirror language modes depending on selected script engine.
* resources/.../restlet/codemirror/
    include just the files from codemirror-2.23.zip necessary for the above
    plus the CodeMirror LICENSE file (MIT)
    plus a README linking to the codemirror website
2012-03-31 18:35:06 +11:00
Noah Levitt 9626b8454a * EditRepresentation.java
make edit page (for cxml mainly) prettier with big textarea (see Kenji's comment on HER-1993 dated 2012-03-27 18:03)
2012-03-30 11:44:28 -07:00
Noah Levitt 335b32aecc * ExtractorHTML.java
some fixup on RELEVANT_TAG_EXTRACTOR javadoc comment
2012-03-20 15:14:14 -07:00
Noah Levitt 4eab6461ed Merge remote-tracking branch 'adam-miller/master' 2012-03-20 15:06:58 -07:00
Adam Miller a108795d50 HER-1998 - Adding comments, adjusting test case to prevent subclasses from failing new tests 2012-03-19 18:19:00 -07:00
Noah Levitt ea42c0c653 * EngineResource.java
acceptRepresentation() - "I'm sure" parameter is now "im_sure"
2012-03-19 17:30:18 -07:00
Adam Miller 1a333fbd35 HER-1998 - ExtractorHTML modified to parse html inside conditional comments. Still ignores normal comments 2012-03-16 14:36:15 -07:00
Noah Levitt f7505e6673 More HTML cleanup. Html5 doctype declaration, get rid of <br> and
xml-style "/>" tag endings, other cleanup to make it validate.
2012-03-15 18:47:18 -07:00
Noah Levitt 52703c6eb4 * ArchiveUtils.java
doubleToString() - return "NaN" for NaN (inf symbol didn't quite make sense to me)
2012-03-15 18:46:01 -07:00
Noah Levitt 7cde3b4f17 Merge branch 'master' of github.com:internetarchive/heritrix3 into html-cleanup 2012-03-15 15:59:10 -07:00
Kenji Nagahashi 64f24d4c66 HER-1993: In JobResource, write error message instead of throwing RuntimeException when it failed to read
job log or crawl log.
2012-03-09 16:48:11 -08:00
Kenji Nagahashi f05287c8ca restored deleted space after header "Crawl Log" 2012-03-09 16:29:52 -08:00
Noah Levitt e7d75cbb5a * AbstractPersistProcessor.java, PersistProcessor.java, PersistStoreProcessor.java
refactor some common persist processor code into new class AbstractPersistProcessor
2012-03-06 16:20:58 -08:00
Kenji Nagahashi adc82c1c6f Second round of HTML cleanup:
- added missing <html> and <body>
- fixed non-working CSS link, added CSS link to all pages
- replaced layout by <br> and &nbsp; with structural tags with CSS
- removed superfluous output, notably in BeanBrowseResource
- <code>-ify variable names in ScriptResource
- added special handling of NaN in doubleToString()
2012-03-03 00:46:56 -08:00
Kenji Nagahashi 46c335c8a3 BeanBrowseResource: fixed malformed HTML. 2012-02-29 13:22:33 -08:00
Noah Levitt 1d05df7fbe * StatisticsTracker.java
writeReportFile() - reduce log level from warning to info when reusing existing report (normal after crawl is finished)
2012-02-29 11:02:39 -08:00
Noah Levitt 7af7cbf18c Merge branch 'master' of github.com:internetarchive/heritrix3 2012-02-10 15:51:15 -08:00
Noah Levitt db801b3e0f Fix problems handling crawl job paths with characters that require url-encoding, since we sometimes refer to them with file: urls
* PathSharingContext.java, CrawlJob.java
    use java.net.URI to convert paths to and from file: urls
2012-02-10 15:48:09 -08:00