Noah Levitt
3dce1e909d
Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.)
2012-06-05 17:55:41 -07:00
Noah Levitt
e965421281
* WorkQueueFrontier.java
...
remove unused variable
2012-06-05 16:26:00 -07:00
Noah Levitt
3bebf10674
* ScriptedDecideRule.java
...
getEngine() - tiny cosmetic fixups
2012-05-31 16:29:33 -07:00
Travis Wellman
392a4b3fd0
correct synchronization and threadlocal logic
...
Previously the whole method was synchronized even if threadlocals
were used (which are thread safe). And previously the sharedEngine
was set and returned only when it /was not/ needed ie when
getIsolateThreads() returned true.
Now getEngine does what it says it does.
2012-05-31 16:09:42 -07:00
Noah Levitt
1015c5fa00
Fix HER-1973 Regression in queue budget overrides
...
* WorkQueueFrontier.java
findEligibleURI() - apply sheet overrides before refreshing queue budgets
2012-05-15 17:35:49 -07:00
Noah Levitt
809981ef02
* PersistProcessor.java
...
populatePersistEnvFromLog() - handle exception deserializing line
2012-05-10 19:00:27 -07:00
Noah Levitt
9fffa8fc53
**/pom.xml
...
bump version to 3.1.2-SNAPSHOT
2012-05-01 18:17:33 -07:00
Noah Levitt
43d57dcaf4
* README.txt
...
update for 3.1.1
2012-05-01 18:05:38 -07:00
Noah Levitt
79c40346fc
**/pom.xml
...
* bump version number to 3.1.1 for release
2012-05-01 18:01:06 -07:00
Noah Levitt
a7c63b8d00
* FetchHTTP.java
...
remove redundant nonsensical comment
2012-05-01 17:53:52 -07:00
Gordon Mohr
e4f12bb20f
Merge pull request #6 from ato/codemirror
...
HER-2001: Use the CodeMirror editor for crawl config and script console
(Thanks Alex!)
2012-04-25 15:04:10 -07:00
Noah Levitt
42e759c01d
List correct defaults for BdbModule cacheSize and cachePercentage with an explanatory comment.
2012-04-25 11:10:56 -07:00
Noah Levitt
a2880c4306
HER-1990 large robots.txt problematic
...
* Robotstxt.java
enforce max size of robots.txt
2012-04-20 13:57:53 -07:00
Noah Levitt
52d5307a9b
Merge branch 'her-1948'
2012-04-20 12:33:19 -07:00
Noah Levitt
8d0a1678de
* effective_tld_names.dat
...
update to latest from publicsuffix.org
2012-04-20 12:25:57 -07:00
Noah Levitt
3fed078817
Merge pull request #9 from kngenie/publicsuffixes2
...
Publicsuffixes2
2012-04-20 12:19:47 -07:00
Noah Levitt
da28258f90
Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948)
2012-04-19 18:16:53 -07:00
Noah Levitt
4a64c6ba59
Revert "Altered FetchStats to also keep track of the number of URIs that have"
...
This reverts commit 4c1740ce1e
2012-04-18 15:53:49 -07:00
Noah Levitt
9551c774af
Merge branch 'master' into http-status-deciderule
2012-04-18 15:47:24 -07:00
Noah Levitt
8e853c5f8d
HER-2005 existing content length decide rule is not flexible enough
...
* ResponseContentLengthDecideRule.java
new predicated decide rule that will ACCEPT or REJECT a uri if the content body is within a specified size range
2012-04-18 13:15:15 -07:00
Noah Levitt
0a4d56b638
Fix for HER-2004 SEVERE error in FetchWhois on some government domains
...
* FetchWhois.java
addWhoisLinks() - handle IllegalStateException from InternetDomainName.topPrivateDomain()
2012-04-18 10:37:24 -07:00
Kenji Nagahashi
27d621f21a
renamed PublicSuffixes2 to PublicSuffixes.
...
deleted debug println.
2012-04-18 10:14:51 -07:00
Kenji Nagahashi
a0d52e6c17
PublicSuffixes2Test still had tests for PublicSuffix. fixed.
2012-04-18 10:14:51 -07:00
Kenji Nagahashi
b2d62a81ff
trial rewrite of PublicSuffixes for small memory footprint (no significant performance gain)
2012-04-18 10:14:50 -07:00
Noah Levitt
31df3a8f03
Merge branch 'master' into speculative-links
2012-04-13 18:16:08 -07:00
Travis Wellman
52eef84658
note about preselector.recheckScope for use of IpAddressSetDecideRule
2012-04-12 17:34:21 -07:00
Travis Wellman
54b1e245eb
gitignore for maven build artifacts
2012-04-12 17:32:57 -07:00
Travis Wellman
563da14330
Merge pull request #7 from travisfw/IpAddressSetDecideRule
...
Ip address set decide rule
2012-04-12 16:52:00 -07:00
Travis Wellman
57d2f42954
formatting
2012-04-12 16:50:22 -07:00
Travis Wellman
64e9cc367b
remove println, make class public
2012-04-12 16:46:16 -07:00
Travis Wellman
2a3945ff79
Merge branch 'master' of github.com:internetarchive/heritrix3 into IpAddressSetDecideRule
2012-04-02 18:17:44 -07:00
Travis Wellman
f9f9e62f4c
Reject or accept matching a set of IP addresses.
2012-04-02 18:16:23 -07:00
Alex Osborne
f19a72efaf
HER-2001: Use the CodeMirror editor for crawl config and script console
...
This enables syntax-highlighting, line-numbering and auto-indent for the crawl
configuration editor and the script console (for BeanShell, JavaScript and
Groovy).
* BaseResource.java
add a more generic getStaticRef() in addition to getStylesheetRef()
* EditRepresentation.java
wrap textarea with CodeMirror in purexml mode.
disable the "save changes" button until config is edited.
dynamically resize CodeMirror edit element to fit window height.
* ScriptResource.java
wrap textarea with CodeMirror.
switch CodeMirror language modes depending on selected script engine.
* resources/.../restlet/codemirror/
include just the files from codemirror-2.23.zip necessary for the above
plus the CodeMirror LICENSE file (MIT)
plus a README linking to the codemirror website
2012-03-31 18:35:06 +11:00
Noah Levitt
9626b8454a
* EditRepresentation.java
...
make edit page (for cxml mainly) prettier with big textarea (see Kenji's comment on HER-1993 dated 2012-03-27 18:03)
2012-03-30 11:44:28 -07:00
Noah Levitt
335b32aecc
* ExtractorHTML.java
...
some fixup on RELEVANT_TAG_EXTRACTOR javadoc comment
2012-03-20 15:14:14 -07:00
Noah Levitt
4eab6461ed
Merge remote-tracking branch 'adam-miller/master'
2012-03-20 15:06:58 -07:00
Adam Miller
a108795d50
HER-1998 - Adding comments, adjusting test case to prevent subclasses from failing new tests
2012-03-19 18:19:00 -07:00
Noah Levitt
ea42c0c653
* EngineResource.java
...
acceptRepresentation() - "I'm sure" parameter is now "im_sure"
2012-03-19 17:30:18 -07:00
Adam Miller
1a333fbd35
HER-1998 - ExtractorHTML modified to parse html inside conditional comments. Still ignores normal comments
2012-03-16 14:36:15 -07:00
Noah Levitt
f7505e6673
More HTML cleanup. Html5 doctype declaration, get rid of <br> and
...
xml-style "/>" tag endings, other cleanup to make it validate.
2012-03-15 18:47:18 -07:00
Noah Levitt
52703c6eb4
* ArchiveUtils.java
...
doubleToString() - return "NaN" for NaN (inf symbol didn't quite make sense to me)
2012-03-15 18:46:01 -07:00
Noah Levitt
7cde3b4f17
Merge branch 'master' of github.com:internetarchive/heritrix3 into html-cleanup
2012-03-15 15:59:10 -07:00
Kenji Nagahashi
64f24d4c66
HER-1993: In JobResource, write error message instead of throwing RuntimeException when it failed to read
...
job log or crawl log.
2012-03-09 16:48:11 -08:00
Kenji Nagahashi
f05287c8ca
restored deleted space after header "Crawl Log"
2012-03-09 16:29:52 -08:00
Noah Levitt
e7d75cbb5a
* AbstractPersistProcessor.java, PersistProcessor.java, PersistStoreProcessor.java
...
refactor some common persist processor code into new class AbstractPersistProcessor
2012-03-06 16:20:58 -08:00
Kenji Nagahashi
adc82c1c6f
Second round of HTML cleanup:
...
- added missing <html> and <body>
- fixed non-working CSS link, added CSS link to all pages
- replaced layout by <br> and with structural tags with CSS
- removed superfluous output, notably in BeanBrowseResource
- <code>-ify variable names in ScriptResource
- added special handling of NaN in doubleToString()
2012-03-03 00:46:56 -08:00
Kenji Nagahashi
46c335c8a3
BeanBrowseResource: fixed malformed HTML.
2012-02-29 13:22:33 -08:00
Noah Levitt
1d05df7fbe
* StatisticsTracker.java
...
writeReportFile() - reduce log level from warning to info when reusing existing report (normal after crawl is finished)
2012-02-29 11:02:39 -08:00
Noah Levitt
7af7cbf18c
Merge branch 'master' of github.com:internetarchive/heritrix3
2012-02-10 15:51:15 -08:00
Noah Levitt
db801b3e0f
Fix problems handling crawl job paths with characters that require url-encoding, since we sometimes refer to them with file: urls
...
* PathSharingContext.java, CrawlJob.java
use java.net.URI to convert paths to and from file: urls
2012-02-10 15:48:09 -08:00
Noah Levitt