* BasicProfileTest.java, SelfTestBase.java, CrawlControllerTest.java, PrecedenceLoader.java, MigrateH1to3Tool.java, CrawlerLoggerModule.java, StatisticsTracker.java, CheckpointUtils.java, BdbUriUniqFilter.java, ARCWriterProcessorTest.java, WARCWriterProcessorTest.java, PersistProcessor.java, WriterPoolProcessor.java, PrefixFinderTest.java, StoredQueueTest.java, FileUtilsTest.java, ObjectIdentityBdbManualCacheTest.java, ObjectIdentityBdbCacheTest.java, ObjectPlusFilesOutputStream.java, TestUtils.java, TmpDirTestCase.java, Engine.java
Replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir). In these cases the calls were either already in a spot where the possible IOException would be handled appropriately, or the line was trivially moved into such a block.
* ActionDirectory.java, Engine.java
replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure
* BdbModule.java
setup() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException" - conveniently the place where this method is called was already in a try block that catches IOException
* Checkpoint.java
generateFrom() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException"
* CheckpointService.java
move call to Checkpoint.generateFrom() inside existing try block since it now can throw IOException
* Recorder.java
ensure(File) - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure
new Recorder(File,String,int,int) - call ensure() on the correct object, the containing directory; and remove redundant call to ensure()
fix mistake in r7240 - 'val.setIdentityCache()' was no longer being called the first time through, when the object comes from the 'supplier' (thanks Gordon)
* WorkQueueFrontier.java
(HER-1925) avoid 2 possible null pointer dereferences
* CrawlController.java
getState() - change declared return type from Object to State
* CrawlerLoggerModule.java
(HER-1932) remove unused, unset field "reports"
* BdbCookieStorage.java
elide pointless extra variable
* FetchHTTP.java, DownloadURLConnection.java, ProcessUtils.java
(HER-1933) use Arrays.toString() for logged arrays
* CrawlServer.java
- remove unused field robotstxtChecksum
- updateRobots() - avoid reinventing existing utility class InstanceofPredicate
* ExternalGeoLookupInterface.java
(HER-1938) extend Serializable, since ExternalGeoLocationDecideRule is declared Serializable and has a ExternalGeoLookupInterface field
* DecideRuleSequence.java
(HER-1937) make field fileLogger transient
* PersistLogProcessor.java
(HER-1924) remove field recoveryCheckpoint shadowing same field in superclass Processor
* ExtractorUniversal.java
(HER-1920) use return value of potentialTLD.toLowerCase() as it appears was intended
* ARCWriterProcessor.java
(HER-1928) avoid possible null pointer dereference
* S3URLConnection.java
(HER-1933) set S3ServiceException as cause of rethrown IOException, and do not put stacktrace array in message
* ObjectIdentityMemCache.java
(HER-1929) avoid possible null pointer dereference
* .classpath
more source jar references, other cleanup
* CookieSpecBase.java
match(String, int, String, boolean, SortedMap) - use InternetDomainName.name() instead of .toString(), since the latter doesn't return the plain old domain name
* Cookie.java
javadoc typo
* ActionDirectory.java, CrawlerLoggerModule.java, StatisticsTracker.java, SurtPrefixedDecideRule.java, WriterPoolProcessor.java, profile-crawler-beans.cxml
use camelcase ${launchId}
* CheckpointService.java, JobResource.java
rename getAvailableCheckpointDirectories() to findAvailableCheckpointDirectories() so that it's not treated as a bean property
* WriterPoolProcessor.java, WriterPoolSettingsData.java, WriterPoolMember.java
rename getOutputDirs() to calcOutputDirs() so that it's not treated as a bean property
* ConfigPath.java
let ConfigPathConfigurer to do interpolation of ${launchId}
* ConfigFile.java
let ConfigPathConfigurer do the snapshotting of config file
* CrawlJob.java, PathSharingContext.java
remove launchDir initialization out of CrawlJob into PathSharingContext
* ConfigPathConfigurer.java
- remove special handling of WriterPoolProcessor store paths; instead, look in beans for ConfigPaths within Iterables
- have each ConfigPath hold reference to this ConfigPathConfigurer to use for interpolating ${launchId} and snapshotting config files
* Recorder
object to unsupported content-encodings when first set
* FetchHTTP
note as annotation unsupported content-encodings
* Link
not as annotation when base URI is used for absent via
* ExtractorHTML
downgrade char-sequence reading problem (often a chunking problem) to WARNING from SEVERE
* CrawlJob.java
decided to go with plain timestamp17 as launch id, no "launch-" prefix
* ConfigPathConfigurer.java
refactored special remembering of WriterPoolProcessor storePaths out of fixupPaths()
* UriProcessingFormatter, Preformatter, GenerationFileHandler
improve preformat-outside-synchronized optimization so that the LogRecord/CrawlURI doesn't linger until next displaces it
* CrawlURI
(processingCleanup) null more of last-processing-run collected values
* WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java
change type of storePaths to List<ConfigPath> and handle appropriately
* ConfigPathConfigurer.java
fixupPaths() - old code did not touch WriterPoolProcessor storePaths, since they're deeply nested inside the bean, but they need to be remembered for later interpolation of ${launch-id}, so add special handling
* HardLinker.java
renamed FilesystemLinkMaker.java
* FilesystemLinkMaker.java
add support for symbolic links
* CLibrary.java
new method symlink()
* BdbModule.java
use new class name FilesystemLinkMaker
* CrawlJob.java
at crawl launch, create launch directory launch-{timestamp17}, copy cxml there, symlink "current" to launch dir, inform ConfigPaths
* ConfigPath.java
interpolate ${launch-id} in configured paths
* ConfigFile.java
obtainReader() - snapshot config files to launch dir when they are read
* ActionDirectory.java
default doneDir now ${launch-id}/actions-done
actOn() - symlink from old style done dir action/done to done files
* SurtPrefixedDecideRule.java
default surtsDumpFile now ${launch-id}/surts.dump
pathsFixedUp() - this gets called at build time, but we don't want anything written to disk until launch time, so remove call to dumpSurtPrefixSet() here
* CrawlerLoggerModule.java
default logs dir now ${launch-id}/logs
* StatisticsTracker.java
default reports dir now ${launch-id}/reports
* WriterPoolProcessor.java
default writer base path now ${launch-id}
* profile-crawler-beans.cxml
update to reflect new default paths under launch dirs
* PropertyUtils.java
fix javadoc typo
* WriterPool
make new LARGEST_MAX_ACTIVE (255) the capacity of the reuse-pool, so that the maxActive property may vary up to that and still work; also handle more gracefully the failure of trying to use a larger number than the capacity (writer gets closed early rather than left hanging open)
* pom.xml, .classpath
update references to necessary spring-3.0.5 packages
* SeedModule
merge rather than replace event listeners, so that (now later) autowiring doesn't clobber the self-insertion done by anonymous (non-top-level) SurtPrefixedDecideRule bean
* HeritrixLifecycleProcessor, PathScharingContext
use this new non-default LifecycleProcessor to avoid new Spring3 behavior of auto-start()ing context on refresh(build)
* CrawlController
example of using @Value annotation to set default value: will offer benefits for auto-discovery of defaults for configuration interface, or enforcing maximally-explicit configurations (see [HER-1897])
* profile-crawler-beans.cxml, selftest-crawler-beans.cxml
update templates with spring3 preamble/boilerplate
[HER-1910] H3: job-related pages very slow to render (pause mid-way through) in large crawls
* BdbModule
option to reuse bdb data on StoredQueue-creation
* Checkpoint
new saveWriter, loadReader utilities for creating extra checkpoint files (for the long list of ready queues)
* BdbFrontier
on checkpoint, remember ordered list of inProcess/ready/snoozed queues (so same queues are active upon resume)
on resume, reuse old retired/inactive StoredQueue data, and reload active queues from newly-saved list
* StoredQueue
restore tailIndex properly when resuing old data
more-efficient size() calculation (for HER-1910)
* HardLinker.java
makeHardLink() - wrap unix link() and windows CreateHardLink() and choose depending on platform
* BdbModule.java
doCheckpoint() - use HardLinker.makeHardLink()
* GenericReplayCharSequence.java
decode() - read another character to check for more content, rather than consult BufferedReader.ready(), since the latter sometimes returns false even when there is more to read
* ReplayCharSequence, GenericReplayCharSequence
use Charset instance rather than String-name
* Recorder
use Charset instance ratehr than String-name
(getContentReplayCharSequence) avoid reusing cached ReplayCharSequence when encoding-in-use has changed since it was created
(getContentReplayPrefixString) allow requested specific charset-interpretation
* ExtractorHTML, ExtractorXML
work with Charset instances
double-check that in-content-declarations are self-consistent before using
recycle matcher instances
change minor charset problems/decisions to a crawl.log annotation
* ExtractorXML.java
if charset not spec'd in http header look for <?xml encoding=""?>
* ExtractorHTML.java
lookForEncodingInContent() -
1. look for <meta http-equiv="content-type"...>
2. if not found then look for <meta charset="">
3. if not found then <?xml encoding=""...?>
* Recorder.java
setCharacterEncoding() - If new encoding is different from old encoding,
close replayCharSequence and set to null, which will trigger recreation on
next retrieval.
* UriUtils.java
NAIVE_LIKELY_URI_PATTERN - revert change and unpublicize
NAIVE_URI_EXCEPTIONS - add some mimetype strings that came up in testing
isLikelyFalsePositive() - unpublicize
* ExtractorXML.java
XML_URI_EXTRACTOR - do not use NAIVE_LIKELY_URI_PATTERN
shouldExtract() - check for mimetype application/vnd.openxmlformats which is not xml
processXml() - use UriUtils.isLikelyUri()
* ExtractorXML.java
instead of considering only strings that start with http(s):, consider all
strings that match UriUtils.NAIVE_LIKELY_URI_PATTERN (and, as before,
constitute the entirety of the xml tag content or attribute value)
* UriUtils.java
- NAIVE_LIKELY_URI_PATTERN - add quotes to the excluded characters so it
does not eat the closing quote when matching xml attribute values, and
make visibility public for use in ExtractorXML
- isLikelyUri() - refactor false positive check into new method
isLikelyFalsePositive() so that it can be used in ExtractorXML avoiding
redundant check against NAIVE_LIKELY_URI_PATTERN
(specifically, when they're in the query string)
* UURIFactory.java
fixup() - replace \ with / up to the first ? in the url, instead of
anywhere in the url
and update comments
* UURIFactoryTest.java
backslash test
* BdbModule.java
new setting, useHardLinkCheckpoints, default true
doCheckpoint now creates hard-links to active log files from the in-environment checkpoint-dir
doRecover now relinks from checkpoint-dir files, when available, in preference to whatever is already in the env-dir
* pom.xml
bring in JNA for native calls
* CLibrary
wrapper for the link() function
* GZIPMembersInputStream
override OpenJDK7GZIPInputStream.readTrailer to not rely on available() as indicator of end-of-stream
fix membersIterator hasNext() to also not rely on available() as indicator of end-of-stream
* ArchiveReader
change innerHasNext to peek one byte (using mark/reset) rather than rely on available() as indicator of end-of-stream
remove redundant getInputStream accessor in favor of field-name-matching getIn/setIn
* ArchiveReaderFactory, (W)ARCReaderFactory
remove no-longer-necessary RepositionableStream references
use getIn() rather than getInputStream()
[HER-728] Offer replay stream that has been un-chunked (whether because response was HTTP/1.1 or used chunked in HTTP/1.0 against spec)
[HER-1876] Offer HTTP/1.1 option - for chunked transfer-encoding (but not persistent connections)
* FetchHTTP
add 'acceptCompression' and 'useHTTP11' properties, both default false
* Recorder
track whether recorded-input is transfer-encoded (chunked) or content-encoded (gzip etc)
offer alternate replay streams for
(1) raw 'messageBody';
(2) entity (un-chunked if necessary)
(3) content (decompressed if necessary)
always content & GenericReplayCharSequence for CharSequence replays
* GenericReplayCharSequence
always use a stream (rather than random-access buffer)
always decode to prefix buffer first, so short content never touches disk no matter the encoding
* InMemoryReplayCharSequence
deleted; 'Generic' now works similarly for small content and anyway random-access for single-byte-encodings is now rarely possible (given deconding streams)
* RecordingInputStream, RecordingOutputStream
adjust for changed stream names, functionality moved to Recorder
* ReplayCharSequence
use Charset instances rather than names
* ReplayInputStream
add convenience constructor (and tmp-file-destroy) for copying any other inputStream into a seekable ReplayInputStream
* OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java
backport the working GZIP implementation from OpenJDK7, with minimal
renames/imports/private-to-protected changes to enable member-at-a-time reading
* GZIPMembersInputStream
base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug
override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode
* GZIPMembersInputStreamTest, ARCWriterTest
test tweaks
ensure mark()ing more frequently, when reset to before current position won't be needed
* ArchiveReaderFactory
avoid misinterpreting a "c:" local path as a URI scheme
nlevitt