and equivalent for other sites. Usually we find these links through
normal link extraction, but we have the info here, so we may as well use
it to make sure.
* warc-writer-chain:
write youtube-dl json to the warc
rename method so as not to conflict with Processor
make WARCRecordBuilder an interface
revisits only for http and ftp
accept any BaseWARCWriterProcessor
use WARCWriterChainProcessor for these tests
default chain in code
oops, handle https too
same test as for WARCWriterProcessor
configurable warc writer chain
* ydl:
whoops, better spawn the thread first
read stderr and stdout in separate threads...
do not drop any `CrawlURI.data` between processing
nothing private ever
quiet org.mortbay.log (jetty?) logging
making everything work
ExtractorYoutubeDL
Update changelog for 3.4.0-20190418
[maven-release-plugin] prepare for next development iteration
[maven-release-plugin] prepare release 3.4.0-20190418
set of frontier management changes to support CrawlHQ module
Remove suffix from warcWriter since it is no longer used.
Add CHANGELOG; address #233.
Without this change (or other measures), we sometimes get nulls in the
ExtractorYoutubeDL log for containing page information. We'll run this
on QA for a while and see if it causes any problems.
nlevitt [1:59 PM]
https://github.com/internetarchive/heritrix3/blob/master/modules/src/main/java/org/archive/modules/CrawlURI.java#L878
drops some stuff from `CrawlURI.data` after processing a uri, even if it needs to be processed again
there is a list of keys that shouldn’t be dropped (`persistentKeys`), but it is final and private
so if you’re writing your own heritrix module and you want to keep some information in CrawlURI.data, it usually works, except when the url is processed more than once (like when it needs a prereq like robots.txt the first time)
in practice it seems that most data is persisted, that is, most commonly used keys are in `persistentKeys`
in a crawl with pretty standard configuration i’m mostly seeing `prerequisite-uri` dropped and occasionally `fetch-completed-time` and `fetch-began-time` being dropped
i’m highly skeptical of the value of dropping keys at all and i’m tempted to get rid of this entirely, make all the keys persistent in other words
soliciting feedback (edited)
anjackson [2:39 PM]
My immediate reaction is HARD AGREE. It looks like Really Old Code though (https://github.com/internetarchive/heritrix3/blame/7d3eff5269142c77fa4b988396153f4c29d16caa/modules/src/main/java/org/archive/modules/CrawlURI.java#L878)
so the reasons for doing so may have been lost in time.
Hm, looking at usage: https://github.com/internetarchive/heritrix3/blob/a60b2ef3875ad47f57b0c6b3c0b19f86c40a12f7/engine/src/main/java/org/archive/crawler/frontier/WorkQueueFrontier.java#L954-L955
engine/src/main/java/org/archive/crawler/frontier/WorkQueueFrontier.java:954-955
curi.processingCleanup(); // lose state that shouldn't burden
// retry
I guess there's a concern that there may be state in there that is set during a fetch and may cause problems if the same CrawlURI is deferred?
But I'm not aware of anything in the fetch chain that behaves like that.
nlevitt [3:02 PM]
oh, i missed `CrawlURI.addDataPersistentMember(String)` et al. still...
There is already a clause for this in logging.properties, but it's using
log4j. It was dumping stack traces every time the client was dubious of
heritrix's self-signed certificate.
Why do we have so many identical log4j.xml's? 🤷♂️
* master:
replace System.err.println with logger.info
Revert "Upgrade httpclient to 4.5.7 and handle cookies more compliantly"
Removing outdated test.
Disable questionalbe test.
Avoid deprecated flag.
Supply an iterator, for #245
Updated POM to use latest version.
Update README.md
Handle missing closing paren in srcset descriptor
Teach jericho extractor srcset
Don't run srcset test against jericho, it doesn't handle it
Handle commas more compliantly when parsing srcset
Ensure we start parsing full lines, for #239.
These changes come from a private fork of H3, originally made by Kenji
Nagahashi, to create org.archive.crawler.frontier.PullingBdbFrontier,
which we intend to merge into 'contrib' of the official version in the
near future.
* master:
Revert "Upgrade httpclient to 4.5.7 and handle cookies more compliantly"
Removing outdated test.
Disable questionalbe test.
Avoid deprecated flag.
Supply an iterator, for #245
Updated POM to use latest version.
* master:
use constant from rethinkdb lib for default port
Update README.md
Handle missing closing paren in srcset descriptor
Teach jericho extractor srcset
Don't run srcset test against jericho, it doesn't handle it
Handle commas more compliantly when parsing srcset
Ensure we start parsing full lines, for #239.
Noah Levitt