 Noah Levitt
|
98d0901531
|
moving org.archive.net.PublicSuffixes to ia-web-commons
|
2013-12-05 16:13:30 -08:00 |
|
|
Noah Levitt
|
552794dc89
|
don't let the exception from commit df2a313797 derail the whole crawl job
|
2013-11-26 16:08:38 -08:00 |
|
|
Noah Levitt
|
1163847644
|
try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted
|
2013-11-21 18:51:16 -08:00 |
|
|
Noah Levitt
|
8e1125d062
|
bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson)
|
2013-11-20 09:18:31 -08:00 |
|
|
Noah Levitt
|
d93b5449be
|
new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc
|
2013-11-19 11:05:14 -08:00 |
|
|
Noah Levitt
|
7089caf3c4
|
remove unused engine.css
|
2013-11-15 19:11:19 -08:00 |
|
|
Noah Levitt
|
119e565acc
|
Adam's responsive UI
|
2013-11-15 18:59:14 -08:00 |
|
|
Noah Levitt
|
2b87219496
|
oops, avoid npe
|
2013-11-15 17:59:34 -08:00 |
|
|
Noah Levitt
|
df2a313797
|
throw exception when host cache returns 0.0.0.0 for any host (trying to track down a strange problem)
|
2013-11-15 17:20:55 -08:00 |
|
|
Ilya Kreymer
|
e2f4b7287f
|
Merge branch 'master' of github.com:internetarchive/heritrix3
|
2013-11-01 15:44:12 -07:00 |
|
|
Ilya Kreymer
|
8633ed0ec3
|
removed ThreadLocalHttpConnectionManager -- it has been moved to ia-web-commons
|
2013-11-01 15:43:46 -07:00 |
|
|
Noah Levitt
|
b137022b79
|
Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.)
|
2013-10-28 17:35:55 -07:00 |
|
|
Noah Levitt
|
38a12b2008
|
avoid NPE resuming from checkpoint if "stats" is {} in warcWriter checkpoint json
|
2013-10-11 14:44:34 -07:00 |
|
|
Noah Levitt
|
a1f84d3cc4
|
HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility
|
2013-10-10 17:57:18 -07:00 |
|
|
Noah Levitt
|
7ad90b3126
|
Merge branch 'master' of github.com:internetarchive/heritrix3
|
2013-09-12 12:04:32 -07:00 |
|
|
Noah Levitt
|
57311db117
|
use libs version ${project.version}
|
2013-09-10 15:24:58 -07:00 |
|
|
Noah Levitt
|
140ba3e89e
|
tweak io block size in FileUtils.appendTo(), referencing performance metrics documented in gnu coreutils (used e.g. for /bin/cat)
|
2013-09-08 12:41:48 -07:00 |
|
|
Noah Levitt
|
254811ff7f
|
commented-out default for new setting forgetAllButLatest on CheckpointService
|
2013-09-08 12:39:54 -07:00 |
|
|
Noah Levitt
|
f096609a50
|
avoid infinite loop spitting out stack traces on BeansException on startup
|
2013-09-08 00:11:54 -07:00 |
|
|
Noah Levitt
|
a00141eef7
|
avoid deadlock in case of BeansException at crawl startup
|
2013-09-08 00:10:27 -07:00 |
|
|
Noah Levitt
|
071cbe99cd
|
include urlsWritten in checkpoint, and be backward compatible with old checkpoints
|
2013-09-07 23:40:17 -07:00 |
|
|
Noah Levitt
|
c49b465d82
|
custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test
|
2013-09-07 19:40:11 -07:00 |
|
|
Noah Levitt
|
52d72e661d
|
part of HER-2048 - WARCWriterProcessor stats resume from checkpoint
|
2013-09-07 11:35:03 -07:00 |
|
|
Noah Levitt
|
e0f78976d7
|
part of HER-2048 - restore BdbServerCache from checkpoint
|
2013-09-07 10:53:22 -07:00 |
|
|
Noah Levitt
|
fd71833050
|
avoid NPE when attempting to terminate already-finished job
|
2013-09-07 09:57:33 -07:00 |
|
|
Noah Levitt
|
d01bc4ed55
|
when resuming from checkpoint, make next checkpoint serial number follow serial number of resumed checkpoint
|
2013-09-07 09:57:07 -07:00 |
|
|
Noah Levitt
|
d61171f076
|
CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size
|
2013-09-07 09:55:58 -07:00 |
|
|
Noah Levitt
|
b670a3c2a7
|
Merge branch 'master' into always-resumable
|
2013-09-06 23:32:32 -07:00 |
|
|
Noah Levitt
|
25d4de186f
|
don't log warning in normal case
|
2013-09-06 22:04:56 -07:00 |
|
|
Noah Levitt
|
cc0b63066b
|
Merge branch 'master' into always-resumable
|
2013-09-03 18:38:55 -07:00 |
|
|
Noah Levitt
|
34cb015ecf
|
adjust PublicSuffixesTest for updated public suffixes list
|
2013-09-03 18:38:44 -07:00 |
|
|
Noah Levitt
|
59596c7043
|
prevent redundant checkpoints when crawl is stopping
|
2013-09-03 14:24:41 -07:00 |
|
|
Noah Levitt
|
0ab8eee24e
|
forgetting old checkpoints - roll up previous checkpointed CrawlerJournals (e.g. frontier.recover.gz) into new checkpoint
|
2013-08-29 10:56:12 -07:00 |
|
|
Noah Levitt
|
73f6b20e48
|
Merge branch 'master' into always-resumable
|
2013-08-28 18:31:42 -07:00 |
|
|
Noah Levitt
|
981cf5bc9d
|
update public suffixes list
|
2013-08-28 18:31:28 -07:00 |
|
|
Noah Levitt
|
1315717002
|
Merge branch 'master' into always-resumable
|
2013-08-28 18:14:03 -07:00 |
|
|
Noah Levitt
|
418c7138a9
|
handle case where there is no whois server for a domain
|
2013-08-28 18:13:53 -07:00 |
|
|
Noah Levitt
|
e8c5083d1a
|
handle case where there is no whois server for a domain
|
2013-08-28 18:13:33 -07:00 |
|
|
Noah Levitt
|
43bb42d346
|
BdbModule#doCheckpoint - support forgetting all but latest checkpoint
|
2013-08-28 11:22:37 -07:00 |
|
|
Noah Levitt
|
8e7c26bc8f
|
fix setForgetAllButLatest
|
2013-08-28 10:58:32 -07:00 |
|
|
Noah Levitt
|
2fe7d8cd45
|
some progress on forgetting old checkpoints when checkpointing anew
|
2013-08-27 18:39:25 -07:00 |
|
|
Noah Levitt
|
77a12e63b8
|
make special "host" "whois:" appear correctly in hosts report
|
2013-08-27 17:39:07 -07:00 |
|
|
Noah Levitt
|
4abfcb01b0
|
HER-1895 support HTTP Refresh header
|
2013-08-26 18:33:02 -07:00 |
|
|
Noah Levitt
|
d1bb1a4637
|
further improve performance of BdbUriUniqFilter.forgetAllSchemeAuthorityMatching() by skipping conversion of key data to long and comparing bytes directly
|
2013-08-21 11:44:46 -07:00 |
|
|
Noah Levitt
|
a06c141740
|
rename BdbUriUniqFilter.forgetSchemeHost() to forgetAllSchemeAuthorityMatching(), improve performance
|
2013-08-21 11:31:05 -07:00 |
|
|
Noah Levitt
|
a6958951df
|
Reset Recorder state when uri processing is finished (had been seeing cases where a url's entry in crawl.log would sometimes have nonzero size value if the url was erroring out and RecordingOutputStream.open() was never called)
|
2013-08-12 19:28:28 -07:00 |
|
|
Noah Levitt
|
68adaa70fd
|
need to close the recorder here so it can calculate the content length properly
|
2013-08-09 19:14:58 -07:00 |
|
|
Noah Levitt
|
705a375daf
|
uses of UriUtils.isLikelyUri() in Extractor{HTML,SWF,XML} with UriUtils.isVeryLikelyUri() to reap the benefits of HER-1523 improvements (should address archive-it issue ARI-3492)
|
2013-08-09 18:17:11 -07:00 |
|
|
Noah Levitt
|
8c64cba9ae
|
TransclusionDecideRule.java - consider form submission "S" hop as navlink like "L", not transcluded
|
2013-08-05 16:05:02 -07:00 |
|
|
Noah Levitt
|
f4f3b9be38
|
add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs
|
2013-07-26 17:50:36 -07:00 |
|