Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#781/merge

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 6795b3bfba [HER-1834] log-bloat, memory problems when hopsPath grows endlessly with no max-hops limit set * CrawlURI MAX_HOPS_DISPLAYED constant, 50 build new CrawlURI pathFromSeed value via extendHopsPath, which shows only MAX_HOPS_DISPLAYED hops, and precedes string with int value of hops unshown * CrawlURITest test for extendHopsPath gojomo 2011-01-19 02:30:42 +00:00
• f0f8aeb9b9 * WARCWriter remove that pesky reference to a 'modules' class (usu. available in dev setup, but not in ordered formal builds) gojomo 2011-01-19 01:25:42 +00:00
• 57edfde618 [HER-1859] ExtractorJS improper handling of Javascript string literal escapes * ExtractorJS apply JS literal string deescaping before interpretation as link * ExtractorJSTest 3 basic tests, including one for this issue * Link provide logging and fallback for case where via may be null; risk identified in creation of test cases could present again if a '.js' file is ever provided as a (via-less) seed gojomo 2011-01-18 23:34:35 +00:00
• 0af0d3cc6a * WARCWriterProcessor.java addStats() - use putIfAbsent() to avoid race conditions; move creation of stats map to constructor to avoid another race condition * WARCWriter.java switch back to thread-unsafe data structures for stats since object is used in one thread at a time according to javadoc nlevitt 2011-01-18 04:02:42 +00:00
• 823aaf0154 Add Javadoc class comment notes about (potentially) left-open-by-design ReplayCharSequence instances gojomo 2011-01-18 01:32:26 +00:00
• 799e0d9ff8 Remove errant import that eclipse added to satisfy a javadoc reference, but that breaks build nlevitt 2011-01-17 23:06:11 +00:00
• cf597b2796 Fix fix on commented out scope.logToFile - default value is false nlevitt 2011-01-17 22:53:43 +00:00
• c966301100 Fix commented out scope.logToFile nlevitt 2011-01-17 22:51:33 +00:00
• 69429f2d4e Missed one file r7051 commit * ExtractorJS.java do not close ReplayCharSequence nlevitt 2011-01-17 21:27:39 +00:00
• 50bb39a839 * WARCWriterProcessor.java initialize urlsWritten (fix npe, oops) nlevitt 2011-01-15 19:46:42 +00:00
• 72932b8ca8 Rename and comment on WARCWriter stats bits to clarify their use as temporary accumulators on behalf of WARCWriterProcessor. nlevitt 2011-01-15 04:17:49 +00:00
• 9a34201c8e Make warc writer stats thread-safe. nlevitt 2011-01-15 04:01:04 +00:00
• 6de68e3e36 Let Recorder remember its replayCharSequence, and ToeThread close it when uri processing is finished. * Recorder.java getReplayCharSequence() - remember ReplayCharSequence, obtain new one if none remembered or remembered has been closed endReplays() - close replayCharSequence * ToeThread.java run() - call Recorder.endReplays() when finished with uri * ReplayCharSequence.java isOpen() - return false if close() has been called * GenericReplayCharSequence.java, InMemoryReplayCharSequence.java implement isOpen() * KeyWordProcessor.java, ExtractorHTML.java, HTTPContentDigest.java, ExtractorCSS.java, ExtractorXML.java do not close ReplayCharSequence nlevitt 2011-01-15 03:13:27 +00:00
• ded64befb5 W/ARCWriter-related refactoring * WriterPoolSettingsData useful defaults for frequentFlushes and writeBufferSize gojomo 2011-01-15 02:43:08 +00:00
• cbd65d8350 [HER-1848] Add option to disable link-extraction from 404 error pages * AbstractFrontier add new/renamed ExtractorParameters methods gojomo 2011-01-15 01:32:54 +00:00
• 6f095b409c [HER-1848] Add option to disable link-extraction from 404 error pages * FetchStatusCodes add 404 constant * ExtractorParameters rename isIndependentExtractor (which seems to describe the parameters object) to getExtractIndependently (the paramters' imperative to extractors) add getExtract404s * Extractor add default ExtractorParameters impl remove redundant shouldProcess * ContentExtractor consult getExtract404s, if false skip extraction of 404s gojomo 2011-01-15 01:21:09 +00:00
• cd398d4c46 W/ARCWriter-related refactoring * Arc2Warc, Warc2Arc adapt to new constructors, settings-object * MiserOutputStream stream to monitor position, optionally suppress flushes * WriterPoolMember use MiserOutputStream rather than special file-access to find compressed offsets use shared settings object rather than copying multiple values * WriterPoolSettings add settings for frequentFlushes and writeBufferSize for IO optimization * ARCReader, ARCWriter, ARCWriterPool adapt to new constructors, settings-object * WriterPoolSettingsData impl of WriterPoolSettings for testing and adhoc use * WARCWriter adapt to new constructors, settings-object remove checkSize rollover to new file (done elsewhere) remove extraneous id-generator code/static-method * WARCWriterPool adapt to new constructors, settings-object * WARCWriterPoolSettings, WARCWriterPoolSettingsData add recordIDGenerator setting, impl class for testing/adhoc use * WARCWriterTest adapt to new constructors, settings-object * Generator, GeneratorFactory removed in favor of RecordIDGenerator interface * RecordIDGenerator common interface for classes that can provide WARC record IDs * UUIDGenerator, UUIDGeneratorTest adapt to derive from RecordIDGenerator avoid throwing exceptions * ARCWriterPoolTest, ARCWriterTest adapt to new constructors, settings-object * WARCWriterProcessor do checkSize file-rollover here, so related records don't span WARCs calculate processor's totalBytesWritten via position offset changes (as before stats-additions) delegate record-ID generation * WriterPoolProcessor add frequentFlushes and writeBufferSize from new WARCWriterPoolSettings interface gojomo 2011-01-14 23:26:31 +00:00
• c5717f80d4 * DisposableStoredSortedMap better class comment, error handling gojomo 2011-01-14 22:09:21 +00:00
• df1d064598 * StoredQueue.java protect isEmpty() like size() for access after backing BDB closed gojomo 2011-01-14 22:07:01 +00:00
• 402b9a45a6 * RobotstxtTest add test case for robots.txt with extraneous leading whitespace and missing blank lines gojomo 2011-01-12 23:45:49 +00:00
• 898689ab38 * WriterPoolMember.java restore void return value of write(*) methods * WriterPoolMember.java do not expect return from write(*) methods nlevitt 2011-01-12 08:38:56 +00:00
• fd212669af [HER-1856] remove redundant profile-crawler-beans.cxml nlevitt 2011-01-12 01:11:36 +00:00
• ef9d2e8c0e [HER-1789] stats from WARCWriterProcessor * WriterPoolMember.java write(*), copyFrom() - return number of (uncompressed) bytes written, for convenience * WARCWriter.java tally by record type: number of records written, content bytes, total bytes, and (possibly compressed) size on disk * WARCWriterProcessor.java keep totals and write report to processors-report.txt nlevitt 2011-01-12 00:54:01 +00:00
• 5fe28f2ecc [HER-1858] H3: frontier futureUris (rescheduling) unreliable for otherwise empty frontier/queue * WorkQueue.java add noteExhausted() for emptied queues * WorkQueueFrontier.java add synchronization around checkFutures() activity use noteExhausted when queue is totally empty gojomo 2011-01-12 00:21:40 +00:00
• 9e75ff0019 * GenerationFileHandler disable flush-minimizing optimization until its effect on watching logs in web UI can be minimized gojomo 2011-01-12 00:18:28 +00:00
• 463d295c24 * SurtPrefixedDecideRule.java buildSurtPrefixSet() - close the reader obtained from surtsSource when we're finished with it nlevitt 2011-01-11 20:46:15 +00:00
• 36515bdbf6 Part of HER-1857. Rename textSource->surtsSource, and make old surtsSourceFile a facade for surtsSource. nlevitt 2011-01-11 20:42:05 +00:00
• 64a25031e4 * engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml synch with engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml nlevitt 2011-01-09 21:33:37 +00:00
• 4ee1e09fd3 * SurtPrefixedDecideRule.java new property textSource, an org.archive.io.ReadSource, allowing specification inline in cxml as well as in external file * profile-crawler-beans.cxml commented out clauses demonstrating use of textSource nlevitt 2011-01-09 21:10:37 +00:00
• 8b510facc5 * CrawlerLoggerModule.java setupSimpleLog() - use existing utility method for timestamp format nlevitt 2011-01-07 23:34:08 +00:00
• 14f4d01be6 * Scoper.java use new logToFile paradigm a la DecideRuleSequence nlevitt 2011-01-07 02:38:11 +00:00
• dd439dac3b Fixes and improvements to to DecideRuleSequence/logToFile. * DecideRuleSequence.java - use Spring Lifecycle start() to initialize logToFile - inject logger module as SimpleFileLoggerProvider, since we're in heritrix-modules and don't have access to CrawlerLoggerModule from heritrix-engine * CrawlerLoggerModule.java - implement SimpleFileLoggerProvider - setupSimpleLog() - use 'T' instead of '+' between date and time in timestamp * SimpleFileLoggerProvider.java new interface with one method, setupSimpleLog() * Scoper.java do not call scope.start()/scope.stop(), these are handled using Spring Lifecycle now * DecideRule.java remove unused, unneeded start()/stop() nlevitt 2011-01-07 02:09:31 +00:00
• bdd7f7e1ab Fix exceptions at launch, post-crawl-finish * KryoBinding.java always default to registrationOptional, for now * StatisticsTracker.java explicitly set valueClass of sourceDistribution to ConcurrentHashMap * StoredQueue.java answer 0 for size() when backing DB closed gojomo 2011-01-07 01:24:38 +00:00
• 93cf655bed Add logToFile support to DecideRuleSequence. File is logs/{spring-bean-id}.log, format: [timestamp] [decisive-rule-num] [decisive-rule-class] [decision] [uri] * Scoper.java start(),stop() - call scope.start()/scope.stop() * CrawlerLoggerModule.java setupSimpleLog() - new logToFile paradigm * DecideRule.java start(),stop() - noop * DecideRuleSequence.java logToFile support * profile-crawler-beans.cxml <!-- <property name="logToFile" /> --> nlevitt 2011-01-06 18:39:19 +00:00
• 641f193e17 * ExtractorXML.java shouldExtract(CrawlUri) - Avoid java.lang.IndexOutOfBoundsException checking for xml preamble in brief/empty url content. nlevitt 2011-01-03 22:41:00 +00:00
• ea49e55c89 [HER-1850] IllegalStateException from WorkQueueFrontier.wakeQueues in BlockIterator * WorkQueueFrontier.java double in-memory snoozedClassQueues size protect all adds/removes to snoozedOverflow in synchronized blocks gojomo 2010-12-09 22:28:20 +00:00
• c1b8138d4d [HER-1762] H3 performance tuning Charset lookup sync de-bottlenecking * org/apache/commons/httpclient/EncodingUtil.java use explicit Charset instances or cached to minimize lookups * LaxURLCodec, BdbMultipleWorkQueues use explicit Charset instances gojomo 2010-12-06 19:47:49 +00:00
• 7678326121 remove obsolete renamed classes gojomo 2010-12-06 19:43:16 +00:00
• 35e015698c [HER-1847] DocumentLengthT(h)resholdDecideRule spelling, naming * (Not)ExceedsDocumentLengthT(h)resholdDecideRule improve comments, rename to: ResourceLongerThanDecideRule ResourceNoLongerThanDecideRule gojomo 2010-12-06 18:39:19 +00:00
• 2546bf5a01 correct misspelling of 'Threshold' gojomo 2010-12-06 18:13:39 +00:00
• f9a59f56a0 * .classpath reference to guava library gojomo 2010-12-03 01:30:28 +00:00
• e3181ffd37 * StoredQueue.java simplify to reduce synchronization intricacy; possibly reduce contention gojomo 2010-12-03 01:29:49 +00:00
• 85e4123b3c * TopNSet refactor to narrow synchronized areas gojomo 2010-12-03 01:03:41 +00:00
• a5256f38ee * commons/pom.xml dependency on guava java libraries gojomo 2010-12-03 00:59:32 +00:00
• 0500cdc8c1 * FetchStatusDecideRule.java eliminate unused gojomo 2010-12-03 00:40:09 +00:00
• 72682be382 * org.apache.commons.httpclient.util.EncodingUtil added in prep for Chaset synchronization bottleneck workarounds gojomo 2010-11-23 02:00:38 +00:00
• 1c8f9374f6 [HER-1845] recovery process generates duplicated F+ entries in new recovery log * AbstractFrontier remove redundant logging gojomo 2010-11-23 01:25:17 +00:00
• e90eff9c84 [HER-1843] Possible int overflow in CheckpointService.java * CheckpointService Made checkpointIntervalMinutes setter accept a long instead of an int (was missed when the class variable was made a long). While it was non-harmful the way it was, this is more in line with Java conventions. kristinn_sig 2010-11-18 08:53:56 +00:00
• 675d71d557 [HER-1843] Possible int overflow in CheckpointService.java * CheckpointService Conversion from mintues to milliseconds is now done using all longs, preventing an int overflow. Removed errant semi-colon. kristinn_sig 2010-11-18 00:46:06 +00:00
• ba4b01261e [HER-1838] H3: job-not-found 404 not very clear * EngineApplication better message for simple 404 (contributed by Matt Warhaftig) gojomo 2010-11-16 23:17:37 +00:00
• 04656f56c8 More synchronization-bottleneck-minimizing * GenerationFileHandler override flush to only flush 1/100th of the time gojomo 2010-11-16 02:03:55 +00:00
• 600a4ca326 Fix selftest failures where empty 'profile' directory may not exist (eg git) * SelfTestBase ...and make a tmp empty directory instead gojomo 2010-11-16 01:01:31 +00:00
• c170390c0d Fix selftest failures where empty 'profile' directory may not exist (eg git) * SelfTestBase only copy profile directory if it exists gojomo 2010-11-16 00:58:13 +00:00
• cb7f96682b Remove synchronization on frontier methods; rely on safe queues, synchronization-per-WorkQueue instead * AbstractFrontier remove inbound/outbound queues; simplify managerThread perform findEligible/schedule/receive/finish immediately * WorkQueueFrontier eliminate holdQueues setting synchronize sendToQueue on target queue make findEligibleUri reentrant; rely on [Blocking|Stored]Queue thread-safety make arriving at front of ready trigger for 'active'/budget-session simplify inactiveQueues management simplify waking from overflow * WorkQueue redefine 'active' as in-budget-session rename 'held' as 'managed' synchronize major methods, relying on allQueues cache that operations on same intended queue use identical instance split over-budget test to isOverSessionBudget and isOverTotalBudget have toString show classKey * BdbFrontier checkpoint fixes: remember nextOrdinal, rotate frontier-recover log, reset queues on recovery consistencyCheck method for probing state during stress tests gojomo 2010-11-15 22:56:03 +00:00
• 881c829fe5 Checkpoint fixes/improvements * BdbModule store lengths in JDB manifest; limit extension of last JDB on recover * CrawlerJournal rotate frontier-recover on checkpoints like other logs gojomo 2010-11-15 22:37:58 +00:00
• c8074bdbf3 Improve ability to force-terminate crawls * RecordingInputStream.java check for interrupt on each socket-timeout * CrawlController.java on second requestCrawlStop, interrupt threads via ToePool.cleanup * ToePool.java adjust for new earlier/repeated cleanup * ToeThread.java better warnings/recovery on forced-interrupts gojomo 2010-11-15 22:26:36 +00:00
• 9fb8374b67 * ObjectIdentityBdbCache comment-out debug output gojomo 2010-11-15 21:05:29 +00:00
• f2edeea28b * StoredQueue.java added debugging info and possible fix for occasionally-seen peek-timing issue gojomo 2010-11-15 21:03:15 +00:00
• d4f5004b9c Lessen synchronization bottleneck on crawl.log * UriProcessingFormatter.java allow pre-caching of a specific LogRecord's formatted version, outside the synchronized publish() * GenerationFileHandler.java force a format before publish(), to benefit above (a small hit in other cases where it's redundant) gojomo 2010-11-15 21:01:37 +00:00
• 1501e1ddd1 * ScriptResource.java flush rather than close, avoiding console dump gojomo 2010-11-15 20:59:45 +00:00
• 6a2716ce00 Reduce synchronization bottlenecks * FrontierJournal, CrawlerJournal do barest minimum inside 'synchronized' method gojomo 2010-11-12 19:37:52 +00:00
• 6f67f9f2ba * StoredQueue improved debugging output for seldom-seen internal inconsistency gojomo 2010-11-12 19:35:35 +00:00
• f44610d79a Remove synchronization bottlenecks * HttpState remove not-strictly-necessary 'synchronized' guards gojomo 2010-11-12 19:34:47 +00:00
• ea797a99f5 remove unnecessary SuppressWarnings gojomo 2010-11-11 06:00:54 +00:00
• 301b943f03 Consolidate ModuleAttributeConstants into CoreAttributeConstants * (many) distinction between these two constant-collecting classes was fuzzy (to the point that they already referred to each other), and they were both in same subproject/package already as well; so, merged constants into the larger, older class gojomo 2010-11-09 23:47:37 +00:00
• 6452013af7 Frontier throughput experiments & fixed, continued * WorkQueueFrontier (findEligibleURI) avoid outbound.capacity-sensitive activation, which under a race created by recent changes led to infinite recursion here * AbstractFrontier (next) when nothing is immediately ready, try adding one-at-a-time to outbound, rather than fillOutbound() gojomo 2010-11-09 23:32:12 +00:00
• ad57749dea * WorkQueueFrontier.java debugging output for reported findEligibleURI deep-recursion gojomo 2010-11-08 23:45:19 +00:00
• b5eb4f3c1e * AbstractFrontier reduce size of inbound now that many threads will opportunistically clear it gojomo 2010-11-08 23:44:43 +00:00
• 484afeeea4 Avoid PermGen OOMEs using serailization constructor magic * AutoKryo cache constructors created to deserialize classes that otherwise lack the right no-arg constructors gojomo 2010-11-08 23:43:42 +00:00
• 3dc5f04c06 Avoid deadlock on terminate() of already-paused/stopping crawl * AbstractFrontier (managementTasks) don't synchronize around possibly-blocking take(), only around process() gojomo 2010-11-08 21:06:58 +00:00
• 3be9941f76 Resolve deadlock seen after latest changes * AbstractFrontier (drainInbound) don't assume current thread will be able to take() full batch; poll() and exit early if other threads have drained queue first gojomo 2010-11-08 18:54:03 +00:00
• f12e940edd Discard 'CredentialAvatar' -- no longer needed with Spring settings -- use Credential instances directly gojomo 2010-11-06 03:45:00 +00:00
• e6c0f71175 * ObjectIdentityBdbCache debugging output -- report low-mem ref-clearing gojomo 2010-11-06 02:25:22 +00:00
• c7398ce463 Experiment to work-around Frontier.managerThread bottleneck * AbstractFrontier, WorkQueueFrontier synchronize around all InEvent/findEligibleURI actions, so that drainInbound and fillOutbound may be called outside managerThread whenever a ToeThread would block on enqueue() or next(), try the appropriate catch-up method before blocking remove assertions no longer true with activity happening outside managerThread gojomo 2010-11-06 02:14:39 +00:00
• 7fdb7cb384 tests cleanup * UURI return to custom-serialization rather than externalization * CachedBdbMap remove obsolete class * StripSessionCFIDs, StripSessionIDs update patterns to restore intended case-insensitivity * StripSessionCFIDsTest put expected, actual in expected order gojomo 2010-11-06 01:26:20 +00:00
• 54e7e28fa6 remove stray imports breaking build gojomo 2010-11-05 23:31:36 +00:00
• c56eb5d343 Experiment to assist large seed-loads * TextSeedModule.java in large seed loads, try to force finalization, just in case it's lagging gojomo 2010-11-05 23:27:43 +00:00
• 0627a67dce Minimize transient garbage volume by optimized serialization based on the 'Kryo' library * AutoKryo extension of Kryo to allow classes to control their own registration, trigger registration of associated classes, and deserialize classes without no-arg constructors * KryoBinding binding for use with BDB that uses AutoKryo serialization for a 2X-4X reduction in byte[] size * UURI improved serialization via Externalizable and Kryo's CustomSerialization methods * BdbModule discard deprecated CachedBDBMap option (getObjectCache) extend with both declaredClass and valueClass (for when map values are specializations of the declared type, as with frontier.allQueues) adjust type declarations * ObjectIdentityBdbCache use KryoBinding rather than SerialBinding * CachedBdbMapTest discarded * BdbFrontier, BdbServerCache, StatisticsTracker adjust type declarations, objectCache creation * BdbMultipleWorkQueues use KryoBinding rather than (Recycling)SerialBinding * BdbWorkQueue, CrawlServer, CrawlHost, CrawlURI add autoregister support * LinkContext public for kryo registration gojomo 2010-11-05 23:23:58 +00:00
• f4a01f1155 remove outdated adler32 workaround gojomo 2010-11-05 22:42:27 +00:00
• 4b7ada3930 [HER-1839] Upgrade BDB to 4.1.6; ensure HER-1628 really fixed * commons/pom.xml, .classpath update BDB jar to 4.1.6 gojomo 2010-11-05 22:33:47 +00:00
• 6164939fc4 Minimize transient garbage creation identified by allocation profiling * CrawlURI use ArrayList and indexed access rather than LinkedList and iterator instances for overlays gojomo 2010-11-05 22:32:05 +00:00
• f4d69b37da Minimize transient garbage creation identified by allocation profiling * UURIFactory, canonicalize/**, PathologicalPathDecideRule, ExtractorHTML use recycled matchers via TextUtils * CrawlURI, KeyedProperties, OverlayContext, SheetOverlaysManager use ArrayList and indexed access rather than LinkedList and iterator instances gojomo 2010-11-05 21:34:53 +00:00
• acedca1788 Part of [HER-1836] let more than one extractor run on a single url - new setting "independentExtractors" - when enabled, extractors run regardless of whether other extractors have run * profile-crawler-beans.cxml, AbstractFrontier.java, ExtractorParameters.java, Extractor.java new setting independentExtractors * ContentExtractor.java shouldProcess() - respect independentExtractors nlevitt 2010-10-26 23:49:42 +00:00
• dc056d73f4 [HER-1832] URI without fetch-attempt causes either RuntimeException (H1) or nonsensical history info (H3) in FetchHistoryProcessor * FetchHistoryProcessor.java skip history-storage if no evidence of fetch-attempt (A_FETCH_START_TIME data) gojomo 2010-10-26 00:32:18 +00:00
• 3a362002c7 [HER-1833] many crawls want to suppress DNS-server's provision of a default local domain * FetchDNS.java append '.' to domain name for lookup, anchoring it as FQDN gojomo 2010-10-25 23:50:12 +00:00
• d392341141 * ExtractorXML.java shouldExtract() - choose to extract also if content starts with <?xml ... nlevitt 2010-10-19 02:53:07 +00:00
• b75f604bf9 [HER-1836] let more than one extractor run on a single url? * ContentExtractor.java shouldProcess() - do not check uri.hasBeenLinkExtracted() * CrawlURI.java update javadoc comments on hasLinkBeenExtracted() nlevitt 2010-10-19 02:08:54 +00:00
• d72e6cd1d0 [HER-1830] H3: improve sheet examples in default configuration CXML * **/profile-crawler-beans.xml only 1 example SurPrefixesSheetAssociation gojomo 2010-10-01 00:16:23 +00:00
• 13d8c4f851 [HER-1830] H3: improve sheet examples in default configuration CXML * **/profile-crawler-beans.xml example SurPrefixesSheetAssociation gojomo 2010-09-30 23:01:25 +00:00
• a757140cf6 [HER-1831] change version numbering convention to be less confusing * **/pom.xml change 3.1.1-SNAPSHOT to 3.0.1-SNAPSHOT gojomo 2010-09-30 19:48:35 +00:00
• f1cd7dd527 [HER-1827] Have Frontier report include top X longest queues * WorkQueueFrontier.java improve effective precision of top-n-queues by also updating on dequeues gojomo 2010-09-28 22:42:22 +00:00
• 8f28d493a2 [HER-1827] Have Frontier report include top X longest queues * Histotable.java expose static method useful for frequency-reporting * TopNSet.java, TopNSet.java improve handling of case where updates may decrease frequency counts add convenience methods for common needs * WorkQueueFrontier.java replace single longestActiveQueue with largestQueues TopNSet, of configurable size make max-queues-to-report-per-category configurable gojomo 2010-09-28 22:12:28 +00:00
• 74d3fb54ce HER-1828 ExtractorXML misses https urls * ExtractorXml.java adjust XML_URI_EXTRACTOR regex to also look for https urls nlevitt 2010-09-25 07:48:46 +00:00
• de32bdb053 [HER-1825] H3: make default base queue precedence a more useful default * BaseQueuePrecedencePolicy.java make default '3' (leaving space for 2 'higher') rather than '1' gojomo 2010-09-24 04:34:20 +00:00
• 63c8de4045 fix for [HER-1824] ExtractorXML doesn't properly unescape * ExtractorXML.java processXML() - unescape extracted urls with commons-lang unescapeXml() nlevitt 2010-09-23 02:03:48 +00:00
• 7909954b8f [HER-1823] H3: adding SheetAssociations mid-crawl won't affect already queued CrawlURIs (sheet liveness) * SheetsOverlayManager.java (applyOverlaysTo) clear previous info so redundant sheets don't accrete gojomo 2010-09-21 11:07:33 +00:00
• 1ce23ecbcd [HER-747] Ensure 'retired' status for queues is operator-controllable [HER-1768] Changing queue totalBudget not retiring queue as expected * CrawlURI.java (includesRetireDirective) internalize to CrawlURI * AbstractFrontier.java allow re-enqueuing of untried URI with retire-driective clean up terminology/method-names * WorkQueue.java, WorkQueueFrontier.java, BdbFrontier.java adjust budget/sessionBalance handling to always count up, always consult latest values, always refresh values from latest URI (processFinish) reorganize to reduce redundancy * DispositionProcessor.java offer forceRetire setting * QuotaEnforcer.java set failed-status *or* forceRetire, not both [HER-1823] H3: adding SheetAssociations mid-crawl won't affect already queued CrawlURIs (sheet liveness) * SheetsOverlayManager.java (applyOverlaysTo) rename, unify, reapply whenever called to ensure liveness * CandidatesProcessor.java adjust to rename gojomo 2010-09-21 10:55:07 +00:00
• c701cd391a [HER-1822] H3: bean browser in non-paused crawl sometimes raises exceptions, doesn't work * JobRelatedResource.java add more property names to blacklist route all relevant requests through new utilitiy getPropertyDescriptors method that patches instances as necessary to suppress problems gojomo 2010-09-21 10:28:29 +00:00
• 966646871c [HER-1727] (W)ARC filename uniqueness guarantees: adminport option, process_id option(?), JVM-global-repeat-suppression * ArchiveUtilsTest.java tests for new HER-1727 guaranteed-no-repeat-timestamp functionality gojomo 2010-09-20 13:51:13 +00:00
• 153e709fe5 * WriterPoolProcessor.java eliminate nonsensical type check gojomo 2010-09-15 03:52:11 +00:00