Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#750

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 2d3f569f15 * InterruptibleCharSequenceTest.java better output to diagnose intermittent failure gojomo 2009-10-06 03:58:22 +00:00
• 3654e330cc * (many) harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits) gojomo 2009-10-06 03:47:38 +00:00
• d86f5f7403 * (many) harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits) gojomo 2009-10-06 03:45:44 +00:00
• dbc1c1b411 * KeyWordProcessor.java update shouldProcess() to be based on via CrawlURI's type gojomo 2009-10-06 02:28:47 +00:00
• 4ce8aef47a Remove obsolete test class gojomo 2009-10-06 01:40:22 +00:00
• 6769cdb2dc Fix inadvertent build-breaking upward dependency from modules to engine * modules//ProcessorChain.java send updates to ChainStatusReceiver interface * engine//ToeThread.java implement ChainStatusReceiver interface gojomo 2009-10-06 01:01:32 +00:00
• 92da7ab8cc Incremental check-in with DOM search and XPath interrogation of scope node from existing order XML. szznax 2009-10-06 00:36:00 +00:00
• 047a157fe8 [HER-1605] Refactor parameterizable policies out of frontier * selftest-crawler.beans.cxml, FormAuthSelfTest.java, Precedence[134]SelfTest.java update self-tests for new chains refactoring gojomo 2009-10-05 23:44:30 +00:00
• 13c9cc1915 [HER-1605] Refactor parameterizable policies out of frontier * defaults.xml update bundled profile for new chains refactoring * CrawlController.java move to three processor-chains rather than one * Frontier.java (loadSeeds) removed * ToeThread.java delegate most processing-loop to FetchChain and DispositionChain * AbstractFrontier.java, WorkQueueFrontier.java move policies/calculation out to processors * CandidatesProcessor.java new processor for DispositionChain that runs every outlink through CandidateChain * CrawlStateUpdater.java -> DispositionProcessor.java rename, expand to prep CrawlURI for frontier * FrontierScheduler.java deleted; use CandidatesProcessor/CandidateChain * LinksScoper.java deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper (only temporarily retained for ease of comparison) * CandidateScoper.java simple single-URI scope-testing for CandidateChain * FrontierPreparer.java precalculate all frontier-policies in CandidateChain, before scheduling * PreconditionEnforcer.java ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere * ProcessorsReport.java update for 3-chains of Processors * SheetOverlaysManager.java (applyOverridesTo) moved here for broader use * CandidateChain.java, FetchChain.java, DispositionChain.java role-specific subclasses of ProcessorChain (suitable for type-based autowiring) * CrawlURI.java new fields/accessors of use to new chains/frontier * PostProcessor.java deleted; skip-to-'postprocessing' is now skip-to-end-of-chain * ProcessorChain.java take-on control loop formerly in ToeThread * ProcessResult.java absorb ProcessStatus eliminate problematic STUCK result gojomo 2009-10-05 23:41:59 +00:00
• 30c729edb3 More ProcessorURI -> CrawlURI updates gojomo 2009-10-05 23:25:12 +00:00
• b7d1a79b4d More ProcessorURI -> CrawlURI updates gojomo 2009-10-05 23:08:12 +00:00
• 8f7ad4eae4 work for [HER-1605] Refactor parameterizable policies out of frontier * QueueAssignmentPolicy.java, URIAuthorityBasedQueueAssignmentPolicy.java move 'force-queue-assignment' setting from frontier to policy gojomo 2009-10-05 23:04:37 +00:00
• 10275d7fa1 More ProcessorURI -> CrawlURI updates gojomo 2009-10-05 23:01:40 +00:00
• c7ed3e775d Misc * ArchiveUtils.java (checkContinue) utility method relocated from ToeThread for converting interrupt status to exception * ClassKeyMatchesRegExpDecideRule.java properly set up crawlController property for auto-wiring gojomo 2009-10-05 22:59:46 +00:00
• 16538f175c Cleanup, refactoring * (many) update license notice to Apache where appropriate eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package) other comment/warning/unused code cleanup gojomo 2009-10-05 22:49:19 +00:00
• c04c5dbdbc Improve test robustness * Engine.java wait another second before testing for job wrap-up gojomo 2009-10-05 22:25:44 +00:00
• 62690268b7 Initial checkin for [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml szznax 2009-10-05 20:00:56 +00:00
• ba5bc1a20c Fix unit tests (carried over from H1) * WARCWriterTest.java (testSpaceInUrl),(testTabInURL): don't expect exception; expect nothing-written szznax 2009-10-05 18:35:06 +00:00
• 86974e2e1d Cleanup post-fix * ObjectIdentityBdbCache.java with late ref-enqueuing confirmed as cause of problems, cleaning out unnecessary steps, improving comments gojomo 2009-10-04 06:15:35 +00:00
• 75676ac905 Attempt at fixing build-box-only warnings * ObjectIdentityBdbCache.java explicitly page-out lingering entry; temporary System.err logging to see if problem is momentary lag in RefQueue-appearance gojomo 2009-10-04 05:14:59 +00:00
• c05044d51b Eliminate double-database close log messages * BdbModule.java split openDatabase() to openManagedDatabase() (auto-closed) and plain openDatabase() (caller-closes) set TempStoredSortedMap to use unmanaged openDatabase * (others) use openManagedDatabase() gojomo 2009-10-04 05:06:14 +00:00
• c3914ea943 Attempt at fixing build-box-only warnings * ObjectIdentityBdbCache.java add earlier check/better logging for error state gojomo 2009-10-04 03:31:45 +00:00
• 839e895a8d Attempt at fixing build-box-only warnings * ObjectIdentityBdbCache.java synchronize critical sections on memMap, diskMap rather than enclosing object gojomo 2009-10-03 23:55:59 +00:00
• 3cb6b59b8c [HER-769] Support case where millions of seeds. * BdbModule.java add temp-stored-map service method remove unused secondary-db support * TempStoredSortedMap.java StoredSortedMap that can destroy its underlying database after temporary use * StatisticsTracker.java update various sorted-by-decreasing-frequency methods to use temp StoredMaps, with duplicate keys that are negative counts * (Multiple)Report.java update to use new duplicate-keyed frequency maps gojomo 2009-10-03 23:28:11 +00:00
• 97778493e2 As part of fix for [HER-1671] WARCWriterProcessor error when writing problematic (illegal control character?) outlink catch IllegalArgumentException in ANVLRecord.addLabelValue() to allow WARCWriter to continue processing, and move WARCWriter.createRecordHeader() into its own try-catch block to with logging prevent fouling record when problems occur creating header. * ANVLRecord.java catch IllegalArgumentException in addLabelValue and log warning to give WARCWriter to continue processing * WARCWriter.java move createRecordHeader() into its own try-catch block to log severe problem writing record without fouling the WARC, and have baseCharacterCheck() throw IllegalArgumentException in accord with ANVLRecord.addLabelValue * WARCWriterTest.java throw IllegalArgumentException in accord with WARCWriter.createRecordHeader() szznax 2009-10-03 00:41:59 +00:00
• 03df2a9a99 As part of fix for [HER-1671] WARCWriterProcessor error when writing problematic (illegal control character?) outlink - percent encode control-characters in 'userinfo' part of URI as we do for uriPath and uriQuery, so we don't raise exception when adding outlinks to metadata record * UURIFactory.java ensureMinimalEscaping() on 'userinfo' in UURIFactory.fixupAuthority() szznax 2009-10-02 01:39:44 +00:00
• 677a816db4 Fix for [HER-1681] FetchFTP module doesn't set content-begin mark correctly. Bug reported and fix suggested by Tomas Ukkonen. * FetchFTP.java saveToRecorder(): call recorder.markContentBegin() after recorder.outputWrap() szznax 2009-10-01 02:21:17 +00:00
• fef7a93bad As part of fix for [HER-1634] verify/port/update latest WARC implementation in 1.14.3/heritrix2 to latest ISO document, heritrix3 - in warcinfo record, "robots" field should be "robots: classic" rather than "robots: CLASSIC" * WARCWriterProcessor.java made robots entry in metadata lower-case szznax 2009-09-30 17:06:53 +00:00
• 3e65e09927 As part of fix for [HER-1634] verify/port/update latest WARC implementation in 1.14.3/heritrix2 to latest ISO document, heritrix3 - update REVISIT profile URIs per WARC spec (May 2009) * WARCConstants.java PROFILE_REVISIT_IDENTICAL_DIGEST = "http://netpreserve.org/warc/1.0/revisit/identical-payload-digest" PROFILE_REVISIT_NOT_MODIFIED = "http://netpreserve.org/warc/1.0/revisit/server-not-modified" szznax 2009-09-29 21:52:33 +00:00
• 1951930842 Improve test robustness in face of unpredictable GC * ObjectIdentityBdbCache.java (pageOutStaleEntries) package-protection for unit-test visibility * ObjectIdentityBdbCacheTest.java explicit pageOutStaleEntries, better assert-fail message gojomo 2009-09-29 20:20:14 +00:00
• 5a0fe30366 Quiet JUnite test issues * BdbModuleTest.java, CachedBdbMapTest.java use putIfAbsent() in preference to put() (which logs warning) * BdbModule.java register big-cache DBs in implementation-specific methods, so always auto-closed on env close gojomo 2009-09-29 19:22:23 +00:00
• 99020c4981 [HER-1672] parallelQueues with overlays - prerequisites not landing in right queues (causes massive DNS-fetch repeats) * PreconditionEnforcer.java, CrawlServer.java in testing found some ordering issues affecting robots-scheduling if parallelQueues>1 reordered/refactored to put more responsibility in CrawlServer to resolve gojomo 2009-09-29 00:47:38 +00:00
• 1cfcf8e0a5 As part of fix for [HER-1634] verify/port/update latest WARC implementation in $ fold in H3 updates to http transaction constant for revisit records. * CoreAttributeConstants.java removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$ * ModelAttributeConstants.java removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$ * FetchHistoryProcessor.java check for HTTP transaction using CrawlURI.isHttpTransaction() use HttpMethod in saveHeader() instead of HttpMethodBase * WARCWriterProcessor.java check for HTTP transaction using CrawlURI.isHttpTransaction() use HttpMethod in saveHeader() instead of HttpMethodBase szznax 2009-09-29 00:41:57 +00:00
• c2b71a6f95 * WriterPoolProcessor.java fix serialization test: make servercache field transient gojomo 2009-09-26 04:41:09 +00:00
• 1c1a6501ba [HER-1672] parallelQueues with overlays - prerequisites not landing in right queues (causes massive DNS-fetch repeats) * CrawlURI.java (getPolicyBasisUURI) return either own UURI, or -- for prereqs -- via UURI, as basis for overlays/policy calculations * URIAuthorityBasedQueueAssignmentPolicy.java use getPolicyBasisUURI for queue-name decisions * SheetOverlaysManager.java use getPolicyBasisUURI for surt-based overlay decisions gojomo 2009-09-26 04:18:25 +00:00
• 7bd6017448 * ObjectIdentityBdbCache.java avoid error if size() called after close() gojomo 2009-09-26 04:12:23 +00:00
• 71aa2ad51d * StatisticsTracker.java, HostsReport.java rename getReverseSortedHostsDistribution() so it isn't misinterpreted as beans property gojomo 2009-09-26 04:09:20 +00:00
• a285a65576 * CachedBdbMap.java remove superfluous (build-breaking) imports gojomo 2009-09-25 22:50:43 +00:00
• 7521375128 [HER-1677] threads stuck in CachedBdbMap.get/_getMem alternate simplified implementation of our object-cache need * ObjectIdentityCache.java new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache * Supplier.java trivial interface for deferred-provision of new instance * CachedBdbMap.java implement ObjectIdentityCache, mainly via getOrUse() * ObjectIdentityMemCache.java trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation * ObjectIdentityBdbCache.java BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap * BdbModule.java refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle * EnhancedEnvironment.java convenience test-environment method * (others) update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap gojomo 2009-09-25 21:18:14 +00:00
• e7ab73dd5d [HER-1677] threads stuck in CachedBdbMap.get/_getMem alternate simplified implementation of our object-cache need * ObjectIdentityCache.java new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache * Supplier.java trivial interface for deferred-provision of new instance * CachedBdbMap.java implement ObjectIdentityCache, mainly via getOrUse() * ObjectIdentityMemCache.java trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation * ObjectIdentityBdbCache.java BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap * BdbModule.java refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle * EnhancedEnvironment.java convenience test-environment method * (others) update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap gojomo 2009-09-25 21:16:53 +00:00
• 78dd7f2086 Fix for [HER-1648] harmonize declared WARC versions in protocol, warcinfo record metadata WARConstants.java update WARC_VERSION=1.0 according to ISO 28500:2009 as of May 2009 WARCWriterProcessor.java update format and conformsTo fields according to ISO 28500:2009, as of May 2009, as described at http://bibnum.bnf.fr/WARC/, latest draft as of November 2008 format: WARC File Format 1.0 conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf szznax 2009-09-23 18:42:25 +00:00
• 2bd9334627 [HER-1677] threads stuck in CachedBdbMap.get/_getMem * CachedBdbMap.java make all WARNING logs include stacks add WARNINGs if memMap.remove() has no expected effect verify reported expunction, WARN and try to fix if memMap expunction not evident adjust indentation/bracketing gojomo 2009-09-22 21:16:21 +00:00
• 142c72fc04 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) * SurtPrefixedDecideRule.java pass only portion after directive ('+' or '-') to SurtPrefixSet * SurtPrefixSet.java better comment/param name gojomo 2009-09-16 01:05:23 +00:00
• ee1c9e2ce4 [HER-1607] handle persist state when crawl starts up * PersistProcessor.java add readOnly option for environment only opened for copying-from * PersistLoadProcessor.java make start() safe to call redundantly (per Lifecycle contract) gojomo 2009-09-12 00:22:06 +00:00
• 163dcb7826 [HER-1607] handle persist state when crawl starts up * BdbModule.java fail-fast if not started properly; stop with fewer errors in problem situations * CrawlController.java continue past runtimeException on stop * CrawlJob.java less redundant error-reporting * AbstractFrontier.java consider PAUSE or FINISH valid targets after unexpected exception * StatisticsTracker.java avoid promoting runtimeException in report-generating up * PersistOnlineProcessor.java delegate DB-closing to BdbModule to avoid double-close gojomo 2009-09-11 02:26:04 +00:00
• d5e088ed6a * WorkQueueFrontier.java improve frontier report: move longest-queue to top, add labels/counters to long lists of queue groups gojomo 2009-09-10 22:40:30 +00:00
• d955e31963 [HER-1676] improve bundled profiles to match best-practice, be self-documenting * defaults.xml add REJECTing SurtPrefixedDecideRule and MatchesListRegExpDecideRule gojomo 2009-09-10 22:39:26 +00:00
• 1c451021d1 * PagedRepresentation.java make 'end' link go to true end, not end-as-of-moment-of-rendering gojomo 2009-09-10 21:57:21 +00:00
• 1e3ec9fe35 * SheetAssociation.java initialize list of names for easier in-crawl construction/insertion gojomo 2009-09-10 21:56:44 +00:00
• a564b7a867 [HER-1675] FetchStats/CrawlSubstats not tallying fetchResponses correctly; as a result QuotaEnforcer doesn't work * FetchStats.java increment fetchResponses appropriately gojomo 2009-09-09 00:12:54 +00:00
• 0b10862c8d Make during-crawl sheets/overlay changes easier * SheetOverlaysManager.java convenience methods for during-crawl changes: (getOrCreateSheet) (deleteSheet) (putSheetOverlay) (removeSheetOverlay) * Sheet.java improved comments gojomo 2009-09-03 06:33:25 +00:00
• c88744ab27 Improve bean-browse UI * BeanBrowseResource.java coerce edits to type of value they're replacing; makes for safer mid-crawl edits of values in maps * JobRelatedResource.java improve rendering/clickability of map, indexed objects gojomo 2009-09-03 03:32:07 +00:00
• f26803bd4f UI tweaks * JobResource.java make jaob main page begin with job title -- easier to distinguish tabs with different jobs in them gojomo 2009-09-03 03:27:05 +00:00
• 635d27b390 More action directory capabilities * ActionDirectory.java execute scripts found in action directory, determining language by their suffix gojomo 2009-09-03 03:15:54 +00:00
• e03329a035 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) * SurtPrefixedDecideRule.java allow seeds to be null (as when rule is intentionally completely independent of seeds; may also need to disqualify rule from Spring autowiring in such a case) gojomo 2009-09-02 23:24:30 +00:00
• 21d90f2a3a UI annoyances * ReportGenResource.java ensure redirect after regeneration of report doesn't show stale cached version gojomo 2009-09-02 23:15:18 +00:00
• d4eabb9631 Improve log display * JobResource.java form proper alerts-rev-tail link for any file ending ".log", use rev-tail-paged view gojomo 2009-09-01 22:42:52 +00:00
• 81a658daa3 Update BDB-JE to 3.3.87 * commons/pom.xml, .classpath update version reference gojomo 2009-09-01 22:40:21 +00:00
• f0421e1f13 * ActionDirectory.java also include 'Fi ' lines -- for recoveries of recoveries gojomo 2009-08-28 17:08:59 +00:00
• ad937c58e5 [HER-1662] use fully qualified hostname * FetchHTTP.java replace one more InetAddress.getHostName() with InetAddress.getCanonicalHostName() (for consistency) gojomo 2009-08-28 06:56:20 +00:00
• d253ff1ace [HER-1662] use fully qualified hostname * WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java, InetAddressUtil.java replace all InetAddress.getHostName() with InetAddress.getCanonicalHostName() (for consistency) gojomo 2009-08-27 23:35:07 +00:00
• 211cd7c659 [HER-1669] slow crawl/contention n WARCWriterProcessor * CrawlController.java increase default size of in-memory recording buffers outBuffer (requests): 4K -> 16K inBuffer (responses): 64K -> 512K gojomo 2009-08-27 23:13:11 +00:00
• 8ba6d47a93 Apache License gojomo 2009-08-27 23:10:38 +00:00
• dc89ce0d95 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) * SurtPrefixSet.java refactor to allow add-directives under outside control * SurtPrefixedDecideRule.java extend nonseedLine() mechanism to trigger off '+' or '-' depending on decision being ACCEPT or REJECT gojomo 2009-08-27 01:40:26 +00:00
• 3292c8b7f6 Apache License, import cleanup gojomo 2009-08-27 01:34:11 +00:00
• 608a3ddc8b [HER-1607] handle persist state when crawl starts up * PersistProcessor.java, PersistLoadProcessor.java port [HER-1586] preloadSource setting/refactoring to H3 * IoUtils.java (getBufferedReader) utility method * BdbModule.java explicit passthrough of settable deferredWrite property gojomo 2009-08-27 01:32:08 +00:00
• beb858369b Apache License gojomo 2009-08-27 01:27:28 +00:00
• f7af3e016b [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? * ReplayCharSequence.java, return to ISO8859_1 as fallback-encoding gojomo 2009-08-27 00:20:37 +00:00
• 783ecfd9d7 [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? * ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java refactor to put fallback-encoding in single place gojomo 2009-08-27 00:12:15 +00:00
• 3f9ecc4dfa * PagedRepresentation.java fix negative estimated size error for log-tail viewing gojomo 2009-08-27 00:10:48 +00:00
• c07a70eb21 Fix SubElement IllegalArgumentException (via WarcWriterProcessor.writeMetadata) * SubElement.java accept zero-length strings as valid values * WarcWriterProcessor.java don't write 'via' or 'hopsFromSeed' unless non-blank gojomo 2009-08-19 21:15:40 +00:00
• 5c93d7bd28 Rename DecideResult.PASS to DecideResult.NONE for clarity * DecideResult.java + references PASS was confusable with ACCEPT; NONE better conveys "no decision" gojomo 2009-08-14 21:31:11 +00:00
• b78fce045b Fix selftests hanging due to pauseAtStart default true * selftest-crawler-beans.cxml make selftests not pauseAtStart gojomo 2009-08-13 10:03:56 +00:00
• d654bb5651 [HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' * ActionDirectory.java support for .seeds, .recover, .include, .schedule, .force files * CrawlController.java change default for pause-at-start to true; ensure enters PAUSED rather than PAUSING * Frontier.java (importRecoverFormat) support method for .recover, .include, .schedule, .force additions (considerIncluded) change to accept CrawlURI * AbstractFrontier.java (importRecoverFormat) support method for .recover, .include, .schedule, .force additions * BdbFrontier.java remove too-strict assertion * FrontierJournal.java rename recover-log 'frontier.recover.gz' (so that '.recover' is a dot-suffix) * WorkQueueFrontier.java (considerIncluded) change to accept CrawlURI gojomo 2009-08-12 01:50:51 +00:00
• bba028ce21 * StatisticsTracker.java, SeedsReport.java remove redundant method gojomo 2009-08-12 01:35:19 +00:00
• a37a1846b7 Move getBufferedReader() -> IoUtils * CrawlerJournal.java, IoUtils.java move getBufferedReader to IoUtils * FrontierJournal.java add new 'included' tag for log lines ("Fi") * (others) update for relocated method gojomo 2009-08-11 02:17:20 +00:00
• 8b95e7fb36 CachedBdbMap cleanup - intended commit comment * CachedBdbMap.java discard code to open/manage own BDB Environments (and related constructor) cleanup comments/FIXMEs * BdbModule.java return CachedBdbMap, rather than more general ConcurrentMap, in case requester wants it * CachedBdbMapTest.java get instance via BdbModule (to manage env/classCatalog) gojomo 2009-08-11 02:10:28 +00:00
• 8d09ee38ae CachedBdbMap cleanup gojomo 2009-08-11 02:07:22 +00:00
• 023efe4aba [HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' * ActionDirectory.java initial implementation; handle '.seeds' files gojomo 2009-08-06 21:33:31 +00:00
• 0400d81ee5 support for... [HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' [HER-769] Support case where millions of seeds. * AbstractFrontier.java move sourceTagSeeds setting to SeedModule trigger and schedule seeds via SeedModule's announcements and SeedListener protocols * StatisticsTracker.java implement SeedListener, use processedSeedRecords as list of all seeds * ProcessorURI.java, DefaultProcessorURI.java (getURI) convenience accessor for plain URI string * SurtPrefixedDecideRule.java use seeds as prefixes via SeedListener notifications, rather than separate scan of seeds source text * SeedListener.java simplify to notifications of new seed, and nonseed text mixed with seeds (as with SURT directives) * SeedModule.java methods for SeedListener interface, announceSeeds, actOn(File) * TextSeedModule.java eliminate interators; add methods for announcing seeds and scanning seed text sources (including arbitrary Files) * PrefixSet.java, SurtPrefixSet.java derive from ConcurrentSkipListSet; eliminate now-unneeded extra synchronization and clone()s offer considerAsDirective() for direct application of directive Strings gojomo 2009-08-06 21:24:04 +00:00
• 4d06b239ee Push CrawlURI and related constants down to modules subproject/package (many) gojomo 2009-08-06 20:22:00 +00:00
• 0ebac2f115 * CachedBdbMapTest.java more timing/heap voodoo for test robustness gojomo 2009-08-06 20:11:04 +00:00
• 64d3fd212f [HER-1659] make default WARC size comply with spec; adjust default pool size for fewer odd-sized (W)ARCs * ArchiveFileConstants.java remove superfluous constant * WriterPool.java DEFAULT_MAX_ACTIVE -> 1 * WriterPoolMember.java update constant use; remove deprecated+unused methods * WriterPoolProcessor.java pull storePaths up from subclasses, but set default via subclass method set defaultMaxSize via subclass method * WARCWriterProcessor.java, ARCWriterProcessor.java (getDefaultMaxFileSize), (getDefaultStorePaths) as appropriate gojomo 2009-08-06 00:55:27 +00:00
• 272ae19961 Cleanup * CachedBdbMap.java remove debug output * TestUtils.java add info logging * CachedBdbMapTest.java make more robust against prior heap usage, different platforms gojomo 2009-08-04 06:55:31 +00:00
• 4f0961c371 Fix NPE * URIAuthorityBasedQueueAssignmentPolicy.java tolerate null getRawPathQuery response gojomo 2009-08-04 01:31:38 +00:00
• c196a1de77 [HER-1658] CachedBdbMaps not expunging as expected (especially StatisticsTracker.processedSeedsRecords) * CachedBdbMap.java do expunge on put(), replace() add low-memory-sensitive 'canary' to force expunge even if otherwise untriggered * CachedBdbMapTest.java add test of idle expunge in low-memory conditions * CrawlURI.java (getURI) direct access to String URI (don't reuse toString() functionally) * SeedRecord.java support for updating record with later/repeat report * StatisticsTracker.java remove put()s for processedSeedRecords, hostsLastFinished * BdbFrontier.java (getQueueFor) remove put, do in concurrent-compliant manner gojomo 2009-08-04 00:06:22 +00:00
• 1c984753f0 Work for [HER-1656] add multiple-queues-per-host ('parallelQueues') capabilities to QueueAssignmentPolicies * LongToIntConsistentHash.java + Test expand range/distribution tests correct problems with phony '-1' bucket being created/returned gojomo 2009-07-31 22:55:09 +00:00
• 5bb8596fab Work for [HER-1656] add multiple-queues-per-host ('parallelQueues') capabilities to QueueAssignmentPolicies * LongToIntConsistentHash.java + Test consistent-hashing utility class * URIAuthorityBasedQueueAssignmentPolicy.java shared superclass for the hostname and surtauthority QAPs setting for 'deferToPrevious' -- avoid changing assignments setting for 'parallelQueues' -- when > 1, consistent-hash URIs across that many separate queues (with numerical suffix) * SurtAuthorityQueueAssignmentPolicy.java, HostnameQueueAssignmentPolicy.java refactor to derive from URIAuthorityBasedQueueAssignmentPolicy * QueueAssignmentPolicy.java add Apache license * Hop.java add String representation for convenience gojomo 2009-07-28 23:02:11 +00:00
• d2a82228a3 TODO/FIXME comments where things don't seem quite right but aren't definitively buggy * CachedBdbMap.java reevaluate if clear() should include close() * CrawlURI.java investigate possible confusion in meaning of isPrerequisite() gojomo 2009-07-28 22:56:54 +00:00
• 5027fb9aa0 * AbstractFrontier.java, WrokQueueFrontier.java remove outdated comments; use isEmpty() gojomo 2009-07-28 22:47:01 +00:00
• 478e425b34 Work for [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? * FetchHTTP.java really disable the WARNING logging, now gojomo 2009-07-28 22:46:04 +00:00
• d8ed3822e7 compartmentalize IOExceptions * RecordingInputStream.java restore expected nulling of closed 'in' gojomo 2009-07-28 22:01:50 +00:00
• 597c910422 Work for [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? * FetchHTTP.java catch charset problems early and add as crawl.log annotation, fallback to default without raising log/alert/WARNING-level notice gojomo 2009-07-28 01:01:09 +00:00
• a2940bc23e compartmentalize IOExceptions * RecordingInputStream.java make sure one exception (intermittently seen on Windows re: 'mapped section') doesn't leave things in foul state for next try in same thread gojomo 2009-07-28 00:54:06 +00:00
• 4cd9eb2e41 Work for [HER-1654] suspicious Cookie bottlenecks: synchronization, growth * Cookie.java simple non-contending String compare (fixed similarly upstream in HttpClient; see https://cwiki.apache.org/jira/browse/HTTPCLIENT-645 ) gojomo 2009-07-28 00:48:36 +00:00
• 917dfc2bd3 Fix ConfigPath inconsistencies in reporting, relative paths, fixup-ordering * ConfigPathConfigurer.java work in two passes: one (BeanPostProcessor) collects all candidates for fixup, two (ApplicationListener) actually performs fixup suppress fixup of own base path, preventing circulare ref/StackOverflow * ConfigPath.java change sense of 'merge' to merge into receiver * SurtPrefixedDecideRule.java use ConfigFile, use 'merge' in assignment * CrawlerLoggerModule.java use merge in assignment * CrawlJob.java (getBeanpathTarget) utility method * JobResource.java respect unset ConfigPaths * ConfigPathEdito.java, ConfigFileEditor.java better comments, ConfigFile variant gojomo 2009-07-24 20:22:06 +00:00
• 60ebd6cfdb * CrawlController.java better handle case where crawl is discarded before launched gojomo 2009-07-24 00:27:57 +00:00
• c3cc7cc7e5 * JobResource.java complete rename of printLinked to printLinkedFile gojomo 2009-07-24 00:27:27 +00:00
• 0b577387af improve display of known ConfigPaths * ConfigPathConfigurer.java key by propPath * JobResource.java display by "propPath: assigned name" * WriterPoolProcessor.java fix typo gojomo 2009-07-24 00:06:34 +00:00
• 3d573a8d5f Resolve remaining broken links to logs outside job dir * JobResource.java refactor, improve printLinkedFile * EngineApplication.java remove debug output gojomo 2009-07-24 00:00:28 +00:00