Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#781/merge

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 99020c4981 [HER-1672] parallelQueues with overlays - prerequisites not landing in right queues (causes massive DNS-fetch repeats) * PreconditionEnforcer.java, CrawlServer.java in testing found some ordering issues affecting robots-scheduling if parallelQueues>1 reordered/refactored to put more responsibility in CrawlServer to resolve gojomo 2009-09-29 00:47:38 +00:00
• 1cfcf8e0a5 As part of fix for [HER-1634] verify/port/update latest WARC implementation in $ fold in H3 updates to http transaction constant for revisit records. * CoreAttributeConstants.java removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$ * ModelAttributeConstants.java removed A_HTTP_TRANSACTION, in favor checking CrawlURI method with isHttpTran$ * FetchHistoryProcessor.java check for HTTP transaction using CrawlURI.isHttpTransaction() use HttpMethod in saveHeader() instead of HttpMethodBase * WARCWriterProcessor.java check for HTTP transaction using CrawlURI.isHttpTransaction() use HttpMethod in saveHeader() instead of HttpMethodBase szznax 2009-09-29 00:41:57 +00:00
• c2b71a6f95 * WriterPoolProcessor.java fix serialization test: make servercache field transient gojomo 2009-09-26 04:41:09 +00:00
• 1c1a6501ba [HER-1672] parallelQueues with overlays - prerequisites not landing in right queues (causes massive DNS-fetch repeats) * CrawlURI.java (getPolicyBasisUURI) return either own UURI, or -- for prereqs -- via UURI, as basis for overlays/policy calculations * URIAuthorityBasedQueueAssignmentPolicy.java use getPolicyBasisUURI for queue-name decisions * SheetOverlaysManager.java use getPolicyBasisUURI for surt-based overlay decisions gojomo 2009-09-26 04:18:25 +00:00
• 7bd6017448 * ObjectIdentityBdbCache.java avoid error if size() called after close() gojomo 2009-09-26 04:12:23 +00:00
• 71aa2ad51d * StatisticsTracker.java, HostsReport.java rename getReverseSortedHostsDistribution() so it isn't misinterpreted as beans property gojomo 2009-09-26 04:09:20 +00:00
• a285a65576 * CachedBdbMap.java remove superfluous (build-breaking) imports gojomo 2009-09-25 22:50:43 +00:00
• 7521375128 [HER-1677] threads stuck in CachedBdbMap.get/_getMem alternate simplified implementation of our object-cache need * ObjectIdentityCache.java new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache * Supplier.java trivial interface for deferred-provision of new instance * CachedBdbMap.java implement ObjectIdentityCache, mainly via getOrUse() * ObjectIdentityMemCache.java trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation * ObjectIdentityBdbCache.java BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap * BdbModule.java refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle * EnhancedEnvironment.java convenience test-environment method * (others) update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap gojomo 2009-09-25 21:18:14 +00:00
• e7ab73dd5d [HER-1677] threads stuck in CachedBdbMap.get/_getMem alternate simplified implementation of our object-cache need * ObjectIdentityCache.java new interface, far less than (Concurrent)Map, for big seems-like-in-memory object-cache * Supplier.java trivial interface for deferred-provision of new instance * CachedBdbMap.java implement ObjectIdentityCache, mainly via getOrUse() * ObjectIdentityMemCache.java trivial ConcurrentHashMap-based all-in-memory ObjectIdentityCache implementation * ObjectIdentityBdbCache.java BDB-backed ObjectIdentityCache implementation, carved from CachedBdbMap * BdbModule.java refactor utility methods to offer CachedBdbMap or ObjectIdentityBdbCache instances, via source toggle * EnhancedEnvironment.java convenience test-environment method * (others) update to use ObjectIdentityCache/ObjectIdentityMemCache in place of ConcurrentMap/ConcurrentHashMap gojomo 2009-09-25 21:16:53 +00:00
• 78dd7f2086 Fix for [HER-1648] harmonize declared WARC versions in protocol, warcinfo record metadata WARConstants.java update WARC_VERSION=1.0 according to ISO 28500:2009 as of May 2009 WARCWriterProcessor.java update format and conformsTo fields according to ISO 28500:2009, as of May 2009, as described at http://bibnum.bnf.fr/WARC/, latest draft as of November 2008 format: WARC File Format 1.0 conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf szznax 2009-09-23 18:42:25 +00:00
• 2bd9334627 [HER-1677] threads stuck in CachedBdbMap.get/_getMem * CachedBdbMap.java make all WARNING logs include stacks add WARNINGs if memMap.remove() has no expected effect verify reported expunction, WARN and try to fix if memMap expunction not evident adjust indentation/bracketing gojomo 2009-09-22 21:16:21 +00:00
• 142c72fc04 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) * SurtPrefixedDecideRule.java pass only portion after directive ('+' or '-') to SurtPrefixSet * SurtPrefixSet.java better comment/param name gojomo 2009-09-16 01:05:23 +00:00
• ee1c9e2ce4 [HER-1607] handle persist state when crawl starts up * PersistProcessor.java add readOnly option for environment only opened for copying-from * PersistLoadProcessor.java make start() safe to call redundantly (per Lifecycle contract) gojomo 2009-09-12 00:22:06 +00:00
• 163dcb7826 [HER-1607] handle persist state when crawl starts up * BdbModule.java fail-fast if not started properly; stop with fewer errors in problem situations * CrawlController.java continue past runtimeException on stop * CrawlJob.java less redundant error-reporting * AbstractFrontier.java consider PAUSE or FINISH valid targets after unexpected exception * StatisticsTracker.java avoid promoting runtimeException in report-generating up * PersistOnlineProcessor.java delegate DB-closing to BdbModule to avoid double-close gojomo 2009-09-11 02:26:04 +00:00
• d5e088ed6a * WorkQueueFrontier.java improve frontier report: move longest-queue to top, add labels/counters to long lists of queue groups gojomo 2009-09-10 22:40:30 +00:00
• d955e31963 [HER-1676] improve bundled profiles to match best-practice, be self-documenting * defaults.xml add REJECTing SurtPrefixedDecideRule and MatchesListRegExpDecideRule gojomo 2009-09-10 22:39:26 +00:00
• 1c451021d1 * PagedRepresentation.java make 'end' link go to true end, not end-as-of-moment-of-rendering gojomo 2009-09-10 21:57:21 +00:00
• 1e3ec9fe35 * SheetAssociation.java initialize list of names for easier in-crawl construction/insertion gojomo 2009-09-10 21:56:44 +00:00
• a564b7a867 [HER-1675] FetchStats/CrawlSubstats not tallying fetchResponses correctly; as a result QuotaEnforcer doesn't work * FetchStats.java increment fetchResponses appropriately gojomo 2009-09-09 00:12:54 +00:00
• 0b10862c8d Make during-crawl sheets/overlay changes easier * SheetOverlaysManager.java convenience methods for during-crawl changes: (getOrCreateSheet) (deleteSheet) (putSheetOverlay) (removeSheetOverlay) * Sheet.java improved comments gojomo 2009-09-03 06:33:25 +00:00
• c88744ab27 Improve bean-browse UI * BeanBrowseResource.java coerce edits to type of value they're replacing; makes for safer mid-crawl edits of values in maps * JobRelatedResource.java improve rendering/clickability of map, indexed objects gojomo 2009-09-03 03:32:07 +00:00
• f26803bd4f UI tweaks * JobResource.java make jaob main page begin with job title -- easier to distinguish tabs with different jobs in them gojomo 2009-09-03 03:27:05 +00:00
• 635d27b390 More action directory capabilities * ActionDirectory.java execute scripts found in action directory, determining language by their suffix gojomo 2009-09-03 03:15:54 +00:00
• e03329a035 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) * SurtPrefixedDecideRule.java allow seeds to be null (as when rule is intentionally completely independent of seeds; may also need to disqualify rule from Spring autowiring in such a case) gojomo 2009-09-02 23:24:30 +00:00
• 21d90f2a3a UI annoyances * ReportGenResource.java ensure redirect after regeneration of report doesn't show stale cached version gojomo 2009-09-02 23:15:18 +00:00
• d4eabb9631 Improve log display * JobResource.java form proper alerts-rev-tail link for any file ending ".log", use rev-tail-paged view gojomo 2009-09-01 22:42:52 +00:00
• 81a658daa3 Update BDB-JE to 3.3.87 * commons/pom.xml, .classpath update version reference gojomo 2009-09-01 22:40:21 +00:00
• f0421e1f13 * ActionDirectory.java also include 'Fi ' lines -- for recoveries of recoveries gojomo 2009-08-28 17:08:59 +00:00
• ad937c58e5 [HER-1662] use fully qualified hostname * FetchHTTP.java replace one more InetAddress.getHostName() with InetAddress.getCanonicalHostName() (for consistency) gojomo 2009-08-28 06:56:20 +00:00
• d253ff1ace [HER-1662] use fully qualified hostname * WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java, InetAddressUtil.java replace all InetAddress.getHostName() with InetAddress.getCanonicalHostName() (for consistency) gojomo 2009-08-27 23:35:07 +00:00
• 211cd7c659 [HER-1669] slow crawl/contention n WARCWriterProcessor * CrawlController.java increase default size of in-memory recording buffers outBuffer (requests): 4K -> 16K inBuffer (responses): 64K -> 512K gojomo 2009-08-27 23:13:11 +00:00
• 8ba6d47a93 Apache License gojomo 2009-08-27 23:10:38 +00:00
• dc89ce0d95 [HER-1668] support '-' SURT directives in seed files analogous to '+' (if a REJECTing SurtPrefixedDecideRule is present) * SurtPrefixSet.java refactor to allow add-directives under outside control * SurtPrefixedDecideRule.java extend nonseedLine() mechanism to trigger off '+' or '-' depending on decision being ACCEPT or REJECT gojomo 2009-08-27 01:40:26 +00:00
• 3292c8b7f6 Apache License, import cleanup gojomo 2009-08-27 01:34:11 +00:00
• 608a3ddc8b [HER-1607] handle persist state when crawl starts up * PersistProcessor.java, PersistLoadProcessor.java port [HER-1586] preloadSource setting/refactoring to H3 * IoUtils.java (getBufferedReader) utility method * BdbModule.java explicit passthrough of settable deferredWrite property gojomo 2009-08-27 01:32:08 +00:00
• beb858369b Apache License gojomo 2009-08-27 01:27:28 +00:00
• f7af3e016b [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? * ReplayCharSequence.java, return to ISO8859_1 as fallback-encoding gojomo 2009-08-27 00:20:37 +00:00
• 783ecfd9d7 [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? * ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java refactor to put fallback-encoding in single place gojomo 2009-08-27 00:12:15 +00:00
• 3f9ecc4dfa * PagedRepresentation.java fix negative estimated size error for log-tail viewing gojomo 2009-08-27 00:10:48 +00:00
• c07a70eb21 Fix SubElement IllegalArgumentException (via WarcWriterProcessor.writeMetadata) * SubElement.java accept zero-length strings as valid values * WarcWriterProcessor.java don't write 'via' or 'hopsFromSeed' unless non-blank gojomo 2009-08-19 21:15:40 +00:00
• 5c93d7bd28 Rename DecideResult.PASS to DecideResult.NONE for clarity * DecideResult.java + references PASS was confusable with ACCEPT; NONE better conveys "no decision" gojomo 2009-08-14 21:31:11 +00:00
• b78fce045b Fix selftests hanging due to pauseAtStart default true * selftest-crawler-beans.cxml make selftests not pauseAtStart gojomo 2009-08-13 10:03:56 +00:00
• d654bb5651 [HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' * ActionDirectory.java support for .seeds, .recover, .include, .schedule, .force files * CrawlController.java change default for pause-at-start to true; ensure enters PAUSED rather than PAUSING * Frontier.java (importRecoverFormat) support method for .recover, .include, .schedule, .force additions (considerIncluded) change to accept CrawlURI * AbstractFrontier.java (importRecoverFormat) support method for .recover, .include, .schedule, .force additions * BdbFrontier.java remove too-strict assertion * FrontierJournal.java rename recover-log 'frontier.recover.gz' (so that '.recover' is a dot-suffix) * WorkQueueFrontier.java (considerIncluded) change to accept CrawlURI gojomo 2009-08-12 01:50:51 +00:00
• bba028ce21 * StatisticsTracker.java, SeedsReport.java remove redundant method gojomo 2009-08-12 01:35:19 +00:00
• a37a1846b7 Move getBufferedReader() -> IoUtils * CrawlerJournal.java, IoUtils.java move getBufferedReader to IoUtils * FrontierJournal.java add new 'included' tag for log lines ("Fi") * (others) update for relocated method gojomo 2009-08-11 02:17:20 +00:00
• 8b95e7fb36 CachedBdbMap cleanup - intended commit comment * CachedBdbMap.java discard code to open/manage own BDB Environments (and related constructor) cleanup comments/FIXMEs * BdbModule.java return CachedBdbMap, rather than more general ConcurrentMap, in case requester wants it * CachedBdbMapTest.java get instance via BdbModule (to manage env/classCatalog) gojomo 2009-08-11 02:10:28 +00:00
• 8d09ee38ae CachedBdbMap cleanup gojomo 2009-08-11 02:07:22 +00:00
• 023efe4aba [HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' * ActionDirectory.java initial implementation; handle '.seeds' files gojomo 2009-08-06 21:33:31 +00:00
• 0400d81ee5 support for... [HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' [HER-769] Support case where millions of seeds. * AbstractFrontier.java move sourceTagSeeds setting to SeedModule trigger and schedule seeds via SeedModule's announcements and SeedListener protocols * StatisticsTracker.java implement SeedListener, use processedSeedRecords as list of all seeds * ProcessorURI.java, DefaultProcessorURI.java (getURI) convenience accessor for plain URI string * SurtPrefixedDecideRule.java use seeds as prefixes via SeedListener notifications, rather than separate scan of seeds source text * SeedListener.java simplify to notifications of new seed, and nonseed text mixed with seeds (as with SURT directives) * SeedModule.java methods for SeedListener interface, announceSeeds, actOn(File) * TextSeedModule.java eliminate interators; add methods for announcing seeds and scanning seed text sources (including arbitrary Files) * PrefixSet.java, SurtPrefixSet.java derive from ConcurrentSkipListSet; eliminate now-unneeded extra synchronization and clone()s offer considerAsDirective() for direct application of directive Strings gojomo 2009-08-06 21:24:04 +00:00
• 4d06b239ee Push CrawlURI and related constants down to modules subproject/package (many) gojomo 2009-08-06 20:22:00 +00:00
• 0ebac2f115 * CachedBdbMapTest.java more timing/heap voodoo for test robustness gojomo 2009-08-06 20:11:04 +00:00
• 64d3fd212f [HER-1659] make default WARC size comply with spec; adjust default pool size for fewer odd-sized (W)ARCs * ArchiveFileConstants.java remove superfluous constant * WriterPool.java DEFAULT_MAX_ACTIVE -> 1 * WriterPoolMember.java update constant use; remove deprecated+unused methods * WriterPoolProcessor.java pull storePaths up from subclasses, but set default via subclass method set defaultMaxSize via subclass method * WARCWriterProcessor.java, ARCWriterProcessor.java (getDefaultMaxFileSize), (getDefaultStorePaths) as appropriate gojomo 2009-08-06 00:55:27 +00:00
• 272ae19961 Cleanup * CachedBdbMap.java remove debug output * TestUtils.java add info logging * CachedBdbMapTest.java make more robust against prior heap usage, different platforms gojomo 2009-08-04 06:55:31 +00:00
• 4f0961c371 Fix NPE * URIAuthorityBasedQueueAssignmentPolicy.java tolerate null getRawPathQuery response gojomo 2009-08-04 01:31:38 +00:00
• c196a1de77 [HER-1658] CachedBdbMaps not expunging as expected (especially StatisticsTracker.processedSeedsRecords) * CachedBdbMap.java do expunge on put(), replace() add low-memory-sensitive 'canary' to force expunge even if otherwise untriggered * CachedBdbMapTest.java add test of idle expunge in low-memory conditions * CrawlURI.java (getURI) direct access to String URI (don't reuse toString() functionally) * SeedRecord.java support for updating record with later/repeat report * StatisticsTracker.java remove put()s for processedSeedRecords, hostsLastFinished * BdbFrontier.java (getQueueFor) remove put, do in concurrent-compliant manner gojomo 2009-08-04 00:06:22 +00:00
• 1c984753f0 Work for [HER-1656] add multiple-queues-per-host ('parallelQueues') capabilities to QueueAssignmentPolicies * LongToIntConsistentHash.java + Test expand range/distribution tests correct problems with phony '-1' bucket being created/returned gojomo 2009-07-31 22:55:09 +00:00
• 5bb8596fab Work for [HER-1656] add multiple-queues-per-host ('parallelQueues') capabilities to QueueAssignmentPolicies * LongToIntConsistentHash.java + Test consistent-hashing utility class * URIAuthorityBasedQueueAssignmentPolicy.java shared superclass for the hostname and surtauthority QAPs setting for 'deferToPrevious' -- avoid changing assignments setting for 'parallelQueues' -- when > 1, consistent-hash URIs across that many separate queues (with numerical suffix) * SurtAuthorityQueueAssignmentPolicy.java, HostnameQueueAssignmentPolicy.java refactor to derive from URIAuthorityBasedQueueAssignmentPolicy * QueueAssignmentPolicy.java add Apache license * Hop.java add String representation for convenience gojomo 2009-07-28 23:02:11 +00:00
• d2a82228a3 TODO/FIXME comments where things don't seem quite right but aren't definitively buggy * CachedBdbMap.java reevaluate if clear() should include close() * CrawlURI.java investigate possible confusion in meaning of isPrerequisite() gojomo 2009-07-28 22:56:54 +00:00
• 5027fb9aa0 * AbstractFrontier.java, WrokQueueFrontier.java remove outdated comments; use isEmpty() gojomo 2009-07-28 22:47:01 +00:00
• 478e425b34 Work for [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? * FetchHTTP.java really disable the WARNING logging, now gojomo 2009-07-28 22:46:04 +00:00
• d8ed3822e7 compartmentalize IOExceptions * RecordingInputStream.java restore expected nulling of closed 'in' gojomo 2009-07-28 22:01:50 +00:00
• 597c910422 Work for [HER-1655] Excessive charset/character-encoding exceptions; is fallback behavior correct? * FetchHTTP.java catch charset problems early and add as crawl.log annotation, fallback to default without raising log/alert/WARNING-level notice gojomo 2009-07-28 01:01:09 +00:00
• a2940bc23e compartmentalize IOExceptions * RecordingInputStream.java make sure one exception (intermittently seen on Windows re: 'mapped section') doesn't leave things in foul state for next try in same thread gojomo 2009-07-28 00:54:06 +00:00
• 4cd9eb2e41 Work for [HER-1654] suspicious Cookie bottlenecks: synchronization, growth * Cookie.java simple non-contending String compare (fixed similarly upstream in HttpClient; see https://cwiki.apache.org/jira/browse/HTTPCLIENT-645 ) gojomo 2009-07-28 00:48:36 +00:00
• 917dfc2bd3 Fix ConfigPath inconsistencies in reporting, relative paths, fixup-ordering * ConfigPathConfigurer.java work in two passes: one (BeanPostProcessor) collects all candidates for fixup, two (ApplicationListener) actually performs fixup suppress fixup of own base path, preventing circulare ref/StackOverflow * ConfigPath.java change sense of 'merge' to merge into receiver * SurtPrefixedDecideRule.java use ConfigFile, use 'merge' in assignment * CrawlerLoggerModule.java use merge in assignment * CrawlJob.java (getBeanpathTarget) utility method * JobResource.java respect unset ConfigPaths * ConfigPathEdito.java, ConfigFileEditor.java better comments, ConfigFile variant gojomo 2009-07-24 20:22:06 +00:00
• 60ebd6cfdb * CrawlController.java better handle case where crawl is discarded before launched gojomo 2009-07-24 00:27:57 +00:00
• c3cc7cc7e5 * JobResource.java complete rename of printLinked to printLinkedFile gojomo 2009-07-24 00:27:27 +00:00
• 0b577387af improve display of known ConfigPaths * ConfigPathConfigurer.java key by propPath * JobResource.java display by "propPath: assigned name" * WriterPoolProcessor.java fix typo gojomo 2009-07-24 00:06:34 +00:00
• 3d573a8d5f Resolve remaining broken links to logs outside job dir * JobResource.java refactor, improve printLinkedFile * EngineApplication.java remove debug output gojomo 2009-07-24 00:00:28 +00:00
• ace6d95ab8 Fix race conditions/unclean shutdown around 'teardown' (esp. if crawl running) * CrawlController.java (isFinished) added * CrawlJob.java rename reset to teardown wait for CrawlController to report finished before discarding AppContext if not finished in time, fail reporting 'false' * JobResource.java show Flash if teardown did not succeed; user must retry * BdbUriUniqFilter.java don't throw DatabaseException from simple accessor gojomo 2009-07-23 22:49:42 +00:00
• ef0a9f89d4 Fix risk of destruction/confusion from WriteTarget#getWriter * WriteTarget.java, ReadSource.java renamed Writer/Reader accessors to avoid bean-style 'get-' naming improve comments * ConfigFile.java, ConfigString.java, ScriptedProcessor.java, ScriptedDecideRule.java, SurtPrefixedDecideRule.java, TextSeedModule.java reflect above change gojomo 2009-07-23 20:30:48 +00:00
• e0d748cd3b improve comment gojomo 2009-07-23 20:28:29 +00:00
• 52a0895c6b * JobResource.java rename 'discard' to 'teardown' enable browse/edit of paths outside jobdir via new /anypath/ service * EngineResource.java, Engine.java enable addition of job dirs from outside main jobs directory * EngineApplication.java register new /engine/anypath service for outside-job-dir paths gojomo 2009-07-22 15:59:30 +00:00
• 4d7c4834d8 * ScriptResource.java offer better ack of lines-executed (even if they generate no output) gojomo 2009-07-22 15:53:49 +00:00
• 1fc2880106 * SurtPrefixedDecideRule.java make surt-config files ConfigPaths, so they can be job-relative gojomo 2009-07-22 15:51:58 +00:00
• 1cea6dc412 * TextSeedModule.java (addSeed) synchronize, just in case a thread contention is causing problems gojomo 2009-07-22 15:49:59 +00:00
• d6496b993a * SheetOverlaysManager.java make empty map modifiable, to ease during-crawl additions add accessors for other fields, to ease browsing gojomo 2009-07-20 20:09:08 +00:00
• e1563a7239 (W)ARCs not being written compressed by default * WriterPoolProcessor set 'compress' field default to true gojomo 2009-07-20 17:15:41 +00:00
• e8a8639277 initial checkin: this class is useful for determining why ArchiveReader fails for problematic W/ARC files szznax 2009-07-17 21:52:21 +00:00
• 9fe716cf8f copy to h3 Fix for [HER-1568] ARCReader fails on some newer Alexa ARC files * ARCConstants.java added ArcRecordErrors enum for known reading errors * ARCReaderFactory.java only check for 'LX' extension in GZIP extra field * ARCRecord.java added errors, hasErrors(), and getErrors() szznax 2009-07-17 21:51:24 +00:00
• f82432ca9d Improve conciseness/clarity of ReplayInputStream/ReplayCharSequence usage/cleanup * ReplayCharSequence.java implement Closeable * ArchiveUtils.java (closeQuietly) for ReplayCharSequence * ExtractorCSS.java, ExtractorDOC.java, ExtractorHTML.java, ExtractorJS.java, ExtractorXML.java, HTTPContentDigest.java, CrawlServer.java, MirrorWriterProcessor.java, WARCWriterProcessor.java cover more with finally blocks; use closeQuietly() variants gojomo 2009-07-15 23:30:22 +00:00
• 20a1b19a6a * ScriptedProcessor.java add intended ApplicationContextAware interface gojomo 2009-07-15 23:24:13 +00:00
• 9ed41e9ea2 Remedy potential filehandle leak * RecordingOutputStream.java close ReplayInputStream used to create GenericReplayCharSequence use 'UTF-8' and not possibly-varying JVm default encoding when none specified gojomo 2009-07-15 23:23:06 +00:00
• c01032a56f Excessive exceptions for unsupported/illegal encodings * GenericReplayCharSequence.java, InMemoryReplayCharSequence.java catch, log, fallback to UTF-8 for problematic encoding names gojomo 2009-07-15 23:20:36 +00:00
• 427b633826 Large seed list = web UI launch spins for hours without returning * CrawlJob.java move the requestCrawlStart into the background/crawl-group thread gojomo 2009-07-14 23:56:53 +00:00
• fd072ffaaa Absolute paths not respected * WriterPoolProcessor.java treat paths beginning '/' as absolute * ARCWriterProcessor.java remove vestigial default-path var/method gojomo 2009-07-14 23:41:00 +00:00
• c721d34fde [HER-1642] fix HOWTO-launch document * HOWTO-Launch-Heritrix.txt removed; info mostly about H2 variants * README.txt updated wiki link, launch info to match H3 * LICENSE.txt updated to Apache2 gojomo 2009-07-13 23:06:23 +00:00
• 8e5b46250e Fix undesired relativization of intended-absolute paths * ConfigPath.java special check for path starting '/'; if so, ignore 'base' gojomo 2009-07-09 22:46:26 +00:00
• 4251e5311f * SelfTestBase.java report full path of unclosed ARC for manual checking -- truly unclosed or timing issue? gojomo 2009-07-08 01:21:49 +00:00
• dd254088e1 [HER-1633] port IsCrossTopmostAssignedSurtHopDecideRule to heritrix3 * HopCrossesAssignmentLevelDomainDecideRule.java port H2 IsCrossTopmostAssignedSurtHopDecideRule to H3 under this new name * PublicSuffixes.java update comment, public method for 'Assignment Level Domain'/'Assignment Level SURT' term * AssignmentLevelSurtQueueAssignmentPolicy renamed TopmostAssignedSurtQueueAssignmentPolicy gojomo 2009-07-08 00:28:56 +00:00
• cb8a374e5d [HER-1636] H3: Failed get of replay char sequence due the OutOfMemoryError: Map failed (port HER-1482) (from nlevitt's H2 commit message:) * InMemoryReplayCharSequence.java new, simple ReplayCharSequence that supports any encoding, keeping everything in memory * GenericReplayCharSequence.java If the encoding supports random access, memory maps the backing file directly; otherwise decodes to UTF-16 and maps that. Supports the first Integer.MAX_VALUE bytes of the file. Maps up to 64M; moves the map around as necessary for larger files. * Latin1ByteReplayCharSequence.java removed, functionality split between InMemoryReplayCharSequence and GenericReplayCharSequence * ReplayCharSequenceTest.java xestHugeReplayCharSequence() - uncomment to test a huge replay char sequence testReplayCharSequenceByteToStringOverflow() - test both UTF-8 and windows-1252 * RecordingOutputStream.java use the new ReplayCharSequences gojomo 2009-07-07 21:53:57 +00:00
• 1128ad638b [HER-1609] wide crawl performance is limited by lock contention in ServerCache.getServerFor(String) * CachedBdbMap.java remove unnecessary import of CrawlController (from engine subproject) gojomo 2009-07-07 01:35:56 +00:00
• 89a55c7fdd * JobResource.java make UI more robust against browser/platform font size variances gojomo 2009-07-06 23:37:30 +00:00
• 149fc414df [HER-1609] wide crawl performance is limited by lock contention in ServerCache.getServerFor(String) * (all, especially CachedBdbMap.java) pbaclace-contributed patch, ported to H3, plus... * StatisticsTracker.java replace LongWrapper with AtomicLong use non-recursive putIfAbsent-based idiom for get-or-create * DefaultServerCache.java clean up host/server get-or-create around non-recursive concurrent idiom * BdbModule.java add expectedConcurrency property & je.lock.nLockTables tuning * TopNSet.java add synchronization, as calling StatisticsTracker method has lost it gojomo 2009-07-06 23:36:17 +00:00
• 4583ff7306 Rename Rrc2617Credential to HttpAuthenticationCredential * (multiple) use more-familiar name for HTTP authentication gojomo 2009-07-06 22:09:52 +00:00
• 56b31490c8 [HER-1574] meta refresh urls not html unescaped * ExtractorHTML.java apply same 1-line fix as H1/H2 gojomo 2009-06-11 23:06:07 +00:00
• 721089044a * **/pom.xml revert version name to SNAPSHOT gojomo 2009-06-11 09:09:43 +00:00
• e8e5fdd9a4 * **/pom.xml rename version to 'alpha' for distinguished 'alpha' identified build (instead of SNAPSHOT/timestamp) gojomo 2009-06-11 08:06:19 +00:00
• ecd93ae458 Fixes from trial guide testing * Heritrix.java fail-fast with helpful error message if launched with Java < 1.6 * EditRepresentation.java "save changes" rather than "update content" * EngineApplication.java customize restlet error to include back-button and full stack (instead of link to HTTP spec!) gojomo 2009-06-10 22:31:44 +00:00
• 27783e87ab Fixes from trial guide testing * StatisticsTracker.java allow Crawl Summary report to complete even pre-launch * BeanBrowseResource.java, ScriptResource.java add page title, backlink to job * EngineResource.java, PagedRepresentation.java better in-page labeling gojomo 2009-06-09 21:59:45 +00:00