Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 3dc5f04c06 Avoid deadlock on terminate() of already-paused/stopping crawl * AbstractFrontier (managementTasks) don't synchronize around possibly-blocking take(), only around process() gojomo 2010-11-08 21:06:58 +00:00
• 3be9941f76 Resolve deadlock seen after latest changes * AbstractFrontier (drainInbound) don't assume current thread will be able to take() full batch; poll() and exit early if other threads have drained queue first gojomo 2010-11-08 18:54:03 +00:00
• f12e940edd Discard 'CredentialAvatar' -- no longer needed with Spring settings -- use Credential instances directly gojomo 2010-11-06 03:45:00 +00:00
• e6c0f71175 * ObjectIdentityBdbCache debugging output -- report low-mem ref-clearing gojomo 2010-11-06 02:25:22 +00:00
• c7398ce463 Experiment to work-around Frontier.managerThread bottleneck * AbstractFrontier, WorkQueueFrontier synchronize around all InEvent/findEligibleURI actions, so that drainInbound and fillOutbound may be called outside managerThread whenever a ToeThread would block on enqueue() or next(), try the appropriate catch-up method before blocking remove assertions no longer true with activity happening outside managerThread gojomo 2010-11-06 02:14:39 +00:00
• 7fdb7cb384 tests cleanup * UURI return to custom-serialization rather than externalization * CachedBdbMap remove obsolete class * StripSessionCFIDs, StripSessionIDs update patterns to restore intended case-insensitivity * StripSessionCFIDsTest put expected, actual in expected order gojomo 2010-11-06 01:26:20 +00:00
• 54e7e28fa6 remove stray imports breaking build gojomo 2010-11-05 23:31:36 +00:00
• c56eb5d343 Experiment to assist large seed-loads * TextSeedModule.java in large seed loads, try to force finalization, just in case it's lagging gojomo 2010-11-05 23:27:43 +00:00
• 0627a67dce Minimize transient garbage volume by optimized serialization based on the 'Kryo' library * AutoKryo extension of Kryo to allow classes to control their own registration, trigger registration of associated classes, and deserialize classes without no-arg constructors * KryoBinding binding for use with BDB that uses AutoKryo serialization for a 2X-4X reduction in byte[] size * UURI improved serialization via Externalizable and Kryo's CustomSerialization methods * BdbModule discard deprecated CachedBDBMap option (getObjectCache) extend with both declaredClass and valueClass (for when map values are specializations of the declared type, as with frontier.allQueues) adjust type declarations * ObjectIdentityBdbCache use KryoBinding rather than SerialBinding * CachedBdbMapTest discarded * BdbFrontier, BdbServerCache, StatisticsTracker adjust type declarations, objectCache creation * BdbMultipleWorkQueues use KryoBinding rather than (Recycling)SerialBinding * BdbWorkQueue, CrawlServer, CrawlHost, CrawlURI add autoregister support * LinkContext public for kryo registration gojomo 2010-11-05 23:23:58 +00:00
• f4a01f1155 remove outdated adler32 workaround gojomo 2010-11-05 22:42:27 +00:00
• 4b7ada3930 [HER-1839] Upgrade BDB to 4.1.6; ensure HER-1628 really fixed * commons/pom.xml, .classpath update BDB jar to 4.1.6 gojomo 2010-11-05 22:33:47 +00:00
• 6164939fc4 Minimize transient garbage creation identified by allocation profiling * CrawlURI use ArrayList and indexed access rather than LinkedList and iterator instances for overlays gojomo 2010-11-05 22:32:05 +00:00
• f4d69b37da Minimize transient garbage creation identified by allocation profiling * UURIFactory, canonicalize/**, PathologicalPathDecideRule, ExtractorHTML use recycled matchers via TextUtils * CrawlURI, KeyedProperties, OverlayContext, SheetOverlaysManager use ArrayList and indexed access rather than LinkedList and iterator instances gojomo 2010-11-05 21:34:53 +00:00
• acedca1788 Part of [HER-1836] let more than one extractor run on a single url - new setting "independentExtractors" - when enabled, extractors run regardless of whether other extractors have run * profile-crawler-beans.cxml, AbstractFrontier.java, ExtractorParameters.java, Extractor.java new setting independentExtractors * ContentExtractor.java shouldProcess() - respect independentExtractors nlevitt 2010-10-26 23:49:42 +00:00
• dc056d73f4 [HER-1832] URI without fetch-attempt causes either RuntimeException (H1) or nonsensical history info (H3) in FetchHistoryProcessor * FetchHistoryProcessor.java skip history-storage if no evidence of fetch-attempt (A_FETCH_START_TIME data) gojomo 2010-10-26 00:32:18 +00:00
• 3a362002c7 [HER-1833] many crawls want to suppress DNS-server's provision of a default local domain * FetchDNS.java append '.' to domain name for lookup, anchoring it as FQDN gojomo 2010-10-25 23:50:12 +00:00
• d392341141 * ExtractorXML.java shouldExtract() - choose to extract also if content starts with <?xml ... nlevitt 2010-10-19 02:53:07 +00:00
• b75f604bf9 [HER-1836] let more than one extractor run on a single url? * ContentExtractor.java shouldProcess() - do not check uri.hasBeenLinkExtracted() * CrawlURI.java update javadoc comments on hasLinkBeenExtracted() nlevitt 2010-10-19 02:08:54 +00:00
• d72e6cd1d0 [HER-1830] H3: improve sheet examples in default configuration CXML * **/profile-crawler-beans.xml only 1 example SurPrefixesSheetAssociation gojomo 2010-10-01 00:16:23 +00:00
• 13d8c4f851 [HER-1830] H3: improve sheet examples in default configuration CXML * **/profile-crawler-beans.xml example SurPrefixesSheetAssociation gojomo 2010-09-30 23:01:25 +00:00
• a757140cf6 [HER-1831] change version numbering convention to be less confusing * **/pom.xml change 3.1.1-SNAPSHOT to 3.0.1-SNAPSHOT gojomo 2010-09-30 19:48:35 +00:00
• f1cd7dd527 [HER-1827] Have Frontier report include top X longest queues * WorkQueueFrontier.java improve effective precision of top-n-queues by also updating on dequeues gojomo 2010-09-28 22:42:22 +00:00
• 8f28d493a2 [HER-1827] Have Frontier report include top X longest queues * Histotable.java expose static method useful for frequency-reporting * TopNSet.java, TopNSet.java improve handling of case where updates may decrease frequency counts add convenience methods for common needs * WorkQueueFrontier.java replace single longestActiveQueue with largestQueues TopNSet, of configurable size make max-queues-to-report-per-category configurable gojomo 2010-09-28 22:12:28 +00:00
• 74d3fb54ce HER-1828 ExtractorXML misses https urls * ExtractorXml.java adjust XML_URI_EXTRACTOR regex to also look for https urls nlevitt 2010-09-25 07:48:46 +00:00
• de32bdb053 [HER-1825] H3: make default base queue precedence a more useful default * BaseQueuePrecedencePolicy.java make default '3' (leaving space for 2 'higher') rather than '1' gojomo 2010-09-24 04:34:20 +00:00
• 63c8de4045 fix for [HER-1824] ExtractorXML doesn't properly unescape * ExtractorXML.java processXML() - unescape extracted urls with commons-lang unescapeXml() nlevitt 2010-09-23 02:03:48 +00:00
• 7909954b8f [HER-1823] H3: adding SheetAssociations mid-crawl won't affect already queued CrawlURIs (sheet liveness) * SheetsOverlayManager.java (applyOverlaysTo) clear previous info so redundant sheets don't accrete gojomo 2010-09-21 11:07:33 +00:00
• 1ce23ecbcd [HER-747] Ensure 'retired' status for queues is operator-controllable [HER-1768] Changing queue totalBudget not retiring queue as expected * CrawlURI.java (includesRetireDirective) internalize to CrawlURI * AbstractFrontier.java allow re-enqueuing of untried URI with retire-driective clean up terminology/method-names * WorkQueue.java, WorkQueueFrontier.java, BdbFrontier.java adjust budget/sessionBalance handling to always count up, always consult latest values, always refresh values from latest URI (processFinish) reorganize to reduce redundancy * DispositionProcessor.java offer forceRetire setting * QuotaEnforcer.java set failed-status *or* forceRetire, not both [HER-1823] H3: adding SheetAssociations mid-crawl won't affect already queued CrawlURIs (sheet liveness) * SheetsOverlayManager.java (applyOverlaysTo) rename, unify, reapply whenever called to ensure liveness * CandidatesProcessor.java adjust to rename gojomo 2010-09-21 10:55:07 +00:00
• c701cd391a [HER-1822] H3: bean browser in non-paused crawl sometimes raises exceptions, doesn't work * JobRelatedResource.java add more property names to blacklist route all relevant requests through new utilitiy getPropertyDescriptors method that patches instances as necessary to suppress problems gojomo 2010-09-21 10:28:29 +00:00
• 966646871c [HER-1727] (W)ARC filename uniqueness guarantees: adminport option, process_id option(?), JVM-global-repeat-suppression * ArchiveUtilsTest.java tests for new HER-1727 guaranteed-no-repeat-timestamp functionality gojomo 2010-09-20 13:51:13 +00:00
• 153e709fe5 * WriterPoolProcessor.java eliminate nonsensical type check gojomo 2010-09-15 03:52:11 +00:00
• 1aca427447 * BdbModule.java change exception-throw to log WARNING during DB-cleanup gojomo 2010-09-10 21:15:20 +00:00
• 1f08c114bb * ToeThread.java tighten INFO logging to FINE gojomo 2010-09-10 21:12:48 +00:00
• 4166316113 * AbstractFrontier.java remove Serializable & related transient declarations -- serialization no longer used for checkpointing nor advisable gojomo 2010-09-10 01:00:19 +00:00
• d6a4d158b0 [HER-1803] H3: make it easier to change robots settings * RobotsPolicy.java simplified unification of RobotsHonoring and RobotsExclusion policy classes; subclasses specialize * ObeyRobotsPolicy.java classic obey-robots-as-declared policy * IgnoreRobotsPolicy.java ignore declared robots policy * CustomRobotsPolicy.java follow custom-crafted policy in robots.txt format * MostFavoredRobotsPolicy.java follow the most-permissive policy available to a set of user-agents; optionally adopt ('masquerade') the user-agent whose rules are chosen * RobotsExclusionPolicy.java, RobotsHonoringPolicy.java deleted * CrawlMetadata.java use new string robotsPolicyName as main determinant of policy maintain map of available policies ('obey','ignore', and any the operator added) * CrawlServer.java retain Robotstxt instance rather than customized RobotsExclusionPolicy * PreconditionEnforcer.java use full CrawlMetadata rather than just UserAgentProvider subset consult RobotsPolicy looked up through CrawlMetadata for allow/disallow decision * DispositionProcessor.java update CrawlServer with Robotstxt only consult Robotstxt for crawl-delay * profile-crawler-beans.cxml (*2), migrate-template-crawler-beans.cxml, selftest-crawler-beans.cxml remove old declarations/examples; add new * ExtractorHTML.java, JerichoExtractorHTML.java consult RobotsPolicy.obeyMetaRobotsNofollow for NOFOLLOW * Robotstxt.java new constructors for new use patterns * **/*Test.java update/remove to match new use patterns gojomo 2010-09-10 00:27:20 +00:00
• 3e43cf7bd5 * LinkContext.java undo inadvertent introduction of class-serialization-format incompatibility gojomo 2010-09-09 23:22:15 +00:00
• bce841be5f * JobResource.java improve message for checkpoint-not-made flash gojomo 2010-09-09 23:21:28 +00:00
• 3861c8caa4 * BdbMultipleWorkQueues.java improve detail of logging of error that's been seen in some checkpoint-resume situations gojomo 2010-09-09 23:20:47 +00:00
• de551ced58 [HER-1801] H3: Unreleased lock causes deadlock when checkpointing * Frontier.java add beginDisposition(), endDisposition() methods * AbstractFrontier.java implement beginDisposition(), endDisposition() methods to maintain dispositions-in-progress lock * BdbFrontier.java acquire exclusive dispositionInProgressLock before allowing checkpoint to begin; release at completion of checkpoint * ToeThread.java call beginDisposition before dispositionChain/finished(); call endDisposition after finished() and on exceptions * DispositionChain.java shed all prior relation to locking/checkpointing gojomo 2010-09-03 02:14:41 +00:00
• d12d078a07 [HER-1800] H3: Incomplete checkpoint listed in combo box * Checkpoint.java remove previous attempt to use Lifecycle to catch error * Checkpointable.java @Autowired(required=false) annotation for setRecoveryCheckpoint * CheckpointService.java HasValidator implementation for post-build validity check; also check at launch (probably redundant) * CheckpointValidator.java ensure validity stamp exists on any present Checkpoint gojomo 2010-09-03 01:03:01 +00:00
• 99ce106a9d * CrawlJob.java tighten logging gojomo 2010-09-03 00:30:33 +00:00
• dbf22ad574 [HER-1480] URIs logged (as -6 or -7) that should never be scheduled * Extractor.java discard special-casing of URI-ignored-scheme gojomo 2010-09-03 00:29:05 +00:00
• bc698c09fc [HER-1480] URIs logged (as -6 or -7) that should never be scheduled * LaxURI.java increase tolerance with regard to path-segments * UURIFactory.java allow digits in scheme regex discard old supported-schemes code * UURIFactoryTest.java tests for roundtrip serialization of problematic hostlike-scheme and bars-in-path * CrawlerLoggerModule.java discard special-casing of URI scheme errors * SchemeNotInSetDecideRule.java rule to apply to URIs of unknown schemes; by default REJECTs those not usually handled by Heritrix * **/profile-crawler-beans.cxml add REJECT SchemeNotInSetDecideRule gojomo 2010-09-03 00:24:13 +00:00
• 0347269785 [HER-1817] cleanup logging.properties / WARNING+ for most 3rd party; SEVERE+ for HttpClient; INFO for project classes * logging.properties streamline to bare essentials; allow INFO for org.archive classes * GenericReplayCharSequence.java, BdbUriUniqFilter.java demote some INFO logging to FINE gojomo 2010-09-03 00:00:46 +00:00
• b9b592e286 * jndi.properties remove obsolete file gojomo 2010-09-02 23:57:31 +00:00
• 9bfa8fc1db * JobResource.java remove stray whitespace gojomo 2010-09-02 23:56:52 +00:00
• ad8941fa23 1-liner fixing argument display for usage help argument echo * Heritrix.java (usage) StringUtils.join the String[] args commit in honor of Travis, who suggested fix gojomo 2010-08-27 00:56:42 +00:00
• 2f78980e5d [HER-1814] remove dependency on apache commons-pool/GenericObjectPool * SelfTestBase.java don't try to UURIify 'fieldesc:' ad-hoc URIs gojomo 2010-08-26 21:56:50 +00:00
• 790469280e [HER-1814] remove dependency on apache commons-pool/GenericObjectPool * WriterPoolMember.java fix ops broken by removal of checkSize at preWriteRecordTasks -- still create if out == null gojomo 2010-08-26 21:19:56 +00:00
• 8d1083922d [HER-1814] remove dependency on apache commons-pool/GenericObjectPool * .classpath, pom.xml, org/apache/commons/pool/impl/* remove references to and patched classes based on commons-pool * WriterPool.java move instance creation/round-robin-pool/dispose into this class close (rather then recycle) extra writers when crawl slows (via tracking time of last need/rollover) * WriterPoolMember.java add isOversize() for external check of ready-to-rollover file avoid rollover before every record (so that set of related WARC records aren't split between files) * ARCWriterPool.java, WARCWriterPool.java remove BasePoolableObjectFactory reference, implement class-appropriate makeWriter() * ARCWriterPoolTest.java adjust for new never-timeout behavior * ARCWriterPoolProcessor.java WARCWriterPoolProcessor.java adjust constructor calls * WriterPoolProcessor.java replace poolMaxWaitMs (which caused eventual complete timeout from waiting for writer) with maxWaitForIdleMs, a much smaller value which only controls how long a thread waits for an reusable writer before considering creating a new one. (If creation isn't allowed, threads will now wait indefinitely for a writer to become available.) gojomo 2010-08-25 23:25:26 +00:00
• 03778ff943 [HER-1727] (W)ARC filename uniqueness guarantees: adminport option, process_id option(?), JVM-global-repeat-suppression * WriterPoolMember.java use '~' rather than '@' and '#' as pid~hostname~port separators, for less risk in shell scripts and URLs gojomo 2010-08-23 22:17:39 +00:00
• e3023218e1 null check; fix broken test gojomo 2010-08-20 04:08:07 +00:00
• 47b28f92f8 fix class reference (and build) gojomo 2010-08-20 00:47:55 +00:00
• b388479d32 [HER-1727] (W)ARC filename uniqueness guarantees: adminport option, process_id option(?), JVM-global-repeat-suppression * Heritrix.java make heritrix.hostname, heritrix.pid, heritrix.port available as global properties * PropertyUtils.java, PropertyUtilsTest.java utility methods to interpolate string values from one or several supplied Properties instances * ArchiveUtils.java utility methods to give timestamps guaranteed larger/different than any previously-issued timestamp gojomo 2010-08-20 00:21:43 +00:00
• c410c90b76 Implementation of [HER-1811] Replace the LowDiskPauseProcessor with something that works kristinn_sig 2010-08-19 10:47:22 +00:00
• afc2a55e49 fix for [HER-1808] uncaught URLDecoder IllegalArgumentException in ExtractorHTML * ExtractorHTML.java still consider values rejected by URLDecoder szznax 2010-08-13 00:37:55 +00:00
• dd4bf9a910 correct canonicalizationPolicy bean template per archive-crawler msg 6649 * profile-crawler-beans.cxml, migrate-template-crawler-beans.cxml surround canonicalizationPolicy list with property element szznax 2010-08-09 21:22:50 +00:00
• a729c041e1 * URIAuthorityBasedQueueAssignmentPolicy.java * SurtAuthorityQueueAssignmentPolicy.java * HostnameQueueAssignmentPolicy.java correct inadvertent 'default' access rather than 'protected' allowing overriding gojomo 2010-08-04 20:26:22 +00:00
• 9a74685065 correct whitespace gojomo 2010-07-27 00:13:16 +00:00
• be954bc9b9 [HER-1804] H3: avoid echoing admin credentials to (possibly world-readable) _out log * Heritrix.java replace echo of exact login/pw with note about whether it came from cmd-line or @file gojomo 2010-07-22 22:55:56 +00:00
• 5f76c7ef56 [HER-1800] H3: Incomplete checkpoint listed in combo box * CheckpointService.java use static method on Checkpoint for validity check * Checkpoint.java throw early-startup exception if attempting resumption of invalid checkpoint gojomo 2010-07-13 02:59:47 +00:00
• 899fb6f404 [HER-1800] H3: Incomplete checkpoint listed in combo box * CheckpointService.java (getAvailableCheckpointDirectories) return List, filtered to leave out directories without a 'valid' stamp; log WARNING to _out so that the attentive operator can manually clean-up/investigate gojomo 2010-07-13 01:19:10 +00:00
• f08c6bd749 [HER-1801] H3: Unreleased lock causes deadlock when checkpointing * DispositionChain.java ensure lock-release after unhandled runtime exceptions with try-finally gojomo 2010-07-12 23:44:36 +00:00
• 7d4ed491be * ReadSourceEditor.java PropertyEditor to ease inline specification (as String/ConfigString) anywhere a ReadSource is expected gojomo 2010-07-09 23:17:06 +00:00
• bba186f333 [HER-1799] H3: expand 'inferred' URI options to optionally include root-page (/) * ExtractorHTTP.java add property inferRootPage; if true (not the default), always consider '/' inferred from all HTTP(S) URIs * Hop.java, LinkContext.java add new hop-type 'I' for inferred URIs; add corresponding LinkContext * ExtractorImpliedURI.java use new 'I'nferred types gojomo 2010-07-09 22:39:22 +00:00
• b4485b05b1 * EngineResource.java show proper separator in path prompt gojomo 2010-07-09 21:55:13 +00:00
• cfc4b347a6 [HER-1796] H3: interval rescheduling becomes erratic after URI rescheduled more than maxRetries times * CrawlURI.java improve comments for fetchAttempts methods (resetForRescheduling) added to clear per-scheduling state * WorkQueueFrontier.java call resetForRescheduling just before scheduling-at-specific-time * FrontierJournal.java, AbstractFrontier.java renamings: prefer 'reenqueue' instead of 'reschedule' for describing simple (non-timed) retries gojomo 2010-07-09 01:45:26 +00:00
• 274e522f85 [HER-1793] BOM in seeds.txt confuses heritrix * RegexpLineIterator.java ignore \ufeff along with leading whitespace gojomo 2010-07-02 17:47:41 +00:00
• f75f95648a [HER-1795] H3: Wrong schemaLocation in .cxml * **/*crawler-beans.cxml update all 2.0 spring schema refs to 2.5 gojomo 2010-07-02 00:58:23 +00:00
• 829a3f78b3 [HER-1794] Extra white spaces in pom.xml * **/pom.xml remove leading blank lines gojomo 2010-07-02 00:51:19 +00:00
• a68293982f Fix a couple of thinkos to really reserve RESERVE_BLOCKS of the intended RESERVE_BLOCK_SIZE nlevitt 2010-06-24 04:24:31 +00:00
• 4c10d4b54f * engine/pom.xml after test OOMEs on build machine, make sure tests use -server and a 256M heap gojomo 2010-06-23 00:47:24 +00:00
• bdc1e078ad * Flash.java protect against empty string, whitespace as well gojomo 2010-06-21 22:08:46 +00:00
• 943ecf3088 [HER-1762] H3 performance tuning * AbstractFrontier.java change config and defaults for 'inbound' and 'outbound': allowing any BlockinqQueue bean, of any size, to be specified if desired * **/profile-crawler-beans.cxml changed commented-out descriptions of defaults to reflect what's used if nothing is specified for 'outbound' and 'inbound' gojomo 2010-06-18 21:17:40 +00:00
• 4875135dc9 [HER-1762] H3 performance tuning * KeyedProperties.java avoid allocation hotspot shown by profiling (creating iterator for empty collection) gojomo 2010-06-18 02:37:42 +00:00
• e83a6eceed follow-up patch for [HER-1764] H3 needs in-UI quit/shutdown option * Flash.java avoid NPE when "flashdrop" cookie not set szznax 2010-06-16 22:02:41 +00:00
• edb70dae06 Fix [HER-1774] TransclusionDecideRule defies expectation when seed redirects with seed-redirects-new-seed enabled * TransclusionDecideRule.java refine further: count 'R's for minimum-hops-necessary-to-transclude, but not for maximum-allowable gojomo 2010-06-16 21:51:50 +00:00
• 2e03aa8cb7 [HER-1764] H3 needs in-UI quit/shutdown option * EngineResource.java add 'exit' control area at bottom of page allow JVM-exit if confirm-box checked and either no jobs built/running or extra confirm boxes for each built job checked * Flash.java fix to allow multiple flashes per page load * JobResource.java, CrawlJob.java move useful status string method to CrawlJob gojomo 2010-06-16 00:51:58 +00:00
• 4c1a4f2e12 remove unneeded import gojomo 2010-06-15 21:34:57 +00:00
• 3c8e72a20e * BloomFilterTest.java disable by renaming ('xest') those tests taking >15mins each gojomo 2010-06-15 21:33:34 +00:00
• 6be649b286 [HER-1783] BloomFilter64bit bit-length bug prevents full bitfield from being used; premature saturation * BloomFilter64bit.java include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes fit a number of problems with int/long overflow and bitwise ops add methods for reporting/testing * BloomFilter.java add methods for reporting/testing * BloomFilterTest.java, BloomFilter64bitTest.java more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming * BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java deleted as buggy or redundant * BenchmarkBlooms.java move to test source dir * BloomUriUniqFilter.java change to accept filter instance (rather than parameters) for added configuration flexibility fix comments * BloomUriUniqFilterTest.java supply filter not paramters gojomo 2010-06-15 21:30:34 +00:00
• b571730b4b Fix [HER-1774] TransclusionDecideRule defies expectation when seed redirects with seed-redirects-new-seed enabled * TransclusionDecideRule.java evaluate(): skip over 'R' hops without counting them as transcludes gojomo 2010-06-14 23:30:22 +00:00
• e1f76dbc07 [HER-1784] default profile MatchesListRegexDecideRule intended to REJECT doesn't * dist/src/main/conf/jobs/profile-defaults/profile-crawler-beans.cxml * engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml add intended REJECT decision for the late (rules[4]) MatchesListRegexDecideRule gojomo 2010-06-14 22:22:34 +00:00
• f9a3c2344c * BloomFilterTest.java explicit deferenced bloom instance to avoid OOME gojomo 2010-06-08 03:38:56 +00:00
• 4e1b47746c * commons/pom.xml increase heap for testing to 1024m to accomodate big bloom bitfield tests gojomo 2010-06-08 03:28:51 +00:00
• e9a11e56c6 HER-1783 BloomFilter64bit bit-length bug prevents full bitfield from being used; premature saturation * BloomFilter64bit.java fix long arithmetic calculating bit-length m in constructor slight optimization of add() * BloomFilterTest.java abstract superclass for bloom impl class tests * BloomFilter64bitTest.java sanity check that set bits, after a handful of adds into default-sized filter, occupy top and bottom 20% of range (which would have caught this bug earlier) gojomo 2010-06-08 02:39:35 +00:00
• e673c7d781 * WorkQueueFrontier.java experimental utility method to force-wake all snoozed (politeness and retry-delayed) queues gojomo 2010-06-04 22:31:16 +00:00
• 8b696da475 * CrawlURI.java dump more info if code that really shouldn't run actually runs gojomo 2010-06-04 22:27:59 +00:00
• 5f83fe75a2 HER-1010 crawl favicon.ico * (self tests) correct expectations to now include /favicon.ico gojomo 2010-06-03 00:31:32 +00:00
• f81a941d97 HER-1010 crawl favicon.ico * ExtractorHTTP.java consider /favicon.ico 'discovered' from every other HTTP fetch * Extractor.java utility method addOutlink() gojomo 2010-06-01 23:36:16 +00:00
• d1f27aaecd removed no-longer-referenced "recoveryDir" property from "frontier" bean szznax 2010-05-27 17:46:44 +00:00
• 72bdad74e9 * pom.xml install source jars to maven repository nlevitt 2010-05-26 04:41:22 +00:00
• 0dbc773fc8 Followup on HER-1728 - treat flashvars strings more like url query strings (key=value&[key=value]*) * ExtractorHTML.java considerQueryStringValues(): split pairs on "&" instead of ";" and url-decode values nlevitt 2010-05-15 02:13:19 +00:00
• f7a9327334 Fix for [HER-1772] NullPointerException in ExtractorHTML related to flashvars * modules/src/main/java/org/archive/modules/extractor/ExtractorHTML.java processGeneralTag(): check for null nameVal nlevitt 2010-05-11 22:11:38 +00:00
• e3c608fab4 HER-1762 H3 performance tuning * ExtractorHTML.java smaller caps on element-name, attribute-name, attribute-value lengths -- costly match-failures on bad/non-HTML may be eating a lot of cycles gojomo 2010-04-29 23:21:38 +00:00
• 95bf82e2d3 followup for HER-1688 remember external job directories * Engine.java suppress WARNING for common case of harmless failure during rescans: job already exists * EngineResource.java suppress rescan on each reload (ODD: steve had already committed this change -- I got the commit email -- but now the repo version is missing it. Sourceforge problems?) gojomo 2010-04-29 23:20:22 +00:00
• fcd0ec6699 HER-1762 H3 performance tuning * WorkQueueFrontier.java override schedule() to keep unique-testing in calling thread; may remove bottleneck shown by time-profiling and thread-observation (most ToeThreads stuck in candidatesProcessor, schedule()ing to frontier) * AbstractFrontier.java add comment about implementation vs. subclass override gojomo 2010-04-27 21:37:24 +00:00
• e10a14df54 HER-1753 offer means of setting login/password that doesn't leak to process info (as when literally specified to '-a' option) * Heritrix.java allow [username:]password to come from file, supplied to '-a' with '@' prefix also, clean up options names/help text to match current behavior also, offer warning about particularly weak (<8 characters, all letters, all numbers) passwords gojomo 2010-04-27 01:36:38 +00:00
• b639873dae Fix broken test * Engine.java tolerate (with return-false-forfailure rather than NPE) non-job directories * EngineTest.java use empty temp job directory rather than (possibly full-of-junk) general junit-tests directory gojomo 2010-04-27 00:44:36 +00:00
• 57c857d5c6 Refinement of fixes for HER-1688 remember external job directories * EngineResource.java avoid calling findJobConfigs() on each page load szznax 2010-04-26 23:33:40 +00:00