Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 9059395617 Make Bloom size configurable without system properties * BloomUriUniqFilter.java make bloom size configurable via bean properties change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps) * BloomUriUniqFilterTest.java update test for property-set sizing * BloomFilter64bit.java improve comment gojomo 2009-11-16 22:42:18 +00:00
• 69bedb8240 a bit more error-handling for input args szznax 2009-11-16 19:31:30 +00:00
• 24463da8bc Fix for [HER-1689] build, launch buttons remain enabled after problem building * CrawlJob.java renamed isContainerOk() to hasApplicationContext() increased granularity of isLaunchable() to offer launch button * JobResource.java more coarse check for hasApplicationContext() to disable build button szznax 2009-11-13 23:23:52 +00:00
• b72750691f [HER-1694] support for storing ftp fetches in warcs, along with improvements to ftp fetching, to match h1 ftp competency nlevitt 2009-11-13 02:58:56 +00:00
• 3b5e9f7c45 Improve pretty printing of arrays, in particular the fetch-history array. nlevitt 2009-11-13 02:21:38 +00:00
• df83c093e9 * PersistLogProcessor.java innerProcess(): persist only "persistent" data, like PersistStoreProcessor nlevitt 2009-11-13 01:51:56 +00:00
• 94e38c5b99 * logging.properties: default PersistLoadProcessor log level INFO to see if dedup info is loaded nlevitt 2009-11-13 01:50:13 +00:00
• 671f21ede6 Include thread id in log line format nlevitt 2009-11-12 02:50:13 +00:00
• 2740cc7bb0 Change the artifactId from commons to heritrix-commons, modules to heritrix-modules, engine to heritrix-engine. nlevitt 2009-11-12 02:48:38 +00:00
• f6b2066d6e [HER-1693] dedup numbers in crawl,host reports - patch contributed by Matt Warhaftig nlevitt 2009-11-12 01:38:28 +00:00
• 9743be5d8c Fix for [HER-1683] h3: inconsistent host classification for dns: urls * DefaultServerCache.java getHostFor(UURI): special case dns uris * ServerCache.java add getHostFor(UURI) to public interface * ServerCacheUtil.java remove now redundant wrapper method * DispositionProcessor.java AbstractFrontier.java IPQueueAssignmentPolicy.java BucketQueueAssignmentPolicy.java PreconditionEnforcer.java QuotaEnforcer.java StatisticsTracker.java ServerCacheTest.java FetchHTTP.java Kw3WriterProcessor.java WriterPoolProcessor.java changed to use serverCache.getHostFor(UURI) (a few cases remain that use getHostFor(String)) nlevitt 2009-11-12 01:28:22 +00:00
• da9eb5970c Fix bug preventing preload source from being loaded. nlevitt 2009-11-12 00:15:31 +00:00
• fe1a852974 Finishing up [HER-1561] * LICENSE.txt: list individual source files that are not (yet) apache licensed * pom.xml: include apache license in <licenses> section nlevitt 2009-11-11 03:23:09 +00:00
• c77de5fc08 [HER-1561] put apache license boilerplate on a couple of stragglers nlevitt 2009-11-11 03:19:06 +00:00
• 3e08d49640 Use json library jar instead of copy&pasted code. nlevitt 2009-11-11 02:30:39 +00:00
• 007e32f4f3 commit project-specific eclipse settings gojomo 2009-11-11 02:05:30 +00:00
• fa166002a5 commit project-specific eclipse settings gojomo 2009-11-11 02:03:38 +00:00
• 91ffdac6d9 [HER-1561] update license to Apache License 2.0 * ScopePlusOneDecideRule.java removing, in favor of SurtPrefixedDecideRule's 'alsoCheckVia' option * RecyclingFastBufferedOutputStream.java use library by deriving as subclass (rather than c&p), now that class is available in an included library (fastutil) gojomo 2009-11-11 01:40:43 +00:00
• ce79e8a76e [HER-1561] update license to Apache License 2.0 * SeekReader.java update to Apache * RecoveryLogMapper.java revert to LGPL pending verification of authors' approval for relicensing gojomo 2009-11-11 00:28:03 +00:00
• dd0ca367c3 [HER-1706] PersistLoadProcessor and PersistStoreProcessor can't share same BdbModule * BdbModule.java allow Database instance reuse if BdbConfig is identical instance * PersistProcessor.java convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG * PersistOnlineProcessor.java use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop() * PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java use HISTORY_DB_CONFIG gojomo 2009-11-10 23:48:10 +00:00
• 0497efbb32 [HER-1561] update license to Apache License 2.0 * (many) update license boilerplate * (few) delete no-longer used classes gojomo 2009-11-10 21:03:27 +00:00
• e3f3402213 * DefaultServerCache.java remove superfluous synchronization gojomo 2009-11-10 01:37:04 +00:00
• c2c0e5116e [HER-1705] 'harmless' low-memory-canary causing heap reference leak/OOME * CachedBdbMap.java, ObjectIdentityBdbCache.java only reinstall canary if map/cache still has 'db' field gojomo 2009-11-09 03:55:12 +00:00
• f5e0d333a2 [HER-1702] console rates (eg URIs/sec) sometimes way off, infinite, NaN (�) * CrawlStatSnapshot.java force double math, preventing premature loss of precision, zeros in denominators gojomo 2009-11-07 20:13:00 +00:00
• 9e0584b021 Followup for [HER-1703] convert all maps that operator might want to change on the fly to concurrent maps * HTTPContentDigest.java avoid using null for overlayable KeyedProperty -- fix broken NPE in test gojomo 2009-11-07 20:10:19 +00:00
• 11048de24c [HER-1703] convert all maps that operator might want to change on the fly to concurrent maps * KeyedProperties.java derive from ConcurrentHashMap * Sheet.java ensure internal map is always a concurrent-map (even when setMap is passed something else) * SheetOverlaysManager.java convert ruleAssociations, sheetNamesBySurt, sheetsByName to concurrent implementation classes * ScriptedProcessor.java, ScriptedDecideRule.java convert sharedMap to concurrent implementation * DecideRuledSheetAssociation.java ensure non-equal associations don't compareTo()==0 (implying equals, causing problems in sorted maps/sets) gojomo 2009-11-07 03:07:51 +00:00
• 06c63ffc5b * FrontierSummaryReport.java allow report even on empty frontier (as at pasue-at-end) gojomo 2009-11-07 03:02:09 +00:00
• bda89f5617 * JobResource.java add descriptive status to TITLE gojomo 2009-11-07 03:00:49 +00:00
• 9b211165df [HER-1698] report generation gives text of path, without showing report * JobResource.java make getHrefPath public static utility method * ReportGenResource.java use getHrefPath so '/anypath/' URLs are used where needed gojomo 2009-11-06 22:18:47 +00:00
• a7a990b638 [HER-1699] pause-at-finish leaves crawl in 'pausing' not 'paused' state * CrawlController.java if frontier pauses and pauseAtFinish is true; enter PAUSED immediately gojomo 2009-11-05 23:19:00 +00:00
• 7cf25a844e * StatisticsTracker.java make expensive seeds-result tracking optional gojomo 2009-10-26 12:08:49 +00:00
• 648ef7deef Fix issues with surt-source, surt-dump file paths * ConfigPathConfigurer, PathFixupListener notify beans when paths fixed * SurtPrefixedDecideRule.java only read source prefixes after path fixup; dump after read or any batch of seeds * SeedModule.java, TextSeedModule.java, SeedListener.java new concludedSeedBatch event * AbstractFrontier.java, StatisticsTracker.java ignore new concluded-batch event * AlertThreadGroup.java, JobResource.java new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger gojomo 2009-10-26 11:29:58 +00:00
• 59003e22ab Fix issues with surt-source, surt-dump file paths * ConfigPathConfigurer, PathFixupListener notify beans when paths fixed * SurtPrefixedDecideRule.java only read source prefixes after path fixup; dump after read or any batch of seeds * SeedModule.java, TextSeedModule.java, SeedListener.java new concludedSeedBatch event * AbstractFrontier.java, StatisticsTracker.java ignore new concluded-batch event * AlertThreadGroup.java, JobResource.java new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger gojomo 2009-10-26 11:27:09 +00:00
• 4ef460ac00 Crawl-distribution-related improvements * URIAuthorityBasedQueueAssignmentPolicy.java choose numbered subqueue on first-path-segment only, so similar URIs land in same subqueue * CrawlMapper.java add queueKey to diversion lines * HashCrawlMapper.java update license gojomo 2009-10-26 09:34:12 +00:00
• 2033c99fea * AbstractFrontier.java promote priority of manageThread (where that is respected) remove stray debug output gojomo 2009-10-23 22:38:21 +00:00
• 69736535a7 * profile-crawler-beans.cxml correct missing <property> tags in example commented-out canonicalization rules gojomo 2009-10-23 22:34:41 +00:00
• 2e31d58687 * ObjectIdentityBdbCache.java replace costly bdb size() operation with maintained counter gojomo 2009-10-23 22:33:36 +00:00
• 06223b2794 BDB JE tuning * BdbModule.java increase je.log.faultReadSize from 2K to 6K -- stats showed many faults (ideal is zero) gojomo 2009-10-23 21:55:06 +00:00
• f0c9608e59 Followup for [HER-1605] Refactor parameterizable policies out of frontier * CandidatesProcessor.java remove misguided lock preventing intended parallelism of pre-enqueue URL prep gojomo 2009-10-21 22:55:06 +00:00
• ea114357fb * FrontierPreparer.java fix inconsistency in canonicalizationPolicy getter/setter names making explicit spring-config impossible gojomo 2009-10-20 21:32:26 +00:00
• efee001bbc Fix for [HER-1688] remember external job directories * Engine.java protect against disappeared jobsDir in findJobConfigs() check for .jobpath files in findJobConfigs() added considerAsJobPath() log added jobs in considerAsJobDirectory() make considerAsJobDirectory() return true when jobConfig (already) exists added leaveJobPathFile() to write .jobpath file for newly added jobs * EngineResource.java call leaveJobPathFile() when job directory added successfully added messageDiv() to make messages more uniform and prominent re-scan jobConfigs on each page load - may obviate need for "rescan" button szznax 2009-10-15 01:03:29 +00:00
• 8311db6a26 minor style change szznax 2009-10-09 16:58:54 +00:00
• 72136dea31 handle null path on "create" action szznax 2009-10-09 00:04:31 +00:00
• 1a136b21d5 Fix for [HER-1419] can't create job in web ui without profile/job to base it on in jobs dir? * Engine.java createNewJobWithDefaults() to write profile-crawler-beans.cxml resoure into new job * EngineResource.java added FORM for new job dir and "create" action * profile-crawler-beans.cxml added resource from dist/src/main/conf/jobs/profile-defaults to bootstrap new jobs szznax 2009-10-08 23:55:42 +00:00
• 58513a8940 [HER-1687] H3 web UI: adding non-existent job directory gives unhelpful NPE * Engine.java return 'false' rather than NPE when given directory non-existent/contains no .cxml * EngineResource.java better handle empty 'add' path; show NACK flash when 'add' has no effect gojomo 2009-10-08 20:57:04 +00:00
• 4a0f809c0d Post-beta-build * **/pom.xml revert version name to SNAPSHOT gojomo 2009-10-07 11:54:12 +00:00
• af9345a3be * **/pom.xml rename version to 'beta' for distinguished 'beta' identified build (instead of SNAPSHOT/timestamp) gojomo 2009-10-07 11:25:20 +00:00
• 64a79ea9e6 Fix issues revealed in extra tests * H1toH3.map, migrate-template-crawler-beans.cxml, .classpath move resources to 'resources' src subdirectory, so included in built JAR move 'resources' directorys to library rather than source directories for dev-time reachability * MigrateH1to3Tool.java handle H1 values that just need capitalization report all 'no rule' situations as 'needs attention' touchup explanations and comments enable uriCanonicalizationRule overrides gojomo 2009-10-07 10:24:05 +00:00
• f114448e6f Fix issues revealed in extra tests * H1toH3.map, migrate-template-crawler-beans.cxml, .classpath move resources to 'resources' src subdirectory, so included in built JAR move 'resources' directorys to library rather than source directories for dev-time reachability * MigrateH1to3Tool.java handle H1 values that just need capitalization report all 'no rule' situations as 'needs attention' touchup explanations and comments enable uriCanonicalizationRule overrides gojomo 2009-10-07 10:23:36 +00:00
• 45f09818ae Fix intermittent ClassCastExceptions on job console * Histotable.java, ToePool.java only tally Strings; convert all histotable keys to strings for sorting gojomo 2009-10-07 10:18:21 +00:00
• f960a018c1 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * MigrateH1to3Tool.java add count of successfully migrated settings gojomo 2009-10-07 05:04:39 +00:00
• 9b3a65ab3a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * MigrateH1to3Tool.java improve output: explain skipped settings, affected paths * H1toH3.map truncate $na to $ gojomo 2009-10-07 05:01:16 +00:00
• 47cbdded9a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * MigrateH1to3Tool.java comment, clean unused code * H1toH3.map text map of {H3-override-beanpath, H1-settings-XPath, H1-default-value} * migrate-template-crawler-beans.cxml version of bundled profile slightly changed for better mapping to H1 (target replace range, seeds.txt, arcWriter) gojomo 2009-10-07 04:28:10 +00:00
• b7631e2b02 Misc fixes from H1-to-H3 testing * BdbModule.java be more robust about 0/-1 as 'don't care' values * AbstractFrontier.java reorder settable properties for ease of discovery * FetchHTTP.java correct type to allow setting property gojomo 2009-10-07 04:22:05 +00:00
• 07adde3a73 Fix impoliteness regression * CrawlURI.java return true calculated politeness delay, not 0! gojomo 2009-10-07 04:20:11 +00:00
• a582596092 Fix bean browser * StatisticsTracker.java, SeedsReport.java avoid naming method as getter, causing bean browser problems gojomo 2009-10-07 04:19:29 +00:00
• f097c07d51 Updates to be merged ... szznax 2009-10-07 02:06:06 +00:00
• 21353cc26d more profile cleanup * profile-crawler-beans.cxml add overlooked AbstractFrontier settings gojomo 2009-10-07 01:39:30 +00:00
• 791df1b78e [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * H1toH3.map order-XPath-to-override-key map for translating simple H1 settings to H3 overrides gojomo 2009-10-07 01:23:37 +00:00
• 78b3b0101f [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * H1toH3.map order-XPath-to-override-key map for translating simple H1 settings to H3 overrides gojomo 2009-10-07 01:14:44 +00:00
• 4cf3c0b151 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * H1toH3.map order-XPath-to-override-key map for translating simple H1 settings to H3 overrides gojomo 2009-10-07 01:11:12 +00:00
• 329204b9ff [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * MigrateSettings.java, MigrateH1toH3Tool.java renamings, 'flatten' method for building map * H1toH3.map order-XPath-to-override-key map for translating simple H1 settings to H3 overrides gojomo 2009-10-07 01:07:33 +00:00
• eb659a9fad more profile cleanup * profile-crawler-beans.cxml add alsoCheckVia improve description of commented-out defaults add CrawlLimitEnforcer * SurtPrefixedDecideRule.java remove obsolete property gojomo 2009-10-07 01:05:59 +00:00
• e2ee037786 Working commit... szznax 2009-10-06 22:24:19 +00:00
• 258da53d57 [HER-1676] improve bundled profiles to match best-practice, be self-documenting * profile-crawler-beans.cxml add comments for all unstated default values an operator might want to change * (many) reorder, refactor, rename to better minimize/match simple configuration gojomo 2009-10-06 19:35:05 +00:00
• 6b1d1963d8 Incremental checkin. szznax 2009-10-06 18:06:46 +00:00
• 6f7301d994 [HER-1676] improve bundled profiles to match best-practice, be self-documenting * defaults.xml deleted gojomo 2009-10-06 04:16:52 +00:00
• 3f7d40412b Fix broken unit tests * CandidatesProcessor.java properly set full-via CrawlURI on prereqs gojomo 2009-10-06 04:16:14 +00:00
• 742e4e8cad [HER-1676] improve bundled profiles to match best-practice, be self-documenting * DispositionProcessor.java indirect robots-through-metadata, to obviate explicit declaration in beans.xml gojomo 2009-10-06 04:14:29 +00:00
• e2a61792a7 * (many) harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits) gojomo 2009-10-06 04:09:49 +00:00
• 5c50f9c104 [HER-1676] improve bundled profiles to match best-practice, be self-documenting * profile-crawler-beans.cxml merge 'defaults' back in, beef up comments, strip unnecessary declarations gojomo 2009-10-06 04:07:25 +00:00
• da95932598 * CrawlMetadata.java set default RobotsHonoringPolicy, making explicit declaration unnecessary * RobotsHonoringPolicy.java apache license gojomo 2009-10-06 04:00:58 +00:00
• 2d3f569f15 * InterruptibleCharSequenceTest.java better output to diagnose intermittent failure gojomo 2009-10-06 03:58:22 +00:00
• 3654e330cc * (many) harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits) gojomo 2009-10-06 03:47:38 +00:00
• d86f5f7403 * (many) harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits) gojomo 2009-10-06 03:45:44 +00:00
• dbc1c1b411 * KeyWordProcessor.java update shouldProcess() to be based on via CrawlURI's type gojomo 2009-10-06 02:28:47 +00:00
• 4ce8aef47a Remove obsolete test class gojomo 2009-10-06 01:40:22 +00:00
• 6769cdb2dc Fix inadvertent build-breaking upward dependency from modules to engine * modules//ProcessorChain.java send updates to ChainStatusReceiver interface * engine//ToeThread.java implement ChainStatusReceiver interface gojomo 2009-10-06 01:01:32 +00:00
• 92da7ab8cc Incremental check-in with DOM search and XPath interrogation of scope node from existing order XML. szznax 2009-10-06 00:36:00 +00:00
• 047a157fe8 [HER-1605] Refactor parameterizable policies out of frontier * selftest-crawler.beans.cxml, FormAuthSelfTest.java, Precedence[134]SelfTest.java update self-tests for new chains refactoring gojomo 2009-10-05 23:44:30 +00:00
• 13c9cc1915 [HER-1605] Refactor parameterizable policies out of frontier * defaults.xml update bundled profile for new chains refactoring * CrawlController.java move to three processor-chains rather than one * Frontier.java (loadSeeds) removed * ToeThread.java delegate most processing-loop to FetchChain and DispositionChain * AbstractFrontier.java, WorkQueueFrontier.java move policies/calculation out to processors * CandidatesProcessor.java new processor for DispositionChain that runs every outlink through CandidateChain * CrawlStateUpdater.java -> DispositionProcessor.java rename, expand to prep CrawlURI for frontier * FrontierScheduler.java deleted; use CandidatesProcessor/CandidateChain * LinksScoper.java deprecated; use CandidatesProcessor/CandidateChain/CandidateScoper (only temporarily retained for ease of comparison) * CandidateScoper.java simple single-URI scope-testing for CandidateChain * FrontierPreparer.java precalculate all frontier-policies in CandidateChain, before scheduling * PreconditionEnforcer.java ProcessorURI->CrawlURI; take on some prerequisite preparation previously deferred to elsewhere * ProcessorsReport.java update for 3-chains of Processors * SheetOverlaysManager.java (applyOverridesTo) moved here for broader use * CandidateChain.java, FetchChain.java, DispositionChain.java role-specific subclasses of ProcessorChain (suitable for type-based autowiring) * CrawlURI.java new fields/accessors of use to new chains/frontier * PostProcessor.java deleted; skip-to-'postprocessing' is now skip-to-end-of-chain * ProcessorChain.java take-on control loop formerly in ToeThread * ProcessResult.java absorb ProcessStatus eliminate problematic STUCK result gojomo 2009-10-05 23:41:59 +00:00
• 30c729edb3 More ProcessorURI -> CrawlURI updates gojomo 2009-10-05 23:25:12 +00:00
• b7d1a79b4d More ProcessorURI -> CrawlURI updates gojomo 2009-10-05 23:08:12 +00:00
• 8f7ad4eae4 work for [HER-1605] Refactor parameterizable policies out of frontier * QueueAssignmentPolicy.java, URIAuthorityBasedQueueAssignmentPolicy.java move 'force-queue-assignment' setting from frontier to policy gojomo 2009-10-05 23:04:37 +00:00
• 10275d7fa1 More ProcessorURI -> CrawlURI updates gojomo 2009-10-05 23:01:40 +00:00
• c7ed3e775d Misc * ArchiveUtils.java (checkContinue) utility method relocated from ToeThread for converting interrupt status to exception * ClassKeyMatchesRegExpDecideRule.java properly set up crawlController property for auto-wiring gojomo 2009-10-05 22:59:46 +00:00
• 16538f175c Cleanup, refactoring * (many) update license notice to Apache where appropriate eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package) other comment/warning/unused code cleanup gojomo 2009-10-05 22:49:19 +00:00
• c04c5dbdbc Improve test robustness * Engine.java wait another second before testing for job wrap-up gojomo 2009-10-05 22:25:44 +00:00
• 62690268b7 Initial checkin for [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml szznax 2009-10-05 20:00:56 +00:00
• ba5bc1a20c Fix unit tests (carried over from H1) * WARCWriterTest.java (testSpaceInUrl),(testTabInURL): don't expect exception; expect nothing-written szznax 2009-10-05 18:35:06 +00:00
• 86974e2e1d Cleanup post-fix * ObjectIdentityBdbCache.java with late ref-enqueuing confirmed as cause of problems, cleaning out unnecessary steps, improving comments gojomo 2009-10-04 06:15:35 +00:00
• 75676ac905 Attempt at fixing build-box-only warnings * ObjectIdentityBdbCache.java explicitly page-out lingering entry; temporary System.err logging to see if problem is momentary lag in RefQueue-appearance gojomo 2009-10-04 05:14:59 +00:00
• c05044d51b Eliminate double-database close log messages * BdbModule.java split openDatabase() to openManagedDatabase() (auto-closed) and plain openDatabase() (caller-closes) set TempStoredSortedMap to use unmanaged openDatabase * (others) use openManagedDatabase() gojomo 2009-10-04 05:06:14 +00:00
• c3914ea943 Attempt at fixing build-box-only warnings * ObjectIdentityBdbCache.java add earlier check/better logging for error state gojomo 2009-10-04 03:31:45 +00:00
• 839e895a8d Attempt at fixing build-box-only warnings * ObjectIdentityBdbCache.java synchronize critical sections on memMap, diskMap rather than enclosing object gojomo 2009-10-03 23:55:59 +00:00
• 3cb6b59b8c [HER-769] Support case where millions of seeds. * BdbModule.java add temp-stored-map service method remove unused secondary-db support * TempStoredSortedMap.java StoredSortedMap that can destroy its underlying database after temporary use * StatisticsTracker.java update various sorted-by-decreasing-frequency methods to use temp StoredMaps, with duplicate keys that are negative counts * (Multiple)Report.java update to use new duplicate-keyed frequency maps gojomo 2009-10-03 23:28:11 +00:00
• 97778493e2 As part of fix for [HER-1671] WARCWriterProcessor error when writing problematic (illegal control character?) outlink catch IllegalArgumentException in ANVLRecord.addLabelValue() to allow WARCWriter to continue processing, and move WARCWriter.createRecordHeader() into its own try-catch block to with logging prevent fouling record when problems occur creating header. * ANVLRecord.java catch IllegalArgumentException in addLabelValue and log warning to give WARCWriter to continue processing * WARCWriter.java move createRecordHeader() into its own try-catch block to log severe problem writing record without fouling the WARC, and have baseCharacterCheck() throw IllegalArgumentException in accord with ANVLRecord.addLabelValue * WARCWriterTest.java throw IllegalArgumentException in accord with WARCWriter.createRecordHeader() szznax 2009-10-03 00:41:59 +00:00
• 03df2a9a99 As part of fix for [HER-1671] WARCWriterProcessor error when writing problematic (illegal control character?) outlink - percent encode control-characters in 'userinfo' part of URI as we do for uriPath and uriQuery, so we don't raise exception when adding outlinks to metadata record * UURIFactory.java ensureMinimalEscaping() on 'userinfo' in UURIFactory.fixupAuthority() szznax 2009-10-02 01:39:44 +00:00
• 677a816db4 Fix for [HER-1681] FetchFTP module doesn't set content-begin mark correctly. Bug reported and fix suggested by Tomas Ukkonen. * FetchFTP.java saveToRecorder(): call recorder.markContentBegin() after recorder.outputWrap() szznax 2009-10-01 02:21:17 +00:00
• fef7a93bad As part of fix for [HER-1634] verify/port/update latest WARC implementation in 1.14.3/heritrix2 to latest ISO document, heritrix3 - in warcinfo record, "robots" field should be "robots: classic" rather than "robots: CLASSIC" * WARCWriterProcessor.java made robots entry in metadata lower-case szznax 2009-09-30 17:06:53 +00:00