Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#750

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• eb9bf5a8e2 BDBJE updates, refactoring * commons/pom.xml, .classpath update to BDB-JE 4.0.71 * BdbModule.java (getStoredQueue) added (getStoredMap) made more general (openDatabase) all databases now 'managed' * TempStoredSortedMap.java -> DisposableStoredSortedMap.java renamed * StatisticsTracker.java, **Report.java use new name * StoredQueue.java, StoredQueueTest.java move to org.archive.bdb package * BdbMultipleWorkQueues.java, PrefixFinderTest.java update for BDB-JE 4 changes * BdbFrontier.java use getStoredQueue drop unneeded custom serialization support * WorkQueueFrontier.java drop unneeded custom serialization support gojomo 2009-11-23 04:41:32 +00:00
• 8251620481 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * profile-crawler-beans.cxml*2, migrate-template-crawler-beans.cxml add CheckpointService as recommended bean gojomo 2009-11-20 18:35:13 +00:00
• f9d56e3f6b move json dependency to commons from engine gojomo 2009-11-20 02:03:27 +00:00
• 25a568fff7 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * JobResource.java suppress checkpoint-selector after terminate * Checkpoint.java adapt to work without direct reference to CheckpointService * CheckpointService.java configure Checkpoint without making it dependent on CheckpointService gojomo 2009-11-20 01:24:57 +00:00
• daab317c94 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls processor-related bean checkpoint support: * Checkpoint.java move to commons for proper dependency graph * (others) recognize above gojomo 2009-11-20 01:10:32 +00:00
• dfacba51e5 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls UI support * JobResource.java enable 'checkpoint' button as appropriate display active recovery-checkpoint or select bos of recovery-options when available * CrawlJob.java (getCheckpointService) accessor gojomo 2009-11-20 01:02:14 +00:00
• 50ee78f6d2 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * BdbFrontier.java implement Checkpointable; save state via JSON or BDB; restore queues-of-queues on recovery * WorkQueueFrontier.java (deactivateQueue) (retireQueue) protected * BdbUriUniqFilter.java implement Checkpointable; save state via JSON or BDB; restore * WorkQueue.java make peekItem transient -- should not be held through checkpoint make retired protected * Frontier.java, AbstractFrontier.java remove/update obsolete code gojomo 2009-11-20 00:59:00 +00:00
• be17d5ed33 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * CrawlController.java remove older checkpoint-functionality passthrough implement Checkpointable so behavior can vary during recovery only trigger annnounce of seeds on 'cold' (non-recovery) start gojomo 2009-11-20 00:51:59 +00:00
• 6481081697 * DefaultWriterPoolSettings.java remove obsolete serialization/checkpoint support gojomo 2009-11-20 00:50:17 +00:00
• ac6b58cb0e [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * Checkpoint.java (generateFrom) version more suitable for unit testing gojomo 2009-11-20 00:49:31 +00:00
• fbd3af35ac [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * PersistLogProcessor.java rotate logs on checkpoint * CrawlerJournal.java, FrontierJournal.java rotate for checkpoint support without full Checkpointable gojomo 2009-11-20 00:47:54 +00:00
• 8591002573 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * BdbModule.java implement Checkpointable; save/restore consistent sets of jdb log files * BdbModuleTest.java testDoCheckpoint gojomo 2009-11-20 00:45:07 +00:00
• f01ad9ada8 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls stats & logging checkpoint support * StatisticsTracker.java implement Checkpointable; save small state in JSON, recycle BDB data when recovering * CrawlerLoggerModule.java implement Checkpointable; rotate logs on checkpoint * GenerationFileHandler.java avoid clobbering leftover logs from earlier futures via moveAside * CrawlStatSnapshot.java (sameProgressAs) test for lack-of-progress * ObjectIdentityMemCache.java, TopNSet.java offer deeper access to assist checkpoint/resume gojomo 2009-11-19 23:18:15 +00:00
• b63c2bdc77 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls processor-related bean checkpoint support: * ToeThread.java use getBeanName gojomo 2009-11-19 23:12:47 +00:00
• 124d1524d9 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls processor-related bean checkpoint support: * Processor.java implement Checkpointable and BeanNameAware; methods to support JSON state-saving * DispositionChain.java implement Checkpointable and during-checkpoint disposition-lockout for state stability gojomo 2009-11-19 23:10:13 +00:00
• 2bb65e9200 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls processor-related bean checkpoint support: * Processor.java implement Checkpointable and BeanNameAware; methods to support JSON state-saving * DispositionChain.java implement Checkpointable and during-checkpoint disposition-lockout for state stability * **Extractor**.java pull common reporting & checkpointing up to Extractor.java * WriterPoolProcessor.java implement Checkpointable; freeze and roll writer-pool during checkpoint * PersistLogProcessor.java roll log on checkpoint * TextSeedModule.java, SeedModule.java eliminate obsolete (and probably unnecessary) checkpoint support * (others) adapt to above changes gojomo 2009-11-19 22:52:39 +00:00
• bdf4437ca2 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls core checkpointing system classes: * Checkpointable.java basic interface for checkpoint aware/capable beans * Checkpoint.java one Checkpoint, either in progress or recovery -- a name + storage directory * CheckpointService.java helper bean to enable, trigger checkpoints and recovery * Checkpointer.java(*2), CheckpointInputStream.java, CheckpointRecovery.java DefaultCheckpointRecovery.java, RecoverAction.java (deleted) no longer used gojomo 2009-11-19 22:45:07 +00:00
• e42e2ed7f6 Cleanup/consolidation/expansion of *Utils classes * IoUtils.java remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils * ArchiveUtils.java, FileUtils.java receive relocated methods; eliminated deprecated or unused methods * JSONUtils.java new class to collect common JSON actions * Iteratorable.java wrap Iterator as Iterable for foreach usage * (many) update to use alternate utils methods gojomo 2009-11-19 22:39:53 +00:00
• 300ce15a06 * EditRepresentation.java add filename to TITLE to help identify browser tabs gojomo 2009-11-19 21:58:12 +00:00
• b97319fb07 reformat gojomo 2009-11-19 21:57:05 +00:00
• 2cb91e33f5 Fix or suppress warnings gojomo 2009-11-19 21:55:43 +00:00
• f9e1191144 Fix build - cleanup leak of incomplete code to prior commit gojomo 2009-11-18 20:53:41 +00:00
• be81d3733a [HER-1710] on problem loading persist log, close reader nlevitt 2009-11-18 03:02:28 +00:00
• 56055d8236 [HER-1709] h3 - dumpPendingAtClose doesn't work * CrawlController.java avoid unnecessarily early/redundant close of logs files (and checkpointer cleanup) * WorkQueueFrontier.java add stop() that does close() * BdbMultipleWorkQueues.java add cursor.close() * AbstractFrontier.java, BdbFrontier.java use closeQuietly gojomo 2009-11-18 00:29:37 +00:00
• 148c7332c4 Merge ServerCache+ServerCacheUtil abstract superclass ServerCache * ServerCache.java make abstract, implement getHostFor(UURI) and getServerFor(UURI) * DefaultServerCache.java extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI) * ServerCacheUtil.java removed * other classes change usage of ServerCacheUtil to ServerCache nlevitt 2009-11-17 06:10:42 +00:00
• e57ad68230 * AbstractFrontier.java implement ApplicationListener like it seems to want to, being that it has a method onApplicationEvent() nlevitt 2009-11-17 04:53:11 +00:00
• f9843974bc Update commons-net to 2.0 to incorporate fix to this bug - http://issues.apache.org/jira/browse/NET-36 - we saw the problem manifest in an archive-it crawl. * ClientFTP.java override buggy org.apache.commons.net.ftp.FTP.getReplyStrings() with fixed version, see https://issues.apache.org/jira/browse/NET-257 * .classpath, commons/pom.xml update dependency to 2.0 jar nlevitt 2009-11-17 02:49:57 +00:00
• 92f3bfe0ec check for exactly 2 args, tightened up SAXException handling by making sourceOrderXmlDom a protected field, cleaned up unused import and whitespace. szznax 2009-11-16 23:45:04 +00:00
• 9059395617 Make Bloom size configurable without system properties * BloomUriUniqFilter.java make bloom size configurable via bean properties change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps) * BloomUriUniqFilterTest.java update test for property-set sizing * BloomFilter64bit.java improve comment gojomo 2009-11-16 22:42:18 +00:00
• 69bedb8240 a bit more error-handling for input args szznax 2009-11-16 19:31:30 +00:00
• 24463da8bc Fix for [HER-1689] build, launch buttons remain enabled after problem building * CrawlJob.java renamed isContainerOk() to hasApplicationContext() increased granularity of isLaunchable() to offer launch button * JobResource.java more coarse check for hasApplicationContext() to disable build button szznax 2009-11-13 23:23:52 +00:00
• b72750691f [HER-1694] support for storing ftp fetches in warcs, along with improvements to ftp fetching, to match h1 ftp competency nlevitt 2009-11-13 02:58:56 +00:00
• 3b5e9f7c45 Improve pretty printing of arrays, in particular the fetch-history array. nlevitt 2009-11-13 02:21:38 +00:00
• df83c093e9 * PersistLogProcessor.java innerProcess(): persist only "persistent" data, like PersistStoreProcessor nlevitt 2009-11-13 01:51:56 +00:00
• 94e38c5b99 * logging.properties: default PersistLoadProcessor log level INFO to see if dedup info is loaded nlevitt 2009-11-13 01:50:13 +00:00
• 671f21ede6 Include thread id in log line format nlevitt 2009-11-12 02:50:13 +00:00
• 2740cc7bb0 Change the artifactId from commons to heritrix-commons, modules to heritrix-modules, engine to heritrix-engine. nlevitt 2009-11-12 02:48:38 +00:00
• f6b2066d6e [HER-1693] dedup numbers in crawl,host reports - patch contributed by Matt Warhaftig nlevitt 2009-11-12 01:38:28 +00:00
• 9743be5d8c Fix for [HER-1683] h3: inconsistent host classification for dns: urls * DefaultServerCache.java getHostFor(UURI): special case dns uris * ServerCache.java add getHostFor(UURI) to public interface * ServerCacheUtil.java remove now redundant wrapper method * DispositionProcessor.java AbstractFrontier.java IPQueueAssignmentPolicy.java BucketQueueAssignmentPolicy.java PreconditionEnforcer.java QuotaEnforcer.java StatisticsTracker.java ServerCacheTest.java FetchHTTP.java Kw3WriterProcessor.java WriterPoolProcessor.java changed to use serverCache.getHostFor(UURI) (a few cases remain that use getHostFor(String)) nlevitt 2009-11-12 01:28:22 +00:00
• da9eb5970c Fix bug preventing preload source from being loaded. nlevitt 2009-11-12 00:15:31 +00:00
• fe1a852974 Finishing up [HER-1561] * LICENSE.txt: list individual source files that are not (yet) apache licensed * pom.xml: include apache license in <licenses> section nlevitt 2009-11-11 03:23:09 +00:00
• c77de5fc08 [HER-1561] put apache license boilerplate on a couple of stragglers nlevitt 2009-11-11 03:19:06 +00:00
• 3e08d49640 Use json library jar instead of copy&pasted code. nlevitt 2009-11-11 02:30:39 +00:00
• 007e32f4f3 commit project-specific eclipse settings gojomo 2009-11-11 02:05:30 +00:00
• fa166002a5 commit project-specific eclipse settings gojomo 2009-11-11 02:03:38 +00:00
• 91ffdac6d9 [HER-1561] update license to Apache License 2.0 * ScopePlusOneDecideRule.java removing, in favor of SurtPrefixedDecideRule's 'alsoCheckVia' option * RecyclingFastBufferedOutputStream.java use library by deriving as subclass (rather than c&p), now that class is available in an included library (fastutil) gojomo 2009-11-11 01:40:43 +00:00
• ce79e8a76e [HER-1561] update license to Apache License 2.0 * SeekReader.java update to Apache * RecoveryLogMapper.java revert to LGPL pending verification of authors' approval for relicensing gojomo 2009-11-11 00:28:03 +00:00
• dd0ca367c3 [HER-1706] PersistLoadProcessor and PersistStoreProcessor can't share same BdbModule * BdbModule.java allow Database instance reuse if BdbConfig is identical instance * PersistProcessor.java convert shared BdbConfig for history db to static constant HISTORY_DB_CONFIG * PersistOnlineProcessor.java use HISTORY_DB_CONFIG, don't close possibly-shared DB on stop() * PrecedenceLoader.java, PreloadedUriPrecedencePolicy.java use HISTORY_DB_CONFIG gojomo 2009-11-10 23:48:10 +00:00
• 0497efbb32 [HER-1561] update license to Apache License 2.0 * (many) update license boilerplate * (few) delete no-longer used classes gojomo 2009-11-10 21:03:27 +00:00
• e3f3402213 * DefaultServerCache.java remove superfluous synchronization gojomo 2009-11-10 01:37:04 +00:00
• c2c0e5116e [HER-1705] 'harmless' low-memory-canary causing heap reference leak/OOME * CachedBdbMap.java, ObjectIdentityBdbCache.java only reinstall canary if map/cache still has 'db' field gojomo 2009-11-09 03:55:12 +00:00
• f5e0d333a2 [HER-1702] console rates (eg URIs/sec) sometimes way off, infinite, NaN (�) * CrawlStatSnapshot.java force double math, preventing premature loss of precision, zeros in denominators gojomo 2009-11-07 20:13:00 +00:00
• 9e0584b021 Followup for [HER-1703] convert all maps that operator might want to change on the fly to concurrent maps * HTTPContentDigest.java avoid using null for overlayable KeyedProperty -- fix broken NPE in test gojomo 2009-11-07 20:10:19 +00:00
• 11048de24c [HER-1703] convert all maps that operator might want to change on the fly to concurrent maps * KeyedProperties.java derive from ConcurrentHashMap * Sheet.java ensure internal map is always a concurrent-map (even when setMap is passed something else) * SheetOverlaysManager.java convert ruleAssociations, sheetNamesBySurt, sheetsByName to concurrent implementation classes * ScriptedProcessor.java, ScriptedDecideRule.java convert sharedMap to concurrent implementation * DecideRuledSheetAssociation.java ensure non-equal associations don't compareTo()==0 (implying equals, causing problems in sorted maps/sets) gojomo 2009-11-07 03:07:51 +00:00
• 06c63ffc5b * FrontierSummaryReport.java allow report even on empty frontier (as at pasue-at-end) gojomo 2009-11-07 03:02:09 +00:00
• bda89f5617 * JobResource.java add descriptive status to TITLE gojomo 2009-11-07 03:00:49 +00:00
• 9b211165df [HER-1698] report generation gives text of path, without showing report * JobResource.java make getHrefPath public static utility method * ReportGenResource.java use getHrefPath so '/anypath/' URLs are used where needed gojomo 2009-11-06 22:18:47 +00:00
• a7a990b638 [HER-1699] pause-at-finish leaves crawl in 'pausing' not 'paused' state * CrawlController.java if frontier pauses and pauseAtFinish is true; enter PAUSED immediately gojomo 2009-11-05 23:19:00 +00:00
• 7cf25a844e * StatisticsTracker.java make expensive seeds-result tracking optional gojomo 2009-10-26 12:08:49 +00:00
• 648ef7deef Fix issues with surt-source, surt-dump file paths * ConfigPathConfigurer, PathFixupListener notify beans when paths fixed * SurtPrefixedDecideRule.java only read source prefixes after path fixup; dump after read or any batch of seeds * SeedModule.java, TextSeedModule.java, SeedListener.java new concludedSeedBatch event * AbstractFrontier.java, StatisticsTracker.java ignore new concluded-batch event * AlertThreadGroup.java, JobResource.java new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger gojomo 2009-10-26 11:29:58 +00:00
• 59003e22ab Fix issues with surt-source, surt-dump file paths * ConfigPathConfigurer, PathFixupListener notify beans when paths fixed * SurtPrefixedDecideRule.java only read source prefixes after path fixup; dump after read or any batch of seeds * SeedModule.java, TextSeedModule.java, SeedListener.java new concludedSeedBatch event * AbstractFrontier.java, StatisticsTracker.java ignore new concluded-batch event * AlertThreadGroup.java, JobResource.java new mechanism for threads not part of AlertThreadGroup to temporarily redirect their logging to provided (job) logger gojomo 2009-10-26 11:27:09 +00:00
• 4ef460ac00 Crawl-distribution-related improvements * URIAuthorityBasedQueueAssignmentPolicy.java choose numbered subqueue on first-path-segment only, so similar URIs land in same subqueue * CrawlMapper.java add queueKey to diversion lines * HashCrawlMapper.java update license gojomo 2009-10-26 09:34:12 +00:00
• 2033c99fea * AbstractFrontier.java promote priority of manageThread (where that is respected) remove stray debug output gojomo 2009-10-23 22:38:21 +00:00
• 69736535a7 * profile-crawler-beans.cxml correct missing <property> tags in example commented-out canonicalization rules gojomo 2009-10-23 22:34:41 +00:00
• 2e31d58687 * ObjectIdentityBdbCache.java replace costly bdb size() operation with maintained counter gojomo 2009-10-23 22:33:36 +00:00
• 06223b2794 BDB JE tuning * BdbModule.java increase je.log.faultReadSize from 2K to 6K -- stats showed many faults (ideal is zero) gojomo 2009-10-23 21:55:06 +00:00
• f0c9608e59 Followup for [HER-1605] Refactor parameterizable policies out of frontier * CandidatesProcessor.java remove misguided lock preventing intended parallelism of pre-enqueue URL prep gojomo 2009-10-21 22:55:06 +00:00
• ea114357fb * FrontierPreparer.java fix inconsistency in canonicalizationPolicy getter/setter names making explicit spring-config impossible gojomo 2009-10-20 21:32:26 +00:00
• efee001bbc Fix for [HER-1688] remember external job directories * Engine.java protect against disappeared jobsDir in findJobConfigs() check for .jobpath files in findJobConfigs() added considerAsJobPath() log added jobs in considerAsJobDirectory() make considerAsJobDirectory() return true when jobConfig (already) exists added leaveJobPathFile() to write .jobpath file for newly added jobs * EngineResource.java call leaveJobPathFile() when job directory added successfully added messageDiv() to make messages more uniform and prominent re-scan jobConfigs on each page load - may obviate need for "rescan" button szznax 2009-10-15 01:03:29 +00:00
• 8311db6a26 minor style change szznax 2009-10-09 16:58:54 +00:00
• 72136dea31 handle null path on "create" action szznax 2009-10-09 00:04:31 +00:00
• 1a136b21d5 Fix for [HER-1419] can't create job in web ui without profile/job to base it on in jobs dir? * Engine.java createNewJobWithDefaults() to write profile-crawler-beans.cxml resoure into new job * EngineResource.java added FORM for new job dir and "create" action * profile-crawler-beans.cxml added resource from dist/src/main/conf/jobs/profile-defaults to bootstrap new jobs szznax 2009-10-08 23:55:42 +00:00
• 58513a8940 [HER-1687] H3 web UI: adding non-existent job directory gives unhelpful NPE * Engine.java return 'false' rather than NPE when given directory non-existent/contains no .cxml * EngineResource.java better handle empty 'add' path; show NACK flash when 'add' has no effect gojomo 2009-10-08 20:57:04 +00:00
• 4a0f809c0d Post-beta-build * **/pom.xml revert version name to SNAPSHOT gojomo 2009-10-07 11:54:12 +00:00
• af9345a3be * **/pom.xml rename version to 'beta' for distinguished 'beta' identified build (instead of SNAPSHOT/timestamp) gojomo 2009-10-07 11:25:20 +00:00
• 64a79ea9e6 Fix issues revealed in extra tests * H1toH3.map, migrate-template-crawler-beans.cxml, .classpath move resources to 'resources' src subdirectory, so included in built JAR move 'resources' directorys to library rather than source directories for dev-time reachability * MigrateH1to3Tool.java handle H1 values that just need capitalization report all 'no rule' situations as 'needs attention' touchup explanations and comments enable uriCanonicalizationRule overrides gojomo 2009-10-07 10:24:05 +00:00
• f114448e6f Fix issues revealed in extra tests * H1toH3.map, migrate-template-crawler-beans.cxml, .classpath move resources to 'resources' src subdirectory, so included in built JAR move 'resources' directorys to library rather than source directories for dev-time reachability * MigrateH1to3Tool.java handle H1 values that just need capitalization report all 'no rule' situations as 'needs attention' touchup explanations and comments enable uriCanonicalizationRule overrides gojomo 2009-10-07 10:23:36 +00:00
• 45f09818ae Fix intermittent ClassCastExceptions on job console * Histotable.java, ToePool.java only tally Strings; convert all histotable keys to strings for sorting gojomo 2009-10-07 10:18:21 +00:00
• f960a018c1 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * MigrateH1to3Tool.java add count of successfully migrated settings gojomo 2009-10-07 05:04:39 +00:00
• 9b3a65ab3a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * MigrateH1to3Tool.java improve output: explain skipped settings, affected paths * H1toH3.map truncate $na to $ gojomo 2009-10-07 05:01:16 +00:00
• 47cbdded9a [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * MigrateH1to3Tool.java comment, clean unused code * H1toH3.map text map of {H3-override-beanpath, H1-settings-XPath, H1-default-value} * migrate-template-crawler-beans.cxml version of bundled profile slightly changed for better mapping to H1 (target replace range, seeds.txt, arcWriter) gojomo 2009-10-07 04:28:10 +00:00
• b7631e2b02 Misc fixes from H1-to-H3 testing * BdbModule.java be more robust about 0/-1 as 'don't care' values * AbstractFrontier.java reorder settable properties for ease of discovery * FetchHTTP.java correct type to allow setting property gojomo 2009-10-07 04:22:05 +00:00
• 07adde3a73 Fix impoliteness regression * CrawlURI.java return true calculated politeness delay, not 0! gojomo 2009-10-07 04:20:11 +00:00
• a582596092 Fix bean browser * StatisticsTracker.java, SeedsReport.java avoid naming method as getter, causing bean browser problems gojomo 2009-10-07 04:19:29 +00:00
• f097c07d51 Updates to be merged ... szznax 2009-10-07 02:06:06 +00:00
• 21353cc26d more profile cleanup * profile-crawler-beans.cxml add overlooked AbstractFrontier settings gojomo 2009-10-07 01:39:30 +00:00
• 791df1b78e [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * H1toH3.map order-XPath-to-override-key map for translating simple H1 settings to H3 overrides gojomo 2009-10-07 01:23:37 +00:00
• 78b3b0101f [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * H1toH3.map order-XPath-to-override-key map for translating simple H1 settings to H3 overrides gojomo 2009-10-07 01:14:44 +00:00
• 4cf3c0b151 [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * H1toH3.map order-XPath-to-override-key map for translating simple H1 settings to H3 overrides gojomo 2009-10-07 01:11:12 +00:00
• 329204b9ff [HER-1641] migration tool to convert Heritrix 1.14.x order.xml to H3 crawler-beans.cxml * MigrateSettings.java, MigrateH1toH3Tool.java renamings, 'flatten' method for building map * H1toH3.map order-XPath-to-override-key map for translating simple H1 settings to H3 overrides gojomo 2009-10-07 01:07:33 +00:00
• eb659a9fad more profile cleanup * profile-crawler-beans.cxml add alsoCheckVia improve description of commented-out defaults add CrawlLimitEnforcer * SurtPrefixedDecideRule.java remove obsolete property gojomo 2009-10-07 01:05:59 +00:00
• e2ee037786 Working commit... szznax 2009-10-06 22:24:19 +00:00
• 258da53d57 [HER-1676] improve bundled profiles to match best-practice, be self-documenting * profile-crawler-beans.cxml add comments for all unstated default values an operator might want to change * (many) reorder, refactor, rename to better minimize/match simple configuration gojomo 2009-10-06 19:35:05 +00:00
• 6b1d1963d8 Incremental checkin. szznax 2009-10-06 18:06:46 +00:00
• 6f7301d994 [HER-1676] improve bundled profiles to match best-practice, be self-documenting * defaults.xml deleted gojomo 2009-10-06 04:16:52 +00:00
• 3f7d40412b Fix broken unit tests * CandidatesProcessor.java properly set full-via CrawlURI on prereqs gojomo 2009-10-06 04:16:14 +00:00
• 742e4e8cad [HER-1676] improve bundled profiles to match best-practice, be self-documenting * DispositionProcessor.java indirect robots-through-metadata, to obviate explicit declaration in beans.xml gojomo 2009-10-06 04:14:29 +00:00
• e2a61792a7 * (many) harmonize our usage on '[Rr]egex' as the abbrieviation for regular-expression (replacing RegExp/etc.; 'regex' is 30-40% more prevalent by google hits) gojomo 2009-10-06 04:09:49 +00:00
• 5c50f9c104 [HER-1676] improve bundled profiles to match best-practice, be self-documenting * profile-crawler-beans.cxml merge 'defaults' back in, beef up comments, strip unnecessary declarations gojomo 2009-10-06 04:07:25 +00:00
• da95932598 * CrawlMetadata.java set default RobotsHonoringPolicy, making explicit declaration unnecessary * RobotsHonoringPolicy.java apache license gojomo 2009-10-06 04:00:58 +00:00