Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 153e709fe5 * WriterPoolProcessor.java eliminate nonsensical type check gojomo 2010-09-15 03:52:11 +00:00
• 1aca427447 * BdbModule.java change exception-throw to log WARNING during DB-cleanup gojomo 2010-09-10 21:15:20 +00:00
• 1f08c114bb * ToeThread.java tighten INFO logging to FINE gojomo 2010-09-10 21:12:48 +00:00
• 4166316113 * AbstractFrontier.java remove Serializable & related transient declarations -- serialization no longer used for checkpointing nor advisable gojomo 2010-09-10 01:00:19 +00:00
• d6a4d158b0 [HER-1803] H3: make it easier to change robots settings * RobotsPolicy.java simplified unification of RobotsHonoring and RobotsExclusion policy classes; subclasses specialize * ObeyRobotsPolicy.java classic obey-robots-as-declared policy * IgnoreRobotsPolicy.java ignore declared robots policy * CustomRobotsPolicy.java follow custom-crafted policy in robots.txt format * MostFavoredRobotsPolicy.java follow the most-permissive policy available to a set of user-agents; optionally adopt ('masquerade') the user-agent whose rules are chosen * RobotsExclusionPolicy.java, RobotsHonoringPolicy.java deleted * CrawlMetadata.java use new string robotsPolicyName as main determinant of policy maintain map of available policies ('obey','ignore', and any the operator added) * CrawlServer.java retain Robotstxt instance rather than customized RobotsExclusionPolicy * PreconditionEnforcer.java use full CrawlMetadata rather than just UserAgentProvider subset consult RobotsPolicy looked up through CrawlMetadata for allow/disallow decision * DispositionProcessor.java update CrawlServer with Robotstxt only consult Robotstxt for crawl-delay * profile-crawler-beans.cxml (*2), migrate-template-crawler-beans.cxml, selftest-crawler-beans.cxml remove old declarations/examples; add new * ExtractorHTML.java, JerichoExtractorHTML.java consult RobotsPolicy.obeyMetaRobotsNofollow for NOFOLLOW * Robotstxt.java new constructors for new use patterns * **/*Test.java update/remove to match new use patterns gojomo 2010-09-10 00:27:20 +00:00
• 3e43cf7bd5 * LinkContext.java undo inadvertent introduction of class-serialization-format incompatibility gojomo 2010-09-09 23:22:15 +00:00
• bce841be5f * JobResource.java improve message for checkpoint-not-made flash gojomo 2010-09-09 23:21:28 +00:00
• 3861c8caa4 * BdbMultipleWorkQueues.java improve detail of logging of error that's been seen in some checkpoint-resume situations gojomo 2010-09-09 23:20:47 +00:00
• de551ced58 [HER-1801] H3: Unreleased lock causes deadlock when checkpointing * Frontier.java add beginDisposition(), endDisposition() methods * AbstractFrontier.java implement beginDisposition(), endDisposition() methods to maintain dispositions-in-progress lock * BdbFrontier.java acquire exclusive dispositionInProgressLock before allowing checkpoint to begin; release at completion of checkpoint * ToeThread.java call beginDisposition before dispositionChain/finished(); call endDisposition after finished() and on exceptions * DispositionChain.java shed all prior relation to locking/checkpointing gojomo 2010-09-03 02:14:41 +00:00
• d12d078a07 [HER-1800] H3: Incomplete checkpoint listed in combo box * Checkpoint.java remove previous attempt to use Lifecycle to catch error * Checkpointable.java @Autowired(required=false) annotation for setRecoveryCheckpoint * CheckpointService.java HasValidator implementation for post-build validity check; also check at launch (probably redundant) * CheckpointValidator.java ensure validity stamp exists on any present Checkpoint gojomo 2010-09-03 01:03:01 +00:00
• 99ce106a9d * CrawlJob.java tighten logging gojomo 2010-09-03 00:30:33 +00:00
• dbf22ad574 [HER-1480] URIs logged (as -6 or -7) that should never be scheduled * Extractor.java discard special-casing of URI-ignored-scheme gojomo 2010-09-03 00:29:05 +00:00
• bc698c09fc [HER-1480] URIs logged (as -6 or -7) that should never be scheduled * LaxURI.java increase tolerance with regard to path-segments * UURIFactory.java allow digits in scheme regex discard old supported-schemes code * UURIFactoryTest.java tests for roundtrip serialization of problematic hostlike-scheme and bars-in-path * CrawlerLoggerModule.java discard special-casing of URI scheme errors * SchemeNotInSetDecideRule.java rule to apply to URIs of unknown schemes; by default REJECTs those not usually handled by Heritrix * **/profile-crawler-beans.cxml add REJECT SchemeNotInSetDecideRule gojomo 2010-09-03 00:24:13 +00:00
• 0347269785 [HER-1817] cleanup logging.properties / WARNING+ for most 3rd party; SEVERE+ for HttpClient; INFO for project classes * logging.properties streamline to bare essentials; allow INFO for org.archive classes * GenericReplayCharSequence.java, BdbUriUniqFilter.java demote some INFO logging to FINE gojomo 2010-09-03 00:00:46 +00:00
• b9b592e286 * jndi.properties remove obsolete file gojomo 2010-09-02 23:57:31 +00:00
• 9bfa8fc1db * JobResource.java remove stray whitespace gojomo 2010-09-02 23:56:52 +00:00
• ad8941fa23 1-liner fixing argument display for usage help argument echo * Heritrix.java (usage) StringUtils.join the String[] args commit in honor of Travis, who suggested fix gojomo 2010-08-27 00:56:42 +00:00
• 2f78980e5d [HER-1814] remove dependency on apache commons-pool/GenericObjectPool * SelfTestBase.java don't try to UURIify 'fieldesc:' ad-hoc URIs gojomo 2010-08-26 21:56:50 +00:00
• 790469280e [HER-1814] remove dependency on apache commons-pool/GenericObjectPool * WriterPoolMember.java fix ops broken by removal of checkSize at preWriteRecordTasks -- still create if out == null gojomo 2010-08-26 21:19:56 +00:00
• 8d1083922d [HER-1814] remove dependency on apache commons-pool/GenericObjectPool * .classpath, pom.xml, org/apache/commons/pool/impl/* remove references to and patched classes based on commons-pool * WriterPool.java move instance creation/round-robin-pool/dispose into this class close (rather then recycle) extra writers when crawl slows (via tracking time of last need/rollover) * WriterPoolMember.java add isOversize() for external check of ready-to-rollover file avoid rollover before every record (so that set of related WARC records aren't split between files) * ARCWriterPool.java, WARCWriterPool.java remove BasePoolableObjectFactory reference, implement class-appropriate makeWriter() * ARCWriterPoolTest.java adjust for new never-timeout behavior * ARCWriterPoolProcessor.java WARCWriterPoolProcessor.java adjust constructor calls * WriterPoolProcessor.java replace poolMaxWaitMs (which caused eventual complete timeout from waiting for writer) with maxWaitForIdleMs, a much smaller value which only controls how long a thread waits for an reusable writer before considering creating a new one. (If creation isn't allowed, threads will now wait indefinitely for a writer to become available.) gojomo 2010-08-25 23:25:26 +00:00
• 03778ff943 [HER-1727] (W)ARC filename uniqueness guarantees: adminport option, process_id option(?), JVM-global-repeat-suppression * WriterPoolMember.java use '~' rather than '@' and '#' as pid~hostname~port separators, for less risk in shell scripts and URLs gojomo 2010-08-23 22:17:39 +00:00
• e3023218e1 null check; fix broken test gojomo 2010-08-20 04:08:07 +00:00
• 47b28f92f8 fix class reference (and build) gojomo 2010-08-20 00:47:55 +00:00
• b388479d32 [HER-1727] (W)ARC filename uniqueness guarantees: adminport option, process_id option(?), JVM-global-repeat-suppression * Heritrix.java make heritrix.hostname, heritrix.pid, heritrix.port available as global properties * PropertyUtils.java, PropertyUtilsTest.java utility methods to interpolate string values from one or several supplied Properties instances * ArchiveUtils.java utility methods to give timestamps guaranteed larger/different than any previously-issued timestamp gojomo 2010-08-20 00:21:43 +00:00
• c410c90b76 Implementation of [HER-1811] Replace the LowDiskPauseProcessor with something that works kristinn_sig 2010-08-19 10:47:22 +00:00
• afc2a55e49 fix for [HER-1808] uncaught URLDecoder IllegalArgumentException in ExtractorHTML * ExtractorHTML.java still consider values rejected by URLDecoder szznax 2010-08-13 00:37:55 +00:00
• dd4bf9a910 correct canonicalizationPolicy bean template per archive-crawler msg 6649 * profile-crawler-beans.cxml, migrate-template-crawler-beans.cxml surround canonicalizationPolicy list with property element szznax 2010-08-09 21:22:50 +00:00
• a729c041e1 * URIAuthorityBasedQueueAssignmentPolicy.java * SurtAuthorityQueueAssignmentPolicy.java * HostnameQueueAssignmentPolicy.java correct inadvertent 'default' access rather than 'protected' allowing overriding gojomo 2010-08-04 20:26:22 +00:00
• 9a74685065 correct whitespace gojomo 2010-07-27 00:13:16 +00:00
• be954bc9b9 [HER-1804] H3: avoid echoing admin credentials to (possibly world-readable) _out log * Heritrix.java replace echo of exact login/pw with note about whether it came from cmd-line or @file gojomo 2010-07-22 22:55:56 +00:00
• 5f76c7ef56 [HER-1800] H3: Incomplete checkpoint listed in combo box * CheckpointService.java use static method on Checkpoint for validity check * Checkpoint.java throw early-startup exception if attempting resumption of invalid checkpoint gojomo 2010-07-13 02:59:47 +00:00
• 899fb6f404 [HER-1800] H3: Incomplete checkpoint listed in combo box * CheckpointService.java (getAvailableCheckpointDirectories) return List, filtered to leave out directories without a 'valid' stamp; log WARNING to _out so that the attentive operator can manually clean-up/investigate gojomo 2010-07-13 01:19:10 +00:00
• f08c6bd749 [HER-1801] H3: Unreleased lock causes deadlock when checkpointing * DispositionChain.java ensure lock-release after unhandled runtime exceptions with try-finally gojomo 2010-07-12 23:44:36 +00:00
• 7d4ed491be * ReadSourceEditor.java PropertyEditor to ease inline specification (as String/ConfigString) anywhere a ReadSource is expected gojomo 2010-07-09 23:17:06 +00:00
• bba186f333 [HER-1799] H3: expand 'inferred' URI options to optionally include root-page (/) * ExtractorHTTP.java add property inferRootPage; if true (not the default), always consider '/' inferred from all HTTP(S) URIs * Hop.java, LinkContext.java add new hop-type 'I' for inferred URIs; add corresponding LinkContext * ExtractorImpliedURI.java use new 'I'nferred types gojomo 2010-07-09 22:39:22 +00:00
• b4485b05b1 * EngineResource.java show proper separator in path prompt gojomo 2010-07-09 21:55:13 +00:00
• cfc4b347a6 [HER-1796] H3: interval rescheduling becomes erratic after URI rescheduled more than maxRetries times * CrawlURI.java improve comments for fetchAttempts methods (resetForRescheduling) added to clear per-scheduling state * WorkQueueFrontier.java call resetForRescheduling just before scheduling-at-specific-time * FrontierJournal.java, AbstractFrontier.java renamings: prefer 'reenqueue' instead of 'reschedule' for describing simple (non-timed) retries gojomo 2010-07-09 01:45:26 +00:00
• 274e522f85 [HER-1793] BOM in seeds.txt confuses heritrix * RegexpLineIterator.java ignore \ufeff along with leading whitespace gojomo 2010-07-02 17:47:41 +00:00
• f75f95648a [HER-1795] H3: Wrong schemaLocation in .cxml * **/*crawler-beans.cxml update all 2.0 spring schema refs to 2.5 gojomo 2010-07-02 00:58:23 +00:00
• 829a3f78b3 [HER-1794] Extra white spaces in pom.xml * **/pom.xml remove leading blank lines gojomo 2010-07-02 00:51:19 +00:00
• a68293982f Fix a couple of thinkos to really reserve RESERVE_BLOCKS of the intended RESERVE_BLOCK_SIZE nlevitt 2010-06-24 04:24:31 +00:00
• 4c10d4b54f * engine/pom.xml after test OOMEs on build machine, make sure tests use -server and a 256M heap gojomo 2010-06-23 00:47:24 +00:00
• bdc1e078ad * Flash.java protect against empty string, whitespace as well gojomo 2010-06-21 22:08:46 +00:00
• 943ecf3088 [HER-1762] H3 performance tuning * AbstractFrontier.java change config and defaults for 'inbound' and 'outbound': allowing any BlockinqQueue bean, of any size, to be specified if desired * **/profile-crawler-beans.cxml changed commented-out descriptions of defaults to reflect what's used if nothing is specified for 'outbound' and 'inbound' gojomo 2010-06-18 21:17:40 +00:00
• 4875135dc9 [HER-1762] H3 performance tuning * KeyedProperties.java avoid allocation hotspot shown by profiling (creating iterator for empty collection) gojomo 2010-06-18 02:37:42 +00:00
• e83a6eceed follow-up patch for [HER-1764] H3 needs in-UI quit/shutdown option * Flash.java avoid NPE when "flashdrop" cookie not set szznax 2010-06-16 22:02:41 +00:00
• edb70dae06 Fix [HER-1774] TransclusionDecideRule defies expectation when seed redirects with seed-redirects-new-seed enabled * TransclusionDecideRule.java refine further: count 'R's for minimum-hops-necessary-to-transclude, but not for maximum-allowable gojomo 2010-06-16 21:51:50 +00:00
• 2e03aa8cb7 [HER-1764] H3 needs in-UI quit/shutdown option * EngineResource.java add 'exit' control area at bottom of page allow JVM-exit if confirm-box checked and either no jobs built/running or extra confirm boxes for each built job checked * Flash.java fix to allow multiple flashes per page load * JobResource.java, CrawlJob.java move useful status string method to CrawlJob gojomo 2010-06-16 00:51:58 +00:00
• 4c1a4f2e12 remove unneeded import gojomo 2010-06-15 21:34:57 +00:00
• 3c8e72a20e * BloomFilterTest.java disable by renaming ('xest') those tests taking >15mins each gojomo 2010-06-15 21:33:34 +00:00
• 6be649b286 [HER-1783] BloomFilter64bit bit-length bug prevents full bitfield from being used; premature saturation * BloomFilter64bit.java include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes fit a number of problems with int/long overflow and bitwise ops add methods for reporting/testing * BloomFilter.java add methods for reporting/testing * BloomFilterTest.java, BloomFilter64bitTest.java more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming * BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java deleted as buggy or redundant * BenchmarkBlooms.java move to test source dir * BloomUriUniqFilter.java change to accept filter instance (rather than parameters) for added configuration flexibility fix comments * BloomUriUniqFilterTest.java supply filter not paramters gojomo 2010-06-15 21:30:34 +00:00
• b571730b4b Fix [HER-1774] TransclusionDecideRule defies expectation when seed redirects with seed-redirects-new-seed enabled * TransclusionDecideRule.java evaluate(): skip over 'R' hops without counting them as transcludes gojomo 2010-06-14 23:30:22 +00:00
• e1f76dbc07 [HER-1784] default profile MatchesListRegexDecideRule intended to REJECT doesn't * dist/src/main/conf/jobs/profile-defaults/profile-crawler-beans.cxml * engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml add intended REJECT decision for the late (rules[4]) MatchesListRegexDecideRule gojomo 2010-06-14 22:22:34 +00:00
• f9a3c2344c * BloomFilterTest.java explicit deferenced bloom instance to avoid OOME gojomo 2010-06-08 03:38:56 +00:00
• 4e1b47746c * commons/pom.xml increase heap for testing to 1024m to accomodate big bloom bitfield tests gojomo 2010-06-08 03:28:51 +00:00
• e9a11e56c6 HER-1783 BloomFilter64bit bit-length bug prevents full bitfield from being used; premature saturation * BloomFilter64bit.java fix long arithmetic calculating bit-length m in constructor slight optimization of add() * BloomFilterTest.java abstract superclass for bloom impl class tests * BloomFilter64bitTest.java sanity check that set bits, after a handful of adds into default-sized filter, occupy top and bottom 20% of range (which would have caught this bug earlier) gojomo 2010-06-08 02:39:35 +00:00
• e673c7d781 * WorkQueueFrontier.java experimental utility method to force-wake all snoozed (politeness and retry-delayed) queues gojomo 2010-06-04 22:31:16 +00:00
• 8b696da475 * CrawlURI.java dump more info if code that really shouldn't run actually runs gojomo 2010-06-04 22:27:59 +00:00
• 5f83fe75a2 HER-1010 crawl favicon.ico * (self tests) correct expectations to now include /favicon.ico gojomo 2010-06-03 00:31:32 +00:00
• f81a941d97 HER-1010 crawl favicon.ico * ExtractorHTTP.java consider /favicon.ico 'discovered' from every other HTTP fetch * Extractor.java utility method addOutlink() gojomo 2010-06-01 23:36:16 +00:00
• d1f27aaecd removed no-longer-referenced "recoveryDir" property from "frontier" bean szznax 2010-05-27 17:46:44 +00:00
• 72bdad74e9 * pom.xml install source jars to maven repository nlevitt 2010-05-26 04:41:22 +00:00
• 0dbc773fc8 Followup on HER-1728 - treat flashvars strings more like url query strings (key=value&[key=value]*) * ExtractorHTML.java considerQueryStringValues(): split pairs on "&" instead of ";" and url-decode values nlevitt 2010-05-15 02:13:19 +00:00
• f7a9327334 Fix for [HER-1772] NullPointerException in ExtractorHTML related to flashvars * modules/src/main/java/org/archive/modules/extractor/ExtractorHTML.java processGeneralTag(): check for null nameVal nlevitt 2010-05-11 22:11:38 +00:00
• e3c608fab4 HER-1762 H3 performance tuning * ExtractorHTML.java smaller caps on element-name, attribute-name, attribute-value lengths -- costly match-failures on bad/non-HTML may be eating a lot of cycles gojomo 2010-04-29 23:21:38 +00:00
• 95bf82e2d3 followup for HER-1688 remember external job directories * Engine.java suppress WARNING for common case of harmless failure during rescans: job already exists * EngineResource.java suppress rescan on each reload (ODD: steve had already committed this change -- I got the commit email -- but now the repo version is missing it. Sourceforge problems?) gojomo 2010-04-29 23:20:22 +00:00
• fcd0ec6699 HER-1762 H3 performance tuning * WorkQueueFrontier.java override schedule() to keep unique-testing in calling thread; may remove bottleneck shown by time-profiling and thread-observation (most ToeThreads stuck in candidatesProcessor, schedule()ing to frontier) * AbstractFrontier.java add comment about implementation vs. subclass override gojomo 2010-04-27 21:37:24 +00:00
• e10a14df54 HER-1753 offer means of setting login/password that doesn't leak to process info (as when literally specified to '-a' option) * Heritrix.java allow [username:]password to come from file, supplied to '-a' with '@' prefix also, clean up options names/help text to match current behavior also, offer warning about particularly weak (<8 characters, all letters, all numbers) passwords gojomo 2010-04-27 01:36:38 +00:00
• b639873dae Fix broken test * Engine.java tolerate (with return-false-forfailure rather than NPE) non-job directories * EngineTest.java use empty temp job directory rather than (possibly full-of-junk) general junit-tests directory gojomo 2010-04-27 00:44:36 +00:00
• 57c857d5c6 Refinement of fixes for HER-1688 remember external job directories * EngineResource.java avoid calling findJobConfigs() on each page load szznax 2010-04-26 23:33:40 +00:00
• 5034c7d96e correct spaces-for-tabs gojomo 2010-04-26 22:28:43 +00:00
• c4946d04d0 HER-1688 remember external job directories * EngineResource.java, Engine.java refactor for less branching/redundancy gojomo 2010-04-26 22:22:09 +00:00
• 14b8649aa9 minor cleanup/clarification on review szznax 2010-04-26 21:49:41 +00:00
• 27cca4da60 Refinement of HER-1737 excessive job.log WARNINGs for misreported encoding (FLV reported as text/html;charset=UTF-8 * ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java store actual first exception for reference * ExtractorHTML.java report exception in nonfatal-log, rather than WARNING (which was still polluting job log more than desired -- but should still be logged to allow future investigation of sites where encoding issues may exist) gojomo 2010-04-24 02:22:24 +00:00
• 9fc5e42cfe Fix failing test * InMemoryReplayCharSequence.java ensure redecode after problem restarts at right position gojomo 2010-04-23 22:57:50 +00:00
• 792d8e528b HER-1766 The use of the MatchesFilePatternDecideRule with the CUSTOM preset causes a fatal error (infinite loop) * MatchesFilePatternDecideRule.java fix getRegex() to super.getRegex() gojomo 2010-04-23 22:16:25 +00:00
• 2855572278 ? gojomo 2010-04-23 04:20:12 +00:00
• a7caceb8ad HER-1737 excessive job.log WARNINGs for misreported encoding (FLV reported as text/html;charset=UTF-8 * ReplayCharSequence.java new interface method: getDecoderExceptionCount(), reporting >0 if any decoding problems were encountered and silently replaced * GenericReplayCharSequence.java count decoding problems, and only log as FINE rather than WARNING * InMemoryReplayCharSequence.java try decoding with reporting; if any problems note at least one then decode without reporting errors * ExtractorHTML.java after using ReplayCharSequence, log WARNING if nonzero decoding errors gojomo 2010-04-23 04:11:53 +00:00
• 67bf73026f remove obsolete/moved example properties gojomo 2010-04-23 04:05:31 +00:00
• cb5ec19b6c HER-1765 quiet excessive INFO level logging * GenericReplayCharSequence.java * Scoper.java * FetchFTP.java * FetchHTTP.java demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl gojomo 2010-04-23 00:44:25 +00:00
• 3f0059c8a8 * BeanBrowseResource.java remove extraneous close() that seemed to be causing (harmless but distracting) console error stacks after web hits gojomo 2010-04-23 00:41:25 +00:00
• 9fd648f3ef HER-1467 Allow periodic checkpointing and checkpoint-copy-logs to be set/unset during crawl * CheckpointService.java reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl * CheckpointSuccessEvent.java, CrawlJob.java report successes as new kind of ApplicationEvent; propagagte to job log * BdbModule.java remove obsolete unconsulted checkpointCopyLogs setting gojomo 2010-04-23 00:15:44 +00:00
• ca32d0d448 HER-1692 Cannot create profile with same name as job * JobResource.java if destination of copy-job fails, show flash error gojomo 2010-04-22 01:59:41 +00:00
• c1207582c4 HER-1728 Heritrix could be enhanced to parse flashvars parameter to correctly extract the links * ExtractorHTMLTest.java tests for embed/@flshvars, param[@name=flashvars]/@value likely-URI discovery * ExtractorHTML.java move likely-URI testing to UriUtils add FLASHVARS support * JerichoExtractorHTML.java use UriUtils ikely-URI testing add FLASHVARS support gojomo 2010-04-22 01:27:24 +00:00
• f933dc922d HER-1763 unify/improve/document likely-URI string tests * UriUtils.java move likely-URI heuristics here for better consistency, reuse, testing, improvement * UriUtilsTest.java testing for UriUtils * ExtractorJS.java move likely-URI support to UriUtils; use new test * ExtractoSWF.java use new UriUtils likely-URI testing * ExtractorHTML.java move likely-URI testing to UriUtils, use new test * JerichoExtractorHTML.java use new UriUtils likely-URI testing gojomo 2010-04-22 01:08:14 +00:00
• 72fb1c55bd Move TLD list to resource file for easier drop-in updating * ArchiveUtils.java read TLDs from tlds-alpha-by-domain.txt * ArchiveUtilsTest.java verify TLDs read * tlds-alpha-by-domain.txt from http://data.iana.org/TLD/tlds-alpha-by-domain.txt gojomo 2010-04-22 01:05:23 +00:00
• f018a2e6f1 * StatisticsTracker.java correct outdated comment gojomo 2010-04-13 22:47:19 +00:00
• affa3b672c HER-1620 Tolerate trailing '*' in robots disallow (& allow?) rules * Robotstxt.java trim single trailing '*' if present (assume unintentional or redundant) gojomo 2010-04-13 22:46:23 +00:00
• 9e12e4abe5 test/deprecation cleanup * CachedBdbMap.java @deprecated * BdbModule.java note (unused) 'CBM' option is deprecated * ObjectIdentityBdbCache.java tighten logging to FINE * ObjectIdentityBdbCacheTest.java fix test that was only trying to log, rather than fail, problems gojomo 2010-04-13 02:15:53 +00:00
• a82838cd27 HER-1756 improved crawl status reporting: more definitive start-after-setup, conclusion-after-logging of crawling * AbstractFrontier.java call new finalTasks just before announcing FINISH state * WorkQueueFrontier.java eliminate closeQueue * BdbFrontier.java move dumpAllPendingToLog from closeQueue (triggered by CrawlController FINISHED) to finalTasks (before Frontier FINISH) replace closeQueue with close() override gojomo 2010-04-13 02:10:56 +00:00
• 8be7be2983 fix race causing intermittent test failures on MacOS * CrawlJob.java synchronize methods which mutate or change behavior based on field 'ac'; previously, some self-tests were exitting/failing immediately because they were not seeing a non-null 'ac' value. gojomo 2010-04-13 00:55:09 +00:00
• 1698412130 HER-1450 Reading at EOF using read(byte[]) causes position pointer to decrement. * ARCWriterTest.java one more test to ensure the problem with read(void) was exercised/fixed gojomo 2010-04-09 21:25:26 +00:00
• cc6fb841d9 part of fix for [HER-1450] Reading at EOF using read(byte[]) causes position pointer to decrement. * ArchiveRecord.java do not decrement position after EOR, port from H1 * ARCWriterTest.java added testArchiveRecordEORConsistent(), testArchiveRecordMarkSupport() port from H1 szznax 2010-04-09 18:25:16 +00:00
• 647a0b3edf HER-1754 treat robots.txt non-response as 404 (optionally?) * CrawlServer.java enable a specific kind of connect-lost (no response given) to be counted as 'deemed 404' on robots retries over a minimum * FetchStatusCodes.java add a synthetic code: -404, S_DEEMED_NOT_FOUND * FetchHTTP.java better distinguish CONNECT_FAILED and CONNECT_LOST * DispositionProcessor.java treat CONNECT_LOST same as _FAILED; simplify a test gojomo 2010-04-02 03:55:12 +00:00
• 693f050fb7 successfully adding external job dir is ACK flash kind szznax 2010-03-29 15:56:09 +00:00
• fd5644f109 HER-1627 followup: show error rather than match faulReadSize * PersistProcessor.java rename setupEnvironment to setupCopyEnvironment to make limited purpose of env clear; catch and rethrow IllegalArgumentException with hint that second-open may be cause * PersistLoadProcessor.java catch and display IllegalArgumentException same as other failure-to-open SEVERE but proceedable exceptions * PrecedenceLoader.java update method names gojomo 2010-03-26 01:02:48 +00:00
• 37c32b4e5c Maven cleanup to enable warningless build on fresh machine with Maven2.2.0: * engine/pom.xml restore a reference to a (new version of a) necessary servlet library * pom.xml remove deprecated {pom.artifactId}; add needed javadoc module version decl * .classpath update ref to currently-used servlet jar gojomo 2010-03-23 22:57:27 +00:00
• 9f77412581 part of fix for HER-1627 incorrect source tag in crawl.log (of persist-loaded crawl) * EnhancedEnvironment.java in getClassCatalog(), when read-only env, insist on read-only catalog * PersistLoadProcessor.java remove existing curi members from prior to prevent clobbering source * PersistProcessor.java set historyDbConfig read-only, set je.log.faultReadSize for compatability with BDBModule.setup() szznax 2010-03-23 20:52:40 +00:00
• 20b8a7bc90 mention url in question in debug level logging nlevitt 2010-03-19 18:39:14 +00:00
• af85fed7f4 part of fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses * Arc2Warc.java reverted writing ARC namedFields, deferred for further analysis as part of HER-1750 szznax 2010-03-12 23:44:07 +00:00