Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 31edae9753 * .classpath source jar for commons-cli Noah Levitt 2011-10-21 15:43:21 -07:00
• 5735839252 Post 3.1.0 * **/pom.xml switch version to "3.1.1-SNAPSHOT" git-svn nlevitt 2011-10-21 19:00:37 +00:00
• 8c99fc947c Prep for 3.1.0 release * **/pom.xml bump version-id to "3.1.0" * README.txt refer to exactly 3.1.0 release notes nlevitt 2011-10-21 17:08:04 +00:00
• e8d4ec73c7 * bin/heritrix when starting in background, make better effort to write pid to $HERITRIX_HOME/heritrix.pid on successful start, plus this change: - echo -n "date Starting heritrix" + echo -n "date Heritrix starting (pid $HERITRIX_PID)" nlevitt 2011-10-21 16:54:40 +00:00
• e9a96ebc12 * profile-crawler-beans.cxml commented-out ReschedulingProcessor nlevitt 2011-10-21 01:20:32 +00:00
• db049be78d * logging.properties revert unintentional changes nlevitt 2011-10-21 00:52:55 +00:00
• d96e086eeb * logging.properties fix typo nlevitt 2011-10-21 00:48:20 +00:00
• 32680b8f73 Fix for HER-1960 kryo buffer can hog memory, lead to oome (bug report, patch from Kenji) * KryoBinding.java wrap ObjectBuffers with WeakReference so that can be garbage collected if necessary (when recreated they'll be back at the default size of 16k) nlevitt 2011-10-20 02:21:54 +00:00
• 351a89c5a9 Fix java.lang.IllegalArgumentException from CharBuffer.allocate() in GenericReplayCharSequence.decode() Reported by Kenji who says, "My guess is that root cause is Recorder.calcRecommendedCharBufferSize(): return Math.min(inStream.getRecordedBufferLength()/2,(int)inStream.getSize()); URL above returns HTML larger than 5GB (infinite smileys!! what the heck), and (int)intStream.getSize() became negative." * Recorder.java - return Math.min(inStream.getRecordedBufferLength()/2,(int)inStream.getSize()); + return (int) Math.min(inStream.getRecordedBufferLength()/2, inStream.getSize()); nlevitt 2011-10-13 15:47:02 +00:00
• 9fb7b0f5ae Fix for HER-1958 race condition on frontier inactive queues * WorkQueueFrontier.java activateInactiveQueue(), deactivateQueue() - synchronize around updates of inactiveQueuesByPrecedence and highestPrecedenceWaiting nlevitt 2011-10-13 01:47:25 +00:00
• 189a557af8 Tweak of last checkin * Heritrix.java instanceMain() - move logging of java vendor/version ahead of version check, since the information could be particularly helpful when version check doesn't pass nlevitt 2011-10-12 18:32:14 +00:00
• 61240573b3 * Heritrix.java instanceMain() - log java vendor, version at startup nlevitt 2011-10-12 18:20:27 +00:00
• a09c8e69b8 * Recorder.java remove TODOs that have already been done nlevitt 2011-10-12 17:52:45 +00:00
• 73e6d21fd2 * profile-crawler-beans.cxml fix small typo nlevitt 2011-10-12 17:49:34 +00:00
• 270faa30c8 Remove empty java package directories. nlevitt 2011-10-12 17:46:44 +00:00
• 10b3686ba8 Fix race condition which can happen with multiple concurrent "launch"es - can end up with duplicate ToePools like before r7268, probably other problems * CrawlJob.java launch() - synchronize this method - sort of a blunt fix but probably safest and shouldn't affect performance nlevitt 2011-10-07 23:18:43 +00:00
• 19e94f07d1 Fix race condition encountered by Travis and Adam where two ToeThread pools are created, one on "launch" and one on "unpause" * CrawlController.java requestCrawlResume() - do not create ToePool (creation here appeared to be inherited cruft), paused crawl must already have toe pool nlevitt 2011-10-07 01:54:29 +00:00
• b4e6f43052 Unset script engine variables after scripts complete, allowing memory to be freed up sooner - see also r7252 * ScriptResource.java, ActionDirectory.java, ScriptedDecideRule.java nlevitt 2011-10-06 18:01:01 +00:00
• b898d55cea Certain invalid "Crawl-delay" values, e.g. "/", in robots.txt cause split() to return empty string, resulting in ArrayIndexOutOfBoundsException - reported by John Lekashman * Robotstxt.java initializeFromReader() - catch ArrayIndexOutOfBoundsException and ignore -- matches existing handling of other invalid crawl-delay values (catch NumberFormatException and ignore) nlevitt 2011-10-06 17:20:13 +00:00
• a76f9818f9 Fix HER-1955 Some annoying interaction between new creation of latest link and disk full java bean * DiskSpaceMonitor.java checkAvailableSpace() - really do ignore non-existent paths, as comment and log message claim is done nlevitt 2011-09-30 20:07:15 +00:00
• c5de0048d1 "Fix" for HER-1951 ExtractorJS produces many false positives * UriUtils.java isLikelyFalsePositive() - check for unusual characters, likely mimetypes, and a couple of other common traps * UriUtilsTest.java add some tests for this code nlevitt 2011-09-30 00:48:00 +00:00
• a577f1eb0b * PathSharingContext.java start() - restore this override to call initLaunchDir() (how did I miss that), then super.start() nlevitt 2011-09-29 23:12:26 +00:00
• 1390024902 * BdbModuleTest.java testDoCheckpoint() - call BdbModule.destroy() in addition to stop() to close resources nlevitt 2011-09-29 21:37:34 +00:00
• e88d1d9df7 CrawlController FINISHED doesn't guarantee we're ready for shutdown, so add new boolean and event for that purpose. Elaboration: CrawlJob needs all beans to receive FINISHED event before teardown. Can't easily guarantee CrawlJob receives the event last. Actually, spring provides a way, org.springframework.core.Ordered, but to make a bean receive events last, *every other bean* must implement Ordered. So this way is simpler. * CrawlController.java new member variable isStopComplete and event StopCompleteEvent * CrawlJob.java rely on cc.isStopComplete() or StopCompleteEvent to indicate ready for teardown nlevitt 2011-09-29 17:22:34 +00:00
• de9975ee23 * CrawlJob.java instantiateContainer() - do not call teardown() on exception from ac.refresh() - can throw IllegalStateException teardown() - put all uses of variable cc inside not-null check nlevitt 2011-09-29 16:09:44 +00:00
• 747fbc0a0f Fix for HER-1954 bdb closed at crawl finish instead of teardown -- refactor crawl finish and teardown * PathSharingContext.java start(), doStart(), stop(), doStop() - remove these overloaded methods, because the spring bug they were working around seems to be fixed, not seeing any problems with cyclical dependencies - might be this one https://jira.springsource.org/browse/SPR-7266 doClose() - remove because superclass version seems to work fine (this method generally wasn't being called anyway, though it would be now with other changes in this checkin) * CrawlJob.java refactor teardown to call close() on the ApplicationContext, which calls destroy() on any beans that implement DisposableBean - this is now the way to have beans do stuff at teardown * CrawlController.java send FINISHED crawl state event after calling appCtx.stop() so that isFinished() can indicate ready-ness for teardown * BdbModule.java move close() to teardown, i.e. implementation of DisposableBean.destroy(); remove shutdown hook and rely on teardown; related tweaks * WorkQueueFrontier.java, CrawlerLoggerModule.java, BdbUriUniqFilter.java move close() to teardown * CrawlMapper.java, AbstractFrontier.java, PreloadedUriPrecedencePolicy.java, FetchWhois.java, FetchHTTP.java, PersistLogProcessor.java, WriterPoolProcessor.java add comments about cleanup that maybe should wait until teardown * UriUniqFilter.java remove incorrect(?) comment nlevitt 2011-09-29 00:39:55 +00:00
• de856f10a6 Fix possible memory leak (encountered in h1, see r7255) * ToeThread.java run() - set local variable curi=null when finished with it, because the jvm seems to be holding on to it even after the containing block is finished nlevitt 2011-09-28 00:22:53 +00:00
• f704ca5bc7 Fix HER-1953 all api xml elements are camelcase except sizeTotalsReport fields * CrawledBytesHistotable.java camelcase notModified, dupByHash, notModifiedCount, dupByHashCount, novelCount * CrawlJob.java sizeTotalsReportData() - camelcase totalCount nlevitt 2011-09-28 00:19:44 +00:00
• 6adc3c285f * Heritrix.java usage() - out.print("Your arguments were: "+StringUtils.join(args, ' ')); + out.println("Your arguments were: "+StringUtils.join(args, ' ')); nlevitt 2011-09-27 04:08:58 +00:00
• f242522efc Fix for HER-1950 URIAuthorityBasedQueueAssignmentPolicy KeyedProperties shadows superclass QueueAssignmentPolicy KeyedProperties, breaking sheet overlay of properties defined in superclass * URIAuthorityBasedQueueAssignmentPolicy.java remove field kp, accessor getKeyedProperties(); instead inherit these from superclass QueueAssignmentPolicy nlevitt 2011-09-20 23:52:59 +00:00
• 54aca17757 Removing very old abandoned line of development, package org.archive.extractor nlevitt 2011-09-14 21:34:26 +00:00
• 12411395a8 * ScriptedProcessor.java innerProcess() - unset engine variable "curi" after script completes, allowing its memory to be freed up sooner nlevitt 2011-09-14 02:30:02 +00:00
• 4eef2d9902 Revert change in r7240 - different return type makes recovery from checkpoints from before the change fail * CrawlController.java getState() - return Object nlevitt 2011-09-13 00:48:06 +00:00
• 5a4e72d902 Couple of synchronizations for HER-1943 WorkQueue: inconsistent synchronization using fields active, lastCost, peekItem, wakeTime * WorkQueue.java considerActive(), unpeek() - make synchronized nlevitt 2011-09-12 22:49:32 +00:00
• 48b0ddfb03 Fix for HER-1942 CheckpointService: inconsistent synchronization around use of isRunning. Doubtful if isRunning thing is a real problem, but it looks like a good idea to synchronize this method anyway, in case there are concurrent calls to it. * CheckpointService.java setRecoveryCheckpointByName() - mark synchronized nlevitt 2011-09-12 22:00:54 +00:00
• 5934e10073 Fix for HER-1939 ScriptedDecideRule: field sharedEngine non-transient and non-serializable. * ScriptedDecideRule.java make field sharedEngine transient nlevitt 2011-09-12 21:48:27 +00:00
• f20b3eee30 Fix HER-1935 Many calls to File.mkdirs() and other file/dir methods don't check the return value. Also fix bug where pointless empty directories were created in scratch dir. * BasicProfileTest.java, SelfTestBase.java, CrawlControllerTest.java, PrecedenceLoader.java, MigrateH1to3Tool.java, CrawlerLoggerModule.java, StatisticsTracker.java, CheckpointUtils.java, BdbUriUniqFilter.java, ARCWriterProcessorTest.java, WARCWriterProcessorTest.java, PersistProcessor.java, WriterPoolProcessor.java, PrefixFinderTest.java, StoredQueueTest.java, FileUtilsTest.java, ObjectIdentityBdbManualCacheTest.java, ObjectIdentityBdbCacheTest.java, ObjectPlusFilesOutputStream.java, TestUtils.java, TmpDirTestCase.java, Engine.java Replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir). In these cases the calls were either already in a spot where the possible IOException would be handled appropriately, or the line was trivially moved into such a block. * ActionDirectory.java, Engine.java replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure * BdbModule.java setup() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException" - conveniently the place where this method is called was already in a try block that catches IOException * Checkpoint.java generateFrom() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException" * CheckpointService.java move call to Checkpoint.generateFrom() inside existing try block since it now can throw IOException * Recorder.java ensure(File) - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure new Recorder(File,String,int,int) - call ensure() on the correct object, the containing directory; and remove redundant call to ensure() nlevitt 2011-09-12 20:12:10 +00:00
• f7cc187b09 * ANVLRecord.java remove accidentally committed test code nlevitt 2011-09-12 18:22:55 +00:00
• d2306fced6 Fix for HER-1934 ANVLRecord: clone() method does not call super.clone() * ANVLRecord.java clone() - return (ANVLRecord) super.clone() - this does the right thing nlevitt 2011-09-12 18:21:21 +00:00
• 0a28717500 Fix for HER-1922 ThreadLocalHttpConnectionManager: possible memory leak due to non-static inner class. * ThreadLocalHttpConnectionManager.java make nested class ConnectionInfo static so it doesn't reference instance of containing class nlevitt 2011-09-12 17:53:25 +00:00
• 40ff3affa3 * ObjectIdentityMemCache.java fix mistake in r7240 - 'val.setIdentityCache()' was no longer being called the first time through, when the object comes from the 'supplier' (thanks Gordon) nlevitt 2011-09-12 17:09:29 +00:00
• 221e63dbd0 Fix for HER-1921 BucketQueueAssignmentPolicy: absolute value of hashCode can be Integer.MIN_VALUE. * BucketQueueAssignmentPolicy.java getClassKey() - cast value to long before taking absolute value nlevitt 2011-09-12 17:02:54 +00:00
• 24f4b715ec Various little cleanups including several discovered by Aaron using an automated tool. * WorkQueueFrontier.java (HER-1925) avoid 2 possible null pointer dereferences * CrawlController.java getState() - change declared return type from Object to State * CrawlerLoggerModule.java (HER-1932) remove unused, unset field "reports" * BdbCookieStorage.java elide pointless extra variable * FetchHTTP.java, DownloadURLConnection.java, ProcessUtils.java (HER-1933) use Arrays.toString() for logged arrays * CrawlServer.java - remove unused field robotstxtChecksum - updateRobots() - avoid reinventing existing utility class InstanceofPredicate * ExternalGeoLookupInterface.java (HER-1938) extend Serializable, since ExternalGeoLocationDecideRule is declared Serializable and has a ExternalGeoLookupInterface field * DecideRuleSequence.java (HER-1937) make field fileLogger transient * PersistLogProcessor.java (HER-1924) remove field recoveryCheckpoint shadowing same field in superclass Processor * ExtractorUniversal.java (HER-1920) use return value of potentialTLD.toLowerCase() as it appears was intended * ARCWriterProcessor.java (HER-1928) avoid possible null pointer dereference * S3URLConnection.java (HER-1933) set S3ServiceException as cause of rethrown IOException, and do not put stacktrace array in message * ObjectIdentityMemCache.java (HER-1929) avoid possible null pointer dereference * .classpath more source jar references, other cleanup nlevitt 2011-09-12 02:12:44 +00:00
• e38f0a73ba * profile-crawler-beans.cxml <bean id="fetchHttp" class="org.archive.modules.fetcher.FetchHTTP"> + <!-- <property name="useHTTP11" value="false" /> --> nlevitt 2011-09-11 17:02:32 +00:00
• d3a6dc1f58 Fix for HER-1946 Mid-Crawl Adjustment of frontier.queueTotalBudget does not propagate to retired queues. * WorkQueueFrontier.java findEligibleURI() - propagate queueTotalBudget (also sessionBudget) to ensure the settings are current right before checking if queue is over budget nlevitt 2011-09-09 23:23:15 +00:00
• 794c41ebd3 Patch from Kenji to fix "weird 'NoSuchElement' exception retrieving XML response from H3 web interface. It appeared to be concurrency related.... Probably Histotable was accessed concurrently from my JMX interface and web interface and it left Histotable in an inconsistent state. Much less likely without JMX monitoring, but it could also happen with H3 without JMX" (Kenji) * Histotable.java make tally(K,long) synchronized nlevitt 2011-09-01 21:04:56 +00:00
• e6de9cd96b * Sheet.java prime() - fix exception triggered in process of handling earlier real exception, which was masking the real exception - see http://tech.groups.yahoo.com/group/archive-crawler/message/7303 nlevitt 2011-08-30 22:41:37 +00:00
• cdb6fb3a9a * PersistProcessor.java copyPersistEnv() - improve logging on history entries by marshalling them as json nlevitt 2011-08-30 22:04:17 +00:00
• c70f7ccbc0 * profile-crawler-beans.cxml remove reference to defunct frontier setting "holdQueues", pointed out by Kristinn Sigurðsson nlevitt 2011-08-08 23:42:47 +00:00
• 28aa2ffede * **/pom.xml improve formatting (no substantive changes at all -- try "svn diff -x -ub -c7232") nlevitt 2011-08-05 17:46:57 +00:00
• 178fdd92b6 * **/pom.xml remove references to unused, defunct agilejava maven repository (had been used for xdoc stuff) nlevitt 2011-08-05 17:37:16 +00:00
• ada8cbd198 Fix bug reported by Travis where in certain corner cases where urls are processed in a particular order, a seed can be incorrectly tagged with S_ROBOTS_PREREQUISITE_FAILURE * DispositionProcessor.java innerProcess() - do not update server's robots info if robots fetch has been deferred nlevitt 2011-08-04 01:32:09 +00:00
• 706869e586 Fix handling of server error on robots.txt - change to what appears to be the intended behavior * DispositionProcessor.java innerProcess() - call server.updateRobots(curi) after special ignore-robots short-circuiting of retry cycle rather than before, so that updateRobots() deemed-not-found handling applies * FetchHTTP.java call curi.setHttpMethod() right after creating the HttpMethod, instead of when processing the response; this is needed so that updateRobots() check of fetchType works properly (before this change, fetchType was "UNKNOWN" on server error) * PreconditionEnforcer.java, CrawlServer.java fix typos nlevitt 2011-08-02 23:28:20 +00:00
• 3cfc315de9 Fix bug reported by Travis where each url in action directory ".include" file is reported as "not canonicalized" in heritrix_out, and some end up crawled in spite of being considered included * WorkQueueFrontier.java considerIncluded(CrawlURI) - call FrontierPreparer.prepare(CrawlURI) so that CrawlURI is canonicalized etc before being considered included (similar to what schedule() does) * CrawlURI.java getCanonicalString(), getPolitenessDelay() - use logger instead of System.err nlevitt 2011-08-02 01:07:11 +00:00
• dcbc259776 Fix HER-1917 failure to use good http authentication type if server supports any that heritrix doesn't - problem reported and fix suggested by Adam Wilmer * HttpAuthenticationCredential.java populate() + http.getParams().setParameter(AuthPolicy.AUTH_SCHEME_PRIORITY, + Arrays.asList(AuthPolicy.DIGEST, AuthPolicy.BASIC)); nlevitt 2011-07-30 01:30:44 +00:00
• 26584d44d7 * ExtractorHTMLTest.java testScriptTagWritingScriptType() - As of r7225, ExtractorJS finds a new url - avoid using a strategically placed space. (The fact that UriUtils.isLikelyUri() returns true for the string ");document.write(unescape(" is kind of crazy, but a separate issue) nlevitt 2011-07-27 20:04:54 +00:00
• bda2fbebe4 Fix HER-1873 strings with spaces can confuse ExtractorJS * ExtractorJS.java considerStrings() - backtrack to just ahead of previous closing quote to reconsider it as a possible opening quote * ExtractorJSTest.java minimal test case nlevitt 2011-07-27 18:56:37 +00:00
• b2307e7cf1 Fix HER-1914 cookie loading and saving does not work properly * AbstractCookieStorage.java make cookiesLoadFile a ConfigFile and use obtainReader() so that the file is snapshotted into launch dir saveCookies() - include expiration date, so file format match the format it claims to use, and what loadCookies() expects loadCookies() - handle expiration date field, and actually put each cookie in the cookie map refactor loadCookies() for use with Reader from ConfigFile.obtainReader() fix up javadocs some nlevitt 2011-07-26 19:30:16 +00:00
• 62e1c835dd Fix HER-1913 cookies not being sent * CookieSpecBase.java match(String, int, String, boolean, SortedMap) - use InternetDomainName.name() instead of .toString(), since the latter doesn't return the plain old domain name * Cookie.java javadoc typo nlevitt 2011-07-26 19:07:26 +00:00
• cb7a454d0c Post 3.1.0-RC1 * **/pom.xml switch version back to "3.1.0-SNAPSHOT" nlevitt 2011-07-26 03:42:02 +00:00
• 347a25e417 Prep for 3.1.0-RC1 release * pom.xml, */pom.xml bump version number to 3.1.0-RC1 nlevitt 2011-07-26 02:57:48 +00:00
• 3a4419967d Remove PathFixupListener, which was only used for SurtPrefixedDecideRule initialization, now done differently as of r7218. * ConfigPathConfigurer.java remove calls to PathFixupListener * PathFixupListener.java removed nlevitt 2011-07-25 23:10:02 +00:00
• a103c5f05c * SurtPrefixedDecideRule.java fix timing reading surts source to make sure launch dir is in place for snapshot nlevitt 2011-07-25 22:52:15 +00:00
• 0c8840db56 * .classpath reference more source jars retrieved with "mvn dependency:sources" nlevitt 2011-07-25 21:53:50 +00:00
• efff6feed6 * CrawlJob.java avoid NPE on build-then-terminate without a launch nlevitt 2011-07-25 19:16:31 +00:00
• e2f29992d4 Put hard limit of 5 on seed redirects being treated as new seeds, to avoid endless recrawling of the same url(s) in case of a redirect loop. * CandidatesProcessor.java nlevitt 2011-07-23 02:20:35 +00:00
• 269a763860 [HER-1912] Kryo-deserialization of Robotstxt can consume excess memory by creating multiple copies of RobotsDirective * CrawlServer, Robotstxt move autoregister details to Robotstxt, RobotsDirectives * RobotsDirectives use ReferenceFieldSerializer so that repeated instances of RobotsDirectives in same context are replaced with backrefs * RobotstxtTest (from Kenji) unit test for instance multiplication gojomo 2011-07-22 06:46:39 +00:00
• a490737b59 * StatisticsTracker add 'trackSources' to make hosts-by-source-tag report optional in large crawls, where it is very expensive (and an OOME risk until other changes are also made) gojomo 2011-07-20 23:36:19 +00:00
• a69126ad21 Reduce redundant Pattern instance caching * TextUtils maintain global Pattern soft-cache by regex string key gojomo 2011-07-20 23:04:26 +00:00
• b3e35499ca HER-1812 Spring-ify how reports are handled - based partly on patch from Kristinn Sigurðsson * Report.java make StatisticsTracker stats an argument to write() instead of a field; new properties shouldReportAtEndOfCrawl and shouldReportDuringCrawl, both default true, but configurable if desired, replacing the LIVE_REPORTS/END_REPORTS stuff in StatisticsTracker * StatisticsTracker.java new property "reports", a List<Report>, which defaults (with lazy initialization) to the same reports we've always had, replacing hard-coded lists of reports classes and related code * profile-crawler-beans.cxml default reports list, commented out * SeedsReport.java FrontierSummaryReport.java MimetypesReport.java SourceTagsReport.java FrontierNonemptyReport.java CrawlSummaryReport.java ResponseCodeReport.java HostsReport.java ProcessorsReport.java ToeThreadsReport.java update write(PrintWriter, StatisticsTracker) arguments * JobResource.java replace use of StatisticsTracker.LIVE_REPORTS nlevitt 2011-07-20 17:05:12 +00:00
• e9d53ec487 HER-1791 - a different implementation * IdenticalDigestDecideRule.java fix javadoc typo * CoreAttributeConstants.java "history-good-to-store" is history * RecrawlAttributeConstants.java new "write-tag" constant * PersistProcessor.java remove defunct "history-good-to-store" logic; nadd new option onlyStoreIfWriteTagPresent, and update shouldStore() to respect that option * WriterPoolProcessor.java remove defunct "history-good-to-store" logic; new method copyForwardWriteTagIfDupe(); override innerRejectProcess() to call copyForwardWriteTagIfDupe() * ARCWriterProcessor.java, WARCWriterProcessor.java remove defunct "history-good-to-store" logic; add writeTag to history on write; call copyForwardWriteTagIfDupe() on skipIdenticalDigest nlevitt 2011-07-18 20:22:48 +00:00
• 421164aaab * ConfigPathConfigurer log as WARNING when snapshot not possible because launch directory not yet available gojomo 2011-07-16 17:49:11 +00:00
• 0676ded3f2 * CrawlJob.java fix logging of job events nlevitt 2011-07-16 00:48:14 +00:00
• f13b393dab * PersistLogProcessor.java put default persist log path in launch directory nlevitt 2011-07-16 00:44:13 +00:00
• e819d197fe [HER-915] (contrib) add http proxy authentication contributed by Adam Wilmer * FetchHTTP.java properties, setup for http proxy user/password * profile-crawler-beans.cxml commented-out example settings for proxy auth properties gojomo 2011-07-15 21:57:20 +00:00
• 4221aee537 fix selftests * selftest-crawler-beans.cxml write logs, arcs to old constant locations, for ease of post-test verification gojomo 2011-07-15 21:34:59 +00:00
• b90aa89e4b More on HER-1901 - fix build by refactoring launch dir initialization into PathSharingContext; refactor config path interpolation mostly into ConfigPathConfigurer; various tweaks * ActionDirectory.java, CrawlerLoggerModule.java, StatisticsTracker.java, SurtPrefixedDecideRule.java, WriterPoolProcessor.java, profile-crawler-beans.cxml use camelcase ${launchId} * CheckpointService.java, JobResource.java rename getAvailableCheckpointDirectories() to findAvailableCheckpointDirectories() so that it's not treated as a bean property * WriterPoolProcessor.java, WriterPoolSettingsData.java, WriterPoolMember.java rename getOutputDirs() to calcOutputDirs() so that it's not treated as a bean property * ConfigPath.java let ConfigPathConfigurer to do interpolation of ${launchId} * ConfigFile.java let ConfigPathConfigurer do the snapshotting of config file * CrawlJob.java, PathSharingContext.java remove launchDir initialization out of CrawlJob into PathSharingContext * ConfigPathConfigurer.java - remove special handling of WriterPoolProcessor store paths; instead, look in beans for ConfigPaths within Iterables - have each ConfigPath hold reference to this ConfigPathConfigurer to use for interpolating ${launchId} and snapshotting config files nlevitt 2011-07-15 19:39:32 +00:00
• 1bfbec8772 fix build; move Preformatter to package (commons) where first referenced gojomo 2011-07-15 19:02:14 +00:00
• 2e10d07da2 [HER-741] Make extractors interrogate for charset * ExtractorHTML fix reflexive probe to look as many characters in (1000) as initially gojomo 2011-07-14 06:53:56 +00:00
• be12d1e8f3 calmer logging (fewer alerts in job.log/alerts.log/heritrix_out.log) * Recorder object to unsupported content-encodings when first set * FetchHTTP note as annotation unsupported content-encodings * Link not as annotation when base URI is used for absent via * ExtractorHTML downgrade char-sequence reading problem (often a chunking problem) to WARNING from SEVERE gojomo 2011-07-14 06:35:07 +00:00
• 5e2e458469 * StoredQueue (size) more robust against concurrent emptying gojomo 2011-07-14 06:24:14 +00:00
• 9eb0c137e8 HER-1901 twiddles * CrawlJob.java decided to go with plain timestamp17 as launch id, no "launch-" prefix * ConfigPathConfigurer.java refactored special remembering of WriterPoolProcessor storePaths out of fixupPaths() nlevitt 2011-07-14 02:03:01 +00:00
• 6a3072ef46 more rapidly release refs for finished CrawlURIs * UriProcessingFormatter, Preformatter, GenerationFileHandler improve preformat-outside-synchronized optimization so that the LogRecord/CrawlURI doesn't linger until next displaces it * CrawlURI (processingCleanup) null more of last-processing-run collected values gojomo 2011-07-14 01:14:04 +00:00
• c84e87c648 More HER-1901 * CrawlJob.java job log in launch dir covering just that launch nlevitt 2011-07-13 20:47:08 +00:00
• 4d966ae70b More on HER-1901 - support ${launch-id} interpolation on W/ARCWriterProcessor storePaths * WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java change type of storePaths to List<ConfigPath> and handle appropriately * ConfigPathConfigurer.java fixupPaths() - old code did not touch WriterPoolProcessor storePaths, since they're deeply nested inside the bean, but they need to be remembered for later interpolation of ${launch-id}, so add special handling nlevitt 2011-07-13 20:18:51 +00:00
• d5b299d1ef * CrawlJob.java initLaunchId() - didn't mean for this method to be public nlevitt 2011-07-13 20:13:13 +00:00
• c39c927447 * profile-crawler-beans.cxml runWhileEmpty defaults to false nlevitt 2011-07-13 19:20:45 +00:00
• 4250e07de9 HER-1901 timestamped subdirectory for each launch * HardLinker.java renamed FilesystemLinkMaker.java * FilesystemLinkMaker.java add support for symbolic links * CLibrary.java new method symlink() * BdbModule.java use new class name FilesystemLinkMaker * CrawlJob.java at crawl launch, create launch directory launch-{timestamp17}, copy cxml there, symlink "current" to launch dir, inform ConfigPaths * ConfigPath.java interpolate ${launch-id} in configured paths * ConfigFile.java obtainReader() - snapshot config files to launch dir when they are read * ActionDirectory.java default doneDir now ${launch-id}/actions-done actOn() - symlink from old style done dir action/done to done files * SurtPrefixedDecideRule.java default surtsDumpFile now ${launch-id}/surts.dump pathsFixedUp() - this gets called at build time, but we don't want anything written to disk until launch time, so remove call to dumpSurtPrefixSet() here * CrawlerLoggerModule.java default logs dir now ${launch-id}/logs * StatisticsTracker.java default reports dir now ${launch-id}/reports * WriterPoolProcessor.java default writer base path now ${launch-id} * profile-crawler-beans.cxml update to reflect new default paths under launch dirs * PropertyUtils.java fix javadoc typo nlevitt 2011-07-13 19:18:55 +00:00
• a8682ff1ff * TextSeedModule (actOn) use ArchiveUtils.getBufferedReader so that seeds-files via action directory may optionally be gzipped (indicated by .gz suffix) gojomo 2011-07-09 02:13:52 +00:00
• d742434b4e [HER-1911] make writer-pool 'maxActive' setting adjustable mid-crawl * WriterPool make new LARGEST_MAX_ACTIVE (255) the capacity of the reuse-pool, so that the maxActive property may vary up to that and still work; also handle more gracefully the failure of trying to use a larger number than the capacity (writer gets closed early rather than left hanging open) gojomo 2011-07-08 21:59:05 +00:00
• a41809b8a7 [HER-1908] H3: upgrade to Spring 3 * pom.xml, .classpath update references to necessary spring-3.0.5 packages * SeedModule merge rather than replace event listeners, so that (now later) autowiring doesn't clobber the self-insertion done by anonymous (non-top-level) SurtPrefixedDecideRule bean * HeritrixLifecycleProcessor, PathScharingContext use this new non-default LifecycleProcessor to avoid new Spring3 behavior of auto-start()ing context on refresh(build) * CrawlController example of using @Value annotation to set default value: will offer benefits for auto-discovery of defaults for configuration interface, or enforcing maximally-explicit configurations (see [HER-1897]) * profile-crawler-beans.cxml, selftest-crawler-beans.cxml update templates with spring3 preamble/boilerplate gojomo 2011-07-08 04:31:54 +00:00
• aa15d293e8 [HER-1909] H3: slow checkpoint resume, locking out web UI during resume [HER-1910] H3: job-related pages very slow to render (pause mid-way through) in large crawls * BdbModule option to reuse bdb data on StoredQueue-creation * Checkpoint new saveWriter, loadReader utilities for creating extra checkpoint files (for the long list of ready queues) * BdbFrontier on checkpoint, remember ordered list of inProcess/ready/snoozed queues (so same queues are active upon resume) on resume, reuse old retired/inactive StoredQueue data, and reload active queues from newly-saved list * StoredQueue restore tailIndex properly when resuing old data more-efficient size() calculation (for HER-1910) gojomo 2011-07-08 04:30:32 +00:00
• 2b0d967b32 Resolve 'Browse Beans' issue after built, pre-launch * JobRelatedResource catch InvalidPropertyException, render as red error message rather than fouling entire trace of beans to web UI * StatisticsTracker rename getSnapshots to listSnapshots to prevent interpretation as (invalid) bean-property gojomo 2011-06-27 22:25:12 +00:00
• 041554fa5d Fix for HER-1906 checkpointing gives error on Windows * HardLinker.java makeHardLink() - wrap unix link() and windows CreateHardLink() and choose depending on platform * BdbModule.java doCheckpoint() - use HardLinker.makeHardLink() nlevitt 2011-06-24 19:24:28 +00:00
• 39091dc82c Fix tests * TextSeedModule remove unintended @Required annotation gojomo 2011-06-22 23:09:43 +00:00
• 842f495ee9 [HER-1905] H3: allow crawling to begin while large seed list still loading * TextSeedModule refactor announceSeeds to occur in background thread if non-default blockAwaitingSeedLines value is set signal CountDownLatch on each line, allowing calling thread to proceed at right count * profile-crawler-beans.cxml commented-out blockAwaitingSeedLines default settings (-1, meaning wait for all seed lines) gojomo 2011-06-22 21:21:26 +00:00
• b9b64a6029 [HER-1776] H3: holding empty crawl for more URIs (other than 'paused') * profile-crawler-beans.cxml replace obsolete pauseAtEmpty example property with replacement runWhileEmpty gojomo 2011-06-22 21:16:39 +00:00
• 9d2b293d4c [HER-1904] H3: OOME after a few hours of previously non-problematic crawl * Recorder use min(raw-buffer size, actual recorded byte size) for char buffer size * TextUtils on recycleMatcher, dereference previous 'input' gojomo 2011-06-20 20:32:14 +00:00
• d3475fadbf * BdbMultipleWorkQueues.java getFrom() - support null marker to start from first key, as appeared to be the intention nlevitt 2011-06-20 18:06:41 +00:00
• d57b7751ae * SelfTestBase.java verifyProgressStatistics() - expect "RUNNING" instead of "RESUMED" per r7180 nlevitt 2011-06-15 18:10:37 +00:00