Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• adc82c1c6f Second round of HTML cleanup: - added missing <html> and <body> - fixed non-working CSS link, added CSS link to all pages - replaced layout by <br> and &nbsp; with structural tags with CSS - removed superfluous output, notably in BeanBrowseResource - <code>-ify variable names in ScriptResource - added special handling of NaN in doubleToString() Kenji Nagahashi 2012-03-03 00:46:56 -08:00
• 46c335c8a3 BeanBrowseResource: fixed malformed HTML. Kenji Nagahashi 2011-05-11 01:25:41 -07:00
• 1d05df7fbe * StatisticsTracker.java writeReportFile() - reduce log level from warning to info when reusing existing report (normal after crawl is finished) Noah Levitt 2012-02-29 11:02:39 -08:00
• 7af7cbf18c Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2012-02-10 15:51:15 -08:00
• db801b3e0f Fix problems handling crawl job paths with characters that require url-encoding, since we sometimes refer to them with file: urls * PathSharingContext.java, CrawlJob.java use java.net.URI to convert paths to and from file: urls Noah Levitt 2012-02-10 15:48:09 -08:00
• d83ba3070b Merge branch 'master' of https://github.com/travisfw/heritrix3 Noah Levitt 2012-02-09 16:43:19 -08:00
• cfac7ead91 Fix for HER-1985 H3: SurtPrefixDecideRule forgets learned/seed-derived/seed-directive SURT prefixes in checkpoint-resume * SurtPrefixedDecideRule.java make class implement Checkpointable, save surt prefixes to json on checkpoint and load them on recover * profile-crawler-beans.cxml make main SurtPrefixedDecideRule a top-level bean so that it can be checkpointed * Checkpoint.java, CheckpointService.java add some FINE level logging during checkpointing and recovery Noah Levitt 2012-02-09 10:05:07 -08:00
• e90fc2965b Sort script engines alphabetically so that they appear consistently. Note that the name of the engine may be different than the name of the language it parses. A good example is the Rhino script engine that parses javascript. Travis Wellman 2012-01-26 00:37:19 -08:00
• a652a9afd0 * profile-crawler-beans.cxml commented-out default value for new CandidatesProcessor settings processErrorOutlinks Noah Levitt 2012-01-20 11:30:24 -08:00
• 9dbe673bc3 * UriUtils.java some improvements to likely-uri detection based on crawl log analysis Noah Levitt 2012-01-20 11:23:45 -08:00
• 4e05c89156 Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2012-01-19 13:41:14 -08:00
• 803f9ca979 HER-1984 save script state - implement by adding a map to the application context for arbitrary use, and make sure the app context is available in all scripting environments * PathSharingContext.java new member variable ConcurrentHashMap data and accessor getData() * ScriptedProcessor.java, ScriptedDecideRule.java make appCtx available to scripts; also remove unused member sharedMap * ActionDirectory.java formatting fix Noah Levitt 2012-01-19 13:31:57 -08:00
• 20502f54d5 add processErrorOutlinks setting to CandidatesProcessor default false is old behavior, skip candidates-handling of outlinks from response codes <200 and >=400 if true, these outlinks will be treated the same as others (get scope-tested and enqueued) gojomo 2012-01-18 01:23:26 -08:00
• 4384312647 reuse pre-allocated shared HTMLLinkContext instance. Kenji Nagahashi 2011-01-25 14:06:43 -08:00
• 55ec01c4e4 Fix for HER-1983 mangled dns records in w/arcs * FetchDNS.java recordDNS() - use local variable for dummy read buffer, instead of member variable that was shared among threads, since RecordingInputStream.read() used the passed buffer as a work area Noah Levitt 2012-01-11 14:38:57 -08:00
• d3ea17baf5 [HER-1976] query-portion ignored when checking against robots.txt rules (using only path) (RobotsPolicyTest) added for case that failed pre-fix (RobotsTxtTest) added case similar to problem (but was already OK at RobotsTxt level) (RobotsPolicy, *RobotsPolicy) rename method to reflect new behavior gojomo 2012-01-05 23:37:00 -08:00
• 4829160c84 [HER-1981] H3: overlay setting of alternate queue-precedence (via BaseQueuePrecedencePolicy.basePrecedence) doesn't stick (CrawlURI) don't null overlayNames in processingCleanup (WorkQueueFrontier) move queue precedence recalc to only in handleQueue, not all (post-wake) reenqueues gojomo 2012-01-05 23:01:54 -08:00
• 4ffb6865e1 [HER-1976] H3: query-portion ignored when checking against robots.txt rules (using only path) * RobotsPolicy.java: use UURI.getPathQuery -- like H1 does Gordon Mohr 2011-12-22 19:04:19 -08:00
• 4c1740ce1e Altered FetchStats to also keep track of the number of URIs that have an HTTP status between 200 and 299. Christopher Miles 2011-11-17 12:34:05 -05:00
• 86cf3e1fcd Added two new decide rules that make decisions against the HTTP status code for the provided URI. This is likely most useful when you want to keep the size of the stored crawled data down be rejecting 404s, etc. Christopher Miles 2011-11-16 14:53:11 -05:00
• a4380db177 .classpath remove unusued, unneeded dependency asm-3.2.jar, see http://tech.groups.yahoo.com/group/archive-crawler/message/7384 Noah Levitt 2011-11-03 16:15:20 -07:00
• abda8a01fb HER-1963 Unvisited hosts being written twice into H3-generated hosts-report.txt * HostsReport.java write() - remove obsolete code described by this no longer correct comment - "StatisticsTracker doesn't know of zero-completion hosts; so supplement report with those entries from host cache" - StatisticsTracker does know of zero-completion hosts Noah Levitt 2011-11-02 13:01:40 -07:00
• dff143b301 HER-1964 record whois server ip address in warc * FetchWhois.java fetch() - record server ip address in CrawlURI attributes list * CoreAttributeConstants.java new string constant A_WHOIS_SERVER_IP, key for whois server ip address CrawlURI attribute * WARCWriterProcessor.java writeWhoisRecords() - include warc header WARC-IP-Address Noah Levitt 2011-11-01 16:57:22 -07:00
• 48434a6293 Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2011-11-01 16:56:43 -07:00
• 450a4c44c4 * CrawlURI.java markPrerequisite() - call setPrerequisite(true) on the prerequisite CrawlURI Noah Levitt 2011-10-28 09:55:55 -07:00
• 264d3c6f5f * CrawlURI.java markPrerequisite() - call setPrerequisite(true) on the prerequisite CrawlURI Noah Levitt 2011-10-28 09:55:55 -07:00
• 995133dc40 Fix for HER-1962 NPE from missing sheet * SheetOverlaysManager.java getOverlayMap(String) - return null if sheet missing instead of triggering npe * KeyedProperties.java get(String) - check for null return value from getOverlayMap() and log warning Noah Levitt 2011-10-27 10:40:30 -07:00
• 9b418bbb76 * LICENSE.txt revert previous commit - we still distribute jars and licenses in the binary distribution (duh) Noah Levitt 2011-10-21 23:10:25 -07:00
• 239f616c17 * **/.gitignore dummy files to make git keep these directories which are expected by unit tests Noah Levitt 2011-10-21 22:40:44 -07:00
• 23028ccab1 * dist/LICENSE.txt remove clause about lib/ directory, which doesn't apply to h3 (XXX replace with something about maven downloading dependencies?) Noah Levitt 2011-10-21 22:39:22 -07:00
• e8cdfcd82f * README.txt symlink to dist/README.txt just to have one in the root of the project Noah Levitt 2011-10-21 15:56:33 -07:00
• 31edae9753 * .classpath source jar for commons-cli Noah Levitt 2011-10-21 15:43:21 -07:00
• 5735839252 Post 3.1.0 * **/pom.xml switch version to "3.1.1-SNAPSHOT" git-svn nlevitt 2011-10-21 19:00:37 +00:00
• 8c99fc947c Prep for 3.1.0 release * **/pom.xml bump version-id to "3.1.0" * README.txt refer to exactly 3.1.0 release notes nlevitt 2011-10-21 17:08:04 +00:00
• e8d4ec73c7 * bin/heritrix when starting in background, make better effort to write pid to $HERITRIX_HOME/heritrix.pid on successful start, plus this change: - echo -n "date Starting heritrix" + echo -n "date Heritrix starting (pid $HERITRIX_PID)" nlevitt 2011-10-21 16:54:40 +00:00
• e9a96ebc12 * profile-crawler-beans.cxml commented-out ReschedulingProcessor nlevitt 2011-10-21 01:20:32 +00:00
• db049be78d * logging.properties revert unintentional changes nlevitt 2011-10-21 00:52:55 +00:00
• d96e086eeb * logging.properties fix typo nlevitt 2011-10-21 00:48:20 +00:00
• 32680b8f73 Fix for HER-1960 kryo buffer can hog memory, lead to oome (bug report, patch from Kenji) * KryoBinding.java wrap ObjectBuffers with WeakReference so that can be garbage collected if necessary (when recreated they'll be back at the default size of 16k) nlevitt 2011-10-20 02:21:54 +00:00
• 351a89c5a9 Fix java.lang.IllegalArgumentException from CharBuffer.allocate() in GenericReplayCharSequence.decode() Reported by Kenji who says, "My guess is that root cause is Recorder.calcRecommendedCharBufferSize(): return Math.min(inStream.getRecordedBufferLength()/2,(int)inStream.getSize()); URL above returns HTML larger than 5GB (infinite smileys!! what the heck), and (int)intStream.getSize() became negative." * Recorder.java - return Math.min(inStream.getRecordedBufferLength()/2,(int)inStream.getSize()); + return (int) Math.min(inStream.getRecordedBufferLength()/2, inStream.getSize()); nlevitt 2011-10-13 15:47:02 +00:00
• 9fb7b0f5ae Fix for HER-1958 race condition on frontier inactive queues * WorkQueueFrontier.java activateInactiveQueue(), deactivateQueue() - synchronize around updates of inactiveQueuesByPrecedence and highestPrecedenceWaiting nlevitt 2011-10-13 01:47:25 +00:00
• 189a557af8 Tweak of last checkin * Heritrix.java instanceMain() - move logging of java vendor/version ahead of version check, since the information could be particularly helpful when version check doesn't pass nlevitt 2011-10-12 18:32:14 +00:00
• 61240573b3 * Heritrix.java instanceMain() - log java vendor, version at startup nlevitt 2011-10-12 18:20:27 +00:00
• a09c8e69b8 * Recorder.java remove TODOs that have already been done nlevitt 2011-10-12 17:52:45 +00:00
• 73e6d21fd2 * profile-crawler-beans.cxml fix small typo nlevitt 2011-10-12 17:49:34 +00:00
• 270faa30c8 Remove empty java package directories. nlevitt 2011-10-12 17:46:44 +00:00
• 10b3686ba8 Fix race condition which can happen with multiple concurrent "launch"es - can end up with duplicate ToePools like before r7268, probably other problems * CrawlJob.java launch() - synchronize this method - sort of a blunt fix but probably safest and shouldn't affect performance nlevitt 2011-10-07 23:18:43 +00:00
• 19e94f07d1 Fix race condition encountered by Travis and Adam where two ToeThread pools are created, one on "launch" and one on "unpause" * CrawlController.java requestCrawlResume() - do not create ToePool (creation here appeared to be inherited cruft), paused crawl must already have toe pool nlevitt 2011-10-07 01:54:29 +00:00
• b4e6f43052 Unset script engine variables after scripts complete, allowing memory to be freed up sooner - see also r7252 * ScriptResource.java, ActionDirectory.java, ScriptedDecideRule.java nlevitt 2011-10-06 18:01:01 +00:00
• b898d55cea Certain invalid "Crawl-delay" values, e.g. "/", in robots.txt cause split() to return empty string, resulting in ArrayIndexOutOfBoundsException - reported by John Lekashman * Robotstxt.java initializeFromReader() - catch ArrayIndexOutOfBoundsException and ignore -- matches existing handling of other invalid crawl-delay values (catch NumberFormatException and ignore) nlevitt 2011-10-06 17:20:13 +00:00
• a76f9818f9 Fix HER-1955 Some annoying interaction between new creation of latest link and disk full java bean * DiskSpaceMonitor.java checkAvailableSpace() - really do ignore non-existent paths, as comment and log message claim is done nlevitt 2011-09-30 20:07:15 +00:00
• c5de0048d1 "Fix" for HER-1951 ExtractorJS produces many false positives * UriUtils.java isLikelyFalsePositive() - check for unusual characters, likely mimetypes, and a couple of other common traps * UriUtilsTest.java add some tests for this code nlevitt 2011-09-30 00:48:00 +00:00
• a577f1eb0b * PathSharingContext.java start() - restore this override to call initLaunchDir() (how did I miss that), then super.start() nlevitt 2011-09-29 23:12:26 +00:00
• 1390024902 * BdbModuleTest.java testDoCheckpoint() - call BdbModule.destroy() in addition to stop() to close resources nlevitt 2011-09-29 21:37:34 +00:00
• e88d1d9df7 CrawlController FINISHED doesn't guarantee we're ready for shutdown, so add new boolean and event for that purpose. Elaboration: CrawlJob needs all beans to receive FINISHED event before teardown. Can't easily guarantee CrawlJob receives the event last. Actually, spring provides a way, org.springframework.core.Ordered, but to make a bean receive events last, *every other bean* must implement Ordered. So this way is simpler. * CrawlController.java new member variable isStopComplete and event StopCompleteEvent * CrawlJob.java rely on cc.isStopComplete() or StopCompleteEvent to indicate ready for teardown nlevitt 2011-09-29 17:22:34 +00:00
• de9975ee23 * CrawlJob.java instantiateContainer() - do not call teardown() on exception from ac.refresh() - can throw IllegalStateException teardown() - put all uses of variable cc inside not-null check nlevitt 2011-09-29 16:09:44 +00:00
• 747fbc0a0f Fix for HER-1954 bdb closed at crawl finish instead of teardown -- refactor crawl finish and teardown * PathSharingContext.java start(), doStart(), stop(), doStop() - remove these overloaded methods, because the spring bug they were working around seems to be fixed, not seeing any problems with cyclical dependencies - might be this one https://jira.springsource.org/browse/SPR-7266 doClose() - remove because superclass version seems to work fine (this method generally wasn't being called anyway, though it would be now with other changes in this checkin) * CrawlJob.java refactor teardown to call close() on the ApplicationContext, which calls destroy() on any beans that implement DisposableBean - this is now the way to have beans do stuff at teardown * CrawlController.java send FINISHED crawl state event after calling appCtx.stop() so that isFinished() can indicate ready-ness for teardown * BdbModule.java move close() to teardown, i.e. implementation of DisposableBean.destroy(); remove shutdown hook and rely on teardown; related tweaks * WorkQueueFrontier.java, CrawlerLoggerModule.java, BdbUriUniqFilter.java move close() to teardown * CrawlMapper.java, AbstractFrontier.java, PreloadedUriPrecedencePolicy.java, FetchWhois.java, FetchHTTP.java, PersistLogProcessor.java, WriterPoolProcessor.java add comments about cleanup that maybe should wait until teardown * UriUniqFilter.java remove incorrect(?) comment nlevitt 2011-09-29 00:39:55 +00:00
• de856f10a6 Fix possible memory leak (encountered in h1, see r7255) * ToeThread.java run() - set local variable curi=null when finished with it, because the jvm seems to be holding on to it even after the containing block is finished nlevitt 2011-09-28 00:22:53 +00:00
• f704ca5bc7 Fix HER-1953 all api xml elements are camelcase except sizeTotalsReport fields * CrawledBytesHistotable.java camelcase notModified, dupByHash, notModifiedCount, dupByHashCount, novelCount * CrawlJob.java sizeTotalsReportData() - camelcase totalCount nlevitt 2011-09-28 00:19:44 +00:00
• 6adc3c285f * Heritrix.java usage() - out.print("Your arguments were: "+StringUtils.join(args, ' ')); + out.println("Your arguments were: "+StringUtils.join(args, ' ')); nlevitt 2011-09-27 04:08:58 +00:00
• f242522efc Fix for HER-1950 URIAuthorityBasedQueueAssignmentPolicy KeyedProperties shadows superclass QueueAssignmentPolicy KeyedProperties, breaking sheet overlay of properties defined in superclass * URIAuthorityBasedQueueAssignmentPolicy.java remove field kp, accessor getKeyedProperties(); instead inherit these from superclass QueueAssignmentPolicy nlevitt 2011-09-20 23:52:59 +00:00
• 54aca17757 Removing very old abandoned line of development, package org.archive.extractor nlevitt 2011-09-14 21:34:26 +00:00
• 12411395a8 * ScriptedProcessor.java innerProcess() - unset engine variable "curi" after script completes, allowing its memory to be freed up sooner nlevitt 2011-09-14 02:30:02 +00:00
• 4eef2d9902 Revert change in r7240 - different return type makes recovery from checkpoints from before the change fail * CrawlController.java getState() - return Object nlevitt 2011-09-13 00:48:06 +00:00
• 5a4e72d902 Couple of synchronizations for HER-1943 WorkQueue: inconsistent synchronization using fields active, lastCost, peekItem, wakeTime * WorkQueue.java considerActive(), unpeek() - make synchronized nlevitt 2011-09-12 22:49:32 +00:00
• 48b0ddfb03 Fix for HER-1942 CheckpointService: inconsistent synchronization around use of isRunning. Doubtful if isRunning thing is a real problem, but it looks like a good idea to synchronize this method anyway, in case there are concurrent calls to it. * CheckpointService.java setRecoveryCheckpointByName() - mark synchronized nlevitt 2011-09-12 22:00:54 +00:00
• 5934e10073 Fix for HER-1939 ScriptedDecideRule: field sharedEngine non-transient and non-serializable. * ScriptedDecideRule.java make field sharedEngine transient nlevitt 2011-09-12 21:48:27 +00:00
• f20b3eee30 Fix HER-1935 Many calls to File.mkdirs() and other file/dir methods don't check the return value. Also fix bug where pointless empty directories were created in scratch dir. * BasicProfileTest.java, SelfTestBase.java, CrawlControllerTest.java, PrecedenceLoader.java, MigrateH1to3Tool.java, CrawlerLoggerModule.java, StatisticsTracker.java, CheckpointUtils.java, BdbUriUniqFilter.java, ARCWriterProcessorTest.java, WARCWriterProcessorTest.java, PersistProcessor.java, WriterPoolProcessor.java, PrefixFinderTest.java, StoredQueueTest.java, FileUtilsTest.java, ObjectIdentityBdbManualCacheTest.java, ObjectIdentityBdbCacheTest.java, ObjectPlusFilesOutputStream.java, TestUtils.java, TmpDirTestCase.java, Engine.java Replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir). In these cases the calls were either already in a spot where the possible IOException would be handled appropriately, or the line was trivially moved into such a block. * ActionDirectory.java, Engine.java replace calls to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure * BdbModule.java setup() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException" - conveniently the place where this method is called was already in a try block that catches IOException * Checkpoint.java generateFrom() - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir) and add "throws IOException" * CheckpointService.java move call to Checkpoint.generateFrom() inside existing try block since it now can throw IOException * Recorder.java ensure(File) - replace call to File.mkdirs() with FileUtils.ensureWriteableDirectory(dir), and throw IllegalStateException on failure new Recorder(File,String,int,int) - call ensure() on the correct object, the containing directory; and remove redundant call to ensure() nlevitt 2011-09-12 20:12:10 +00:00
• f7cc187b09 * ANVLRecord.java remove accidentally committed test code nlevitt 2011-09-12 18:22:55 +00:00
• d2306fced6 Fix for HER-1934 ANVLRecord: clone() method does not call super.clone() * ANVLRecord.java clone() - return (ANVLRecord) super.clone() - this does the right thing nlevitt 2011-09-12 18:21:21 +00:00
• 0a28717500 Fix for HER-1922 ThreadLocalHttpConnectionManager: possible memory leak due to non-static inner class. * ThreadLocalHttpConnectionManager.java make nested class ConnectionInfo static so it doesn't reference instance of containing class nlevitt 2011-09-12 17:53:25 +00:00
• 40ff3affa3 * ObjectIdentityMemCache.java fix mistake in r7240 - 'val.setIdentityCache()' was no longer being called the first time through, when the object comes from the 'supplier' (thanks Gordon) nlevitt 2011-09-12 17:09:29 +00:00
• 221e63dbd0 Fix for HER-1921 BucketQueueAssignmentPolicy: absolute value of hashCode can be Integer.MIN_VALUE. * BucketQueueAssignmentPolicy.java getClassKey() - cast value to long before taking absolute value nlevitt 2011-09-12 17:02:54 +00:00
• 24f4b715ec Various little cleanups including several discovered by Aaron using an automated tool. * WorkQueueFrontier.java (HER-1925) avoid 2 possible null pointer dereferences * CrawlController.java getState() - change declared return type from Object to State * CrawlerLoggerModule.java (HER-1932) remove unused, unset field "reports" * BdbCookieStorage.java elide pointless extra variable * FetchHTTP.java, DownloadURLConnection.java, ProcessUtils.java (HER-1933) use Arrays.toString() for logged arrays * CrawlServer.java - remove unused field robotstxtChecksum - updateRobots() - avoid reinventing existing utility class InstanceofPredicate * ExternalGeoLookupInterface.java (HER-1938) extend Serializable, since ExternalGeoLocationDecideRule is declared Serializable and has a ExternalGeoLookupInterface field * DecideRuleSequence.java (HER-1937) make field fileLogger transient * PersistLogProcessor.java (HER-1924) remove field recoveryCheckpoint shadowing same field in superclass Processor * ExtractorUniversal.java (HER-1920) use return value of potentialTLD.toLowerCase() as it appears was intended * ARCWriterProcessor.java (HER-1928) avoid possible null pointer dereference * S3URLConnection.java (HER-1933) set S3ServiceException as cause of rethrown IOException, and do not put stacktrace array in message * ObjectIdentityMemCache.java (HER-1929) avoid possible null pointer dereference * .classpath more source jar references, other cleanup nlevitt 2011-09-12 02:12:44 +00:00
• e38f0a73ba * profile-crawler-beans.cxml <bean id="fetchHttp" class="org.archive.modules.fetcher.FetchHTTP"> + <!-- <property name="useHTTP11" value="false" /> --> nlevitt 2011-09-11 17:02:32 +00:00
• d3a6dc1f58 Fix for HER-1946 Mid-Crawl Adjustment of frontier.queueTotalBudget does not propagate to retired queues. * WorkQueueFrontier.java findEligibleURI() - propagate queueTotalBudget (also sessionBudget) to ensure the settings are current right before checking if queue is over budget nlevitt 2011-09-09 23:23:15 +00:00
• 794c41ebd3 Patch from Kenji to fix "weird 'NoSuchElement' exception retrieving XML response from H3 web interface. It appeared to be concurrency related.... Probably Histotable was accessed concurrently from my JMX interface and web interface and it left Histotable in an inconsistent state. Much less likely without JMX monitoring, but it could also happen with H3 without JMX" (Kenji) * Histotable.java make tally(K,long) synchronized nlevitt 2011-09-01 21:04:56 +00:00
• e6de9cd96b * Sheet.java prime() - fix exception triggered in process of handling earlier real exception, which was masking the real exception - see http://tech.groups.yahoo.com/group/archive-crawler/message/7303 nlevitt 2011-08-30 22:41:37 +00:00
• cdb6fb3a9a * PersistProcessor.java copyPersistEnv() - improve logging on history entries by marshalling them as json nlevitt 2011-08-30 22:04:17 +00:00
• c70f7ccbc0 * profile-crawler-beans.cxml remove reference to defunct frontier setting "holdQueues", pointed out by Kristinn Sigurðsson nlevitt 2011-08-08 23:42:47 +00:00
• 28aa2ffede * **/pom.xml improve formatting (no substantive changes at all -- try "svn diff -x -ub -c7232") nlevitt 2011-08-05 17:46:57 +00:00
• 178fdd92b6 * **/pom.xml remove references to unused, defunct agilejava maven repository (had been used for xdoc stuff) nlevitt 2011-08-05 17:37:16 +00:00
• ada8cbd198 Fix bug reported by Travis where in certain corner cases where urls are processed in a particular order, a seed can be incorrectly tagged with S_ROBOTS_PREREQUISITE_FAILURE * DispositionProcessor.java innerProcess() - do not update server's robots info if robots fetch has been deferred nlevitt 2011-08-04 01:32:09 +00:00
• 706869e586 Fix handling of server error on robots.txt - change to what appears to be the intended behavior * DispositionProcessor.java innerProcess() - call server.updateRobots(curi) after special ignore-robots short-circuiting of retry cycle rather than before, so that updateRobots() deemed-not-found handling applies * FetchHTTP.java call curi.setHttpMethod() right after creating the HttpMethod, instead of when processing the response; this is needed so that updateRobots() check of fetchType works properly (before this change, fetchType was "UNKNOWN" on server error) * PreconditionEnforcer.java, CrawlServer.java fix typos nlevitt 2011-08-02 23:28:20 +00:00
• 3cfc315de9 Fix bug reported by Travis where each url in action directory ".include" file is reported as "not canonicalized" in heritrix_out, and some end up crawled in spite of being considered included * WorkQueueFrontier.java considerIncluded(CrawlURI) - call FrontierPreparer.prepare(CrawlURI) so that CrawlURI is canonicalized etc before being considered included (similar to what schedule() does) * CrawlURI.java getCanonicalString(), getPolitenessDelay() - use logger instead of System.err nlevitt 2011-08-02 01:07:11 +00:00
• dcbc259776 Fix HER-1917 failure to use good http authentication type if server supports any that heritrix doesn't - problem reported and fix suggested by Adam Wilmer * HttpAuthenticationCredential.java populate() + http.getParams().setParameter(AuthPolicy.AUTH_SCHEME_PRIORITY, + Arrays.asList(AuthPolicy.DIGEST, AuthPolicy.BASIC)); nlevitt 2011-07-30 01:30:44 +00:00
• 26584d44d7 * ExtractorHTMLTest.java testScriptTagWritingScriptType() - As of r7225, ExtractorJS finds a new url - avoid using a strategically placed space. (The fact that UriUtils.isLikelyUri() returns true for the string ");document.write(unescape(" is kind of crazy, but a separate issue) nlevitt 2011-07-27 20:04:54 +00:00
• bda2fbebe4 Fix HER-1873 strings with spaces can confuse ExtractorJS * ExtractorJS.java considerStrings() - backtrack to just ahead of previous closing quote to reconsider it as a possible opening quote * ExtractorJSTest.java minimal test case nlevitt 2011-07-27 18:56:37 +00:00
• b2307e7cf1 Fix HER-1914 cookie loading and saving does not work properly * AbstractCookieStorage.java make cookiesLoadFile a ConfigFile and use obtainReader() so that the file is snapshotted into launch dir saveCookies() - include expiration date, so file format match the format it claims to use, and what loadCookies() expects loadCookies() - handle expiration date field, and actually put each cookie in the cookie map refactor loadCookies() for use with Reader from ConfigFile.obtainReader() fix up javadocs some nlevitt 2011-07-26 19:30:16 +00:00
• 62e1c835dd Fix HER-1913 cookies not being sent * CookieSpecBase.java match(String, int, String, boolean, SortedMap) - use InternetDomainName.name() instead of .toString(), since the latter doesn't return the plain old domain name * Cookie.java javadoc typo nlevitt 2011-07-26 19:07:26 +00:00
• cb7a454d0c Post 3.1.0-RC1 * **/pom.xml switch version back to "3.1.0-SNAPSHOT" nlevitt 2011-07-26 03:42:02 +00:00
• 347a25e417 Prep for 3.1.0-RC1 release * pom.xml, */pom.xml bump version number to 3.1.0-RC1 nlevitt 2011-07-26 02:57:48 +00:00
• 3a4419967d Remove PathFixupListener, which was only used for SurtPrefixedDecideRule initialization, now done differently as of r7218. * ConfigPathConfigurer.java remove calls to PathFixupListener * PathFixupListener.java removed nlevitt 2011-07-25 23:10:02 +00:00
• a103c5f05c * SurtPrefixedDecideRule.java fix timing reading surts source to make sure launch dir is in place for snapshot nlevitt 2011-07-25 22:52:15 +00:00
• 0c8840db56 * .classpath reference more source jars retrieved with "mvn dependency:sources" nlevitt 2011-07-25 21:53:50 +00:00
• efff6feed6 * CrawlJob.java avoid NPE on build-then-terminate without a launch nlevitt 2011-07-25 19:16:31 +00:00
• e2f29992d4 Put hard limit of 5 on seed redirects being treated as new seeds, to avoid endless recrawling of the same url(s) in case of a redirect loop. * CandidatesProcessor.java nlevitt 2011-07-23 02:20:35 +00:00
• 269a763860 [HER-1912] Kryo-deserialization of Robotstxt can consume excess memory by creating multiple copies of RobotsDirective * CrawlServer, Robotstxt move autoregister details to Robotstxt, RobotsDirectives * RobotsDirectives use ReferenceFieldSerializer so that repeated instances of RobotsDirectives in same context are replaced with backrefs * RobotstxtTest (from Kenji) unit test for instance multiplication gojomo 2011-07-22 06:46:39 +00:00
• a490737b59 * StatisticsTracker add 'trackSources' to make hosts-by-source-tag report optional in large crawls, where it is very expensive (and an OOME risk until other changes are also made) gojomo 2011-07-20 23:36:19 +00:00
• a69126ad21 Reduce redundant Pattern instance caching * TextUtils maintain global Pattern soft-cache by regex string key gojomo 2011-07-20 23:04:26 +00:00