Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• b3e35499ca HER-1812 Spring-ify how reports are handled - based partly on patch from Kristinn Sigurðsson * Report.java make StatisticsTracker stats an argument to write() instead of a field; new properties shouldReportAtEndOfCrawl and shouldReportDuringCrawl, both default true, but configurable if desired, replacing the LIVE_REPORTS/END_REPORTS stuff in StatisticsTracker * StatisticsTracker.java new property "reports", a List<Report>, which defaults (with lazy initialization) to the same reports we've always had, replacing hard-coded lists of reports classes and related code * profile-crawler-beans.cxml default reports list, commented out * SeedsReport.java FrontierSummaryReport.java MimetypesReport.java SourceTagsReport.java FrontierNonemptyReport.java CrawlSummaryReport.java ResponseCodeReport.java HostsReport.java ProcessorsReport.java ToeThreadsReport.java update write(PrintWriter, StatisticsTracker) arguments * JobResource.java replace use of StatisticsTracker.LIVE_REPORTS nlevitt 2011-07-20 17:05:12 +00:00
• e9d53ec487 HER-1791 - a different implementation * IdenticalDigestDecideRule.java fix javadoc typo * CoreAttributeConstants.java "history-good-to-store" is history * RecrawlAttributeConstants.java new "write-tag" constant * PersistProcessor.java remove defunct "history-good-to-store" logic; nadd new option onlyStoreIfWriteTagPresent, and update shouldStore() to respect that option * WriterPoolProcessor.java remove defunct "history-good-to-store" logic; new method copyForwardWriteTagIfDupe(); override innerRejectProcess() to call copyForwardWriteTagIfDupe() * ARCWriterProcessor.java, WARCWriterProcessor.java remove defunct "history-good-to-store" logic; add writeTag to history on write; call copyForwardWriteTagIfDupe() on skipIdenticalDigest nlevitt 2011-07-18 20:22:48 +00:00
• 421164aaab * ConfigPathConfigurer log as WARNING when snapshot not possible because launch directory not yet available gojomo 2011-07-16 17:49:11 +00:00
• 0676ded3f2 * CrawlJob.java fix logging of job events nlevitt 2011-07-16 00:48:14 +00:00
• f13b393dab * PersistLogProcessor.java put default persist log path in launch directory nlevitt 2011-07-16 00:44:13 +00:00
• e819d197fe [HER-915] (contrib) add http proxy authentication contributed by Adam Wilmer * FetchHTTP.java properties, setup for http proxy user/password * profile-crawler-beans.cxml commented-out example settings for proxy auth properties gojomo 2011-07-15 21:57:20 +00:00
• 4221aee537 fix selftests * selftest-crawler-beans.cxml write logs, arcs to old constant locations, for ease of post-test verification gojomo 2011-07-15 21:34:59 +00:00
• b90aa89e4b More on HER-1901 - fix build by refactoring launch dir initialization into PathSharingContext; refactor config path interpolation mostly into ConfigPathConfigurer; various tweaks * ActionDirectory.java, CrawlerLoggerModule.java, StatisticsTracker.java, SurtPrefixedDecideRule.java, WriterPoolProcessor.java, profile-crawler-beans.cxml use camelcase ${launchId} * CheckpointService.java, JobResource.java rename getAvailableCheckpointDirectories() to findAvailableCheckpointDirectories() so that it's not treated as a bean property * WriterPoolProcessor.java, WriterPoolSettingsData.java, WriterPoolMember.java rename getOutputDirs() to calcOutputDirs() so that it's not treated as a bean property * ConfigPath.java let ConfigPathConfigurer to do interpolation of ${launchId} * ConfigFile.java let ConfigPathConfigurer do the snapshotting of config file * CrawlJob.java, PathSharingContext.java remove launchDir initialization out of CrawlJob into PathSharingContext * ConfigPathConfigurer.java - remove special handling of WriterPoolProcessor store paths; instead, look in beans for ConfigPaths within Iterables - have each ConfigPath hold reference to this ConfigPathConfigurer to use for interpolating ${launchId} and snapshotting config files nlevitt 2011-07-15 19:39:32 +00:00
• 1bfbec8772 fix build; move Preformatter to package (commons) where first referenced gojomo 2011-07-15 19:02:14 +00:00
• 2e10d07da2 [HER-741] Make extractors interrogate for charset * ExtractorHTML fix reflexive probe to look as many characters in (1000) as initially gojomo 2011-07-14 06:53:56 +00:00
• be12d1e8f3 calmer logging (fewer alerts in job.log/alerts.log/heritrix_out.log) * Recorder object to unsupported content-encodings when first set * FetchHTTP note as annotation unsupported content-encodings * Link not as annotation when base URI is used for absent via * ExtractorHTML downgrade char-sequence reading problem (often a chunking problem) to WARNING from SEVERE gojomo 2011-07-14 06:35:07 +00:00
• 5e2e458469 * StoredQueue (size) more robust against concurrent emptying gojomo 2011-07-14 06:24:14 +00:00
• 9eb0c137e8 HER-1901 twiddles * CrawlJob.java decided to go with plain timestamp17 as launch id, no "launch-" prefix * ConfigPathConfigurer.java refactored special remembering of WriterPoolProcessor storePaths out of fixupPaths() nlevitt 2011-07-14 02:03:01 +00:00
• 6a3072ef46 more rapidly release refs for finished CrawlURIs * UriProcessingFormatter, Preformatter, GenerationFileHandler improve preformat-outside-synchronized optimization so that the LogRecord/CrawlURI doesn't linger until next displaces it * CrawlURI (processingCleanup) null more of last-processing-run collected values gojomo 2011-07-14 01:14:04 +00:00
• c84e87c648 More HER-1901 * CrawlJob.java job log in launch dir covering just that launch nlevitt 2011-07-13 20:47:08 +00:00
• 4d966ae70b More on HER-1901 - support ${launch-id} interpolation on W/ARCWriterProcessor storePaths * WriterPoolProcessor.java, ARCWriterProcessor.java, WARCWriterProcessor.java change type of storePaths to List<ConfigPath> and handle appropriately * ConfigPathConfigurer.java fixupPaths() - old code did not touch WriterPoolProcessor storePaths, since they're deeply nested inside the bean, but they need to be remembered for later interpolation of ${launch-id}, so add special handling nlevitt 2011-07-13 20:18:51 +00:00
• d5b299d1ef * CrawlJob.java initLaunchId() - didn't mean for this method to be public nlevitt 2011-07-13 20:13:13 +00:00
• c39c927447 * profile-crawler-beans.cxml runWhileEmpty defaults to false nlevitt 2011-07-13 19:20:45 +00:00
• 4250e07de9 HER-1901 timestamped subdirectory for each launch * HardLinker.java renamed FilesystemLinkMaker.java * FilesystemLinkMaker.java add support for symbolic links * CLibrary.java new method symlink() * BdbModule.java use new class name FilesystemLinkMaker * CrawlJob.java at crawl launch, create launch directory launch-{timestamp17}, copy cxml there, symlink "current" to launch dir, inform ConfigPaths * ConfigPath.java interpolate ${launch-id} in configured paths * ConfigFile.java obtainReader() - snapshot config files to launch dir when they are read * ActionDirectory.java default doneDir now ${launch-id}/actions-done actOn() - symlink from old style done dir action/done to done files * SurtPrefixedDecideRule.java default surtsDumpFile now ${launch-id}/surts.dump pathsFixedUp() - this gets called at build time, but we don't want anything written to disk until launch time, so remove call to dumpSurtPrefixSet() here * CrawlerLoggerModule.java default logs dir now ${launch-id}/logs * StatisticsTracker.java default reports dir now ${launch-id}/reports * WriterPoolProcessor.java default writer base path now ${launch-id} * profile-crawler-beans.cxml update to reflect new default paths under launch dirs * PropertyUtils.java fix javadoc typo nlevitt 2011-07-13 19:18:55 +00:00
• a8682ff1ff * TextSeedModule (actOn) use ArchiveUtils.getBufferedReader so that seeds-files via action directory may optionally be gzipped (indicated by .gz suffix) gojomo 2011-07-09 02:13:52 +00:00
• d742434b4e [HER-1911] make writer-pool 'maxActive' setting adjustable mid-crawl * WriterPool make new LARGEST_MAX_ACTIVE (255) the capacity of the reuse-pool, so that the maxActive property may vary up to that and still work; also handle more gracefully the failure of trying to use a larger number than the capacity (writer gets closed early rather than left hanging open) gojomo 2011-07-08 21:59:05 +00:00
• a41809b8a7 [HER-1908] H3: upgrade to Spring 3 * pom.xml, .classpath update references to necessary spring-3.0.5 packages * SeedModule merge rather than replace event listeners, so that (now later) autowiring doesn't clobber the self-insertion done by anonymous (non-top-level) SurtPrefixedDecideRule bean * HeritrixLifecycleProcessor, PathScharingContext use this new non-default LifecycleProcessor to avoid new Spring3 behavior of auto-start()ing context on refresh(build) * CrawlController example of using @Value annotation to set default value: will offer benefits for auto-discovery of defaults for configuration interface, or enforcing maximally-explicit configurations (see [HER-1897]) * profile-crawler-beans.cxml, selftest-crawler-beans.cxml update templates with spring3 preamble/boilerplate gojomo 2011-07-08 04:31:54 +00:00
• aa15d293e8 [HER-1909] H3: slow checkpoint resume, locking out web UI during resume [HER-1910] H3: job-related pages very slow to render (pause mid-way through) in large crawls * BdbModule option to reuse bdb data on StoredQueue-creation * Checkpoint new saveWriter, loadReader utilities for creating extra checkpoint files (for the long list of ready queues) * BdbFrontier on checkpoint, remember ordered list of inProcess/ready/snoozed queues (so same queues are active upon resume) on resume, reuse old retired/inactive StoredQueue data, and reload active queues from newly-saved list * StoredQueue restore tailIndex properly when resuing old data more-efficient size() calculation (for HER-1910) gojomo 2011-07-08 04:30:32 +00:00
• 2b0d967b32 Resolve 'Browse Beans' issue after built, pre-launch * JobRelatedResource catch InvalidPropertyException, render as red error message rather than fouling entire trace of beans to web UI * StatisticsTracker rename getSnapshots to listSnapshots to prevent interpretation as (invalid) bean-property gojomo 2011-06-27 22:25:12 +00:00
• 041554fa5d Fix for HER-1906 checkpointing gives error on Windows * HardLinker.java makeHardLink() - wrap unix link() and windows CreateHardLink() and choose depending on platform * BdbModule.java doCheckpoint() - use HardLinker.makeHardLink() nlevitt 2011-06-24 19:24:28 +00:00
• 39091dc82c Fix tests * TextSeedModule remove unintended @Required annotation gojomo 2011-06-22 23:09:43 +00:00
• 842f495ee9 [HER-1905] H3: allow crawling to begin while large seed list still loading * TextSeedModule refactor announceSeeds to occur in background thread if non-default blockAwaitingSeedLines value is set signal CountDownLatch on each line, allowing calling thread to proceed at right count * profile-crawler-beans.cxml commented-out blockAwaitingSeedLines default settings (-1, meaning wait for all seed lines) gojomo 2011-06-22 21:21:26 +00:00
• b9b64a6029 [HER-1776] H3: holding empty crawl for more URIs (other than 'paused') * profile-crawler-beans.cxml replace obsolete pauseAtEmpty example property with replacement runWhileEmpty gojomo 2011-06-22 21:16:39 +00:00
• 9d2b293d4c [HER-1904] H3: OOME after a few hours of previously non-problematic crawl * Recorder use min(raw-buffer size, actual recorded byte size) for char buffer size * TextUtils on recycleMatcher, dereference previous 'input' gojomo 2011-06-20 20:32:14 +00:00
• d3475fadbf * BdbMultipleWorkQueues.java getFrom() - support null marker to start from first key, as appeared to be the intention nlevitt 2011-06-20 18:06:41 +00:00
• d57b7751ae * SelfTestBase.java verifyProgressStatistics() - expect "RUNNING" instead of "RESUMED" per r7180 nlevitt 2011-06-15 18:10:37 +00:00
• 45a59d3a19 [HER-1776] H3: holding empty crawl for more URIs (other than 'paused') * CrawlController change pauseAtFinish to runWhileEmpty add EMPTY state replace isStateRunning() with isActive() (RUNNING or EMPTY) trigger finish-test off frontier EMPTY rather than PAUSE in logging say 'running' rather than 'resumed' * Frontier add EMPTY state * AbstractFrontier handle EMPTY like RUNNING, transition EMPTY<->RUNNING when appropriate * StatisticsTracker handle EMPTY analogous to RUNNING in logging say 'running' rather than 'resumed' * CrawlJob, DiskSpaceMonitor replace isStateRunning() with isActive() gojomo 2011-06-15 06:01:49 +00:00
• d9a53ff584 * JobRelatedResource (writeNestedNames) use local getPropertyDescriptors(bwrap) to only show safely-displayable properties gojomo 2011-06-14 22:39:21 +00:00
• 88416a150a Tangent revealed by [HER-1898] eritrix 3.1 misses many pages that Heritrix 3.0 catches * Recorder use intended basis for GenericReplayCharSequence in-memory-prefix * RecordingInputStream reveal internal ROS buffer size gojomo 2011-06-14 03:27:21 +00:00
• 78755dd304 HER-1898 - Heritrix 3.1 misses many pages that Heritrix 3.0 catches - reported by Dan Yadgar * GenericReplayCharSequence.java decode() - read another character to check for more content, rather than consult BufferedReader.ready(), since the latter sometimes returns false even when there is more to read nlevitt 2011-06-14 02:11:21 +00:00
• 4ef6c5dc24 [HER-1899] scripting console comes back as XML in webkit browsers (Chrome, Safari) * JobRelatedResource derive from BaseResource for webkit-XML-workaround gojomo 2011-06-13 23:31:36 +00:00
• 19390f270a extra output to determine reason for slow test runs gojomo 2011-06-08 23:56:35 +00:00
• 71ca9262df extra output to determine reason for slow test runs gojomo 2011-06-08 23:49:53 +00:00
• c4636f38ec [HER-741] Make extractors interrogate for charset * FetchHTTP use Charset-instances rather than String names gojomo 2011-06-04 00:25:23 +00:00
• 2043d1801a [HER-741] Make extractors interrogate for charset * ReplayCharSequence, GenericReplayCharSequence use Charset instance rather than String-name * Recorder use Charset instance ratehr than String-name (getContentReplayCharSequence) avoid reusing cached ReplayCharSequence when encoding-in-use has changed since it was created (getContentReplayPrefixString) allow requested specific charset-interpretation * ExtractorHTML, ExtractorXML work with Charset instances double-check that in-content-declarations are self-consistent before using recycle matcher instances change minor charset problems/decisions to a crawl.log annotation gojomo 2011-06-03 21:28:03 +00:00
• 572aa83d39 * CrawlURI make extraInfo declarations adjacent utility test for charset-declared-in-contentType gojomo 2011-06-03 21:17:32 +00:00
• 4ea85ac817 * commons/pom.xml bump test heap back to -Xmx1g * BloomFilterTest restore 495MB-filter test gojomo 2011-05-31 22:28:39 +00:00
• e791c23751 * BloomFilterTest temporarily disable 495MB test until build box has more RAM gojomo 2011-05-28 01:10:33 +00:00
• 292b30af1b remove unnecessary Serializable interface gojomo 2011-05-28 00:02:38 +00:00
• 083e75e30a * CrawlServer.java autoregister ConcurrentSkipListSet gojomo 2011-05-28 00:02:10 +00:00
• 164e6e2999 * commons/pom.xml increase heap for surefire to size (640m) that seems to work on build box nlevitt 2011-05-27 22:27:40 +00:00
• 0cb6e96925 * commons/pom.xml reduce heap for surefire in hopes of speeding build on memory-constrained build box gojomo 2011-05-27 05:17:27 +00:00
• 99a49b2df7 Avoid treating temporarily-DEFERRED robots.txt handling as-if a failure-to-ignore in robots-ignored case * DispositionProcessor treat S_DEFERRED different from other negative error statuses gojomo 2011-05-27 01:24:49 +00:00
• 728951d1ff Fix for HER-1791 - heritrix writes revisit record whether or not previous fetch was archived * CoreAttributeConstants.java new flag A_HISTORY_GOOD_TO_STORE w/ javadoc * WriterPoolProcessor.java shouldWrite() - set A_HISTORY_GOOD_TO_STORE on CrawlURI if we decide to skip writing because of previous fetch with identical digest * WARCWriterProcessor.java write() - set A_HISTORY_GOOD_TO_STORE on CrawlURI on successful write to warc * PersistProcessor.java shouldStore() - change to return true if A_HISTORY_GOOD_TO_STORE is set on the CrawlURI nlevitt 2011-05-26 16:44:53 +00:00
• 3acedc86a2 Different fix for HER-1891 which also fixes HER-1890, two cases of "seed redirect url sometimes '0 NOTCRAWLED' in seeds-report.txt even when crawled" * CandidatesProcessor.java innerProcess() - set force-fetch on outlinks promoted to seeds nlevitt 2011-05-26 16:07:28 +00:00
• 39d6caad8b Revert last fix (r7154,r7155) for HER-1891 in preparation for applying different fix (see discussion on jira issue). * CandidatesProcessor.java nlevitt 2011-05-26 16:04:41 +00:00
• 2d267cf173 HER-843 - crawl.log record of which ARC a capture landed in * WARCWriterProcessor.java write() - on success, add arcFilename to CrawlURI extraInfo nlevitt 2011-05-25 22:23:53 +00:00
• 5e1571f8dd HER-1790 - record warc where url was saved in crawl.log * CrawlURI.java new field JSONObject extraInfo, methods getExtraInfo() and addExtraInfo() * WARCWriterProcessor.java write() - on success, add warcFilename to CrawlURI extraInfo * CrawlerLoggerModule.java new config option logExtraInfo, default false * UriProcessingFormatter.java new field logExtraInfo format() - include CrawlURI extraInfo if logExtraInfo is enabled also include "-" if CrawlURI has no annotations, since this is no longer the last field on the line * profile-crawler-beans.cxml <!-- <property name="logExtraInfo" value="false" /> --> * NonFatalErrorFormatter.java, RuntimeErrorFormatter.java new constructor to handle logExtraInfo since these subclass UriProcessingFormatter nlevitt 2011-05-25 22:11:18 +00:00
• 6827ad4a84 Fix bug introduced in r7154 and revealed by unit tests. * CandidatesProcessor.java innerProcess() - do KeyedProperties overrides at the right time nlevitt 2011-05-25 19:43:12 +00:00
• 29fa5f6563 Fix for HER-1891 seed redirect url sometimes "0 NOTCRAWLED" in seeds-report.txt even when crawled, when original seed also has a regular link to the redirect url * CandidatesProcessor.java innerProcess() - present seed outlinks to the frontier ahead of non-seed outlinks, so that seed version of any duplicated outlink is always the one that's crawled nlevitt 2011-05-25 04:16:53 +00:00
• c2c24d1939 Move "implements CrawlUriReceiver" from WorkQueueFrontier to superclass AbstractFrontier where the method CrawlUriReceiver.receive() is implemented. nlevitt 2011-05-23 22:31:56 +00:00
• e70f0f8a50 Fix HER-1889 should recover from problems unescaping javascript in ExtractorJS * ExtractorJS.java considerStrings() - catch exceptions from StringEscapeUtils.unescapeJavaScript(), log warning, and proceed nlevitt 2011-05-20 00:05:09 +00:00
• d26bee27e8 * Link.java addRelativeToVia() - really use base instead of via when via is missing, as the warning warns will happen nlevitt 2011-05-19 23:43:16 +00:00
• 11b54e7ec1 Fix HER-1888 StringIndexOutOfBoundsException in Recorder.getContentReplayPrefixString() * Recorder.java getContentReplayPrefixString() - return empty string there's no content, to avoid exception nlevitt 2011-05-19 23:19:11 +00:00
• 2a6aca0fcd * Recorder.java getContentReplayPrefixString() - use fallback character encoding if necessary, much like getContentReplayCharSequence() nlevitt 2011-05-03 17:14:28 +00:00
• f7712cc0bf Clarification to help [HER-1885] source-report.txt empty * SourceTagsReport informative note if there's none of the info needed for this report gojomo 2011-04-29 22:21:54 +00:00
• 26f34c1b8c HER-1547 some sites may require an 'Accept' header * FetchHTTP.java "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" by default - this is what my firefox 4.0 for mac sends * profile-crawler-beans.cxml commented-out default value nlevitt 2011-04-25 20:43:17 +00:00
• d159b4e106 HER-741 Make extractors interrogate for charset * ExtractorXML.java if charset not spec'd in http header look for <?xml encoding=""?> * ExtractorHTML.java lookForEncodingInContent() - 1. look for <meta http-equiv="content-type"...> 2. if not found then look for <meta charset=""> 3. if not found then <?xml encoding=""...?> * Recorder.java setCharacterEncoding() - If new encoding is different from old encoding, close replayCharSequence and set to null, which will trigger recreation on next retrieval. nlevitt 2011-04-22 18:43:18 +00:00
• c9c74b5c28 * ExtractorXML.java shouldExtract() - use getContentReplayPrefixString() for <?xml check nlevitt 2011-04-22 00:42:09 +00:00
• a2eaa9d6ac HER-1820 followup * UriUtils.java NAIVE_LIKELY_URI_PATTERN - revert change and unpublicize NAIVE_URI_EXCEPTIONS - add some mimetype strings that came up in testing isLikelyFalsePositive() - unpublicize * ExtractorXML.java XML_URI_EXTRACTOR - do not use NAIVE_LIKELY_URI_PATTERN shouldExtract() - check for mimetype application/vnd.openxmlformats which is not xml processXml() - use UriUtils.isLikelyUri() nlevitt 2011-04-21 17:18:01 +00:00
• 43dae4d84d HER-1884 when html is processed by ExtractorXML, which can happen when it starts with <?xml..., a@href links are treated as embeds * ExtractorXML.java shouldExtract() - return true if content starts with "<?xml" only if it does not also contain "<!doctype html" or "<html" early in the content nlevitt 2011-04-20 20:57:41 +00:00
• c4595fdcc9 * profile-crawler-beans.cxml - <!-- <property name="independentExtractors" value="false" /> --> + <!-- <property name="extractIndependently" value="false" /> --> nlevitt 2011-04-20 20:54:21 +00:00
• 6d6c1059a5 Use Recorder.getContentReplayCharSequence() instead of deprecated getReplayCharSequence() nlevitt 2011-04-20 19:38:12 +00:00
• 630ddb4deb [HER-1883] seeds result info (for seeds report, crawl report tally) lost after checkpoint & resume * SeedRecord.java ensure makeDirty() is called after every updateWith() (but not the initial instantiation) gojomo 2011-04-20 02:38:26 +00:00
• 70d21ca2da HER-1820 more eager xml link extraction * ExtractorXML.java instead of considering only strings that start with http(s):, consider all strings that match UriUtils.NAIVE_LIKELY_URI_PATTERN (and, as before, constitute the entirety of the xml tag content or attribute value) * UriUtils.java - NAIVE_LIKELY_URI_PATTERN - add quotes to the excluded characters so it does not eat the closing quote when matching xml attribute values, and make visibility public for use in ExtractorXML - isLikelyUri() - refactor false positive check into new method isLikelyFalsePositive() so that it can be used in ExtractorXML avoiding redundant check against NAIVE_LIKELY_URI_PATTERN nlevitt 2011-04-20 01:18:00 +00:00
• b064d43db5 Fix HER-1792 backslashes in urls can in some cases really mean backslashes (specifically, when they're in the query string) * UURIFactory.java fixup() - replace \ with / up to the first ? in the url, instead of anywhere in the url and update comments * UURIFactoryTest.java backslash test nlevitt 2011-04-19 02:11:20 +00:00
• 2be8af10a9 * ArchiveReaderFactory comment about possible weakness in checking response to Range:d request gojomo 2011-04-16 08:06:03 +00:00
• 8bc70dd82a [HER-1881] GZIPMembersInputStream.compressedSkip(long) ignores return value of in.skip(offset) * GZIPMembersInputStream use ByteStreams.skipFully to skip desired amount or hit EOF gojomo 2011-04-16 07:43:53 +00:00
• 991a67e0e4 Post 3.1.0-beta * **/pom.xml chnge version-identifier to "3.1.0-SNAPSHOT" gojomo 2011-04-15 22:28:52 +00:00
• 52d00a9917 Prep for 3.1.0-beta * **/pom.xml increment declared version gojomo 2011-04-15 18:09:43 +00:00
• 1a2ec76805 comment typo, missing serialization ID gojomo 2011-04-15 18:08:12 +00:00
• 2df2584573 Fix HER-1880 robots.txt less specific Allow overrides more specific Disallow * RobotsDirectives.java - use plain ConcurrentSkipListSet<String> instead of PrefixSet, to maintain complete list of prefixes, so we can know the longest match - allows(String) - return true if longest matching Allow prefix is longer than or equal to longest matching Disallow prefix * RobotstxtTest.java flip expected result of test of generic Allow against specific Disallow nlevitt 2011-04-14 02:25:43 +00:00
• 75b7b9066c Followup to r7127 - add jna to eclipse classpath * .classpath <classpathentry kind="var" path="M2_REPO/net/java/dev/jna/jna/3.2.3/jna-3.2.3.jar"/> nlevitt 2011-04-13 00:54:26 +00:00
• 722a8549ab [HER-1821] H3: implement hard-link-based BDB/state-dir checkpointing * BdbModule.java new setting, useHardLinkCheckpoints, default true doCheckpoint now creates hard-links to active log files from the in-environment checkpoint-dir doRecover now relinks from checkpoint-dir files, when available, in preference to whatever is already in the env-dir * pom.xml bring in JNA for native calls * CLibrary wrapper for the link() function gojomo 2011-04-12 00:09:51 +00:00
• 6388dc7605 [HER-1878] NullPointerException when parsing (some?) WARC files over slow connections * GZIPMembersInputStream override OpenJDK7GZIPInputStream.readTrailer to not rely on available() as indicator of end-of-stream fix membersIterator hasNext() to also not rely on available() as indicator of end-of-stream * ArchiveReader change innerHasNext to peek one byte (using mark/reset) rather than rely on available() as indicator of end-of-stream remove redundant getInputStream accessor in favor of field-name-matching getIn/setIn * ArchiveReaderFactory, (W)ARCReaderFactory remove no-longer-necessary RepositionableStream references use getIn() rather than getInputStream() gojomo 2011-04-11 22:58:05 +00:00
• bd45720bae improved comments about new naming conventions/default-template gojomo 2011-04-11 22:49:01 +00:00
• f2ad45045b * FetchWhois.java fix whois uri syntax - add missing "/" token nlevitt 2011-04-07 17:17:48 +00:00
• 020ef44551 * FetchWhois.java class-level javadoc including informal specification of whois uri nlevitt 2011-04-07 06:44:06 +00:00
• f4cf64af29 [HER-1053] compressed HTTP fetch: "Accept-encoding: gzip" [HER-728] Offer replay stream that has been un-chunked (whether because response was HTTP/1.1 or used chunked in HTTP/1.0 against spec) [HER-1876] Offer HTTP/1.1 option - for chunked transfer-encoding (but not persistent connections) * (all) update to use new (decoded-as-necessary) content streams/CharSequences gojomo 2011-04-06 20:54:16 +00:00
• 3a53da1073 adapted tests for other changes gojomo 2011-04-06 20:52:06 +00:00
• e29e7dd36a comment typo gojomo 2011-04-06 20:51:40 +00:00
• 9f3b8d11db * ReplayInputStream constructor comment about using destroy() gojomo 2011-04-06 20:51:14 +00:00
• c5eb3ffd53 [HER-1053] compressed HTTP fetch: "Accept-encoding: gzip" [HER-728] Offer replay stream that has been un-chunked (whether because response was HTTP/1.1 or used chunked in HTTP/1.0 against spec) [HER-1876] Offer HTTP/1.1 option - for chunked transfer-encoding (but not persistent connections) * FetchHTTP add 'acceptCompression' and 'useHTTP11' properties, both default false * Recorder track whether recorded-input is transfer-encoded (chunked) or content-encoded (gzip etc) offer alternate replay streams for (1) raw 'messageBody'; (2) entity (un-chunked if necessary) (3) content (decompressed if necessary) always content & GenericReplayCharSequence for CharSequence replays * GenericReplayCharSequence always use a stream (rather than random-access buffer) always decode to prefix buffer first, so short content never touches disk no matter the encoding * InMemoryReplayCharSequence deleted; 'Generic' now works similarly for small content and anyway random-access for single-byte-encodings is now rarely possible (given deconding streams) * RecordingInputStream, RecordingOutputStream adjust for changed stream names, functionality moved to Recorder * ReplayCharSequence use Charset instances rather than names * ReplayInputStream add convenience constructor (and tmp-file-destroy) for copying any other inputStream into a seekable ReplayInputStream gojomo 2011-04-06 20:42:43 +00:00
• 3f98fc248b * ArchiveUtils (readFully) return actual number of bytes read into array * FileUtils (readFullyToFile) simplify; also return total bytes read gojomo 2011-04-06 20:22:30 +00:00
• b79308b2e0 Part of HER-1875 - socket timeout for whois fetcher * FetchWhois.java nlevitt 2011-04-05 01:22:01 +00:00
• 81cda7be59 HER-1875 followup - override SocketFactory to support connect timeout for ftp data connection nlevitt 2011-04-05 01:21:09 +00:00
• 653f24e713 Avoid exception - "org.springframework.beans.NotWritablePropertyException: Invalid property 'timeoutSeconds' of bean class [org.archive.modules.fetcher.FetchFTP]: Bean property 'timeoutSeconds' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?" * FetchFTP.java change setTimeoutSeconds(Integer) to setTimeoutSeconds(int) nlevitt 2011-04-04 20:10:54 +00:00
• 8e7b9a5fdf Fix HER-1875 FetchFTP needs socket timeout * FetchFTP.java setSoTimeoutMs(), getSoTimeoutMs() (init) - set default value of 20000 ms, same as FetchHTTP fetch() - set various timeouts on FTPClient - effectively sets connect timeout and socket timeout on control connection, and socket timeout on data connection (notably, does not set connect timeout on data connection w/ current commons-net) nlevitt 2011-04-04 19:50:56 +00:00
• 1395854f9a [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling * OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java backport the working GZIP implementation from OpenJDK7, with minimal renames/imports/private-to-protected changes to enable member-at-a-time reading * GZIPMembersInputStream base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode * GZIPMembersInputStreamTest, ARCWriterTest test tweaks gojomo 2011-03-30 02:42:26 +00:00
• e6e415b9d9 [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling * OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java backport the working GZIP implementation from OpenJDK7, with minimal renames/imports/private-to-protected changes to enable member-at-a-time reading * GZIPMembersInputStream base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode * GZIPMembersInputStreamTest, ARCWriterTest test tweaks gojomo 2011-03-30 02:42:04 +00:00
• e60fd8b0e1 * XmlMarshaller.java marshalDocument() - SAXException can wrap other exceptions, e.g. EOFException when tcp connection is broken; thus, in case of IOException, rather than throwing a RuntimeException, re-throw the wrapped IOException (unfortunately in that case we still get a massive stack trace in the log, but at least it comes from the appropriate spot and is not too misleading) nlevitt 2011-03-02 01:34:19 +00:00
• 013b5e394d Different fix for HER-1872: additional property preloadSourceUrl. Use either that or preloadSource (a filesystem path), rather than try to handle both possibilities with one string. * PersistLoadProcessor.java * PersistProcessor.java nlevitt 2011-03-02 01:24:28 +00:00
• 17b12ca19b * HostsReport.java url-encode hosts (other than "dns:") in host report, in case of hostname containing whitespace or other unusual characters, which can happen nlevitt 2011-03-01 23:43:58 +00:00
• 3fc3d324e8 * ArchiveReaderFactory avoid misinterpreting a "c:" local path as a URI scheme gojomo 2011-02-26 01:04:26 +00:00
• d97d4aa8cb Fix HER-1872 PersistLoadProcessor preloadSource does not support url (regression from h1) * PersistLoadProcessor.java change preloadSource from ConfigFile to String, allowing the logic in PersistProcessor.copyPersistSourceToHistoryMap() to decide how to load it nlevitt 2011-02-26 00:59:35 +00:00