Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 45a59d3a19 [HER-1776] H3: holding empty crawl for more URIs (other than 'paused') * CrawlController change pauseAtFinish to runWhileEmpty add EMPTY state replace isStateRunning() with isActive() (RUNNING or EMPTY) trigger finish-test off frontier EMPTY rather than PAUSE in logging say 'running' rather than 'resumed' * Frontier add EMPTY state * AbstractFrontier handle EMPTY like RUNNING, transition EMPTY<->RUNNING when appropriate * StatisticsTracker handle EMPTY analogous to RUNNING in logging say 'running' rather than 'resumed' * CrawlJob, DiskSpaceMonitor replace isStateRunning() with isActive() gojomo 2011-06-15 06:01:49 +00:00
• d9a53ff584 * JobRelatedResource (writeNestedNames) use local getPropertyDescriptors(bwrap) to only show safely-displayable properties gojomo 2011-06-14 22:39:21 +00:00
• 88416a150a Tangent revealed by [HER-1898] eritrix 3.1 misses many pages that Heritrix 3.0 catches * Recorder use intended basis for GenericReplayCharSequence in-memory-prefix * RecordingInputStream reveal internal ROS buffer size gojomo 2011-06-14 03:27:21 +00:00
• 78755dd304 HER-1898 - Heritrix 3.1 misses many pages that Heritrix 3.0 catches - reported by Dan Yadgar * GenericReplayCharSequence.java decode() - read another character to check for more content, rather than consult BufferedReader.ready(), since the latter sometimes returns false even when there is more to read nlevitt 2011-06-14 02:11:21 +00:00
• 4ef6c5dc24 [HER-1899] scripting console comes back as XML in webkit browsers (Chrome, Safari) * JobRelatedResource derive from BaseResource for webkit-XML-workaround gojomo 2011-06-13 23:31:36 +00:00
• 19390f270a extra output to determine reason for slow test runs gojomo 2011-06-08 23:56:35 +00:00
• 71ca9262df extra output to determine reason for slow test runs gojomo 2011-06-08 23:49:53 +00:00
• c4636f38ec [HER-741] Make extractors interrogate for charset * FetchHTTP use Charset-instances rather than String names gojomo 2011-06-04 00:25:23 +00:00
• 2043d1801a [HER-741] Make extractors interrogate for charset * ReplayCharSequence, GenericReplayCharSequence use Charset instance rather than String-name * Recorder use Charset instance ratehr than String-name (getContentReplayCharSequence) avoid reusing cached ReplayCharSequence when encoding-in-use has changed since it was created (getContentReplayPrefixString) allow requested specific charset-interpretation * ExtractorHTML, ExtractorXML work with Charset instances double-check that in-content-declarations are self-consistent before using recycle matcher instances change minor charset problems/decisions to a crawl.log annotation gojomo 2011-06-03 21:28:03 +00:00
• 572aa83d39 * CrawlURI make extraInfo declarations adjacent utility test for charset-declared-in-contentType gojomo 2011-06-03 21:17:32 +00:00
• 4ea85ac817 * commons/pom.xml bump test heap back to -Xmx1g * BloomFilterTest restore 495MB-filter test gojomo 2011-05-31 22:28:39 +00:00
• e791c23751 * BloomFilterTest temporarily disable 495MB test until build box has more RAM gojomo 2011-05-28 01:10:33 +00:00
• 292b30af1b remove unnecessary Serializable interface gojomo 2011-05-28 00:02:38 +00:00
• 083e75e30a * CrawlServer.java autoregister ConcurrentSkipListSet gojomo 2011-05-28 00:02:10 +00:00
• 164e6e2999 * commons/pom.xml increase heap for surefire to size (640m) that seems to work on build box nlevitt 2011-05-27 22:27:40 +00:00
• 0cb6e96925 * commons/pom.xml reduce heap for surefire in hopes of speeding build on memory-constrained build box gojomo 2011-05-27 05:17:27 +00:00
• 99a49b2df7 Avoid treating temporarily-DEFERRED robots.txt handling as-if a failure-to-ignore in robots-ignored case * DispositionProcessor treat S_DEFERRED different from other negative error statuses gojomo 2011-05-27 01:24:49 +00:00
• 728951d1ff Fix for HER-1791 - heritrix writes revisit record whether or not previous fetch was archived * CoreAttributeConstants.java new flag A_HISTORY_GOOD_TO_STORE w/ javadoc * WriterPoolProcessor.java shouldWrite() - set A_HISTORY_GOOD_TO_STORE on CrawlURI if we decide to skip writing because of previous fetch with identical digest * WARCWriterProcessor.java write() - set A_HISTORY_GOOD_TO_STORE on CrawlURI on successful write to warc * PersistProcessor.java shouldStore() - change to return true if A_HISTORY_GOOD_TO_STORE is set on the CrawlURI nlevitt 2011-05-26 16:44:53 +00:00
• 3acedc86a2 Different fix for HER-1891 which also fixes HER-1890, two cases of "seed redirect url sometimes '0 NOTCRAWLED' in seeds-report.txt even when crawled" * CandidatesProcessor.java innerProcess() - set force-fetch on outlinks promoted to seeds nlevitt 2011-05-26 16:07:28 +00:00
• 39d6caad8b Revert last fix (r7154,r7155) for HER-1891 in preparation for applying different fix (see discussion on jira issue). * CandidatesProcessor.java nlevitt 2011-05-26 16:04:41 +00:00
• 2d267cf173 HER-843 - crawl.log record of which ARC a capture landed in * WARCWriterProcessor.java write() - on success, add arcFilename to CrawlURI extraInfo nlevitt 2011-05-25 22:23:53 +00:00
• 5e1571f8dd HER-1790 - record warc where url was saved in crawl.log * CrawlURI.java new field JSONObject extraInfo, methods getExtraInfo() and addExtraInfo() * WARCWriterProcessor.java write() - on success, add warcFilename to CrawlURI extraInfo * CrawlerLoggerModule.java new config option logExtraInfo, default false * UriProcessingFormatter.java new field logExtraInfo format() - include CrawlURI extraInfo if logExtraInfo is enabled also include "-" if CrawlURI has no annotations, since this is no longer the last field on the line * profile-crawler-beans.cxml <!-- <property name="logExtraInfo" value="false" /> --> * NonFatalErrorFormatter.java, RuntimeErrorFormatter.java new constructor to handle logExtraInfo since these subclass UriProcessingFormatter nlevitt 2011-05-25 22:11:18 +00:00
• 6827ad4a84 Fix bug introduced in r7154 and revealed by unit tests. * CandidatesProcessor.java innerProcess() - do KeyedProperties overrides at the right time nlevitt 2011-05-25 19:43:12 +00:00
• 29fa5f6563 Fix for HER-1891 seed redirect url sometimes "0 NOTCRAWLED" in seeds-report.txt even when crawled, when original seed also has a regular link to the redirect url * CandidatesProcessor.java innerProcess() - present seed outlinks to the frontier ahead of non-seed outlinks, so that seed version of any duplicated outlink is always the one that's crawled nlevitt 2011-05-25 04:16:53 +00:00
• c2c24d1939 Move "implements CrawlUriReceiver" from WorkQueueFrontier to superclass AbstractFrontier where the method CrawlUriReceiver.receive() is implemented. nlevitt 2011-05-23 22:31:56 +00:00
• e70f0f8a50 Fix HER-1889 should recover from problems unescaping javascript in ExtractorJS * ExtractorJS.java considerStrings() - catch exceptions from StringEscapeUtils.unescapeJavaScript(), log warning, and proceed nlevitt 2011-05-20 00:05:09 +00:00
• d26bee27e8 * Link.java addRelativeToVia() - really use base instead of via when via is missing, as the warning warns will happen nlevitt 2011-05-19 23:43:16 +00:00
• 11b54e7ec1 Fix HER-1888 StringIndexOutOfBoundsException in Recorder.getContentReplayPrefixString() * Recorder.java getContentReplayPrefixString() - return empty string there's no content, to avoid exception nlevitt 2011-05-19 23:19:11 +00:00
• 2a6aca0fcd * Recorder.java getContentReplayPrefixString() - use fallback character encoding if necessary, much like getContentReplayCharSequence() nlevitt 2011-05-03 17:14:28 +00:00
• f7712cc0bf Clarification to help [HER-1885] source-report.txt empty * SourceTagsReport informative note if there's none of the info needed for this report gojomo 2011-04-29 22:21:54 +00:00
• 26f34c1b8c HER-1547 some sites may require an 'Accept' header * FetchHTTP.java "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" by default - this is what my firefox 4.0 for mac sends * profile-crawler-beans.cxml commented-out default value nlevitt 2011-04-25 20:43:17 +00:00
• d159b4e106 HER-741 Make extractors interrogate for charset * ExtractorXML.java if charset not spec'd in http header look for <?xml encoding=""?> * ExtractorHTML.java lookForEncodingInContent() - 1. look for <meta http-equiv="content-type"...> 2. if not found then look for <meta charset=""> 3. if not found then <?xml encoding=""...?> * Recorder.java setCharacterEncoding() - If new encoding is different from old encoding, close replayCharSequence and set to null, which will trigger recreation on next retrieval. nlevitt 2011-04-22 18:43:18 +00:00
• c9c74b5c28 * ExtractorXML.java shouldExtract() - use getContentReplayPrefixString() for <?xml check nlevitt 2011-04-22 00:42:09 +00:00
• a2eaa9d6ac HER-1820 followup * UriUtils.java NAIVE_LIKELY_URI_PATTERN - revert change and unpublicize NAIVE_URI_EXCEPTIONS - add some mimetype strings that came up in testing isLikelyFalsePositive() - unpublicize * ExtractorXML.java XML_URI_EXTRACTOR - do not use NAIVE_LIKELY_URI_PATTERN shouldExtract() - check for mimetype application/vnd.openxmlformats which is not xml processXml() - use UriUtils.isLikelyUri() nlevitt 2011-04-21 17:18:01 +00:00
• 43dae4d84d HER-1884 when html is processed by ExtractorXML, which can happen when it starts with <?xml..., a@href links are treated as embeds * ExtractorXML.java shouldExtract() - return true if content starts with "<?xml" only if it does not also contain "<!doctype html" or "<html" early in the content nlevitt 2011-04-20 20:57:41 +00:00
• c4595fdcc9 * profile-crawler-beans.cxml - <!-- <property name="independentExtractors" value="false" /> --> + <!-- <property name="extractIndependently" value="false" /> --> nlevitt 2011-04-20 20:54:21 +00:00
• 6d6c1059a5 Use Recorder.getContentReplayCharSequence() instead of deprecated getReplayCharSequence() nlevitt 2011-04-20 19:38:12 +00:00
• 630ddb4deb [HER-1883] seeds result info (for seeds report, crawl report tally) lost after checkpoint & resume * SeedRecord.java ensure makeDirty() is called after every updateWith() (but not the initial instantiation) gojomo 2011-04-20 02:38:26 +00:00
• 70d21ca2da HER-1820 more eager xml link extraction * ExtractorXML.java instead of considering only strings that start with http(s):, consider all strings that match UriUtils.NAIVE_LIKELY_URI_PATTERN (and, as before, constitute the entirety of the xml tag content or attribute value) * UriUtils.java - NAIVE_LIKELY_URI_PATTERN - add quotes to the excluded characters so it does not eat the closing quote when matching xml attribute values, and make visibility public for use in ExtractorXML - isLikelyUri() - refactor false positive check into new method isLikelyFalsePositive() so that it can be used in ExtractorXML avoiding redundant check against NAIVE_LIKELY_URI_PATTERN nlevitt 2011-04-20 01:18:00 +00:00
• b064d43db5 Fix HER-1792 backslashes in urls can in some cases really mean backslashes (specifically, when they're in the query string) * UURIFactory.java fixup() - replace \ with / up to the first ? in the url, instead of anywhere in the url and update comments * UURIFactoryTest.java backslash test nlevitt 2011-04-19 02:11:20 +00:00
• 2be8af10a9 * ArchiveReaderFactory comment about possible weakness in checking response to Range:d request gojomo 2011-04-16 08:06:03 +00:00
• 8bc70dd82a [HER-1881] GZIPMembersInputStream.compressedSkip(long) ignores return value of in.skip(offset) * GZIPMembersInputStream use ByteStreams.skipFully to skip desired amount or hit EOF gojomo 2011-04-16 07:43:53 +00:00
• 991a67e0e4 Post 3.1.0-beta * **/pom.xml chnge version-identifier to "3.1.0-SNAPSHOT" gojomo 2011-04-15 22:28:52 +00:00
• 52d00a9917 Prep for 3.1.0-beta * **/pom.xml increment declared version gojomo 2011-04-15 18:09:43 +00:00
• 1a2ec76805 comment typo, missing serialization ID gojomo 2011-04-15 18:08:12 +00:00
• 2df2584573 Fix HER-1880 robots.txt less specific Allow overrides more specific Disallow * RobotsDirectives.java - use plain ConcurrentSkipListSet<String> instead of PrefixSet, to maintain complete list of prefixes, so we can know the longest match - allows(String) - return true if longest matching Allow prefix is longer than or equal to longest matching Disallow prefix * RobotstxtTest.java flip expected result of test of generic Allow against specific Disallow nlevitt 2011-04-14 02:25:43 +00:00
• 75b7b9066c Followup to r7127 - add jna to eclipse classpath * .classpath <classpathentry kind="var" path="M2_REPO/net/java/dev/jna/jna/3.2.3/jna-3.2.3.jar"/> nlevitt 2011-04-13 00:54:26 +00:00
• 722a8549ab [HER-1821] H3: implement hard-link-based BDB/state-dir checkpointing * BdbModule.java new setting, useHardLinkCheckpoints, default true doCheckpoint now creates hard-links to active log files from the in-environment checkpoint-dir doRecover now relinks from checkpoint-dir files, when available, in preference to whatever is already in the env-dir * pom.xml bring in JNA for native calls * CLibrary wrapper for the link() function gojomo 2011-04-12 00:09:51 +00:00
• 6388dc7605 [HER-1878] NullPointerException when parsing (some?) WARC files over slow connections * GZIPMembersInputStream override OpenJDK7GZIPInputStream.readTrailer to not rely on available() as indicator of end-of-stream fix membersIterator hasNext() to also not rely on available() as indicator of end-of-stream * ArchiveReader change innerHasNext to peek one byte (using mark/reset) rather than rely on available() as indicator of end-of-stream remove redundant getInputStream accessor in favor of field-name-matching getIn/setIn * ArchiveReaderFactory, (W)ARCReaderFactory remove no-longer-necessary RepositionableStream references use getIn() rather than getInputStream() gojomo 2011-04-11 22:58:05 +00:00
• bd45720bae improved comments about new naming conventions/default-template gojomo 2011-04-11 22:49:01 +00:00
• f2ad45045b * FetchWhois.java fix whois uri syntax - add missing "/" token nlevitt 2011-04-07 17:17:48 +00:00
• 020ef44551 * FetchWhois.java class-level javadoc including informal specification of whois uri nlevitt 2011-04-07 06:44:06 +00:00
• f4cf64af29 [HER-1053] compressed HTTP fetch: "Accept-encoding: gzip" [HER-728] Offer replay stream that has been un-chunked (whether because response was HTTP/1.1 or used chunked in HTTP/1.0 against spec) [HER-1876] Offer HTTP/1.1 option - for chunked transfer-encoding (but not persistent connections) * (all) update to use new (decoded-as-necessary) content streams/CharSequences gojomo 2011-04-06 20:54:16 +00:00
• 3a53da1073 adapted tests for other changes gojomo 2011-04-06 20:52:06 +00:00
• e29e7dd36a comment typo gojomo 2011-04-06 20:51:40 +00:00
• 9f3b8d11db * ReplayInputStream constructor comment about using destroy() gojomo 2011-04-06 20:51:14 +00:00
• c5eb3ffd53 [HER-1053] compressed HTTP fetch: "Accept-encoding: gzip" [HER-728] Offer replay stream that has been un-chunked (whether because response was HTTP/1.1 or used chunked in HTTP/1.0 against spec) [HER-1876] Offer HTTP/1.1 option - for chunked transfer-encoding (but not persistent connections) * FetchHTTP add 'acceptCompression' and 'useHTTP11' properties, both default false * Recorder track whether recorded-input is transfer-encoded (chunked) or content-encoded (gzip etc) offer alternate replay streams for (1) raw 'messageBody'; (2) entity (un-chunked if necessary) (3) content (decompressed if necessary) always content & GenericReplayCharSequence for CharSequence replays * GenericReplayCharSequence always use a stream (rather than random-access buffer) always decode to prefix buffer first, so short content never touches disk no matter the encoding * InMemoryReplayCharSequence deleted; 'Generic' now works similarly for small content and anyway random-access for single-byte-encodings is now rarely possible (given deconding streams) * RecordingInputStream, RecordingOutputStream adjust for changed stream names, functionality moved to Recorder * ReplayCharSequence use Charset instances rather than names * ReplayInputStream add convenience constructor (and tmp-file-destroy) for copying any other inputStream into a seekable ReplayInputStream gojomo 2011-04-06 20:42:43 +00:00
• 3f98fc248b * ArchiveUtils (readFully) return actual number of bytes read into array * FileUtils (readFullyToFile) simplify; also return total bytes read gojomo 2011-04-06 20:22:30 +00:00
• b79308b2e0 Part of HER-1875 - socket timeout for whois fetcher * FetchWhois.java nlevitt 2011-04-05 01:22:01 +00:00
• 81cda7be59 HER-1875 followup - override SocketFactory to support connect timeout for ftp data connection nlevitt 2011-04-05 01:21:09 +00:00
• 653f24e713 Avoid exception - "org.springframework.beans.NotWritablePropertyException: Invalid property 'timeoutSeconds' of bean class [org.archive.modules.fetcher.FetchFTP]: Bean property 'timeoutSeconds' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?" * FetchFTP.java change setTimeoutSeconds(Integer) to setTimeoutSeconds(int) nlevitt 2011-04-04 20:10:54 +00:00
• 8e7b9a5fdf Fix HER-1875 FetchFTP needs socket timeout * FetchFTP.java setSoTimeoutMs(), getSoTimeoutMs() (init) - set default value of 20000 ms, same as FetchHTTP fetch() - set various timeouts on FTPClient - effectively sets connect timeout and socket timeout on control connection, and socket timeout on data connection (notably, does not set connect timeout on data connection w/ current commons-net) nlevitt 2011-04-04 19:50:56 +00:00
• 1395854f9a [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling * OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java backport the working GZIP implementation from OpenJDK7, with minimal renames/imports/private-to-protected changes to enable member-at-a-time reading * GZIPMembersInputStream base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode * GZIPMembersInputStreamTest, ARCWriterTest test tweaks gojomo 2011-03-30 02:42:26 +00:00
• e6e415b9d9 [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling * OpenJDK7GZIPInputStream.java, OpenJDK7InflatedInputStream.java backport the working GZIP implementation from OpenJDK7, with minimal renames/imports/private-to-protected changes to enable member-at-a-time reading * GZIPMembersInputStream base on OpenJDK7 implementation to avoid pre- & post-6u23 codepaths, and 6u23/24 readHeader bug override readTrailer to set member-end; update comment to reflect end-uncertainty without EOF-per-member-mode * GZIPMembersInputStreamTest, ARCWriterTest test tweaks gojomo 2011-03-30 02:42:04 +00:00
• e60fd8b0e1 * XmlMarshaller.java marshalDocument() - SAXException can wrap other exceptions, e.g. EOFException when tcp connection is broken; thus, in case of IOException, rather than throwing a RuntimeException, re-throw the wrapped IOException (unfortunately in that case we still get a massive stack trace in the log, but at least it comes from the appropriate spot and is not too misleading) nlevitt 2011-03-02 01:34:19 +00:00
• 013b5e394d Different fix for HER-1872: additional property preloadSourceUrl. Use either that or preloadSource (a filesystem path), rather than try to handle both possibilities with one string. * PersistLoadProcessor.java * PersistProcessor.java nlevitt 2011-03-02 01:24:28 +00:00
• 17b12ca19b * HostsReport.java url-encode hosts (other than "dns:") in host report, in case of hostname containing whitespace or other unusual characters, which can happen nlevitt 2011-03-01 23:43:58 +00:00
• 3fc3d324e8 * ArchiveReaderFactory avoid misinterpreting a "c:" local path as a URI scheme gojomo 2011-02-26 01:04:26 +00:00
• d97d4aa8cb Fix HER-1872 PersistLoadProcessor preloadSource does not support url (regression from h1) * PersistLoadProcessor.java change preloadSource from ConfigFile to String, allowing the logic in PersistProcessor.copyPersistSourceToHistoryMap() to decide how to load it nlevitt 2011-02-26 00:59:35 +00:00
• 2f9a8b860d * GZIPMembersInputStream ensure mark()ing more frequently, when reset to before current position won't be needed * ArchiveReaderFactory avoid misinterpreting a "c:" local path as a URI scheme gojomo 2011-02-26 00:59:10 +00:00
• fe5363feb0 * GzipMembersInputStreamTest test with 6 small members gojomo 2011-02-24 22:13:46 +00:00
• 9fef4f1f26 * ArchiveReader better logging of caught/recovery-attempted IOException * GZIPMemberInputStream handling of previously-unrecognized case in pre-JDK6u23: attempt at read() exhausts inflater with no new bytes available (previous nonzero-length read had not indicated inflater finished) gojomo 2011-02-24 19:22:50 +00:00
• bb07080e02 [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling * ArchiveReaderFactoryTest test cases for generic reader-get methods -- one of which had obsolete STREAM_ALL=-1 problem gojomo 2011-02-23 01:09:05 +00:00
• c6a94c2264 [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling * ArchiveReaderFactory discard obsolete -1 flag value creating problems with usual uses gojomo 2011-02-23 00:24:47 +00:00
• f03b1ac689 * URIAuthorityBasedQueueAssignmentPolicy.java rename "whois" queue to "whois..." to avoid conflicting with queue for theoretical hostname "whois" nlevitt 2011-02-19 02:10:39 +00:00
• b9c408cb07 Change form of generic serverless whois url to e.g. whois:archive.org * FetchWhois.java * UURIFactory.java nlevitt 2011-02-19 02:00:18 +00:00
• a6f3b7db25 * ARCWriterTest test inspired by Erik Hetzner's HER-1865 test code gojomo 2011-02-19 00:35:09 +00:00
• 368ff65286 * FetchWhois.java add license preamble nlevitt 2011-02-17 22:52:37 +00:00
• 4991b295bc HER-1645 whois fetcher * URIAuthorityBasedQueueAssignmentPolicy.java whois urls all go in special "whois" queue * PreconditionEnforcer.java, CrawlURI.java move markPrerequisite() method from PreconditionEnforcer to CrawlURI * profile-crawler-beans.cxml commented-out whois fetcher clause * FetchStatusCodes.java new status codes for whois * ServerCache.java getHostFor() - return "whois:" for whois urls, following dns: convention * UURIFactory.java ugly hack for whois urls * SchemeNotInSetDecideRule.java add whois to list of supported schemes * WriterPoolProcessor.java shouldWrite() - we should write successful whois fetches * WARCWriterProcessor.java write whois records * FetchWhois.java the fetcher nlevitt 2011-02-17 22:33:28 +00:00
• 885a25c90c Robustify based on JUnit issues on Windows * ArchiveReader improve warning message * WriterPoolMember attempt pre-delete of target filename, warn on failure, for better odds on some filesystems * ARCWriterTest, WARCWriterTest ensure timely close of readers/writers in case filenames are reused restore originally-intended unique target names gojomo 2011-02-17 01:21:48 +00:00
• 9d588b9978 implement Closeable gojomo 2011-02-17 01:17:35 +00:00
• b86f0afc83 * TopNSet resolve strange type complaint in my Windows Eclipse JDK6u24 setup gojomo 2011-02-17 01:14:32 +00:00
• 61dc661f47 * TopNSet resolve strange type complaint in my Windows Eclipse JDK6u24 setup gojomo 2011-02-17 01:13:37 +00:00
• 548114dbde [HER-1864] cookies are being set and sent back on TLDs/public-suffixes (like '.com') (sometimes called 'supercookies') * CookieSpecBase use Guava InternetDomainName public-suffixes to both (1) prevent storing cookie on public-suffix; (2) prevent looking-up cookies for public-suffixes * BdbCookieStorage correct checkpoint/relaunch behavior: don't reuse persisted cookies on normal relaunches; do reuse on resume-from-checkpoint gojomo 2011-02-16 01:18:57 +00:00
• 5de965590c Update Guava library to r08 * .classpath, commons/pom.xml update reference * ObjectIdentityBdbManualCache use new maximumSize() option on dirtyItems map gojomo 2011-02-16 01:15:13 +00:00
• 9b78770027 [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling * GZIPMembersInputStream new workaround class that offers choice of pre/post JDK6u23 behavior and new accessors for finding member boundary offsets * GZIPMembersInputStreamTest test for above * ArchiveUtils, ARCWriter move static compress utilities to ArchiveUtils * ArchiveReader, (W)ARCReaderFactory, (W)ARCWriterTest adapt to use GZIPMembersInputStream * GzippedInputStream(Test) deleted gojomo 2011-02-16 01:12:25 +00:00
• 9e53f8be7c [HER-1867] make H3 ssl adhoc self-signed certificate valid for longer (appears to expire after 90 days) * Heritrix pass '-validity 3650' to KeyTool gojomo 2011-02-14 22:07:10 +00:00
• 9468cb0c1f Report total bytes, as well as breakdown, in xml version of job page, to match html. * CrawlJob.java sizeTotalsReportData() - add "total" (bytes) and "total-count" (urls) to map * CrawledBytesHistotable.java new method getTotalUrls() to complement getTotalBytes() nlevitt 2011-02-09 01:17:26 +00:00
• f58fdcdba4 * Heritrix use heritrix.home, if available, to find default logging.properties (to let those who launch Heritrix from elsewhere save themselves from HttpClient's copious logging) gojomo 2011-02-05 03:26:02 +00:00
• 244c1cf563 Update to latest restlet 1.1.10. It includes Daniel Truemper's fix for the restlet bug he discovered while using heritrix, which I ran into. http://tech.dir.groups.yahoo.com/group/archive-crawler/message/6338 http://restlet.tigris.org/issues/show_bug.cgi?id=1018 http://restlet.tigris.org/source/browse/restlet?view=rev&revision=6135 * .classpath * engine/pom.xml nlevitt 2011-02-05 00:36:11 +00:00
• 40f7debeaa [HER-1846] IOException due to disk full situation can cause frontier lock-up * WARCWriterProcessor move checkSize into block protected by catch/finally cleanup of errors * WARCWriterProcessorTest test to simulate record-write-failure and verify does not cause pool lockup * WARCWriterTest, WriterPoolMember comment and warnings cleanup gojomo 2011-01-28 03:02:17 +00:00
• a6fd015f6b * commons/pom.xml fix inconsistent line endings (remove ^M's) - a trivial change to poke continuum which is truncating this file nlevitt 2011-01-28 02:04:55 +00:00
• c1a99a8e35 [HER-1861] Add 'first-named' robots policy, which considers rules for more than one user-agent, in order * Robotstxt, RobotstxtTest treat wildcardRules specially, so that it is possible to ask for only explicitly-named matches * MostFavoredRobotsPolicy improve comment, change masquerade default to true * FirstNamedRobotsPolicy, FirstNamedRobotsPolicyTest policy to consider alternate user-agents in order, using the first that has any explicit (non-wildcard) rules declared (or failing that, the original default user-agent) gojomo 2011-01-28 00:48:25 +00:00
• d22ab1cb6d * PreconditionEnforcer prefer 'protected' over 'private' gojomo 2011-01-28 00:22:02 +00:00
• c2027b7a30 * SimpleSelfTest more useful assertEquals gojomo 2011-01-26 03:11:17 +00:00
• a3d4dc5ac2 [HER-1834] log-bloat, memory problems when hopsPath grows endlessly with no max-hops limit set * TooManyHopsDecideRule remove inadvertent reversal of sense of comparison, making default scope reject almost everything (and breaking most self-tests) gojomo 2011-01-26 03:08:32 +00:00
• 8212d1c0dd [HER-1860] H3: Remove dependence on GC/finalization/PhantomReference magic in used ObjectIdentityCache implementation * StatisticsTracker move small stat maps to in-memory ConcurrentMaps update processedSeedRecords for new ObjectIdentityCache shape move hostsDistribution/hostsBytes/hostsLastFinished tracking to CrawlHosts/hostsCache * CrawlSummaryReport gets hosts count from hostsCache * SeedRecord implement IdentityCacheable * FetchStats remember last-success times gojomo 2011-01-20 23:36:50 +00:00
• 6bdc7642b4 * CrawlController ensure final stat snapshot before stop gojomo 2011-01-20 23:29:59 +00:00
• 0b577c1245 [HER-1860] H3: Remove dependence on GC/finalization/PhantomReference magic in used ObjectIdentityCache implementation * IdentityCacheable new interface required of objects stored in ObjectIdentityCaches * IdentityCacheableWrapper wrapper for storing arbitrary objects in ObjectIdentityCaches * ObjectIdentityCache keys now always Strings values now always IdentityCacheables new dirtyKey() method to ensure a key is persisted * ObjectIdentityBdbCache, ObjectIdentityMemCache update for new ObjectIdentityCache shape; still using legacy GC magic * ObjectIdentityBdbManualCache alternate implementation relying on dirtying to ensure persistence use Guava library MapMaker for soft memMap; capped dirtyMap * BdbModule use ObjectIdentityBdbManualCache by default up expectedConcurrency default to 64 * BdbModuleTest, ObjectIdentityBdbCacheTest, ObjectIdentityBdbManualCacheTest update, add tests * Frontier FrontierGroup as IdentityCacheable * AbstractFrontier, WorkQueueFrontier, BdbFrontier touchup queue/group accessors make queue instances dirty whenever mutated * WorkQueue, CrawlHost, CrawlServer IdentityCacheable support; appropriate makeDirty()s * ServerCache, DefaultServerCache update for new ObjectIdentityCache shape gojomo 2011-01-20 23:28:38 +00:00
• d2f28f051c [HER-1834] log-bloat, memory problems when hopsPath grows endlessly with no max-hops limit set * CrawlURI add getHopCount() for total (including tallied overflow) hop count * TooManyHopsDecideRule base decision on getHopCount() gojomo 2011-01-19 07:37:43 +00:00