Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 85118c59c4 Tighten up javascript extraction based on real-world analysis (addresses HER-1523) * UriUtils.java new method isVeryLikelyUri() with tighter heuristic than isLikelyUri() * ExtractorJS.java use UriUtils.isVeryLikelyUri(), and change order of operations to do fixup before call to isVeryLikelyUri(), since it doesn't expect strings with javascript escaping and stuff * StringExtractorTestBase.java handle test data with expected value null, meaning no outlinks expected * ExtractorHTMLTest.java avoid redundancy by using the extractor created in ContentExtractorTestBase.setUp() * ExtractorJSTest.java some new tests Noah Levitt 2012-12-11 16:27:26 -08:00
• b1cd3d30a0 * ExtractorHTMLTest.java makeExtractor() - call setExtractorJS(new ExtractorJS()) so testSpeculativeLinkExtraction() passes Noah Levitt 2012-12-10 17:31:55 -08:00
• 256b16249a * ExtractorJS.java refactor so considerStrings() is not static, allowing it to be overridden in subclasses * ExtractorHTML.java, ExtractorJS.java add @Autowired parameter extractorJS used to process inline javascript, instead of call to static ExtractorJS.considerStrings() Noah Levitt 2012-12-10 16:51:42 -08:00
• 6531733972 H3 module for deduplicating with WBM index (with test case). Added EasyMock to project dependency. Kenji Nagahashi 2012-12-10 14:37:18 -08:00
• b2a90ddc15 * UriUtils.java speculativeFixup() - improve detection of scheme-less intended-absolute-URIs Noah Levitt 2012-12-10 14:10:34 -08:00
• d3dc153c5f * ModuleTestBase.java runTest() - convenience logging of test failures Noah Levitt 2012-12-10 13:56:45 -08:00
• e44cf9ca9b A little cleanup of FetchHTTPTest refactoring Noah Levitt 2012-12-04 18:43:15 -08:00
• 5354960f00 Refactor FetchHTTPTest to use a TestSuite so that starting and shutting down the test http servers happen only once at start and finish respectively Noah Levitt 2012-12-04 17:25:09 -08:00
• 2c0dbed399 .classpath include junit as a regular dependency not managed by eclipse, so source jar can be attached Noah Levitt 2012-12-04 17:20:39 -08:00
• e30cb83540 * FetchHTTPTest.java testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix lets pass through to mimic browser behavior. Noah Levitt 2012-12-04 16:25:09 -08:00
• b981fd7423 * ArchiveReader.java add "implements Closeable" since it already has the close() method Noah Levitt 2012-12-03 17:09:22 -08:00
• 4496c5d755 * FetchHTTPTest.java move to modules/src/test testFormAuth() - remove, there is FormAuthSelfTest.java, and this was the test that would fail in modules/ testHttpBindAddress() - incorporate ed65493db0 - alternate testing of multiple local binds more likely to work more places Noah Levitt 2012-11-30 17:57:36 -08:00
• dc7fea3252 * .classpath add missing libs Noah Levitt 2012-11-30 17:56:53 -08:00
• a20824aea7 Merge branch 'master' into new-fetchhttp-only Noah Levitt 2012-11-30 17:04:28 -08:00
• 92b302f164 * ExtractorMultipleRegexTest.java avoid "constant string too long" compile error Noah Levitt 2012-11-16 19:08:18 -08:00
• 94df568b4f * ExtractorMultipleRegex.java remove temporary performance testing code Noah Levitt 2012-11-16 18:34:28 -08:00
• aa98ec3327 * ExtractorMultipleRegex.java keep cache of groovy Template objects, since they are expensive to create Noah Levitt 2012-11-16 18:30:41 -08:00
• eeee01a52d * ExtractorMultipleRegex.java javadocs Noah Levitt 2012-11-16 16:42:43 -08:00
• 89dfef468c * ExtractorMultipleRegexTest.java test passes now; extractor gets what it can, which is most of the scroll down urls Noah Levitt 2012-11-15 19:18:59 -08:00
• dfd7a8a7d3 * ExtractorMultipleRegexTest.java turns out that __adt parameter can be found near the json blob - most, but not all, expected links are now found Noah Levitt 2012-11-15 18:58:11 -08:00
• 22ea489c57 * ExtractorMultipleRegex.java remove the "fooIndex" thing from available bindings, since it's kinda hacky and turned out not to be needed for our use case Noah Levitt 2012-11-15 18:56:54 -08:00
• 4af58f38e5 * CrawlURI.java outLinks, outCandidates - use LinkedHashSet to ensure predictable order (any reason not to do that?) Noah Levitt 2012-11-15 18:55:37 -08:00
• 75280a13a8 * ExtractorMultipleRegexTest.java working on making test work - first two scroll-down urls are extracted successfully, others fail Noah Levitt 2012-11-15 17:27:41 -08:00
• 13e1f9d64d * ExtractorMultipleRegex.java fix omission from last commit, uriRegex Noah Levitt 2012-11-15 14:01:01 -08:00
• 6ec4b5a973 * ExtractorMultipleRegex.java more refactoring to avoid redundant operations Noah Levitt 2012-11-15 13:55:11 -08:00
• 43dbcbfe81 * ExtractorMultipleRegex.java refactor for readability Noah Levitt 2012-11-15 13:35:16 -08:00
• 3f44a4bc66 Merge branch 'master' into MoreImpliedExtractor Noah Levitt 2012-11-15 12:35:17 -08:00
• 8395e372e4 * ScriptedProcessor.java getEngine() - fix logic to respect isolateThreads setting Noah Levitt 2012-11-15 12:33:48 -08:00
• ba5bca807e * ExtractorMultipleRegex.java use groovy templating facility Noah Levitt 2012-11-15 12:32:42 -08:00
• 6e2d2129fe * ExtractorMultipleRegexTest.java use real use case actual content, actual urls from facebook for testing * ExtractorMultipleRegex.java some comments, little twiddles Noah Levitt 2012-11-15 11:16:57 -08:00
• 1e2d86dc63 * ContentExtractorTestBase.java createRecorder(String,String) - new method for creating test Recorder and specifying charset createRecorder(String) - deprecate (use default charset as before) Noah Levitt 2012-11-15 11:14:25 -08:00
• 1d994493e2 more progress on ExtractorMultipleRegex Noah Levitt 2012-11-14 19:58:43 -08:00
• 1998337645 tmp commit Travis Wellman 2012-11-14 17:58:44 -08:00
• ed65493db0 alternate testing of multiple local binds more likely to work more places gojomo 2012-10-01 16:38:17 -07:00
• b4baea0a13 make maven requested version match eclipse project classpath gojomo 2012-10-01 16:37:31 -07:00
• 37e6ab802a Merge branch 'master' of github.com:internetarchive/heritrix3 gojomo 2012-10-01 13:41:12 -07:00
• 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics * CrawledBytesHistotable.java accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates * ContentDigestHistoryTest.java testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation Noah Levitt 2012-09-27 16:56:14 -07:00
• a1d135edbc * WARCWriterProcessor.java include warcFileOffset in CrawlURI extraInfo (optionally logged in crawl log if logExtraInfo is enabled) Noah Levitt 2012-09-27 16:51:50 -07:00
• e2a1e3bb53 * ContentDigestHistoryTest.java testWarcDedupe() - open the written with with WARCReader, check that each record is as expected Noah Levitt 2012-09-27 14:34:20 -07:00
• a1a5fc7379 * WARCWriterProcessor.java writeRevisitUriAgnosticDigest() - put angle brackets around WARC-Refers-To uri value, since warc record ids appear everywhere with those brackets Noah Levitt 2012-09-27 14:20:00 -07:00
• 7a92e1ff7c * FetchHTTPTest.java shut down test servers in tearDown() (server startup/shutdown happens on every test, inefficient but it doesn't take all that long anyway) * ContentDigestHistoryTest.java shut down test servers in finally block Noah Levitt 2012-09-26 19:29:11 -07:00
• e6fbe45774 Another test for ContentDigestHistory * ContentDigestHistoryTest.java new test testWarcDedupe() fetches two different urls with same content, writes warc records, checks results * FetchHTTPTest.java, WARCWriterProcessorTest.java make test module code public and static for use in ContentDigestHistoryTest.java Noah Levitt 2012-09-26 18:54:48 -07:00
• f9703fb5ef * engine/pom.xml remove incorrect, unnecessary dependency Noah Levitt 2012-09-26 14:55:03 -07:00
• b9cbbbd248 Bunch of unit tests for FetchHTTP * FetchHTTPTest.java a bunch of new tests * FetchHTTP.java make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides) * modules/pom.xml, .classpath dependencies for testing * engine/pom.xml update jetty to 6.1.26 since we require that for the new tests Noah Levitt 2012-09-26 14:41:49 -07:00
• f0b4cfeeac Merge branch 'master' into new-fetchhttp-only Noah Levitt 2012-09-26 12:59:41 -07:00
• ca5802f7ba separated hq from local hertirix3 repo, fixed up layout Kenji Nagahashi 2012-09-26 12:31:55 -07:00
• 3612396ff3 Unit test for content digest history * ContentDigestHistoryTest.java test basics * BdbContentDigestHistory.java code formatting tweak Noah Levitt 2012-09-26 12:03:45 -07:00
• 190e2ff0f3 Fix unit test * WARCWriterTest.java writeRandomHTTPRecord() - fix method, had transposed some lines of code Noah Levitt 2012-09-26 10:58:01 -07:00
• 2e1d81fa80 Fix build * WARCConstants.java, WARCWriter.java remove imports of WARCWriterProcessor from the modules/ area (was only used in javadoc) Noah Levitt 2012-09-26 08:58:22 -07:00
• 50c8a09b63 * BdbContentDigestHistory.java class javadoc Noah Levitt 2012-09-25 16:04:44 -07:00
• 819371f6eb * BdbContentDigestHistory.java store() - avoid clobbering good content digest history with empty one Noah Levitt 2012-09-25 15:28:19 -07:00
• 34252c4949 * WARCWriterProcessor.java writeHttpRecords() - avoid inadvertently creating content digest history map Noah Levitt 2012-09-25 15:26:56 -07:00
• fe1df5883f Some tweaks on maintaining content digest history * CrawlURI.java utility method hasContentDigestHistory() * BdbContentDigestHistory.java load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there * WARCWriterProcessor.java updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record Noah Levitt 2012-09-25 14:35:26 -07:00
• 25a9155298 New parent class AbstractContentDigestHistory for implementations to extend * AbstractContentDigestHistory.java abstract methods load(), store(); non-abstract persistKeyFor(); javadocs * BdbContentDigestHistory.java extend AbstractContentDigestHistory * ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java use AbstractContentDigestHistory Noah Levitt 2012-09-25 13:38:25 -07:00
• b8e14ba1df Rename ContentDigestHistory -> BdbContentDigestHistory, since it's bdb-specific Noah Levitt 2012-09-25 13:12:59 -07:00
• 7965a27b1d Fix infinite loop bug on server returning no response * RecordingSocketInputBuffer.java readLine(CharArrayBuffer) - return -1 (not 0) on end of stream * RecordingHttpClient.java setHttpRequestRetryHandler() - never retry (may want to reevaluate later, old fetcher did retry sometimes) * FetchHTTPTest.java testNoResponse() Noah Levitt 2012-09-13 19:10:11 -07:00
• dd7e0c5bd2 * profile-crawler-beans.cxml replace defunct commons-httpclient BdbCookieStorage with new httpcomponents BdbCookieStore Noah Levitt 2012-09-13 17:58:47 -07:00
• 77559bf81c **/pom.xml bump version number because of api change Noah Levitt 2012-09-13 17:47:26 -07:00
• ef78e584c4 * WARCWriterProcessor.java writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob * WARCConstants.java new header field names Noah Levitt 2012-09-13 17:21:29 -07:00
• a8aae91167 * WARCConstants.java remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field * WARCWriterProcessor.java writeRevisitUriAgnosticDigest() - add some comments Noah Levitt 2012-09-11 18:21:20 -07:00
• 1048536a25 * WARCWriterProcessor.java writeRevisitUriAgnosticDigest() - fix construction of X-WARC-Reference-Location json value Noah Levitt 2012-09-11 17:21:34 -07:00
• 38aefe4d16 Write revisit records for URI-agnostic content digest duplicates. * WARCConstants.java new header keys "WARC-Refers-To" and "X-WARC-Reference-Location" * CrawlURI.java getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value * WARCWriterProcessor.java writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled Noah Levitt 2012-09-11 17:12:24 -07:00
• a04e76f00f Load, store content digest history * WARCRecordInfo.java new fields warcFilename, warcFileOffset * WARCWriter.java temporarily remember warc records written * CrawlURI.java convenience method getContentDigestHistory() * ContentDigestHistory.java start up properly; load into and store from curi.getContentDigestHistory() * RecrawlAttributeConstants.java bunch of hash keys for curi.getContentDigestHistory() * WARCWriterProcessor.java save info in curi.getContentDigestHistory() Noah Levitt 2012-09-11 10:10:49 -07:00
• 2407fc5dd0 Beginning work on deduplication by content digest (irrespective of url) * ContentDigestHistory.java, ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java initial checkin Noah Levitt 2012-09-10 18:18:12 -07:00
• c4faa444a2 * ANVLRecord.java remove unneeded(?) serialVersionUID Noah Levitt 2012-09-10 18:14:15 -07:00
• a414f522f6 Continuing to work on cleaning up WARC writer code: simply creation of WARC record headers. * ANVLRecord.java extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10) * ANVLRecords.java remove (seemed to be an incomplete duplicate of ANVLRecord) * WARCRecordInfo.java addExtraHeader() - convenience method * Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int) Noah Levitt 2012-09-10 18:03:25 -07:00
• 360ac00000 Continuing to work on cleaning up WARC writer code. * WARCRecordInfo.java get rid of constructor with arguments, add setUrl() and setType() * WARCWriter.java get rid of writeRecord() taking a million arguments * Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method Noah Levitt 2012-09-10 17:41:43 -07:00
• dd5e51fd9d Working on cleaning up WARC writer code. * WARCRecordInfo.java new class to hold warc record info to write, so we don't have to pass around all these long lists of variables * WARCConstants.java new enum WARCRecordType replaces bunch of string constants * WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java use WARCRecordInfo, WARCRecordType Noah Levitt 2012-09-10 15:43:37 -07:00
• 395220f5bf get rid of archive-overlays-commons-httpclient Noah Levitt 2012-09-08 01:57:14 -07:00
• 60738faf14 get rid of archive-overlays-commons-httpclient Noah Levitt 2012-09-08 01:31:32 -07:00
• 9699d6a6ed get rid of AbstractFetchHTTP Noah Levitt 2012-09-08 01:17:32 -07:00
• 33df8faaf0 remove LegacyFetchHTTP and its classes Noah Levitt 2012-09-08 00:49:30 -07:00
• 71761a72a7 rename FetchHTTP2 to FetchHTTP Noah Levitt 2012-09-08 00:28:06 -07:00
• bc29599a3b oops forgot to add LegacyFetchHTTP.java ? Noah Levitt 2012-09-08 00:27:14 -07:00
• 917986454b rename FetchHTTP to LegacyFetchHTTP Noah Levitt 2012-09-08 00:25:14 -07:00
• d3acf337cc use ThreadLocal<HttpClient> since the connection manager doesn't like being shared among threads, and who knows what else; basically it seems to be designed for use by a single thread Noah Levitt 2012-09-08 00:11:43 -07:00
• ec7c525c9d use Integer instead of int because null is allowed Noah Levitt 2012-09-07 23:19:52 -07:00
• 24e813c858 test chunked transfer encoding Noah Levitt 2012-09-07 13:28:41 -07:00
• 09f0e53119 testHttp11 Noah Levitt 2012-09-07 11:32:38 -07:00
• a346389a3a twiddles Noah Levitt 2012-09-07 03:10:53 -07:00
• 9a0191a84c ssl trust level support for FetchHTTP2! another thing that doesn't even work properly in old FetchHTTP Noah Levitt 2012-09-07 02:47:54 -07:00
• 0764096eb0 skip slow tests for old FetchHTTP for now Noah Levitt 2012-09-07 02:42:13 -07:00
• f2f0ca59f6 embedded jetty https server Noah Levitt 2012-09-06 19:28:07 -07:00
• da76fe322a Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-09-06 15:51:36 -07:00
• 5b82d9abee Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2012-09-06 15:15:03 -07:00
• 5b35835d53 Unsupported auth scheme causing recoverableProblem with SEVERE level log line and stack trace * FetchHTTP.java getAuthScheme() - handle IllegalStateException thrown by AuthPolicy.getAuthScheme() Noah Levitt 2012-09-06 15:12:56 -07:00
• 948d641b14 socket/connection timeout Noah Levitt 2012-09-06 12:39:44 -07:00
• d11338d620 testFetchTimeout Noah Levitt 2012-09-05 19:24:50 -07:00
• fbd25b789a implement shouldFetchBodyRule Noah Levitt 2012-09-05 14:19:17 -07:00
• d703fce31a add missing license header Noah Levitt 2012-09-05 14:08:26 -07:00
• b19994e3bb Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-09-05 12:08:48 -07:00
• 1a07c6b741 Make unit tests pass * ContentExtractorTestBase.java assertNoSideEffects() - check curi.getAnnotations.isEmpty() instead of equality with Collections.EMPTY_LIST Noah Levitt 2012-09-05 11:19:18 -07:00
• e711ff9506 * JobRelatedResource.java fix code formatting from 01be6d4b Noah Levitt 2012-09-04 18:03:38 -07:00
• c433c78e90 Merge pull request #17 from adam-miller/HER-2018 Noah Levitt 2012-09-04 17:59:43 -07:00
• cd0c169a4d Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-09-04 17:54:03 -07:00
• b704f4eeb7 Fix HER-2021 usedBaseForVia annotation repeated many times * CrawlURI.java getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates * Link.java addRelativeToVia() - only log use of base for via the first time, before the annotation has been added Noah Levitt 2012-09-04 17:49:01 -07:00
• baaf4f8092 testShouldFetchBodyRule Noah Levitt 2012-09-04 13:25:51 -07:00
• ca606588ae test sendIfNoneMatch (etag) Noah Levitt 2012-09-02 15:25:56 -07:00
• c03273230b sendIfModifiedSince Noah Levitt 2012-09-02 13:00:01 -07:00
• f3963d5f89 convenience method CrawlURI.getFetchHistory() Noah Levitt 2012-09-02 12:25:19 -07:00