Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 60738faf14 get rid of archive-overlays-commons-httpclient Noah Levitt 2012-09-08 01:31:32 -07:00
• 9699d6a6ed get rid of AbstractFetchHTTP Noah Levitt 2012-09-08 01:17:32 -07:00
• 33df8faaf0 remove LegacyFetchHTTP and its classes Noah Levitt 2012-09-08 00:49:30 -07:00
• 71761a72a7 rename FetchHTTP2 to FetchHTTP Noah Levitt 2012-09-08 00:28:06 -07:00
• bc29599a3b oops forgot to add LegacyFetchHTTP.java ? Noah Levitt 2012-09-08 00:27:14 -07:00
• 917986454b rename FetchHTTP to LegacyFetchHTTP Noah Levitt 2012-09-08 00:25:14 -07:00
• d3acf337cc use ThreadLocal<HttpClient> since the connection manager doesn't like being shared among threads, and who knows what else; basically it seems to be designed for use by a single thread Noah Levitt 2012-09-08 00:11:43 -07:00
• ec7c525c9d use Integer instead of int because null is allowed Noah Levitt 2012-09-07 23:19:52 -07:00
• 24e813c858 test chunked transfer encoding Noah Levitt 2012-09-07 13:28:41 -07:00
• 09f0e53119 testHttp11 Noah Levitt 2012-09-07 11:32:38 -07:00
• a346389a3a twiddles Noah Levitt 2012-09-07 03:10:53 -07:00
• 9a0191a84c ssl trust level support for FetchHTTP2! another thing that doesn't even work properly in old FetchHTTP Noah Levitt 2012-09-07 02:47:54 -07:00
• 0764096eb0 skip slow tests for old FetchHTTP for now Noah Levitt 2012-09-07 02:42:13 -07:00
• f2f0ca59f6 embedded jetty https server Noah Levitt 2012-09-06 19:28:07 -07:00
• da76fe322a Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-09-06 15:51:36 -07:00
• 5b82d9abee Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2012-09-06 15:15:03 -07:00
• 5b35835d53 Unsupported auth scheme causing recoverableProblem with SEVERE level log line and stack trace * FetchHTTP.java getAuthScheme() - handle IllegalStateException thrown by AuthPolicy.getAuthScheme() Noah Levitt 2012-09-06 15:12:56 -07:00
• 948d641b14 socket/connection timeout Noah Levitt 2012-09-06 12:39:44 -07:00
• d11338d620 testFetchTimeout Noah Levitt 2012-09-05 19:24:50 -07:00
• fbd25b789a implement shouldFetchBodyRule Noah Levitt 2012-09-05 14:19:17 -07:00
• d703fce31a add missing license header Noah Levitt 2012-09-05 14:08:26 -07:00
• b19994e3bb Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-09-05 12:08:48 -07:00
• 1a07c6b741 Make unit tests pass * ContentExtractorTestBase.java assertNoSideEffects() - check curi.getAnnotations.isEmpty() instead of equality with Collections.EMPTY_LIST Noah Levitt 2012-09-05 11:19:18 -07:00
• e711ff9506 * JobRelatedResource.java fix code formatting from 01be6d4b Noah Levitt 2012-09-04 18:03:38 -07:00
• c433c78e90 Merge pull request #17 from adam-miller/HER-2018 Noah Levitt 2012-09-04 17:59:43 -07:00
• cd0c169a4d Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-09-04 17:54:03 -07:00
• b704f4eeb7 Fix HER-2021 usedBaseForVia annotation repeated many times * CrawlURI.java getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates * Link.java addRelativeToVia() - only log use of base for via the first time, before the annotation has been added Noah Levitt 2012-09-04 17:49:01 -07:00
• baaf4f8092 testShouldFetchBodyRule Noah Levitt 2012-09-04 13:25:51 -07:00
• ca606588ae test sendIfNoneMatch (etag) Noah Levitt 2012-09-02 15:25:56 -07:00
• c03273230b sendIfModifiedSince Noah Levitt 2012-09-02 13:00:01 -07:00
• f3963d5f89 convenience method CrawlURI.getFetchHistory() Noah Levitt 2012-09-02 12:25:19 -07:00
• 1fe4781cf5 axLengthBytes, sendRange Noah Levitt 2012-09-02 01:37:00 -07:00
• 2d8687e55c maxFetchKBSec Noah Levitt 2012-09-01 23:05:50 -07:00
• 5cb3b914e8 Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-09-01 22:11:59 -07:00
• d0882a18e6 * AbstractPersistProcessor.java shouldStore() - only save persist record if http, https or ftp Noah Levitt 2012-09-01 22:09:30 -07:00
• 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans * JobRelatedResource.java Added encoding to URL generation in makePresentableMapFor(). Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames() Adam Miller 2012-08-31 15:07:31 -07:00
• 61cbd54059 Merge pull request #16 from adam-miller/HER-2017 Noah Levitt 2012-08-30 16:04:04 -07:00
• 52e70a10ef Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans" * BeanBrowseResource.java replaced "script" string in root node with "beans" Adam Miller 2012-08-30 15:35:27 -07:00
• 3f2295353a test dependencies belong in engine/ now too Noah Levitt 2012-08-30 12:16:25 -07:00
• cdd95cbb1f move FetchHTTP*Test* to engine/ subproject because testFormAuth() makes use of PreconditionEnforcer, part of engine/, and we want to keep that test Noah Levitt 2012-08-30 09:55:20 -07:00
• f502292eef commons-codec was already pulled in from elsewhere Noah Levitt 2012-08-29 19:06:37 -07:00
• 731d382ae0 fetchhttp2 support for proxy auth, somewhat elaborately done, sharing principles and code with http "target" auth Noah Levitt 2012-08-29 17:41:50 -07:00
• 0e02aa3523 check for Proxy-Connection header, but don't worry about the value for now (does it matter?) Noah Levitt 2012-08-29 11:12:49 -07:00
• b112fdd9f7 http proxy support for FetchHTTP2; also, avoid credentials contamination across requests and across threads Noah Levitt 2012-08-29 10:57:28 -07:00
• c17f74a7b0 add check that our little proxy server really handled a request Noah Levitt 2012-08-29 10:06:20 -07:00
• a2d65044f0 progress on http proxy test Noah Levitt 2012-08-28 18:25:53 -07:00
• 6c1b762fe4 remove unnecessary finally blocks unsetting test settings, since each test gets a fresh fetcher Noah Levitt 2012-08-28 16:57:45 -07:00
• aa33887e64 Merge remote-tracking branch 'origin/master' into fetchhttp-hc Noah Levitt 2012-08-28 16:07:35 -07:00
• 0d65d7e446 Fix for HER-2019 fetchHTTP httpBindAddress setting doesn't take effect * FetchHTTP.java use keyed properties for httpBindAddress setting, as expected in configureBindAddress(CrawlURI, HostConfiguration) Noah Levitt 2012-08-28 16:03:17 -07:00
• 2be26573ba httpBindAddress Noah Levitt 2012-08-28 16:00:14 -07:00
• d820fe946e Merge remote-tracking branch 'travisfw/nlevitt-fetchhttp-hc' into fetchhttp-hc Noah Levitt 2012-08-27 18:51:37 -07:00
• 60954dbbc0 set accept compression back to false (shared) Travis Wellman 2012-08-27 18:44:55 -07:00
• bc71578c68 testing setAcceptCompression Travis Wellman 2012-08-27 18:41:49 -07:00
• 9f433cbde4 use eclipse junit thing Noah Levitt 2012-08-27 17:09:49 -07:00
• 6eb1e4cefd Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-08-24 14:28:33 -07:00
• a7a2b3cf7e Make speculative link extractor ignore urls with two consecutive slashes (not preceded by a colon as in http://). We've seen the double slashes lead to crawler traps. * UriUtils.java isLikelyFalsePositive() - consider likely false positive if it contains '//' (but not '://') * UriUtilsTest.java testDoubleSlashFalsePositives() Noah Levitt 2012-08-24 14:03:45 -07:00
• 9c00bd009e form auth Noah Levitt 2012-08-21 16:40:47 -07:00
• bc30eabe72 disable automatic redirect handling Noah Levitt 2012-08-21 15:48:06 -07:00
• 6941842fa8 make http auth work in FetchHTTP2 the new good way, like we did with FetchHTTP Noah Levitt 2012-08-21 15:23:31 -07:00
• f072dfc83c * Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java bump serialVersionUID because of changed method signature on populate() Noah Levitt 2012-08-16 12:28:19 -07:00
• 0144df024e Real fix for HER-2016 FetchHTTP tries every url twice without credentials before sending credentials: Remember WWW-Authenticate headers, aka auth challenges, with CrawlURI, and after successful auth, with the CrawlServer. It's necessary to do this to know what Authorization header to send without getting another 401 first. * CoreAttributeConstants.java new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges * CrawlURI.java add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges * Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java add httpAuthChallenges argument to populate() * HttpAuthenticationCredential.java populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header * CrawlServer.java cache httpAuthChallenges * FetchHTTP.java getAuthScheme() - cache auth challenges with CrawlURI configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate() promoteCredentials() - cache auth challenges with the CrawlServer Noah Levitt 2012-08-16 12:17:01 -07:00
• 201e941b69 tell CrawlURI to remember auth challenges across URI processings; couple other tweaks Noah Levitt 2012-08-16 12:00:17 -07:00
• 883ca25925 remember www-authenticate headers to make http auth work as intended, without lots of extra 401s Noah Levitt 2012-08-15 22:11:34 -07:00
• 5272b0b8c8 make form auth test work and test stuff for real Noah Levitt 2012-08-15 13:53:27 -07:00
• eb3419aa59 deprecate method (GET/POST) in HtmlFormCredential, since PreconditionEnforcer always sets fetch type of login urls to POST (who sends password in a GET request anyway?) Noah Levitt 2012-08-15 13:53:07 -07:00
• fe7eb22b2a oops... was swallowing problems so all junit tests passed! throw that Throwable Noah Levitt 2012-08-15 13:51:22 -07:00
• 5bc1fb0b6d import doesn't belong Noah Levitt 2012-08-14 16:12:22 -07:00
• c57d3ba97a Merge pull request #15 from kngenie/reuse-htmllinkcontext Noah Levitt 2012-08-07 15:41:16 -07:00
• 541c2ecfe4 Merge branch 'master' into reuse-htmllinkcontext Kenji Nagahashi 2012-07-31 12:25:16 -07:00
• 864fea938b working on form auth test Noah Levitt 2012-07-27 19:13:19 -07:00
• d71fbb3a1a stick a basic auth 401 in the middle of testDigestAuth to check it doesn't mess with the digest auth we're working on Noah Levitt 2012-07-26 18:45:29 -07:00
• 3efe8221cf commons-httpclient, don't try to handle 401s internally Noah Levitt 2012-07-26 18:44:30 -07:00
• 4e50c32578 Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-07-20 21:36:23 -07:00
• 3078eac4ab test case for scheme-less relative URL Kenji Nagahashi 2012-07-19 14:11:48 -07:00
• 8fc44b626c added a test case for catching HER-2013 Kenji Nagahashi 2012-06-20 18:27:47 -07:00
• 6f4053cab7 HER-2016 FetchHTTP tries every url twice without credentials before sending credentials * FetchHTTP.java configureHttp() - enable preemptive authentication parameter innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want) Noah Levitt 2012-07-19 13:15:27 -07:00
• 9073d12f96 Revert "HER-2015 credentials cached to server never used" Noah Levitt 2012-07-19 12:36:05 -07:00
• 93c3a706dd Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-07-16 20:05:05 -07:00
• 37a8504fda HER-2015 credentials cached to server never used * FetchHTTP.java populateCredentials() - return true when credentials populated from server cache Noah Levitt 2012-07-16 20:02:56 -07:00
• ea7e7ed304 check that successful basic/digest auth credentials are cached with server and volunteered on subsequent requests Noah Levitt 2012-07-16 19:33:11 -07:00
• cb6be2e9ca license header Noah Levitt 2012-07-16 18:07:11 -07:00
• fe739e8c2e remove unnecessary dependency on httpcomponents from CrawlURI Noah Levitt 2012-07-16 17:45:44 -07:00
• e9b6bb3fd6 Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-07-16 17:33:02 -07:00
• eee6c6cd30 * ArchiveRecord.java fix indentation only (no substantive changes) Noah Levitt 2012-07-16 09:57:24 -07:00
• c108c4c451 HER-2014 Make ArchiveRecord.getPosition() public instead of protected. * ArchiveRecord.java, HeaderedArchiveRecord.java getPosition() - change visibility to public Noah Levitt 2012-07-16 09:54:54 -07:00
• 82c983d2b3 Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-07-15 11:58:41 -07:00
• daac7fb96f more work on http authentication (is it done now?) digest auth test passes Noah Levitt 2012-07-15 11:56:30 -07:00
• c5d740a6d4 remove some dead or redundant code Travis Wellman 2012-07-11 16:57:13 -07:00
• 1eb5ba704e Merge commit '787f7d' Travis Wellman 2012-07-11 13:51:13 -07:00
• d0add8c186 Merge branch 'master' of github.com:internetarchive/heritrix3 Travis Wellman 2012-07-11 13:50:28 -07:00
• 787f7d9234 If the URI doesn't have a path, don't count segments. Travis Wellman 2012-07-11 13:40:01 -07:00
• 9505edc9c3 fix for null paths Travis Wellman 2012-07-10 18:01:53 -07:00
• c1a0525107 suppress unused warnings for serialVersionUid Travis Wellman 2012-07-10 14:12:33 -07:00
• 4ca6e9cc99 revert a change that may break bdb history Travis Wellman 2012-07-03 17:38:43 -07:00
• 73640e0cd8 Merge pull request #13 from travisfw/master Travis Wellman 2012-07-03 13:31:23 -07:00
• 4b047dba64 count path segments by looking at the path Travis Wellman 2012-07-03 13:13:50 -07:00
• 3f58893d1e disassociate Credential type from commons-httpclient library by moving populate() method into utility class Noah Levitt 2012-07-02 17:34:23 -07:00
• 0eaf1c1d3e missed commit Noah Levitt 2012-07-02 17:15:18 -07:00
• 2e5af085bc Merge branch 'fetchhttp-hc' of ssh://desktop-nlevitt.sf.archive.org/~nlevitt/workspace/heritrix3 into fetchhttp-hc Noah Levitt 2012-07-02 17:07:32 -07:00
• 22b327bdc1 progress on basic auth Noah Levitt 2012-07-02 15:38:52 -07:00