Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 1fe4781cf5 axLengthBytes, sendRange Noah Levitt 2012-09-02 01:37:00 -07:00
• 2d8687e55c maxFetchKBSec Noah Levitt 2012-09-01 23:05:50 -07:00
• 5cb3b914e8 Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-09-01 22:11:59 -07:00
• d0882a18e6 * AbstractPersistProcessor.java shouldStore() - only save persist record if http, https or ftp Noah Levitt 2012-09-01 22:09:30 -07:00
• 01be6d4b18 Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans * JobRelatedResource.java Added encoding to URL generation in makePresentableMapFor(). Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames() Adam Miller 2012-08-31 15:07:31 -07:00
• 61cbd54059 Merge pull request #16 from adam-miller/HER-2017 Noah Levitt 2012-08-30 16:04:04 -07:00
• 52e70a10ef Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans" * BeanBrowseResource.java replaced "script" string in root node with "beans" Adam Miller 2012-08-30 15:35:27 -07:00
• 3f2295353a test dependencies belong in engine/ now too Noah Levitt 2012-08-30 12:16:25 -07:00
• cdd95cbb1f move FetchHTTP*Test* to engine/ subproject because testFormAuth() makes use of PreconditionEnforcer, part of engine/, and we want to keep that test Noah Levitt 2012-08-30 09:55:20 -07:00
• f502292eef commons-codec was already pulled in from elsewhere Noah Levitt 2012-08-29 19:06:37 -07:00
• 731d382ae0 fetchhttp2 support for proxy auth, somewhat elaborately done, sharing principles and code with http "target" auth Noah Levitt 2012-08-29 17:41:50 -07:00
• 0e02aa3523 check for Proxy-Connection header, but don't worry about the value for now (does it matter?) Noah Levitt 2012-08-29 11:12:49 -07:00
• b112fdd9f7 http proxy support for FetchHTTP2; also, avoid credentials contamination across requests and across threads Noah Levitt 2012-08-29 10:57:28 -07:00
• c17f74a7b0 add check that our little proxy server really handled a request Noah Levitt 2012-08-29 10:06:20 -07:00
• a2d65044f0 progress on http proxy test Noah Levitt 2012-08-28 18:25:53 -07:00
• 6c1b762fe4 remove unnecessary finally blocks unsetting test settings, since each test gets a fresh fetcher Noah Levitt 2012-08-28 16:57:45 -07:00
• aa33887e64 Merge remote-tracking branch 'origin/master' into fetchhttp-hc Noah Levitt 2012-08-28 16:07:35 -07:00
• 0d65d7e446 Fix for HER-2019 fetchHTTP httpBindAddress setting doesn't take effect * FetchHTTP.java use keyed properties for httpBindAddress setting, as expected in configureBindAddress(CrawlURI, HostConfiguration) Noah Levitt 2012-08-28 16:03:17 -07:00
• 2be26573ba httpBindAddress Noah Levitt 2012-08-28 16:00:14 -07:00
• d820fe946e Merge remote-tracking branch 'travisfw/nlevitt-fetchhttp-hc' into fetchhttp-hc Noah Levitt 2012-08-27 18:51:37 -07:00
• 60954dbbc0 set accept compression back to false (shared) Travis Wellman 2012-08-27 18:44:55 -07:00
• bc71578c68 testing setAcceptCompression Travis Wellman 2012-08-27 18:41:49 -07:00
• 9f433cbde4 use eclipse junit thing Noah Levitt 2012-08-27 17:09:49 -07:00
• 6eb1e4cefd Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-08-24 14:28:33 -07:00
• a7a2b3cf7e Make speculative link extractor ignore urls with two consecutive slashes (not preceded by a colon as in http://). We've seen the double slashes lead to crawler traps. * UriUtils.java isLikelyFalsePositive() - consider likely false positive if it contains '//' (but not '://') * UriUtilsTest.java testDoubleSlashFalsePositives() Noah Levitt 2012-08-24 14:03:45 -07:00
• 9c00bd009e form auth Noah Levitt 2012-08-21 16:40:47 -07:00
• bc30eabe72 disable automatic redirect handling Noah Levitt 2012-08-21 15:48:06 -07:00
• 6941842fa8 make http auth work in FetchHTTP2 the new good way, like we did with FetchHTTP Noah Levitt 2012-08-21 15:23:31 -07:00
• f072dfc83c * Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java bump serialVersionUID because of changed method signature on populate() Noah Levitt 2012-08-16 12:28:19 -07:00
• 0144df024e Real fix for HER-2016 FetchHTTP tries every url twice without credentials before sending credentials: Remember WWW-Authenticate headers, aka auth challenges, with CrawlURI, and after successful auth, with the CrawlServer. It's necessary to do this to know what Authorization header to send without getting another 401 first. * CoreAttributeConstants.java new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges * CrawlURI.java add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges * Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java add httpAuthChallenges argument to populate() * HttpAuthenticationCredential.java populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header * CrawlServer.java cache httpAuthChallenges * FetchHTTP.java getAuthScheme() - cache auth challenges with CrawlURI configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate() promoteCredentials() - cache auth challenges with the CrawlServer Noah Levitt 2012-08-16 12:17:01 -07:00
• 201e941b69 tell CrawlURI to remember auth challenges across URI processings; couple other tweaks Noah Levitt 2012-08-16 12:00:17 -07:00
• 883ca25925 remember www-authenticate headers to make http auth work as intended, without lots of extra 401s Noah Levitt 2012-08-15 22:11:34 -07:00
• 5272b0b8c8 make form auth test work and test stuff for real Noah Levitt 2012-08-15 13:53:27 -07:00
• eb3419aa59 deprecate method (GET/POST) in HtmlFormCredential, since PreconditionEnforcer always sets fetch type of login urls to POST (who sends password in a GET request anyway?) Noah Levitt 2012-08-15 13:53:07 -07:00
• fe7eb22b2a oops... was swallowing problems so all junit tests passed! throw that Throwable Noah Levitt 2012-08-15 13:51:22 -07:00
• 5bc1fb0b6d import doesn't belong Noah Levitt 2012-08-14 16:12:22 -07:00
• c57d3ba97a Merge pull request #15 from kngenie/reuse-htmllinkcontext Noah Levitt 2012-08-07 15:41:16 -07:00
• 541c2ecfe4 Merge branch 'master' into reuse-htmllinkcontext Kenji Nagahashi 2012-07-31 12:25:16 -07:00
• 864fea938b working on form auth test Noah Levitt 2012-07-27 19:13:19 -07:00
• d71fbb3a1a stick a basic auth 401 in the middle of testDigestAuth to check it doesn't mess with the digest auth we're working on Noah Levitt 2012-07-26 18:45:29 -07:00
• 3efe8221cf commons-httpclient, don't try to handle 401s internally Noah Levitt 2012-07-26 18:44:30 -07:00
• 4e50c32578 Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-07-20 21:36:23 -07:00
• 3078eac4ab test case for scheme-less relative URL Kenji Nagahashi 2012-07-19 14:11:48 -07:00
• 8fc44b626c added a test case for catching HER-2013 Kenji Nagahashi 2012-06-20 18:27:47 -07:00
• 6f4053cab7 HER-2016 FetchHTTP tries every url twice without credentials before sending credentials * FetchHTTP.java configureHttp() - enable preemptive authentication parameter innerProcess() - do not call method.setDoAuthentication() (it's for automatic 401 handling which is not what we want) Noah Levitt 2012-07-19 13:15:27 -07:00
• 9073d12f96 Revert "HER-2015 credentials cached to server never used" Noah Levitt 2012-07-19 12:36:05 -07:00
• 93c3a706dd Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-07-16 20:05:05 -07:00
• 37a8504fda HER-2015 credentials cached to server never used * FetchHTTP.java populateCredentials() - return true when credentials populated from server cache Noah Levitt 2012-07-16 20:02:56 -07:00
• ea7e7ed304 check that successful basic/digest auth credentials are cached with server and volunteered on subsequent requests Noah Levitt 2012-07-16 19:33:11 -07:00
• cb6be2e9ca license header Noah Levitt 2012-07-16 18:07:11 -07:00
• fe739e8c2e remove unnecessary dependency on httpcomponents from CrawlURI Noah Levitt 2012-07-16 17:45:44 -07:00
• e9b6bb3fd6 Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-07-16 17:33:02 -07:00
• eee6c6cd30 * ArchiveRecord.java fix indentation only (no substantive changes) Noah Levitt 2012-07-16 09:57:24 -07:00
• c108c4c451 HER-2014 Make ArchiveRecord.getPosition() public instead of protected. * ArchiveRecord.java, HeaderedArchiveRecord.java getPosition() - change visibility to public Noah Levitt 2012-07-16 09:54:54 -07:00
• 82c983d2b3 Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-07-15 11:58:41 -07:00
• daac7fb96f more work on http authentication (is it done now?) digest auth test passes Noah Levitt 2012-07-15 11:56:30 -07:00
• c5d740a6d4 remove some dead or redundant code Travis Wellman 2012-07-11 16:57:13 -07:00
• 1eb5ba704e Merge commit '787f7d' Travis Wellman 2012-07-11 13:51:13 -07:00
• d0add8c186 Merge branch 'master' of github.com:internetarchive/heritrix3 Travis Wellman 2012-07-11 13:50:28 -07:00
• 787f7d9234 If the URI doesn't have a path, don't count segments. Travis Wellman 2012-07-11 13:40:01 -07:00
• 9505edc9c3 fix for null paths Travis Wellman 2012-07-10 18:01:53 -07:00
• c1a0525107 suppress unused warnings for serialVersionUid Travis Wellman 2012-07-10 14:12:33 -07:00
• 4ca6e9cc99 revert a change that may break bdb history Travis Wellman 2012-07-03 17:38:43 -07:00
• 73640e0cd8 Merge pull request #13 from travisfw/master Travis Wellman 2012-07-03 13:31:23 -07:00
• 4b047dba64 count path segments by looking at the path Travis Wellman 2012-07-03 13:13:50 -07:00
• 3f58893d1e disassociate Credential type from commons-httpclient library by moving populate() method into utility class Noah Levitt 2012-07-02 17:34:23 -07:00
• 0eaf1c1d3e missed commit Noah Levitt 2012-07-02 17:15:18 -07:00
• 2e5af085bc Merge branch 'fetchhttp-hc' of ssh://desktop-nlevitt.sf.archive.org/~nlevitt/workspace/heritrix3 into fetchhttp-hc Noah Levitt 2012-07-02 17:07:32 -07:00
• 22b327bdc1 progress on basic auth Noah Levitt 2012-07-02 15:38:52 -07:00
• 1f700e5cac poking around at basic auth, nothing close to working yet Noah Levitt 2012-07-02 10:31:32 -07:00
• 29abbecdf3 make data key "httpResponseHeaders" a constant Noah Levitt 2012-07-02 10:30:27 -07:00
• 8abe871af6 configure logging Noah Levitt 2012-07-02 10:29:19 -07:00
• 4790be501c fix up saving of http response headers in CrawlURI Noah Levitt 2012-06-29 22:00:35 -07:00
• 5140a3f5bb basic-auth test works! Noah Levitt 2012-06-29 18:34:17 -07:00
• 373abbbe81 latest jetty 6.x; starting on basic-auth test Noah Levitt 2012-06-29 17:19:46 -07:00
• 09d1839e6c test sending cookies and ignoring cookies Noah Levitt 2012-06-29 15:42:31 -07:00
• 69d7b0a0cd make jetty server not require anything external; also make it work with maven Noah Levitt 2012-06-29 13:43:54 -07:00
• 72303cc103 make jetty server not require anything external; also make it work with maven Noah Levitt 2012-06-29 12:33:20 -07:00
• 93958abfcd these help classes for fetchhttp2 need to be in modules/ since they use stuff from modules/ Noah Levitt 2012-06-29 11:19:56 -07:00
• c7ba0675cf more license headers Noah Levitt 2012-06-29 07:40:53 -07:00
• 9ef01de7b4 twiddles Noah Levitt 2012-06-29 07:36:36 -07:00
• b6cc860438 refactor to get rid of redundancy Noah Levitt 2012-06-29 07:24:09 -07:00
• 98a4be5eb1 fix user-agent test, add more tests Noah Levitt 2012-06-29 07:05:44 -07:00
• 1a79348cf7 test a few more things Noah Levitt 2012-06-29 02:55:26 -07:00
• 1a1d1b008c allow FetchHTTP2 infrastructure to fall back on default java hostname resolution Noah Levitt 2012-06-29 02:40:34 -07:00
• 1f315a7cc7 unit tests for FetchHTTP and FetchHTTP2 Noah Levitt 2012-06-29 02:04:39 -07:00
• 2653583cdd unit tests for FetchHTTP and FetchHTTP2 Noah Levitt 2012-06-29 01:59:58 -07:00
• 58d827c85b Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-06-28 18:08:39 -07:00
• 6225f6fa0b Fix HER-2013 scheme-less relative URI not resolved correctly * LaxURI.java parseUriReference() - fix logic testing if uri string is a path to respect scheme-less uris even if they contain colons * UURIFactoryTest.java add test for scheme-less relative uri (with colons) Noah Levitt 2012-06-28 16:49:50 -07:00
• 63c6ca59c2 Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-06-25 16:42:34 -07:00
• c40c24e97d Merge pull request #12 from travisfw/generics-fixes Travis Wellman 2012-06-25 11:37:18 -07:00
• 6626b44286 generics warnings fixes Travis Wellman 2012-06-21 18:23:06 -07:00
• 2b0141fd0d bdb cookie store for FetchHTTP2 Noah Levitt 2012-06-20 18:49:57 -07:00
• 49c00c0387 do not rely on ToeThread for the ServerCache, pass it to where it's needed instead Noah Levitt 2012-06-19 13:54:05 -07:00
• 56f0c92690 resolve hostnames from heritrix server cache (the bad old way) Noah Levitt 2012-06-19 13:32:34 -07:00
• 1d0beedf2b use http 1.0 by default, other request configuration stuff Noah Levitt 2012-06-19 12:09:46 -07:00
• 233b72cb34 get charset, encodings from headers Noah Levitt 2012-06-19 11:28:01 -07:00
• de152eaa9e browser-compatibility mode for cookie matching; that's what old FetchHTTP does, and also avoids sending weird Cookie2:$Version1 Noah Levitt 2012-06-18 17:49:09 -07:00
• 3c7bd8afd6 no keep-alive Noah Levitt 2012-06-18 17:06:46 -07:00
• a779e46ac7 set user-agent Noah Levitt 2012-06-18 15:55:07 -07:00