Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 124907341f Merge pull request #152 from ukwa/switch-to-java-7 Noah Levitt 2016-03-08 16:47:58 -08:00
• 8ff09dc769 Switch to Java 1.7 Andrew Jackson 2016-03-08 23:53:59 +00:00
• e1476401ac Merged with upstream changes. Andrew Jackson 2016-03-08 23:50:07 +00:00
• 7f752d8d45 Merge pull request #150 from nlevitt/ari-4638 Adam Miller 2016-03-08 09:26:58 -08:00
• 0dc95ab7f7 Merge pull request #144 from nlevitt/ari-4528 Hunter 2016-02-29 14:24:43 -08:00
• ae413b1d9e Merge pull request #149 from eleclerc/patch-1 Kristinn Sigurðsson 2016-02-19 09:49:34 +00:00
• 9bdafc3c33 remove stray line Noah Levitt 2016-02-18 18:19:44 -08:00
• bc9a6cc640 new parameter extraInfo, "Arbitrary additional information to include in the json payload", on AMQPPublishProcessor; used KeyedProperties so parameters can be overridden with the sheet/overlay system Noah Levitt 2016-02-18 18:17:30 -08:00
• f4b8468564 set custom http headers last, so they override headers set elsewhere (e.g. cookies) Noah Levitt 2016-02-18 18:16:03 -08:00
• a64551296a new boolean forceFetch setting; REQUEST_HEADER_BLACKLIST to exclude certain http headers, since with https://github.com/internetarchive/umbra/pull/55 umbra sends over more stuff Noah Levitt 2016-02-18 18:15:22 -08:00
• 8a6a49f7fc ARI-4528 avoid this exception Noah Levitt 2016-02-17 17:45:21 -08:00
• 5f431de215 Update to ExtractorHTML.java for cond. comments Eric Leclerc 2016-02-17 14:49:50 -05:00
• 847400776b Merge pull request #148 from vonrosen/ARI-4738 Noah Levitt 2016-02-17 11:43:30 -08:00
• 3e219af148 Set source tag to uri as it exists before possible prepending of scheme. Hunter Stern 2016-02-17 08:10:07 -08:00
• b9b2343bc0 Don't canonicalize source tag so that SourceSeedDecideRule will work. Trailing slash of canonicalized seed currently causes it not to work. Hunter Stern 2016-02-16 08:52:16 -08:00
• 7b3c9e51e3 Add support for setting the hop and the prefetch count. Andrew Jackson 2016-02-12 11:20:51 +00:00
• bfbd61edd7 Merge pull request #145 from vonrosen/ARI-4713 Hunter 2016-02-11 08:27:52 -08:00
• cbb5e3ab99 Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions. Noah Levitt 2016-02-09 15:02:11 -08:00
• d80cf8244b Merge pull request #146 from vonrosen/ARI-4656-2 Noah Levitt 2016-02-09 10:48:07 -08:00
• 30ce2f09a0 Take care of possible NullPointerException Hunter Stern 2016-02-09 08:22:39 -08:00
• 44139e7e3b fixes for kafka 0.9 (?) Noah Levitt 2016-02-08 16:08:00 -08:00
• f665ce844f upgrade to kafka 0.9 Noah Levitt 2016-02-08 12:49:47 -08:00
• e7c46753ce Fix formatting Hunter Stern 2016-01-29 18:26:18 -08:00
• e83ff04e86 More fixes for https://webarchive.jira.com/browse/ARI-4656 Hunter Stern 2016-01-29 17:25:06 -08:00
• 088189ee8b Fix up code based on pull requests comments. Add test for urls with spaces in them. Hunter Stern 2016-01-28 15:11:49 -08:00
• 4518bc2da8 Make sure no urls with whitespace and not having good extension slip through Hunter Stern 2016-01-20 14:29:42 -08:00
• 12a88d6d23 Allow spaces in urls extracted from JS. Hunter Stern 2016-01-20 10:44:20 -08:00
• 23885557bb Allow spaces in JavaScript urls, but only if they have a known good file extension Hunter Stern 2016-01-19 18:07:05 -08:00
• 34c90966a8 Merge pull request #143 from nlevitt/form-login-extra-inputs Hunter 2016-01-19 08:37:05 -08:00
• 13322111b5 run received urls through the candidates processor, to check scope and schedule them with the frontier Noah Levitt 2016-01-15 17:46:27 -08:00
• 25a7aa9c8d Merge pull request #141 from vonrosen/ARI-4659 Noah Levitt 2016-01-15 11:13:42 -08:00
• cc97f01656 Merge pull request #123 from kris-sigur/HostsReport Noah Levitt 2016-01-15 10:33:07 -08:00
• 70c688a4f5 Merge pull request #135 from vonrosen/ARI-4562 Noah Levitt 2016-01-15 10:15:23 -08:00
• e92e5f2bb3 Merge branch 'master' into form-login-extra-inputs Noah Levitt 2016-01-15 09:39:24 -08:00
• 02e3a554f0 Merge pull request #142 from nlevitt/form-login-multipart Hunter 2016-01-15 08:19:23 -08:00
• c1abbfa5bd handle login forms with <input type="text"> fields in addition to username (for this to work there must be exactly one input field whose name contains the string "username") Noah Levitt 2016-01-14 18:02:54 -08:00
• 368d586eb1 fix NullPointerException when using old HtmlFormCredential login Noah Levitt 2016-01-14 16:00:08 -08:00
• 3b12a751e1 to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly Noah Levitt 2016-01-14 12:17:19 -08:00
• 309847cc40 Be sure to return at least one of the candidates Hunter SternandNoah Levitt 2016-01-11 16:40:28 -08:00
• 704ec62487 Remove unneeded constant Hunter SternandNoah Levitt 2016-01-07 15:08:02 -08:00
• a2a39fe052 Make sure to check form encoding type when addign form to curi data Hunter SternandNoah Levitt 2016-01-07 15:03:25 -08:00
• 93bca3d55c More changes for https://webarchive.jira.com/browse/ARI-4656 Hunter SternandNoah Levitt 2016-01-07 14:52:00 -08:00
• 4997014b34 More changes Hunter SternandNoah Levitt 2015-12-28 10:06:53 -08:00
• ad329ceea8 Save enctype of form in submission curi Hunter SternandNoah Levitt 2015-12-17 10:40:57 -08:00
• 34cffd512a Add ability to submit forms with enctype="multipart/form-data" Hunter SternandNoah Levitt 2015-12-17 09:46:32 -08:00
• afa4f6535b If multiple username inputs in login form, favor the one with the word 'login' in it. Hunter SternandNoah Levitt 2015-12-07 16:36:49 -08:00
• 5f630e4268 Make regex more inclusive for parsing out youtube videoId and add youtube watch page as relative to base Hunter Stern 2016-01-14 09:03:42 -08:00
• 7bcdf03a78 Merge branch 'master' of github.com:internetarchive/heritrix3 into ARI-4562 Hunter Stern 2016-01-14 08:48:39 -08:00
• eeeea6f0c4 Merge pull request #122 from nlevitt/form-login-checked-boxes Hunter 2016-01-14 08:00:49 -08:00
• bf2a887688 Merge pull request #138 from nlevitt/novel-quotas Adam Miller 2016-01-13 16:25:24 -08:00
• 9e4c7f0024 Merge branch 'master' into novel-quotas Noah Levitt 2016-01-13 13:44:54 -08:00
• b82896927a Merge pull request #132 from nlevitt/amqp-url-receiver-changes Adam Miller 2016-01-13 13:35:43 -08:00
• df5748d1f1 Merge pull request #136 from ukwa/amqp-declare-recieved-seeds Noah Levitt 2016-01-13 13:27:43 -08:00
• 22336fb71b fix stats in unusual case of "failed" fetch with response > 0 (only case where this can happen currently is if basic auth is configured for a url, but fails and url returns "401 Unauthorized") Noah Levitt 2016-01-12 17:34:27 -08:00
• 73be17af06 Disable SNI for a request if that request failed due to an SNI error 'javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name' Hunter Stern 2016-01-12 16:58:48 -08:00
• 3dbf754d56 Merge pull request #137 from nlevitt/seed-limits Adam Miller 2016-01-12 16:39:17 -08:00
• 65a1077425 license header Noah Levitt 2016-01-12 14:59:45 -08:00
• 32dd07375d Merge pull request #139 from nlevitt/robots-sep-sections Neil Minton 2016-01-11 18:39:25 -06:00
• 8918ce99c6 check that sourceTag of CrawlURI actually matches configured sourceTag Noah Levitt 2015-12-15 11:24:53 -08:00
• 7a8a614804 avoid exception in case applyToSubdomains is enabled and configured host is an ip address Noah Levitt 2015-12-15 11:06:33 -08:00
• a95519543d avoid exception in case applyToSubdomains is enabled and CrawlURI host is a nonstandard host like "dns:" in HostQuotaEnforcer Noah Levitt 2015-12-15 11:02:38 -08:00
• e3e6bfeb11 new HostQuotaEnforcer option applyToSubdomains, to additionally apply the quotas (separately) to each subdomain of the configured host Noah Levitt 2015-12-14 18:17:44 -08:00
• 2f82d43110 remove already-outdated stuff from javadoc Noah Levitt 2015-12-14 15:24:08 -08:00
• 6f9c235fd0 fix javadoc Noah Levitt 2015-12-14 15:23:16 -08:00
• 013029d9ad fix annotation copy-paste-o Noah Levitt 2015-12-14 15:22:51 -08:00
• 99c5af0cf2 comment out currently failing test (needs webarchive-commons pull request merged) Noah Levitt 2015-12-14 10:46:50 -08:00
• 2a56886c01 add license header Noah Levitt 2015-12-14 10:46:14 -08:00
• 8deead15ec new class to enforce quotas on a host Noah Levitt 2015-12-11 17:02:32 -08:00
• f1173789f7 fix test now that ContentDigestHistoryTest.shouldProcess() is smarter Noah Levitt 2015-12-11 16:13:17 -08:00
• c05678e296 CrawlLimitEnforcer support for maxWarcNovelBytes and maxWarcNovelUrls, with tests Noah Levitt 2015-12-11 16:08:47 -08:00
• 38931f152a handle multiple clauses for same user agent in robots.txt Noah Levitt 2015-12-11 15:42:16 -08:00
• dfc1dd940e tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests Noah Levitt 2015-12-11 12:49:02 -08:00
• c20a06ec27 Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch Noah Levitt 2015-12-11 11:35:11 -08:00
• 87f70c3e6a WARCLimitEnforcer to enforce limits (by stopping the crawl job) on any of the stats kept by WARCWriterProcessor Noah Levitt 2015-12-11 11:26:11 -08:00
• 294cfa3524 limited test of warc writer stats Noah Levitt 2015-12-11 11:22:03 -08:00
• 9ce12d2dbe add warc stats to CrawlURI.getData() dynamic attributes list after writing records; new method WARCWriterProcessor.getStats() to expose cumulative stats Noah Levitt 2015-12-11 11:21:05 -08:00
• 35a7954882 refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code Noah Levitt 2015-12-11 11:05:44 -08:00
• 566c1a600d remove warning which can be misleading if crawl is configured to write only certain urls to warcs Noah Levitt 2015-12-11 11:01:26 -08:00
• cce07084e5 CrawlLimitEnforcer support for maxNovelBytes and maxNovelUrls, with tests Noah Levitt 2015-12-09 17:15:34 -08:00
• 501efe4a7b QuotaEnforcer support for quotas on novelBytes and novelUrls, with tests Noah Levitt 2015-12-09 15:54:36 -08:00
• ce31cada5f change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable Noah Levitt 2015-12-08 13:45:25 -08:00
• 7929e16d6a fix checkpointing problems with new statsBySource Noah Levitt 2015-12-04 21:33:40 -08:00
• de22096dde SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached Noah Levitt 2015-12-03 17:50:46 -08:00
• 5559a5353d SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds Noah Levitt 2015-12-03 17:43:26 -08:00
• ce4a865764 add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality Noah Levitt 2015-12-03 16:13:34 -08:00
• 0f6d385b24 Hook in submitted seeds properly. Andrew Jackson 2015-11-27 20:51:35 +00:00
• 89cd085e76 Merge pull request #133 from vonrosen/ARI-4565 Noah Levitt 2015-11-10 17:00:23 -08:00
• d3070cef54 Allow KnowledgableExtractorJS to parse out youtube watch from youtube javascript embed Hunter Stern 2015-11-06 11:39:41 -08:00
• 17f58f4bf8 Remove uneeded logger Hunter Stern 2015-11-06 10:42:55 -08:00
• 580391e0a7 Move most of addCookie functionality into AbstractCookieStore Hunter Stern 2015-11-06 10:40:01 -08:00
• 3142bbd485 Revise based on feedback in pull request from nlevitt Hunter Stern 2015-11-02 13:07:11 -08:00
• 44ac1030f2 Make sure to synchronize in addCookie now that domain count limits are imposed and modify unit test to also take into account domain limits Hunter Stern 2015-10-16 17:12:42 -07:00
• d9684e913b Restrict number of cookies per domain to 50 Hunter Stern 2015-10-16 14:05:56 -07:00
• 5ec6c738a6 Revert "Possible fix that probably isn't needed" Hunter Stern 2015-10-13 17:21:28 -07:00
• c6f019a095 Possible fix that probably isn't needed Hunter Stern 2015-10-13 14:18:06 -07:00
• e37b2b42cb avoid spurious logging Noah Levitt 2015-10-08 18:11:09 -07:00
• e4acb6c512 try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability Noah Levitt 2015-10-08 17:22:44 -07:00
• 79ebb0e03c Merge pull request #129 from nlevitt/java8keytool Hunter 2015-10-06 16:09:43 -07:00
• a9c4cb42c9 Merge pull request #130 from nlevitt/stay-alive-statistics-tracker Hunter 2015-10-06 16:09:11 -07:00
• 98d41051b8 Merge pull request #131 from nlevitt/amqp-url-receiver-changes Hunter 2015-10-06 15:52:05 -07:00