Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#750

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 8ddaf41db2 Add crawler-commons dependency. kristinn 2019-07-05 11:27:15 +00:00
• 5a813197c2 Merge branch 'warc-writer-chain' into ait-qa Noah Levitt 2019-06-12 17:42:20 -07:00
• bff33e00e6 write youtube-dl json to the warc Noah Levitt 2019-06-12 17:40:57 -07:00
• 5177b2b6da rename method so as not to conflict with Processor Noah Levitt 2019-06-12 15:17:33 -07:00
• 9ddd281ef2 make WARCRecordBuilder an interface Noah Levitt 2019-06-12 15:04:39 -07:00
• 9e67a8dab4 revisits only for http and ftp Noah Levitt 2019-06-12 14:58:11 -07:00
• e928c218e7 accept any BaseWARCWriterProcessor Noah Levitt 2019-06-12 12:57:27 -07:00
• 7c31b0752a use WARCWriterChainProcessor for these tests Noah Levitt 2019-06-12 12:56:49 -07:00
• 8c4c443a88 default chain in code Noah Levitt 2019-06-12 12:54:00 -07:00
• ece435874e oops, handle https too Noah Levitt 2019-06-12 12:53:27 -07:00
• 870d84740a same test as for WARCWriterProcessor Noah Levitt 2019-06-12 11:04:02 -07:00
• 9435f761a6 configurable warc writer chain Noah Levitt 2019-06-11 13:45:51 -07:00
• 26bfc5dd34 Merge branch 'ydl' into ait-qa Noah Levitt 2019-06-07 15:15:50 -07:00
• 4d6314bc50 use --dump-single-json Noah Levitt 2019-06-07 15:14:32 -07:00
• 25848e9821 cap number of videos per page to 1000 Noah Levitt 2019-06-07 14:24:48 -07:00
• de0c19f934 Merge pull request #257 from nlevitt/ydl Adam Miller 2019-05-22 13:40:50 -07:00
• 40959e7f97 Merge branch 'ydl' into ait-qa Noah Levitt 2019-05-07 11:03:40 -07:00
• 99eebb8101 ugh, get it right Noah Levitt 2019-05-07 11:01:34 -07:00
• 49cb6c2b2b Merge branch 'ydl' into ait-qa Noah Levitt 2019-05-06 15:36:01 -07:00
• 75b33663f9 also annotate and log containing pages Noah Levitt 2019-05-06 15:35:47 -07:00
• aec1443e56 Merge branch 'ydl' into ait-qa Noah Levitt 2019-05-02 18:27:36 -07:00
• 7c66da7be7 whoops, better spawn the thread first Noah Levitt 2019-05-02 15:48:46 -07:00
• b4aa3c9511 read stderr and stdout in separate threads... Noah Levitt 2019-05-02 15:46:14 -07:00
• 37fb6f6b7b do not drop any CrawlURI.data between processing Noah Levitt 2019-04-30 16:20:53 -07:00
• b7de2bd2ae nothing private ever Noah Levitt 2019-04-30 16:20:35 -07:00
• 1bd8b713c6 quiet org.mortbay.log (jetty?) logging Noah Levitt 2019-04-30 16:17:35 -07:00
• 21cfd4b73d making everything work Noah Levitt 2019-04-30 16:16:25 -07:00
• 79a0d34adf ExtractorYoutubeDL Noah Levitt 2019-04-29 15:29:16 -07:00
• 1ed207976c Update README and add LICENSE.txt nruest 2019-04-22 09:17:23 -04:00
• 7d3eff5269 Update changelog for 3.4.0-20190418 Andrew Jackson 2019-04-18 16:17:50 +01:00
• b3961a2f96 [maven-release-plugin] prepare for next development iteration Andrew Jackson 2019-04-18 15:36:28 +01:00
• c7c6141ee1 [maven-release-plugin] prepare release 3.4.0-20190418 3.4.0-20190418 Andrew Jackson 2019-04-18 15:36:20 +01:00
• 8492306241 Merge pull request #238 from ruebot/issue-233 Andy Jackson 2019-04-17 22:50:01 +01:00
• f1683264d5 Merge pull request #251 from nlevitt/trough-dedup Adam Miller 2019-04-15 14:40:13 -07:00
• a60b2ef387 Merge pull request #249 from ruebot/remove-suffix-crawler-bean Noah Levitt 2019-04-10 11:15:37 -07:00
• a20b649576 Merge pull request #253 from dvanduzer/master Noah Levitt 2019-04-10 11:15:08 -07:00
• ec4ec2593a Merge branch 'trough-dedup' into ait-qa Noah Levitt 2019-04-10 11:10:08 -07:00
• f1dae275b4 give me some space Noah Levitt 2019-04-10 11:09:57 -07:00
• 0424dfb51c Merge branch 'trough-dedup' into ait-qa Noah Levitt 2019-04-10 10:50:08 -07:00
• 303ffb486e close the rethinkdb connection! Noah Levitt 2019-04-10 10:43:22 -07:00
• fc16e4ec3e Merge branch 'master' into trough-dedup Noah Levitt 2019-04-10 10:16:45 -07:00
• 6c6e460c8d set of frontier management changes to support CrawlHQ module David Van Duzer 2019-04-08 17:15:24 -06:00
• 2c6ab5a44a replace System.err.println with logger.info Noah Levitt 2019-04-04 17:06:01 -07:00
• de63ed04b1 Merge branch 'master' into ait-qa Noah Levitt 2019-04-01 16:16:21 -07:00
• 0aac882eb4 Merge branch 'master' into ait-qa Noah Levitt 2019-04-01 16:16:03 -07:00
• 5f5d4615c2 fix some trough dedup bugs Noah Levitt 2019-04-01 16:14:22 -07:00
• 5ee7528369 Remove suffix from warcWriter since it is no longer used. nruest 2019-03-29 12:40:51 -04:00
• b68731e2b9 Merge pull request #248 from internetarchive/revert-246-upgrade-httpclient Alex Osborne 2019-03-28 10:19:22 +09:00
• dd37598470 Revert "Upgrade httpclient to 4.5.7 and handle cookies more compliantly" Alex Osborne 2019-03-28 10:14:42 +09:00
• 954043cdf3 Merge pull request #246 from ukwa/upgrade-httpclient Noah Levitt 2019-03-22 10:38:20 -07:00
• 1099c152cf Merge pull request #242 from nlevitt/trough-dedup jkafader 2019-03-21 15:49:17 -07:00
• c596f49c8a use constant from rethinkdb lib for default port Noah Levitt 2019-03-21 15:34:42 -07:00
• 03da8c06cf Merge branch 'master' into upgrade-httpclient Andrew Jackson 2019-03-21 00:06:10 +00:00
• 044f068d1b Removing outdated test. Andrew Jackson 2019-03-21 00:06:05 +00:00
• 629a7adcb6 Disable questionalbe test. Andrew Jackson 2019-03-20 22:02:58 +00:00
• 2fbe603878 Avoid deprecated flag. Andrew Jackson 2019-03-20 22:02:35 +00:00
• e910bb6a5b Supply an iterator, for #245 Andrew Jackson 2019-03-20 21:18:19 +00:00
• 0235a0e553 Updated POM to use latest version. Andrew Jackson 2019-03-20 20:26:51 +00:00
• 77c126b9db Merge branch 'trough-dedup' into ait-qa Noah Levitt 2019-03-20 11:32:10 -07:00
• fd0e86cd64 less alarming logging for normal situation Noah Levitt 2019-03-20 11:31:40 -07:00
• dff29b1685 Merge pull request #243 from internetarchive/srcset Adam Miller 2019-03-19 09:53:23 -07:00
• 0c1c973c97 Merge branch 'trough-dedup' into ait-qa Noah Levitt 2019-03-18 22:39:54 -07:00
• c4ec9d31fe make TroughCrawlLogFeed use TroughClient, and... Noah Levitt 2019-03-18 22:39:29 -07:00
• fcc3bc2940 Merge pull request #244 from mikeizbicki/patch-1 Alex Osborne 2019-03-19 12:41:25 +09:00
• 102b5086eb Update README.md Mike Izbicki 2019-03-18 20:26:15 -07:00
• 7d91a1d4ee Handle missing closing paren in srcset descriptor Alex Osborne 2019-03-16 16:02:48 +09:00
• 2a34fcffd6 Teach jericho extractor srcset Alex Osborne 2019-03-16 15:47:50 +09:00
• e1d93e3308 Don't run srcset test against jericho, it doesn't handle it Alex Osborne 2019-03-16 12:35:57 +09:00
• 90c52c3a25 Handle commas more compliantly when parsing srcset Alex Osborne 2019-03-16 11:50:53 +09:00
• 1ce76345e9 Merge branch 'trough-dedup' into ait-qa Noah Levitt 2019-03-15 14:17:00 -07:00
• af1a325319 fix bugs around crawl finish Noah Levitt 2019-03-15 14:12:50 -07:00
• 7f7fb3cfe8 allow travis-ci jdk7 failures because contrib... Noah Levitt 2019-03-14 16:57:31 -07:00
• b12e34d340 Merge branch 'trough-dedup' into ait-qa Noah Levitt 2019-03-14 15:17:39 -07:00
• 82406f4028 promote dirty segments at crawl finish Noah Levitt 2019-03-14 15:16:56 -07:00
• 1b89420f99 trough dedup! Noah Levitt 2019-03-14 13:47:15 -07:00
• afa88c025e Ensure we start parsing full lines, for #239. Andrew Jackson 2019-03-11 21:35:13 +00:00
• a5e66aa556 Add CHANGELOG; address #233. nruest 2019-03-05 07:45:02 -05:00
• c40616e494 Merge pull request #235 from ukwa/retry-failed-dns Noah Levitt 2019-02-25 14:15:38 -08:00
• 91b18e9bbd Allow failed lookups to expire, for #234. Andrew Jackson 2019-02-21 23:04:40 +00:00
• ad9e74d8ad [maven-release-plugin] prepare for next development iteration Andrew Jackson 2019-02-07 13:52:56 +00:00
• 83c7044b22 [maven-release-plugin] prepare release 3.4.0-20190207 3.4.0-20190207 Andrew Jackson 2019-02-07 13:52:49 +00:00
• 18d3bb23aa Merge pull request #231 from ukwa/synchronized-enqueuecounts Andy Jackson 2019-02-07 13:49:50 +00:00
• cb96a44bdc Merge pull request #230 from ukwa/more-robust-fetch-history-processor Andy Jackson 2019-02-07 13:49:23 +00:00
• 667cf3ac5e As @nlevitt suggestion, a further check. Andrew Jackson 2019-02-06 21:03:49 +00:00
• e8e37751bb Merge branch 'master' into more-robust-fetch-history-processor Andrew Jackson 2019-02-06 20:47:06 +00:00
• 317b19e7e2 [maven-release-plugin] prepare for next development iteration Andrew Jackson 2019-02-05 12:33:08 +00:00
• 9c7d0c299d [maven-release-plugin] prepare release 3.4.0-20190205 3.4.0-20190205 Andrew Jackson 2019-02-05 12:32:57 +00:00
• ab34a54566 [maven-release-plugin] prepare for next development iteration Andrew Jackson 2019-02-05 12:29:33 +00:00
• d1fc5cedb4 [maven-release-plugin] prepare release 3.4.0-20190205-2 Andrew Jackson 2019-02-05 12:29:26 +00:00
• 096ba4e9c3 [maven-release-plugin] prepare for next development iteration Andrew Jackson 2019-02-05 12:15:26 +00:00
• 24ef7eb9d0 Extend to set additional property. Andrew Jackson 2019-02-05 12:14:17 +00:00
• 947bb39aff Use argument syntax. Andrew Jackson 2019-02-05 12:13:20 +00:00
• 0e1a21c9d8 Skip tests during release process (covered by CI). Andrew Jackson 2019-02-05 12:11:29 +00:00
• 765d607e5c Set consistent tag, and include contrib. Andrew Jackson 2019-02-05 11:57:23 +00:00
• 24b72e07df Wrong repo spec. Andrew Jackson 2019-02-05 11:23:18 +00:00
• fae30ed63c Add build profile for deployment to Maven Central. Andrew Jackson 2019-02-05 10:32:40 +00:00
• 1cb399ec18 Avoid headings being treated as lists Andrew Jackson 2019-02-05 10:32:13 +00:00
• d0a146bc66 Clarification about APIs. Andy Jackson 2019-02-05 10:28:18 +00:00
• 021aeeaa20 Swapped original and link to make maintenance simpler. Andrew Jackson 2019-02-05 10:25:54 +00:00
• b08db899dd Tidy up markup and links. Andrew Jackson 2019-02-05 10:23:01 +00:00