Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#781/merge

#782

#783

#784

#785

#786

#787

#787/merge

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 345e927d32 Remove HBase from contrib Alex Osborne 2024-10-25 14:27:38 +09:00
• ff84c8d32f Bump jetty from 9.4.55.v20240627 to 9.4.56.v20240826 Alex Osborne 2024-10-18 22:17:25 +09:00
• 3c649614ba Avoid old versions of commons-io and commons-collections from hbase Alex Osborne 2024-10-18 22:14:51 +09:00
• c25e57a4a1 Bump version to 3.4.1-SNAPSHOT Alex Osborne 2024-10-18 20:54:12 +09:00
• b0fad42e25 Merge pull request #619 from internetarchive/jdk23 Alex Osborne 2024-10-18 20:51:08 +09:00
• 070063ce0a Merge pull request #618 from internetarchive/ftpserver-core-1.2.0 Alex Osborne 2024-10-18 20:45:33 +09:00
• 5cfa5f02df Update CI tests from JDK 22 to 23 Alex Osborne 2024-10-18 20:45:11 +09:00
• 7a3d29ad4d Bump ftpserver-core from 1.1.1 to 1.2.0 Alex Osborne 2024-10-18 20:38:49 +09:00
• 6157645545 Merge pull request #612 from internetarchive/config-extractor-js-default-strict Alex Osborne 2024-10-18 20:32:58 +09:00
• 33f009516f Merge pull request #616 from internetarchive/webarchive-commons-1.1.10 Alex Osborne 2024-10-18 20:32:42 +09:00
• d8aa0902e4 Merge pull request #617 from internetarchive/kafka-3.8.0 Alex Osborne 2024-10-18 20:30:38 +09:00
• 80b174f1fd Bump kafka from 0.9.0.0 to kafka-clients 3.8.0 Alex Osborne 2024-10-18 20:06:23 +09:00
• cbd7f37ed9 Replace littleproxy with jetty-proxy (to remove dependency on log4j) Alex Osborne 2024-10-18 17:24:02 +09:00
• 3d1b7e37ca Remove log4j Alex Osborne 2024-10-15 20:08:53 +09:00
• c797985590 Bump webarchive-commons from 1.1.8 to 1.1.10 Alex Osborne 2024-10-15 19:29:50 +09:00
• f49dbda2b8 Merge pull request #615 from internetarchive/dependabot/maven/org.eclipse.jetty-jetty-server-9.4.55.v20240627 Alex Osborne 2024-10-17 12:02:09 +09:00
• 1d37cbc237 StatisticsSelfTest: Update for new jetty version Alex Osborne 2024-10-17 11:58:13 +09:00
• ececa3b6a3 Bump org.eclipse.jetty:jetty-server dependabot[bot] 2024-10-17 02:46:12 +00:00
• 74cabbe798 Merge pull request #614 from internetarchive/remove-builds.archive.org Alex Osborne 2024-10-17 11:40:20 +09:00
• 76cbe9343d Remove dependency on builds.archive.org Maven repository Alex Osborne 2024-10-17 11:26:43 +09:00
• af280584f0 Set default value for strict. Kristinn Sigurðsson 2024-10-16 09:25:02 +00:00
• 891dbe7615 Re-add restlet repository with new URL Alex Osborne 2024-10-15 19:19:28 +09:00
• 8e568187b6 Merge pull request #611 from internetarchive/dependabot/maven/commons/commons-io-commons-io-2.14.0 Alex Osborne 2024-10-15 18:18:22 +09:00
• 27259d8b82 Bump commons-io:commons-io from 2.7 to 2.14.0 in /commons dependabot[bot] 2024-10-15 07:35:22 +00:00
• 3b06226509 Merge pull request #610 from internetarchive/remove-restlet-repo Alex Osborne 2024-10-09 12:39:34 +09:00
• cb1143e438 Remove maven.restlet.com repository Alex Osborne 2024-10-09 12:29:06 +09:00
• 043dc4e48d Merge pull request #609 from internetarchive/remove-joda-time Alex Osborne 2024-10-09 11:29:15 +09:00
• 3c80dacabe CrawlJob: Replace joda-time with java.time (JSR-310) Alex Osborne 2024-09-11 16:15:07 +09:00
• e02187ea3b Arc2Warc: Replace joda-time with java.time (JSR-310) Alex Osborne 2024-09-11 15:34:50 +09:00
• 2e891bc4bb Merge pull request #608 from internetarchive/cite-navlinks Alex Osborne 2024-09-13 13:19:03 +09:00
• 3a447b5e61 ExtractorHTML: Treat 'cite' attribute as navlink instead of embed manderson 2024-05-28 14:50:57 +10:00
• 791f1c81be Add changelog links for 3.4.0-20240909 Alex Osborne 2024-09-10 12:27:56 +09:00
• 8ac80be9d3 Add 3.4.0-20240909 to CHANGELOG Alex Osborne 2024-09-10 12:24:06 +09:00
• c7c492b507 [maven-release-plugin] prepare for next development iteration Alex Osborne 2024-09-09 15:28:44 +09:00
• 11fcbe71f3 [maven-release-plugin] prepare release 3.4.0-20240909 3.4.0-20240909 Alex Osborne 2024-09-09 15:28:44 +09:00
• ae7a025c35 Merge pull request #607 from internetarchive/dependabot/maven/contrib/com.rabbitmq-amqp-client-5.18.0 Alex Osborne 2024-09-06 11:18:33 +09:00
• ee048f4129 Merge pull request #606 from internetarchive/dependabot/maven/commons/org.springframework-spring-expression-5.3.39 Alex Osborne 2024-09-06 11:17:14 +09:00
• 2cffefc798 Bump com.rabbitmq:amqp-client from 4.8.0 to 5.18.0 in /contrib dependabot[bot] 2024-09-06 02:13:37 +00:00
• 09ad15f711 Bump org.springframework:spring-expression in /commons dependabot[bot] 2024-09-06 02:12:04 +00:00
• 2c9a5c97ab Merge pull request #586 from nla/kryo-5.6.0 Alex Osborne 2024-09-06 11:10:17 +09:00
• ec57e2a8b1 Merge pull request #605 from bnfleb/extractorHTML Alex Osborne 2024-09-06 11:09:45 +09:00
• ca3508f55f Add new HTML tags and attributes for ExtractorHTML (#604) Leslie Bellony 2024-09-04 17:49:40 +02:00
• 9405cb9af1 Merge pull request #598 from kris-sigur/hostname-queues-with-limits Kristinn Sigurðsson 2024-09-03 12:25:20 +00:00
• 9ee8520604 ConfigurableExtractorJS (#602) Kristinn Sigurðsson 2024-09-03 12:24:08 +00:00
• 020c6d77b0 Merge pull request #603 from internetarchive/dependabot/maven/commons/dnsjava-dnsjava-3.6.0 Alex Osborne 2024-09-03 11:46:15 +09:00
• 3ec27f2097 Bump dnsjava:dnsjava from 3.3.1 to 3.6.0 in /commons dependabot[bot] 2024-09-03 01:37:46 +00:00
• c35d72211c Merge pull request #601 from internetarchive/remove-extractor-chrome Alex Osborne 2024-08-20 11:59:55 +09:00
• 2d862c3e28 Merge pull request #600 from internetarchive/adam/restore-has-prerequisite-behavior Adam Miller 2024-08-08 12:05:22 -07:00
• cd3a424176 Merge pull request #597 from kris-sigur/extractorhttp-implicit Kristinn Sigurðsson 2024-08-08 07:37:45 +00:00
• 0faf338f91 Merge pull request #595 from internetarchive/meta-content-name Kristinn Sigurðsson 2024-08-08 07:36:36 +00:00
• 55a179c610 Remove ExtractorChrome Alex Osborne 2024-08-08 07:20:44 +09:00
• ec164ec77f Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts adam/restore-has-prerequisite-behavior Adam Miller 2020-11-07 01:12:56 +00:00
• b22d6ce2e1 Merge pull request #593 from internetarchive/adam/add-crawl-log-logging-to-extractoryoutubedl Adam Miller 2024-08-07 10:17:25 -07:00
• 3ae1300a21 Merge pull request #599 from miku/miku/scriptedprocessor-javadoc-fix Alex Osborne 2024-08-07 17:04:02 +09:00
• 5ced224193 Merge pull request #594 from internetarchive/adam/ait-add-crawl-log-logging-to-extractoryoutubedl master-ait-contrib Adam Miller 2024-07-24 17:16:45 -07:00
• 3128ae5b84 fix: ScriptedProcessor function name in docs Martin Czygan 2024-07-22 14:15:40 +02:00
• 568db3cabc Added queue assignment variants that limit queue name length Kristinn Sigurðsson 2024-07-09 09:36:39 +00:00
• 778a57b3b4 Add a more general support for inferred path discovery Kristinn Sigurðsson 2024-07-08 13:05:56 +00:00
• 0dd3a2506d Apply speculativeFixup before evaluating meta content meta-content-name Kristinn Sigurðsson 2024-07-03 11:09:57 +00:00
• 59635f00f4 chore: fix contrib dependencies adam/add-crawl-log-logging-to-extractoryoutubedl Adam Miller 2024-06-21 23:34:23 +00:00
• 54c6b02040 chore: fix contrib dependencies ait-qa Adam Miller 2024-06-21 22:45:23 +00:00
• eb17155046 fix: commons-codec DigestUtils dependency adam/ait-add-crawl-log-logging-to-extractoryoutubedl Adam Miller 2024-06-21 22:45:23 +00:00
• 8acb9ebc45 refactor: simplify sha1 calculation for yt-dlp json content. Adam Miller 2024-06-20 10:59:21 -07:00
• 13f63cd952 refactor: simplify sha1 calculation for yt-dlp json content. Adam Miller 2024-06-20 10:59:21 -07:00
• e64629cddd refactor: simplify sha1 calculation for yt-dlp json content. Adam Miller 2024-06-20 10:59:21 -07:00
• 37b6ef9cfa fix: re-add imports where are used in this branch, but not others Adam Miller 2024-06-14 00:21:34 +00:00
• b0191c8571 feat: Add unit tests for ExtractorYoutubeDL WARC record building Adam Miller 2024-06-07 17:03:37 -07:00
• 220f098947 Make logging of metadata record optional, but default. Adam Miller 2024-06-06 14:32:45 -07:00
• de4d6c8612 feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL Adam Miller 2024-06-06 14:21:43 -07:00
• 74d6b37779 Update to kryo 5.6.0 Alex Osborne 2024-05-15 16:36:02 +09:00
• 4a103f3e21 Merge pull request #592 from internetarchive/adam/fix-maven-junit3-test-discovery Alex Osborne 2024-06-18 09:57:21 +09:00
• 71911100cf fix: re-add imports where are used in this branch, but not others Adam Miller 2024-06-14 00:21:34 +00:00
• 871db559e0 feat: Add unit tests for ExtractorYoutubeDL WARC record building Adam Miller 2024-06-07 17:03:37 -07:00
• 5483407829 Make logging of metadata record optional, but default. Adam Miller 2024-06-06 14:32:45 -07:00
• 2f3605540d feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL Adam Miller 2024-06-06 14:21:43 -07:00
• cf5ccf7895 fix: maven was not running older junit3 style tests without some extra configuration. Adam Miller 2024-06-07 17:07:38 -07:00
• 55dc23daaf feat: Add unit tests for ExtractorYoutubeDL WARC record building Adam Miller 2024-06-07 17:03:37 -07:00
• d5e2ce6466 Make logging of metadata record optional, but default. Adam Miller 2024-06-06 14:32:45 -07:00
• 3f764f7904 feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL Adam Miller 2024-06-06 14:21:43 -07:00
• e38cd3528e Merge pull request #589 from internetarchive/jdk22 Alex Osborne 2024-05-15 23:17:26 +09:00
• 5424423e7d Add JDK 22 to test matrix Alex Osborne 2024-05-15 23:14:51 +09:00
• 0c251c7c34 Update groovy to 4.0.21 for JDK 22 support Alex Osborne 2024-05-15 23:13:18 +09:00
• a151937d4c Remove workaround for non-https builds.archive.org repository Alex Osborne 2024-05-15 21:03:50 +09:00
• 64c7f0fb1c Merge pull request #572 from BulkSecurityGeneratorProjectV2/fix/JLL/use_https_to_resolve_dependencies_maven Alex Osborne 2024-05-15 15:50:07 +09:00
• f81a987110 Merge pull request #583 from internetarchive/dependabot/maven/commons/org.springframework-spring-expression-5.3.27 Alex Osborne 2024-04-12 16:00:21 +09:00
• 7975bf3d7c Bump org.springframework:spring-expression in /commons dependabot[bot] 2024-04-12 06:58:23 +00:00
• b3d276b6d8 Merge pull request #575 from internetarchive/pdfbox Alex Osborne 2024-04-12 15:49:17 +09:00
• 4552509941 Merge pull request #582 from galgeek/fix-yt-dlp-typo-too Barbara Miller 2024-03-20 10:22:05 -07:00
• 31897ed11a Merge pull request #578 from galgeek/fix-yt-dlp-typo Barbara Miller 2024-03-20 10:19:55 -07:00
• 3996195ab3 update comment Barbara Miller 2024-03-20 10:17:34 -07:00
• 8ce1ba310e Merge branch 'yt-dlp-fix' into fix-yt-dlp-typo Barbara Miller 2024-03-20 10:11:29 -07:00
• baefa0a32d update comment Barbara Miller 2024-03-20 10:09:35 -07:00
• 5505051390 smaller youtube single-file offering Barbara Miller 2024-03-19 16:12:37 -07:00
• dd186df08b smaller youtube single-file offering Barbara Miller 2024-03-19 16:12:37 -07:00
• c9ada60d52 vcodec,res Barbara Miller 2024-03-18 15:38:25 -07:00
• d748f11ac7 Merge branch 'internetarchive:fix-yt-dlp-typo-too' into fix-yt-dlp-typo-too Barbara Miller 2024-03-19 18:39:29 -07:00
• 4053d2fcf2 smaller youtube single-file offering Barbara Miller 2024-03-19 16:12:37 -07:00
• 12691f5d9d Merge branch 'fix-yt-dlp-typo' into ait-qa Barbara Miller 2024-03-19 13:06:58 -07:00
• 547aa7d954 720p Barbara Miller 2024-03-19 12:50:59 -07:00
• e84e125d94 720p fix-yt-dlp-typo-too Barbara Miller 2024-03-19 12:50:59 -07:00