Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#781/merge

#782

#783

#784

#785

#786

#787

#787/merge

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• e84e125d94 720p fix-yt-dlp-typo-too Barbara Miller 2024-03-19 12:50:59 -07:00
• a3d1873f42 Merge branch 'master' into fix-yt-dlp-typo-too Barbara Miller 2024-03-18 17:26:46 -07:00
• 0bac560a9c rm ait updates Barbara Miller 2024-03-18 17:24:26 -07:00
• 3396970d27 vcodec,res Barbara Miller 2024-03-18 15:38:25 -07:00
• 956d3a29c1 vcodec,res Barbara Miller 2024-03-18 15:38:25 -07:00
• 737ea9f673 vcodec,res Barbara Miller 2024-03-18 15:38:25 -07:00
• a230f9e402 Merge pull request #577 from BitBaron/play-nice Neil Minton 2024-03-14 07:04:41 -04:00
• 6de3580794 Modify process array. Neil Minton 2024-03-13 21:32:48 -04:00
• fcbdad584c Modify process array. Neil Minton 2024-03-13 21:32:48 -04:00
• 32d356e967 Update logging and comments. Neil Minton 2024-03-13 12:50:18 -04:00
• 32078dd418 Make video extraction nicer. Neil Minton 2024-03-13 12:49:08 -04:00
• 7eed2bc03c Update logging and comments. Neil Minton 2024-03-13 12:50:18 -04:00
• d608bb0b43 Make video extraction nicer. Neil Minton 2024-03-13 12:49:08 -04:00
• 19dd5c00de Switch ExtractorPDFContent to pdfbox Alex Osborne 2024-02-21 14:47:56 +09:00
• 3700628cb2 Switch PDFParser and ExtractorPDF to pdfbox Alex Osborne 2024-02-21 14:31:10 +09:00
• c467065b79 Add PDFParserTest Alex Osborne 2024-02-21 14:27:28 +09:00
• ca079845c8 Merge pull request #574 from internetarchive/yt-dlp Neil Minton 2024-02-09 06:31:30 -05:00
• 4b390404f0 more updates for newer yt-dlp yt-dlp Barbara Miller 2023-11-07 14:09:13 -08:00
• 75b18e6117 yt-dlp (not youtube-dl) Barbara Miller 2023-09-29 15:41:13 -07:00
• 3bf1c70eef Update ExtractorYoutubeDL.java Barbara Miller 2023-09-06 11:25:12 -07:00
• 8549031e59 Merge pull request #565 from internetarchive/yt-dlp-20230706 Barbara Miller 2024-02-06 10:21:17 -08:00
• ea89c56c1d vuln-fix: Use HTTPS instead of HTTP to resolve deps CVE-2021-26291 Jonathan LeitschuhandModerne 2023-12-16 06:58:50 +00:00
• a1500586a8 more updates for newer yt-dlp Barbara Miller 2023-11-07 14:09:13 -08:00
• e4c9eb210a more updates for newer yt-dlp yt-dlp-20230706 Barbara Miller 2023-11-07 14:09:13 -08:00
• d3ca691f8d Update CI tests from JDK 20 to 21 Alex Osborne 2023-10-11 11:11:39 +09:00
• bd6c6bbff5 Merge pull request #566 from internetarchive/jdk20 Alex Osborne 2023-10-11 11:09:53 +09:00
• e19bdc0af1 yt-dlp (not youtube-dl) Barbara Miller 2023-09-29 15:41:13 -07:00
• e86a5fedfe Revert "Merge branch 'yt-dlp-20230706' into ait-qa" Barbara Miller 2023-09-29 16:11:20 -07:00
• 1074d95e2d Merge branch 'yt-dlp-20230706' into ait-qa Barbara Miller 2023-09-29 15:48:35 -07:00
• bcfed0c1f7 yt-dlp (not youtube-dl) Barbara Miller 2023-09-29 15:41:13 -07:00
• 1a27551e60 Update CI tests from JDK 19 to 20 Alex Osborne 2023-09-19 11:10:28 +09:00
• 1c07e43ade Update groovy to 4.0.15 for JDK 20 support Alex Osborne 2023-09-19 11:08:56 +09:00
• 122417c0fb Update ExtractorYoutubeDL.java Barbara Miller 2023-09-06 11:25:12 -07:00
• d50158c012 Update ExtractorYoutubeDL.java Barbara Miller 2023-09-06 11:25:12 -07:00
• 8563f491a5 Merge pull request #536 from claire9910/oscs_fix_cd5c2voau51t5k2gfi30 Andy Jackson 2023-03-10 13:12:30 +00:00
• a3952be181 Merge pull request #552 from internetarchive/revert-375-fix-last-modified-millis Andy Jackson 2023-03-10 13:05:55 +00:00
• d073b43b48 Revert "Use seconds from the FetchCompletedTime for LastModified time" Andy Jackson 2023-03-10 13:05:39 +00:00
• d948364105 Merge pull request #549 from KBNLresearch/master Andy Jackson 2023-03-10 12:57:53 +00:00
• b25d71339e Merge pull request #551 from internetarchive/revert-445-adds-bdbmodule-max-file-size-option Andy Jackson 2023-03-10 12:53:48 +00:00
• b6af2389fb Revert "Adding configurable MaxLogFileSize option to BdbModule" Andy Jackson 2023-03-10 12:53:35 +00:00
• 87566eac61 Merge pull request #445 from internetarchive/adds-bdbmodule-max-file-size-option Andy Jackson 2023-03-10 11:23:06 +00:00
• 2d0afcdbd7 Merge pull request #375 from phanidv/fix-last-modified-millis Andy Jackson 2023-03-10 11:21:32 +00:00
• cdfcce8deb In case of a crashed job, the crawlsummaryreport says: "Finished - Abnormal exit from crawl while the ApplicationContext still reports running==true. So the console shows "Job is ACTIVE - Running", while the process crashed This change reports the exit status in this case. fedor 2023-03-06 15:54:27 +01:00
• c45d524e74 Bump commons-net from 2.0 to 3.9.0 in /commons dependabot[bot] 2023-01-23 05:44:46 +00:00
• f4280012ae BdbModule: Fix resource leak on job teardown if sync() throws Alex Osborne 2023-01-23 14:38:17 +09:00
• a34d9f3f5e Merge pull request #539 from krakan/dont-warn-for-non-authenticating-calls Alex Osborne 2023-01-23 14:44:16 +09:00
• de5c419d8f Don't log authentication failures when header is missing Jonas Linde 2022-11-16 11:15:50 +01:00
• 4bad4713ba Set correct timestamp for lastFailureTime Jonas Linde 2022-11-16 11:23:54 +01:00
• b9cb43e910 Update groovy to 3.0.13 for JDK 19 support Alex Osborne 2022-11-09 21:11:45 +09:00
• b0fb41e7b5 Update CI tests from JDK 18 to 19 Alex Osborne 2022-11-09 21:01:04 +09:00
• c60af1eba2 Fix typo Christian Clauss 2022-10-24 00:55:18 +02:00
• 7cd4a914c3 update com.itextpdf:itextpdf 5.5.0 to 5.5.12 claire9910 2022-10-15 22:19:49 +08:00
• b328ded92e docs: Add 'Checkpointing' section to operating guide Alex Osborne 2022-09-30 15:31:21 +09:00
• 3c79a0b2b1 Increase DiskSpaceMonitor default pause threshold to 8 GiB Alex Osborne 2022-09-10 23:17:53 +09:00
• f736bf2854 docs: Add DecideRuleSequence to bean-reference Alex Osborne 2022-09-06 18:02:20 +09:00
• c5d9bbd0c0 Fix changelog following tag merge. Andy Jackson 2022-07-28 10:58:48 +01:00
• e83f4f9e02 Reset to SNAPSHOT Andy Jackson 2022-07-28 10:56:59 +01:00
• 9278651b32 Merge tag '3.4.0-20220727' Andy Jackson 2022-07-28 10:53:55 +01:00
• a00a9618a5 Resync changelog. Andy Jackson 2022-07-28 09:06:02 +01:00
• 7c4453cf66 [maven-release-plugin] prepare for next development iteration Andy Jackson 2022-07-27 21:41:40 +01:00
• b0505662d7 [maven-release-plugin] prepare release 3.4.0-20220727 Andy Jackson 2022-07-27 21:36:19 +01:00
• 292d2998cb [maven-release-plugin] prepare release 3.4.0-20220727 3.4.0-20220727 Andy Jackson 2022-07-27 21:36:19 +01:00
• d17a6a8711 Merge pull request #493 from ukwa/master Andy Jackson 2022-07-27 21:26:06 +01:00
• b8b6c5e209 Update changelog. Andy Jackson 2022-07-27 21:24:33 +01:00
• 1a91a2c8c1 Bump jsch from 0.1.52 to 0.1.54 in /commons dependabot[bot] 2022-07-07 14:34:14 +00:00
• 46ea8e1047 Bump spring-core from 5.3.19 to 5.3.20 in /commons dependabot[bot] 2022-07-07 14:26:32 +00:00
• 82dee7c89c Bump jsch from 0.1.52 to 0.1.54 in /modules dependabot[bot] 2022-07-07 14:24:53 +00:00
• c1ba694add Document robotsTxtOnly robots policy in config guide Alex Osborne 2022-07-07 11:54:58 +09:00
• 1209b2d42f Add robotsTxtOnly robots policy Alex Osborne 2022-07-05 15:35:48 +09:00
• 09f75e389a docs/glossary: Add manifest (M) to discovery path Alex Osborne 2022-06-22 15:44:44 +09:00
• 1d490c779b Merge pull request #488 from netarchivesuite/crawluri-npe Alex Osborne 2022-06-08 13:35:23 +09:00
• 702bfa5170 Removed a potential NPE in hashCode method to CrawlURI which was fatal for heritrix Browse Beans functionality. csrster 2022-05-31 08:20:02 +02:00
• e8be86f7cf Bump gson from 2.8.6 to 2.8.9 in /contrib dependabot[bot] 2022-05-24 03:04:06 +00:00
• a35cd44a73 Docs: Add section to config guide about http and form credentials Alex Osborne 2022-05-24 12:01:56 +09:00
• 9644780a54 ExtractorHTML: Add --robots option to main() Alex Osborne 2022-05-11 16:31:48 +09:00
• d377652190 Merge branch 'master' into yt-dlp-for-h3 yt-dlp-for-h3 Barbara Miller 2022-05-10 16:11:34 -07:00
• 916aad631c use yt-dlp instead of youtube-dl Barbara Miller 2022-05-10 14:50:54 -07:00
• 677744e756 use yt-dlp instead of youtube-dl Barbara Miller 2022-05-10 14:50:54 -07:00
• 9bbc38239b use yt-dlp instead of youtube-dl yt-dlp-4-h3 Barbara Miller 2022-05-10 14:50:54 -07:00
• e1e7adf088 Bump spring-core from 5.3.18 to 5.3.19 in /commons dependabot[bot] 2022-04-27 05:12:23 +00:00
• 9fe64a6f0c ExtractorChrome: If disabled only connect the browser process on demand Alex Osborne 2022-04-27 13:20:08 +09:00
• 207adec12c Merge pull request #478 from internetarchive/srcset-fix Alex Osborne 2022-04-27 14:11:09 +09:00
• 914756f155 ExtractorHTML: Add a main() method to run the extractor standalone Alex Osborne 2022-04-22 16:43:37 +09:00
• aac74411a1 ExtractorHTML: Fix srcset by normalizing elementContext() to lowercase Alex Osborne 2022-04-22 16:26:04 +09:00
• 55078c078d Merge pull request #476 from ClemensRobbenhaar/issue211-dns-over-https Alex Osborne 2022-04-10 08:59:59 +09:00
• f5c83ce368 Add support for DNS-over-HTTPS lookups Clemens Klein-Robbenhaar 2022-04-09 23:32:55 +02:00
• ed332bcb9e Add support for DNS-over-HTTPS lookups Clemens Klein-Robbenhaar 2022-04-09 23:26:07 +02:00
• fdde6d87a0 Add support for DNS-over-HTTPS lookups Clemens Klein-Robbenhaar 2022-04-09 23:00:06 +02:00
• 9648c98c77 [misc] rst formatting Clemens Klein-Robbenhaar 2022-04-08 12:14:11 +02:00
• 2144abb2cb Add support for DNS-over-HTTPS lookups Clemens Klein-Robbenhaar 2022-04-08 11:37:39 +02:00
• 8557533d11 ChromeWindow: Ignore WebsocketNotConnectedException in window.close() Alex Osborne 2022-04-05 23:14:57 +09:00
• adc68583cc ChromeWindow: Ignore events once we've requested the window be closed Alex Osborne 2022-04-05 22:46:47 +09:00
• e5d8d53993 ExtractorChrome: Only run on pages with 2xx status codes Alex Osborne 2022-04-05 22:45:03 +09:00
• 42ac612e1a ExtractorChrome: Add a counter to give each recorder a unique basename Alex Osborne 2022-04-05 22:13:17 +09:00
• f2b3722b0b ChromeProcess: Disable features that prompt or cause background traffic Alex Osborne 2022-04-05 17:29:57 +09:00
• e5bff6a06a ExtractorChrome: Add commandLineOptions property Alex Osborne 2022-04-05 17:00:04 +09:00
• 0378e0d123 ChromeProcess: Use the --disable-crash-reporter command-line option Alex Osborne 2022-04-05 16:40:52 +09:00
• f2af65122c ChromeProcess: Add macOS default install path to DEFAULT_EXECUTABLES Alex Osborne 2022-04-05 16:32:20 +09:00
• ce0b5e3a24 Bump spring-beans from 5.3.14 to 5.3.18 in /commons dependabot[bot] 2022-04-06 01:00:29 +00:00
• 0698545eec JDK18 support: Run GitHub CI tests on JDK 18 Alex Osborne 2022-04-01 11:33:50 +09:00