Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#781/merge

#782

#783

#784

#785

#786

#787

#787/merge

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• c3f140b9b5 JDK18 support: Update Groovy to 3.0.10 Alex Osborne 2022-04-01 11:32:56 +09:00
• 971f04dd23 TransclusionDecideRule: Don't treat sitemap links ('M') as transclusions Alex Osborne 2022-03-09 09:37:08 +09:00
• d796491995 FetchHTTPTest: Fix SOCKS server startup race Alex Osborne 2022-03-28 13:26:53 +09:00
• 0dc05408da FetchHTTPTest: Don't test IP binding on macOS utun network interfaces Alex Osborne 2022-03-28 11:40:48 +09:00
• 0d6c04fda5 FetchHTTPTest: Update to junit 4 Alex Osborne 2022-03-28 11:35:45 +09:00
• dd01689ace FetchHTTP: SOCKS5 proxy support Jamie Hoyle 2022-03-28 10:23:55 +09:00
• 69f231b8ab Merge pull request #468 from internetarchive/remove-jna Alex Osborne 2022-03-03 17:16:34 +09:00
• c6d4251e4f Merge pull request #459 from internetarchive/parse-link-rel Alex Osborne 2022-03-03 17:16:22 +09:00
• 98f42de863 Merge pull request #460 from internetarchive/dependabot/maven/commons/org.springframework-spring-core-5.3.14 Alex Osborne 2022-03-03 17:14:46 +09:00
• 71b40cf882 Use Files.createLink() and Files.createSymbolicLink() instead of JNA Alex Osborne 2022-03-02 21:25:02 +09:00
• 568d5cedfe Merge pull request #461 from kris-sigur/seed-redirect-tld Alex Osborne 2022-02-14 07:46:09 +09:00
• 4a75fe1fea Merge pull request #465 from ClemensRobbenhaar/issue464 Alex Osborne 2022-02-14 07:45:15 +09:00
• fae2ccb88d Fix name of parameter in setMaxLogFileSize Clemens Klein-Robbenhaar 2022-02-13 13:01:58 +01:00
• 1c4831d6e9 JerichoExtractorHTML: Determine LINK tag type by parsing REL attribute Alex Osborne 2022-01-25 17:57:32 +09:00
• ab27a48e26 Bump spring-core from 5.3.3 to 5.3.14 in /commons dependabot[bot] 2022-01-27 03:03:08 +00:00
• dcf9b04007 ExtractorHTML: Determine LINK tag type by parsing REL attribute Alex Osborne 2022-01-25 17:57:32 +09:00
• d40a025692 Merge pull request #452 from internetarchive/handle-nulls-in-content-length-header Alex Osborne 2022-01-17 16:20:12 +09:00
• aa4f0f9933 Merge pull request #457 from ClemensRobbenhaar/issues191-https-over-proxy Alex Osborne 2022-01-17 16:18:13 +09:00
• 261fa318b4 FetchHTTPRequest: handle HTTPS requests via proxy Clemens Klein-Robbenhaar 2022-01-10 08:57:22 +01:00
• c4edadf1a8 Configuration guide: clarify usage of ContentTypeMatchesRegexDecideRule Oliver Schihin 2021-12-21 09:33:32 +01:00
• a3d651ba46 FetchHTTP: Handle null characters in the Content-Length header Alex Osborne 2021-12-13 15:41:38 +09:00
• 140211e8c2 Link to the structured documentation in the README Alex Osborne 2021-12-13 14:20:33 +09:00
• 52733b9dc0 README: Fix javadoc badge and add docker badge Alex Osborne 2021-12-13 14:15:21 +09:00
• 65c1025546 Make the docker build manually triggered Alex Osborne 2021-12-13 14:06:25 +09:00
• b73a8d0356 docker ci: set working-directory Alex Osborne 2021-12-13 14:01:45 +09:00
• bad8bbd1ee Move docker-image.yml to .github/workflows Alex Osborne 2021-12-13 13:58:55 +09:00
• 31c9378888 Create docker-image.yml Alex Osborne 2021-12-13 00:58:43 +09:00
• 0f9109ba9f Merge pull request #450 from Querela/360-docker Alex Osborne 2021-12-13 00:50:20 +09:00
• 3a5b758d5e Docs (temp) Erik Körner 2021-12-10 16:39:17 +01:00
• 17c9694039 Add Dockerfile Erik Körner 2021-12-10 15:35:03 +01:00
• d83e799ad9 Merge pull request #448 from ldko/resolve-gid-too-big Alex Osborne 2021-12-04 13:23:21 +09:00
• a0f0ba45e5 Fix files included by assembly Lauren Ko 2021-12-03 16:05:59 -06:00
• 256f8742e9 Fix group id is too big Lauren Ko 2021-12-03 12:20:32 -06:00
• d11d02eeac docs: Add 'Loading Cookies' section to config guide Alex Osborne 2021-11-29 12:11:54 +09:00
• d63cb9af8b Merge pull request #444 from internetarchive/fix-dnsjava-selector-thread Alex Osborne 2021-11-27 12:41:11 +09:00
• c46cfd1c31 Merge pull request #441 from netarchivesuite/iipc-master Alex Osborne 2021-11-02 10:34:26 +09:00
• 1832c5ef89 FetchDNS: Keep dnsjava selector thread out of ToePool Alex Osborne 2021-11-01 15:56:46 +09:00
• 513c5fc487 FetchDNS: Move DNS cache init from CrawlController Alex Osborne 2021-11-01 15:40:29 +09:00
• 69b023199d Add conf to not allow TLDs as seeds found via redirect from other seeds Kristinn Sigurdsson 2021-10-21 09:39:19 +00:00
• 6a5c629cd2 Merge branch 'master' into fix-last-modified-millis Phani Dharmavarapu 2021-10-08 16:21:50 -04:00
• 17d760aa78 Remove obsolete Java < 1.6 version check Alex Osborne 2021-09-30 09:59:28 +09:00
• 698341a1d6 Github action: test against JDK 17 (LTS) instead of 16 Alex Osborne 2021-09-30 09:10:55 +09:00
• c373c8e95e Re-sync changelog for 20210923 Andrew Jackson 2021-09-23 23:58:54 +01:00
• 024a7772b5 [maven-release-plugin] prepare for next development iteration Andrew Jackson 2021-09-23 22:08:33 +01:00
• fedff1e2c7 [maven-release-plugin] prepare release 3.4.0-20210923 3.4.0-20210923 Andrew Jackson 2021-09-23 22:08:30 +01:00
• 16ab42973b Resync changelog. Andrew Jackson 2021-09-23 22:03:51 +01:00
• 2101a8c0f3 Merge pull request #434 from ukwa/safer-cookie-iteration Andy Jackson 2021-09-23 21:35:23 +01:00
• c1b1b11163 SECURITY.md: Use HTTPS for PANDORA contact link Alex Osborne 2021-09-11 11:13:42 +09:00
• 5fcb9f4084 SECURITY.md: add mising 'the' Alex Osborne 2021-09-11 10:59:49 +09:00
• 46d7e06bce Create SECURITY.md Alex Osborne 2021-09-11 10:51:27 +09:00
• 780abb0b1d docs: Update AdoptOpenJDK link as it has been rebranded Adoptium Alex Osborne 2021-09-11 07:53:18 +09:00
• cdfe1b32b4 Adopt is no more, moving to Temurin Geertjan Wielenga 2021-09-10 14:28:45 +02:00
• 3f4535a933 Add safer cookie iteration, as per #427. Andy Jackson 2021-09-06 11:05:12 +01:00
• cf75291024 add unit tests for CollectdSocketWriter.Client [WIP] collectd-socket-stats-writer asmr-hex 2021-08-24 17:26:37 -04:00
• 47eebbc765 Merge pull request #431 from internetarchive/extractor-chrome-bug-fixes Alex Osborne 2021-08-19 08:32:49 +09:00
• 2dbc664e73 introduce CollectdSocketWriter class for polled stats writing [WIP] asmr-hexandAdam Miller 2021-08-18 16:31:41 -04:00
• 3346428bf2 ExtractorChrome: Warn instead of throwing when response headers missing Alex Osborne 2021-08-17 09:20:35 +09:00
• e93e457e19 ChromeWindow: Handle raw headers or headersText being unavailable Alex Osborne 2021-08-17 09:14:59 +09:00
• 6b19daf1d2 ExtractorChrome: Don't capture data: URIs Alex Osborne 2021-08-17 09:07:04 +09:00
• cd30fbcc3c Merge pull request #424 from internetarchive/ui-cleanup Alex Osborne 2021-08-09 14:41:28 +09:00
• c34cb65683 ChromeClient: increase RPC timeout from 10 to 60 seconds Alex Osborne 2021-08-09 14:37:45 +09:00
• 0d46239c2d Update changelog following release. Andrew Jackson 2021-08-03 10:40:01 +01:00
• ddb695abad [maven-release-plugin] prepare for next development iteration Andrew Jackson 2021-08-03 09:56:26 +01:00
• ba44cce7f2 [maven-release-plugin] prepare release 3.4.0-20210803 3.4.0-20210803 Andrew Jackson 2021-08-03 09:56:20 +01:00
• 0b48699948 Update changelog Andrew Jackson 2021-08-03 09:53:30 +01:00
• eb67cfb215 Merge pull request #423 from internetarchive/dont-extract-data-uris Alex Osborne 2021-08-03 07:43:17 +09:00
• 8fe92c3437 Update changlog. Andrew Jackson 2021-08-02 13:36:45 +01:00
• 4d5b2dc3fe Merge pull request #397 from ukwa/upgrade-httpclient-4-5 Andy Jackson 2021-08-02 09:19:30 +01:00
• ed13a2b0c5 UI: Pull duplicate getEngine() methods up to BaseResource Alex Osborne 2021-08-01 16:02:15 +09:00
• 2719d022d5 UI: Refactor duplicated template code into a common render() helper Alex Osborne 2021-08-01 15:50:37 +09:00
• 4233a930f0 UI: Use a single instance of Freemarker for the whole application Alex Osborne 2021-08-01 14:59:27 +09:00
• d72d847d92 Merge pull request #421 from internetarchive/toe-thread-interrupt-fix Alex Osborne 2021-07-31 16:21:14 +09:00
• 82b4360962 Merge pull request #418 from internetarchive/fix-keytool-on-jdk16 Alex Osborne 2021-07-31 16:20:45 +09:00
• 33a5d70209 ExtractorSitemap: Use logUriError() helper like other extractors Alex Osborne 2021-07-30 15:33:50 +09:00
• 1032c0ec40 ExtractorHTML: Avoid allocating strings for data: URIs when possible Alex Osborne 2021-07-30 15:22:38 +09:00
• e33e1ff398 Extractor: ignore data URIs when adding outlinks Alex Osborne 2021-07-30 14:41:02 +09:00
• 91ef23eddf ExtractorPDFContext, ExtractorYoutubeDL: use addOutlink() helper method Alex Osborne 2021-07-30 14:24:10 +09:00
• 466c10b7a4 Merge pull request #416 from internetarchive/extractor-chrome-replay-responses Alex Osborne 2021-07-27 20:26:46 +09:00
• 721e89ca10 ToeThread: ensure currentCuri is finished before exiting Alex Osborne 2021-07-26 15:26:09 +09:00
• f0d9385120 GitHub actions: run test suite on JDK 16 too Alex Osborne 2021-07-25 11:22:09 +09:00
• f4bd5d1015 Upgrade Groovy to latest stable version (3.0.8) for JDK 16 compatibility Alex Osborne 2021-07-25 14:09:37 +09:00
• 636948cb09 KeyTool wrapper: fallback to running keytool as a subprocess on JDK 16+ Alex Osborne 2021-07-25 11:15:39 +09:00
• 014fb2dcad ExtractorChrome: have frontier consider browser-fetched uris included Alex Osborne 2021-07-24 17:25:21 +09:00
• bbae794b47 ExtractorChrome: run extractors on subresources captured by the browser Alex Osborne 2021-07-24 16:27:28 +09:00
• 8ec7ce3363 ExtractorChrome: replay the recorded CrawlURI response to the browser Alex Osborne 2021-07-23 17:38:56 +09:00
• ab19efa9d6 Merge pull request #414 from internetarchive/maven-assembly-plugin-3.3.0 Alex Osborne 2021-07-22 11:21:35 +09:00
• d2d9d4f157 Adding configurable MaxLogFileSize option to BdbModule Adam Miller 2021-07-13 21:29:33 +00:00
• 0dfb119df2 Upgrade maven-assembly-plugin to 3.3.0 to fix file permissions Alex Osborne 2021-07-19 15:37:46 +09:00
• d2d8c490bc Merge pull request #412 from internetarchive/adds-bdbmodule-max-file-size-option Adam Miller 2021-07-16 10:21:56 -07:00
• 523200ada6 Enabled configurable url-matching and extraction for sitemaps. csrster 2021-07-15 08:45:49 +02:00
• 953e7c9399 Adding configurable MaxLogFileSize option to BdbModule adds-bdbmodule-max-file-size-option Adam Miller 2021-07-13 21:29:33 +00:00
• ea42630608 docs: Add decide rules to bean reference Alex Osborne 2021-07-13 21:49:27 +09:00
• 09e3ff3568 Everywhere: Fix dangling JavaDoc comments Alex Osborne 2021-07-12 21:38:48 +09:00
• 2a027a4064 docs: Fallback to field javadoc comments when setter javadoc unavailable Alex Osborne 2021-07-12 19:30:56 +09:00
• 812ea0f720 Merge pull request #411 from internetarchive/chrome-request-capturing Alex Osborne 2021-07-12 17:49:31 +09:00
• c54f18e3ff docs: Add most of the default config beans to the bean reference Alex Osborne 2021-07-12 11:39:06 +09:00
• 4dfe48b1a0 docs: Strip @link and @code javadoc directives from bean reference Alex Osborne 2021-07-12 09:28:04 +09:00
• 4a5fdcfaf0 docs: Add remaining link extractor to the bean reference Alex Osborne 2021-07-12 09:23:08 +09:00
• d10be35f0c docs: Remove duplicate word 'Documentation' from page titles Alex Osborne 2021-07-12 09:10:57 +09:00
• 63bd15535d docs: Fix links accidentally using Markdown syntax Alex Osborne 2021-07-12 09:09:17 +09:00