Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#750

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• b60ae999cc Merge pull request #399 from ukwa/master Andy Jackson 2021-06-17 21:23:39 +01:00
• 2007ddb005 Switch to properties that enforce Java 8 compatibility. Andrew Jackson 2021-06-17 21:22:04 +01:00
• 18e8779398 Updated changelog. Andrew Jackson 2021-06-17 16:42:23 +01:00
• 3423cab6a6 [maven-release-plugin] prepare for next development iteration Andrew Jackson 2021-06-17 14:39:00 +01:00
• 7368ded9c2 [maven-release-plugin] prepare release 3.4.0-20210617 3.4.0-20210617 Andrew Jackson 2021-06-17 14:38:54 +01:00
• a5164dbcea Update changelog. Andrew Jackson 2021-06-17 14:34:22 +01:00
• 998eaa3ed8 Merge branch 'master' into upgrade-httpclient-4-5 Andrew Jackson 2021-06-17 14:32:43 +01:00
• e07b1bb938 Support cookie expiration and extend testing to cover that case. Andrew Jackson 2021-06-17 14:23:42 +01:00
• 2d64e005cb Switch to standard cookie handling. Andrew Jackson 2021-06-17 14:23:20 +01:00
• 1d1d2f5abe Avoid two slf4j implementations. Andrew Jackson 2021-06-17 14:19:12 +01:00
• 4cd416a5e4 Update complier plugin and use compatible flip(). Andrew Jackson 2021-06-17 14:01:04 +01:00
• 4b82d6060c Merge pull request #398 from kris-sigur/sitemap-tweak Andy Jackson 2021-06-17 10:10:58 +01:00
• 81bef22270 Mark nested sitemap links as sitemaps Kristinn Sigurdsson 2021-06-16 13:07:18 +00:00
• d892026731 Expire cookies when requested. Andrew Jackson 2021-06-15 00:31:04 +01:00
• 81fa60d870 Upgrade HTTPClient to 4.5.x for #245 Andrew Jackson 2021-06-14 23:35:46 +01:00
• d0221642c3 Update changelog. Andrew Jackson 2021-06-14 11:04:33 +01:00
• 09c8d5696b Use HTTPS version for Oracle Maven repo. Andrew Jackson 2021-06-14 11:03:32 +01:00
• c403d7b913 Merge pull request #395 from internetarchive/adds-trough-dedup-performance-rework Adam Miller 2021-05-28 16:15:22 -07:00
• 58bf0cf45c Refactor Trough client URL cache put. Limit TroughContentDigestHistory query. adds-trough-dedup-performance-rework Adam Miller 2021-05-28 23:10:11 +00:00
• 169b3cc8ab Merge pull request #393 from ukwa/checkpoint-success-stats-392 Andy Jackson 2021-05-28 14:20:06 +01:00
• 699f107ccd Merge pull request #394 from ukwa/update-ampq-client Andy Jackson 2021-05-28 13:24:23 +01:00
• 921e349939 Update AMPQ client library to address security warning. Andy Jackson 2021-05-28 12:47:59 +01:00
• 5caa83d327 Only update last checkpoint stats if the checkpoint completed, for #392. Andy Jackson 2021-05-28 12:42:17 +01:00
• 773b4a074d Merge pull request #391 from ukwa/master Andy Jackson 2021-05-27 10:27:40 +01:00
• a87937caf9 Sync changelog with release. Andy Jackson 2021-05-27 10:26:50 +01:00
• d3dfb34f83 [maven-release-plugin] prepare for next development iteration Andrew Jackson 2021-05-27 10:03:10 +01:00
• 9dcbb240a6 [maven-release-plugin] prepare release 3.4.0-20210527 3.4.0-20210527 Andrew Jackson 2021-05-27 10:03:04 +01:00
• cf0f4ef144 Merge pull request #390 from ukwa/master Andy Jackson 2021-05-27 09:46:41 +01:00
• 9df4a0dc05 Update changelog. Andy Jackson 2021-05-27 09:45:47 +01:00
• 3b4c442683 Merge pull request #389 from ukwa/update-dependencies-2021-05-26 Andy Jackson 2021-05-26 21:56:21 +01:00
• 669cf1ed2b Merge branch 'master' into update-dependencies Andy Jackson 2021-05-26 13:40:15 +01:00
• a2c65f4618 Update dependencies for security issues, including changes to tests. Andy Jackson 2021-05-26 13:39:30 +01:00
• d175040e33 Merge pull request #386 from ukwa/master Andy Jackson 2021-05-20 22:50:43 +01:00
• 400677a46a Merge branch 'master' of github.com:internetarchive/heritrix3 Andrew Jackson 2021-05-20 22:49:05 +01:00
• 7c72b8d584 Updated changelog. Andrew Jackson 2021-05-20 22:47:56 +01:00
• 8c59cdfbad Merge pull request #385 from internetarchive/adds-trough-dedup-performance-rework Adam Miller 2021-05-20 14:45:40 -07:00
• 7fc648cad4 Add dedup load from in-memory dedup cache db Adam Miller 2021-05-20 21:37:11 +00:00
• d9fe123b2f Merge pull request #384 from ukwa/configure-bdb-je-cleaner-threads Andy Jackson 2021-05-20 22:24:42 +01:00
• 276ced0de3 Allow tuning of BDB-JE evictor and cleaner threads. Andy Jackson 2021-05-20 22:22:00 +01:00
• c0dbfb2a0a Merge pull request #383 from ukwa/upgrade-dnsjava Andy Jackson 2021-05-20 21:05:59 +01:00
• d7869de8d9 Merge pull request #262 from kris-sigur/sitemaps Andy Jackson 2021-05-20 21:05:13 +01:00
• 396467c535 Merge branch 'master' into sitemaps Andy Jackson 2021-05-20 21:03:54 +01:00
• c1bcdd9f9e Merge pull request #341 from internetarchive/noplaylist-ydl Andy Jackson 2021-05-20 21:01:39 +01:00
• b2a07b290f Merge pull request #348 from internetarchive/fixes-leaky-file-handles Andy Jackson 2021-05-20 21:00:31 +01:00
• c7b1dfa198 Merge pull request #368 from webdev4422/patch-1 Andy Jackson 2021-05-20 20:58:41 +01:00
• 045b2516da Update to latest version of dnsjava, for #344 Andy Jackson 2021-05-20 20:44:19 +01:00
• 1b9565ef45 Merge pull request #382 from internetarchive/adds-trough-dedup-performance-rework Adam Miller 2021-05-17 12:45:56 -07:00
• 6fad293759 Ensure we set dedup schema when writing dedup shards Adam Miller 2021-05-17 19:41:27 +00:00
• d1026bdc55 Merge pull request #381 from internetarchive/adds-trough-dedup-performance-rework Adam Miller 2021-05-13 10:39:20 -07:00
• fc402acd06 Merge branch 'ait-qa' into adds-trough-dedup-performance-rework Adam Miller 2021-05-13 10:39:02 -07:00
• 18edb3e413 Refactor crawl log batch posting. Add configuration options for trough dedup batches. Fix posting of trough dedup at end of crawl. Adam Miller 2021-05-13 17:32:26 +00:00
• a09479364a Merge pull request #379 from ldko/bean-url-missing-fix Alex Osborne 2021-05-12 09:31:27 +09:00
• 065e62fc1f Merge pull request #377 from internetarchive/bean-browser-optional-fix Alex Osborne 2021-05-12 09:31:04 +09:00
• ac33e2ef55 Avoid error when bean properties have no url available Lauren Ko 2021-05-11 16:42:33 -05:00
• 71b3065172 Handle empty Optionals when browsing beans Alex Osborne 2021-05-11 21:55:03 +09:00
• 145d46dd8d Remove change to helper class as it can be done in schema Phani Dharmavarapu 2021-04-21 11:49:17 -04:00
• 52093eb25f Only copy source tag if not null. Kristinn Sigurdsson 2021-04-20 13:18:29 +00:00
• 39e9e72b31 Use seconds from the FetchCompletedTime for LastModified time Phani Dharmavarapu 2021-04-14 13:09:38 -04:00
• 4625a78a2c Merge pull request #374 from internetarchive/adds-trough-dedup-performance-rework Adam Miller 2021-04-05 16:49:50 -07:00
• e353ccb097 Merge branch 'ait-qa' into adds-trough-dedup-performance-rework Adam Miller 2021-04-05 16:17:47 -07:00
• 21e84ad015 Merge pull request #373 from internetarchive/revert-367-fixes-trough-crawl-log-feed-synchronization Adam Miller 2021-04-05 16:01:51 -07:00
• d6eb68a265 Revert "Fix error log reporting of batch size for trough crawl logs" revert-367-fixes-trough-crawl-log-feed-synchronization Adam Miller 2021-04-05 16:01:27 -07:00
• 0b09340e84 Fix inconsistency with cached batch sizes and refine logic for forcing final batch posting at end of crawl Adam Miller 2021-04-05 22:56:23 +00:00
• d72175635e Ensure that the final batch is posted during shutdown. Make posting of uncrawled urls optional. Adam Miller 2021-04-05 21:06:56 +00:00
• 466c6795b3 Merge branch 'fixes-trough-crawl-log-feed-synchronization' into adds-trough-dedup-performance-rework Adam Miller 2021-03-10 21:36:33 +00:00
• 54abea3157 Merge branch 'adds-trough-local-dedup-write-cache' into adds-trough-dedup-performance-rework Adam Miller 2021-03-10 21:36:24 +00:00
• 8724b14a9d Replacing some synchronized lists with threadsafe objects and fixing discrepency between crawled and uncrawled lists fixes-trough-crawl-log-feed-synchronization Adam Miller 2021-03-10 21:21:59 +00:00
• 12eef6c70d Merge pull request #367 from internetarchive/fixes-trough-crawl-log-feed-synchronization jkafader 2021-02-26 15:41:03 -08:00
• e700f384d5 Switching to in memory sqlite for dedup cache adds-trough-local-dedup-write-cache Adam Miller 2021-02-26 23:32:20 +00:00
• 5ec7fb87d9 Fix misspell in comments webdev4422 2021-02-26 08:50:13 +02:00
• 88b7eeaf02 On disk - per-seed sqlite dedup Adam Miller 2021-02-25 22:35:08 +00:00
• 00947b0167 Fixing sync issues on uncrawled urls batch as well Adam Miller 2021-02-25 22:31:25 +00:00
• 37ce8d6945 Merge pull request #366 from ato/upstream-spring-5.3.3 master-it Alex Osborne 2021-02-23 15:07:59 +09:00
• e30005128d Merge branch 'adds-trough-local-dedup-write-cache' into adds-trough-dedup-performance-rework Adam Miller 2021-02-23 00:44:34 +00:00
• 09485a5dcd Merge branch 'fixes-trough-crawl-log-feed-synchronization' into adds-trough-dedup-performance-rework Adam Miller 2021-02-23 00:44:23 +00:00
• 86e238e7f7 Create and insert into local sqlite dedup db shards Adam Miller 2021-02-23 00:43:26 +00:00
• 79e56337b2 Fix error log reporting of batch size for trough crawl logs Adam Miller 2021-02-18 23:06:51 +00:00
• 35bb3286f4 Clarify robots protocol support Alex Osborne 2021-02-15 17:29:49 +09:00
• b10b33821f Fix broken robots.txt link Alex Osborne 2021-02-15 17:12:04 +09:00
• 5f52a14afc Upgrade to Spring 5.3.3 Alex Osborne 2021-02-12 20:38:01 +09:00
• 5a64141302 Merge pull request #352 from BitBaron/absolute-meta-urls Alex Osborne 2021-02-15 15:48:50 +09:00
• ba0d51882a Merge pull request #359 from internetarchive/ait-ydl-updates Alex Osborne 2021-02-15 15:41:48 +09:00
• d22ef124d3 Trying to avoid hard locks on updating url caches fixes-trough-client-synchronization Adam Miller 2021-02-02 23:58:48 +00:00
• 6a8b754ba7 Merge branch 'log_sleep' into ait-qa Barbara Miller 2021-02-01 21:24:54 -08:00
• 7e82722da3 log sleep Barbara Miller 2021-02-01 21:24:24 -08:00
• 5e7a20d19a Merge pull request #364 from internetarchive/fixes-trough-crawl-log-feed-synchronization jkafader 2021-01-28 15:31:53 -08:00
• 14be8752e9 Fixing synchronization issues in TroughCrawlLogFeed.java to reduce blocking Adam Miller 2021-01-28 22:54:06 +00:00
• 77e2e9171f Merge pull request #362 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats jkafader 2021-01-14 12:08:10 -08:00
• 2f640fc974 Merge pull request #355 from internetarchive/fixes-prerequisite-status jkafader 2021-01-14 12:08:01 -08:00
• 147467c326 Merge branch 'fix_restlet_npe' into ait-qa Barbara Miller 2021-01-12 18:50:16 -08:00
• b5b204b40a avoid NPE https://webarchive.jira.com/browse/WT-28?focusedCommentId=139095 Barbara Miller 2021-01-04 16:16:39 -08:00
• ff13363c7d avoid NPE https://webarchive.jira.com/browse/WT-28?focusedCommentId=139095 restlet_npe Barbara Miller 2021-01-04 16:16:39 -08:00
• 0649751456 --no-cache-dir, --no-playlist ait-ydl-updates Barbara Miller 2021-01-04 17:53:46 -08:00
• 6ff10be094 Merge pull request #357 from internetarchive/fixes-prerequisite-status Adam Miller 2020-11-12 15:46:16 -08:00
• 6ebb28db18 Merge pull request #356 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats Adam Miller 2020-11-12 15:20:14 -08:00
• 1f357c8daf Fixing race condition on totalBytesWritten fixes-WARCWriterChainProcessor-checkpoint-stats Adam Miller 2020-11-11 00:02:09 +00:00
• e694a4b5c8 Reset CrawlURI status for hasPrerequisite() so that it isn't preserved between attempts fixes-prerequisite-status Adam Miller 2020-11-07 01:12:56 +00:00
• 288b167399 Merge pull request #354 from internetarchive/fixes-WARCWriterChainProcessor-checkpoint-stats jkafader 2020-10-15 13:51:09 -07:00
• 85052d6cb2 Merge branch 'fixes-WARCWriterChainProcessor-checkpoint-stats' into ait-qa Adam Miller 2020-10-09 22:14:23 +00:00
• 5504d492a1 Add checkpoint store/load functionality for warc writer stats Adam Miller 2020-10-09 22:14:08 +00:00