Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#750

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• adac067ea7 Update link to robots.txt meta documentation Kristinn Sigurðsson 2020-09-02 12:15:02 +00:00
• 2505effadc Strip quotes from URL value. Neil Minton 2020-08-06 19:23:50 -04:00
• 87ac0dacb2 Merge pull request #349 from BitBaron/absolute-meta-urls jkafader 2020-08-17 12:38:02 -07:00
• 13a1028e2f Merge branch 'absolute-meta-urls' into ait-qa Neil Minton 2020-08-14 10:59:17 -04:00
• 941c6c50ad Strip quotes from URL value. Neil Minton 2020-08-06 19:23:50 -04:00
• 6b1e88934a Merge branch 'absolute-meta-urls' into ait-qa Neil Minton 2020-08-12 13:16:30 -04:00
• 5cefd18357 Strip quotes from URL value. Neil Minton 2020-08-12 13:16:14 -04:00
• 09884d1f57 Merge branch 'absolute-meta-urls' into ait-qa Neil Minton 2020-08-11 14:02:59 -04:00
• 36f7aea625 Enable parsing of absolute URLs in meta HTML tags. Neil Minton 2020-08-06 19:23:50 -04:00
• 5b014e7971 Merge branch 'absolute-meta-urls' into ait-qa Neil Minton 2020-08-06 19:38:11 -04:00
• 0dd65dfd0b Enable parsing of absolute URLs in meta HTML tags. Neil Minton 2020-08-06 19:23:50 -04:00
• f879226c8e Merge pull request #342 from internetarchive/noplaylist-ydl Adam Miller 2020-08-03 16:40:54 -07:00
• 34bb996e5c Merge pull request #346 from internetarchive/fixes-leaky-file-handles jkafader 2020-08-03 16:22:23 -07:00
• 46100d65b2 Merge pull request #347 from internetarchive/fixes-leaky-file-handles Adam Miller 2020-07-31 15:27:09 -07:00
• a9c0c6588b Manage youtube dl temp files which can be closed in the warc writer. fixes-leaky-file-handles Adam Miller 2020-07-31 22:22:25 +00:00
• ab27ff93e9 Merge pull request #345 from internetarchive/fixes-leaky-file-handles jkafader 2020-07-30 16:37:03 -07:00
• 52a8f345ed Fixing up logging and comments Adam Miller 2020-07-30 23:30:08 +00:00
• 00d1c46d60 Ensure Replay Input Stream and File Channels are closed after writing Adam Miller 2020-07-30 23:19:43 +00:00
• 1a50ff538b Merge branch 'noplaylist-ydl' into ait-qa Barbara Miller 2020-07-29 11:15:02 -07:00
• 1b8e7f73ca youtube-dl --no-playlist noplaylist-ydl Barbara Miller 2020-07-29 11:12:33 -07:00
• 33a3f9a58f Updated changelog based on master. Andrew Jackson 2020-06-01 14:00:59 +01:00
• 2e6b76d517 Merge pull request #333 from dengliming/patch-1 Alex Osborne 2020-06-01 14:05:02 +09:00
• c4644d4e5c Merge pull request #335 from internetarchive/fixes-extractor-multiple-regex-matcher-recycle Alex Osborne 2020-06-01 14:03:11 +09:00
• 812e96d2c0 Merge pull request #336 from internetarchive/revert-334-fixes-ftp-response-record-builder Alex Osborne 2020-06-01 13:56:37 +09:00
• 947507b8ec Revert "Warc convention for storing ftp responses has been to use a WARC reso…" Alex Osborne 2020-06-01 13:56:12 +09:00
• 4763a8a04b Merge pull request #334 from internetarchive/fixes-ftp-response-record-builder Alex Osborne 2020-06-01 13:54:59 +09:00
• f34684d7c6 Merge branch 'ait-qa' of github.com:internetarchive/heritrix3 into ait-qa Barbara Miller 2020-05-29 18:00:14 -07:00
• aea704c9b7 Merge branch 'ffmpeg' into ait-qa Barbara Miller 2020-05-29 17:59:47 -07:00
• c0f57a5f54 use ffmpeg for segmented downloads Barbara Miller 2020-05-29 17:56:56 -07:00
• 60daef90b2 Merge branch 'master' into fixes-extractor-multiple-regex-matcher-recycle fixes-extractor-multiple-regex-matcher-recycle Adam Miller 2020-05-20 16:46:14 +00:00
• c0fe2064a1 Merge branch 'fixes-ftp-response-record-builder' Adam Miller 2020-05-20 16:32:36 +00:00
• 87c349fe13 Re-sync change log. Andrew Jackson 2020-05-18 15:28:56 +01:00
• e215b89963 [maven-release-plugin] prepare for next development iteration Andrew Jackson 2020-05-18 14:08:12 +01:00
• ef38dc08c3 [maven-release-plugin] prepare release 3.4.0-20200518 3.4.0-20200518 Andrew Jackson 2020-05-18 14:08:04 +01:00
• 176f796beb Updated changelog. Andrew Jackson 2020-05-18 14:05:26 +01:00
• b4962d25a8 Remove deprecated sudo setting. dengliming 2020-05-12 22:53:40 +08:00
• f9c4057074 Merge pull request #331 from internetarchive/fixes-ftp-response-record-builder Neil Minton 2020-05-06 12:03:34 -04:00
• 0db661fb36 Warc convention for storing ftp responses has been to use a WARC resource record and not a response record. Changing to be consisten with the default WARCWriterProcessor Adam Miller 2020-05-05 23:09:53 +00:00
• b84d443f00 Merge pull request #328 from morokosi/fix-matchesregexdeciderule Alex Osborne 2020-05-01 12:56:44 +09:00
• ce409c03c1 Merge pull request #329 from internetarchive/ydl-no-cache jkafader 2020-04-30 15:03:23 -07:00
• 0a5d46d927 Merge pull request #330 from internetarchive/less_youtubedl jkafader 2020-04-30 14:31:18 -07:00
• 3a8236a836 Merge branch 'less_youtubedl' into ait-qa Barbara Miller 2020-04-30 11:53:01 -07:00
• aabd16b41e don't youtube-dl receivedFromAMQP less_youtubedl Barbara Miller 2020-04-23 16:28:21 -07:00
• 13ba0b2b67 optimize imports morokosi 2020-04-30 22:29:50 +09:00
• 18bee39a4c fix discarded future value morokosi 2020-04-30 22:27:33 +09:00
• ba9bdcea7d Merge branch 'less_youtubedl' into ait-qa Barbara Miller 2020-04-29 13:34:37 -07:00
• 5cbeb1d5dd try skipping all receivedFromAMQP Barbara Miller 2020-04-29 13:33:38 -07:00
• 147b56dd8f Merge branch 'ydl-no-cache' into ait-qa Barbara Miller 2020-04-27 18:16:28 -07:00
• 633a7cffea no youtube-dl cache dir ydl-no-cache Barbara Miller 2020-04-27 18:15:54 -07:00
• 197c7f900b Merge branch 'less_youtubedl' into ait-qa Barbara Miller 2020-04-24 17:45:52 -07:00
• 1ffe0bc8c6 NPEs begone! Barbara Miller 2020-04-24 17:45:34 -07:00
• b39b68ae57 Merge branch 'less_youtubedl' into ait-qa Barbara Miller 2020-04-24 16:48:54 -07:00
• 9c9da328d5 logging Barbara Miller 2020-04-24 16:47:13 -07:00
• 5aa395333d Merge branch 'less_youtubedl' into ait-qa Barbara Miller 2020-04-23 17:53:20 -07:00
• cf57532dbd try again Barbara Miller 2020-04-23 17:35:01 -07:00
• 479d3b8e5e Merge branch 'less_youtubedl' into ait-qa Barbara Miller 2020-04-23 16:46:12 -07:00
• 5849b2ecf5 fix typos Barbara Miller 2020-04-23 16:45:55 -07:00
• c946cdba80 Merge branch 'less_youtubedl' into ait-qa Barbara Miller 2020-04-23 16:28:55 -07:00
• 0bec2ca59a don't youtube-dl seeds twice Barbara Miller 2020-04-23 16:28:21 -07:00
• 7764f9265f Merge pull request #320 from bnfleb/sftp Alex Osborne 2020-04-24 08:13:39 +09:00
• 8dae196099 Merge pull request #323 from clawia/master Alex Osborne 2020-04-24 07:09:21 +09:00
• 63ca9d4c6a Remove static fields and byte loop + rename variable Leslie Bellony 2020-04-23 17:10:42 +02:00
• aad54c3cb3 rename variable + javadoc Leslie Bellony 2020-04-23 15:43:30 +02:00
• 7a6ea789db Merge pull request #326 from clawia/crawl-status-and-report Alex Osborne 2020-04-22 23:51:36 +09:00
• 5e948883fa Add overrides in JerichoExtractorHTMLTest Clara Wiatrowski 2020-04-22 15:33:23 +02:00
• 0d1f4ea84f Add parsing case data-* Clara Wiatrowski 2020-04-22 11:48:05 +02:00
• cc29bb97d1 Revert ExtractorHTML Clara Wiatrowski 2020-04-22 11:33:24 +02:00
• eea3914059 Add missing import Clara Wiatrowski 2020-04-21 15:57:40 +02:00
• 0f8fbaf8f2 Merge pull request #327 from internetarchive/yt-dl-format-medium-ish Neil Minton 2020-04-20 18:20:00 -04:00
• 4999843ade best medium-ish size yt-dl-format-medium-ish Barbara Miller 2020-04-03 16:18:07 -07:00
• 70a77d931a Merge pull request #325 from internetarchive/yt-dl-format Neil Minton 2020-04-20 16:50:24 -04:00
• f19dacfe67 Document GET /engine Alex Osborne 2020-04-20 18:32:43 +09:00
• f442f76784 Add missing newline Alex Osborne 2020-04-20 12:47:17 +09:00
• 33e637b887 Correct indentation to 3-spaces for reST Alex Osborne 2020-04-20 12:40:51 +09:00
• 75dc16e8ca Document 'Get Job Status' endpoint Alex Osborne 2020-04-20 12:35:49 +09:00
• b61f7e574e Add real crawlStatus in the crawlReport Clara Wiatrowski 2020-04-17 14:53:00 +02:00
• 6710c8dba3 Add parsing for HTML tags (data-src, data-srcset, data-original and data-original-set) Clara Wiatrowski 2020-04-10 15:59:50 +02:00
• ea374b3a99 Merge pull request #317 from internetarchive/fixes-extractor-multiple-regex-matcher-recycle jkafader 2020-04-06 16:04:54 -07:00
• 1ecf69dd64 best medium-ish size Barbara Miller 2020-04-03 16:18:07 -07:00
• 0a2f57fa73 best medium-ish size yt-dl-format Barbara Miller 2020-04-03 16:18:07 -07:00
• 40725372d6 Add support for the SFTP protocol Leslie Bellony 2020-04-01 14:47:34 +02:00
• a384dea2f9 Recycle the Matcher after use. Adam Miller 2020-03-25 22:33:06 +00:00
• fc1de30668 Updated changelog post release Andrew Jackson 2020-03-04 23:21:15 +00:00
• ff05de44b6 [maven-release-plugin] prepare for next development iteration Andrew Jackson 2020-03-04 23:05:41 +00:00
• a1bdcb1be7 [maven-release-plugin] prepare release 3.4.0-20200304 3.4.0-20200304 Andrew Jackson 2020-03-04 23:05:33 +00:00
• 384da2e82f Disable doclint Andrew Jackson 2020-03-04 23:03:47 +00:00
• 799c58e291 Merge branch 'master' of github.com:internetarchive/heritrix3 Andrew Jackson 2020-03-04 22:43:24 +00:00
• fd514b7bf6 Merge pull request #315 from ukwa/fix-oracle-doc-link Alex Osborne 2020-03-05 01:40:39 +09:00
• 9f7887a5fa Merge pull request #314 from hennekey/parameterize-bloom-filter Andy Jackson 2020-03-04 16:36:17 +00:00
• d5fad01122 Re-sync of changelog prior to release. Andrew Jackson 2020-03-04 16:21:21 +00:00
• b4494a7ffb Use the Wayback Machine to repair a link to Oracle docs. Andrew Jackson 2020-03-04 15:42:29 +00:00
• 348b5330bc Utilize the d parameter Tim Hennekey 2020-03-04 09:37:31 -05:00
• e0c82f70e9 Merge pull request #289 from netarchivesuite/bdb-frontier-access Andy Jackson 2020-03-04 13:41:27 +00:00
• 7f80b4fd98 Merge pull request #281 from ukwa/upgrade-bdb-je Andy Jackson 2020-03-04 13:36:03 +00:00
• 977f608c74 Merge pull request #290 from netarchivesuite/crawltrap-regex-timeout Alex Osborne 2020-03-03 19:27:42 +09:00
• e54258756a Fixed regex timeout handling following suggestion https://github.com/internetarchive/heritrix3/pull/290#discussion_r366711640 Colin Rosenthal 2020-03-03 09:02:07 +01:00
• 60e4053180 Commented this test back in to make Travis happy Colin Rosenthal 2020-03-03 08:52:09 +01:00
• 66535370ff Merge pull request #312 from internetarchive/guava-dep Alex Osborne 2020-02-28 10:16:55 +09:00
• 009458f2db Merge branch 'master-ait-contrib' into ait-qa Neil Minton 2020-02-13 15:40:58 -05:00
• 97545664b2 Remove Kafka support. Neil Minton 2020-02-13 15:38:38 -05:00