Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 0aa20a9a68 Merge branch 'master' into use-latest-checkpoint Andrew Jackson 2018-10-12 12:51:06 +01:00
• a81a648996 Merge pull request #213 from ukwa/frontier-threadsafe-workqueues Andy Jackson 2018-10-12 12:48:33 +01:00
• 59ee9b2adc Default to starting anew if no checkpoints are found. Andrew Jackson 2018-10-09 10:28:34 +01:00
• 3b341f79df Provide a simple hook to restart from the latest checkpoint. Andrew Jackson 2018-10-08 14:26:04 +01:00
• fca796265d Move sync to after Null check. Andrew Jackson 2018-09-17 15:25:45 +01:00
• e4973e4375 Attempt to cache dependencies. Andrew Jackson 2018-09-16 16:15:02 +01:00
• 8ffd2c2813 Simplify contrib build. Andrew Jackson 2018-09-14 17:03:25 +01:00
• 2330641931 Attempt to prevent build time-outs. Andrew Jackson 2018-09-14 15:57:50 +01:00
• 09a272088e Add synchronisation around host and server stats. Andrew Jackson 2018-09-13 16:37:18 +01:00
• 55dbe5a25d Avoid overdoing the RAM allocation. Andrew Jackson 2018-09-13 16:36:58 +01:00
• a2af528501 Ensure all WorkQueue modification as serialised across threads. Andrew Jackson 2018-09-13 13:45:10 +01:00
• 992a51aa57 Tests now need more RAM. Andrew Jackson 2018-09-13 13:44:40 +01:00
• a831676196 Merge pull request #209 from ukwa/relative-base-href Noah Levitt 2018-09-11 15:18:48 -07:00
• 9c9d11d272 fix exception starting DecideRuleSequence logging Noah Levitt 2018-07-23 13:08:17 -05:00
• 8384a59b47 Fix up tests to account for new Base URI behaviour. Andrew Jackson 2018-07-06 15:15:53 +01:00
• 8bf23081fa Only set the BaseURI if not set already. Andrew Jackson 2018-07-05 13:34:17 +01:00
• 14148cc7a8 Same fix for JerichoExtractorHTML. Andrew Jackson 2018-07-05 09:40:21 +01:00
• 42181b8585 Test case and fix for internetarchive/heritrix3#208. Andrew Jackson 2018-07-04 16:18:46 +01:00
• 48908ee5b4 Fix link to User Guide Maurice Schleußinger 2018-07-04 15:47:03 +02:00
• da9fcf1af6 Fix typos Alex Osborne 2018-07-04 22:22:15 +09:00
• 2d48b40ea4 Replace API response section with statuscode tag Alex Osborne 2018-07-04 21:46:42 +09:00
• 0cfd62f76e Reformat the rest of the API calls Alex Osborne 2018-07-04 21:43:57 +09:00
• 3b3d579cb8 Add actions to API URLs to distinguish them Alex Osborne 2018-07-04 21:16:53 +09:00
• 1ff31c135e Add explanatory note about documentation Alex Osborne 2018-07-04 19:48:33 +09:00
• 7076dd6388 Try nicer formatting for the first API example Alex Osborne 2018-07-04 18:46:15 +09:00
• ab8a5efea2 Format HTTP request lines in API guide Alex Osborne 2018-07-04 18:24:08 +09:00
• e35eb9e850 Move conventions and REST section to the end of the document Alex Osborne 2018-07-04 17:53:05 +09:00
• 57f465706e Convert html tables to rst tables Alex Osborne 2018-07-04 17:34:19 +09:00
• 90db2e33d5 Add skeleton sphinx docs for readthedocs Alex Osborne 2018-07-04 16:39:59 +09:00
• 05811705ed Point at GitHub wiki for latest release info Kristinn Sigurðsson 2018-05-29 09:42:24 +00:00
• 7308911807 Merge pull request #205 from adam-miller/ParallelQueueRandomAssignment Noah Levitt 2018-05-10 15:18:11 -07:00
• bee53de8b1 Add parameter to allow even distribution for parallel queues. Adam Miller 2017-10-12 21:35:23 +00:00
• 40bfe9f79f Merge pull request #197 from nlevitt/ari-5630 Barbara Miller 2018-01-24 16:31:33 -08:00
• 74c48659b8 Merge branch 'ari-5630' into ait-qa Noah Levitt 2018-01-16 17:13:04 -08:00
• 95759149ae catch exceptions scoping outlinks to stop them from derailing processing of the parent url Noah Levitt 2018-01-16 17:12:24 -08:00
• 5a0326eba1 Merge pull request #196 from kngenie/fix-test-failures-with-bdb Noah Levitt 2017-12-12 14:20:55 -08:00
• a7b7c6cf5a fix for test failures in a workspace on NFS-mounted filesystem Kenji Nagahashi 2017-12-08 17:15:03 -08:00
• 76bf98a4f0 Merge pull request #194 from internetarchive/formsMaxSize Noah Levitt 2017-11-16 11:24:20 -08:00
• 998edf3479 max size for extracted form elements Barbara Miller 2017-11-13 21:20:28 -08:00
• 9169eb831b Merge branch 'formsMaxSize' into ait-qa Barbara Miller 2017-11-15 11:57:52 -08:00
• bac58e8ee6 hardcode 8{ Barbara Miller 2017-11-15 11:57:18 -08:00
• 537f75de77 Merge branch 'formsMaxSize' into ait-qa Barbara Miller 2017-11-15 08:49:52 -08:00
• a2f8557f45 maxSize for extracted form elements Barbara Miller 2017-11-13 21:20:28 -08:00
• feaaf2c934 Merge branch 'formsMaxSize' into ait-qa Barbara Miller 2017-11-13 21:40:47 -08:00
• 6d6fd5cd6a maxSize for extracted form elements Barbara Miller 2017-11-13 21:20:28 -08:00
• deda35eedd Merge pull request #192 from internetarchive/robotstxt-size-limit Noah Levitt 2017-11-01 17:20:30 -07:00
• d6d7f1b162 Fix ArrayIndexOutOfBoundsException when all lines are blank Alex Osborne 2017-11-01 10:04:02 +09:00
• 68ceceedd2 Use simpler read 500k chars and split method suggested by @nlevitt Alex Osborne 2017-11-01 09:29:28 +09:00
• 1fae77c64d Enforce robots.txt character limit per char not per line Alex Osborne 2017-10-31 18:41:00 +09:00
• f0d1c0422b Merge pull request #187 from nlevitt/robots-allow-failed-fetch Barbara Miller 2017-10-13 11:23:54 -07:00
• dd88191bac Merge pull request #190 from ukwa/fix-ci-and-add-java-8 Noah Levitt 2017-10-13 11:09:36 -07:00
• aee83dfe26 Allow DNS settings to be overridden via sheets. Andrew Jackson 2017-10-13 10:29:15 +01:00
• e7ce1f56b4 Ensure disabling JavaDNS always fails over to the InetAddress approach. Andrew Jackson 2017-10-12 23:33:54 +01:00
• d178c689de Add some comments to try to avoid misuse. Andrew Jackson 2017-10-12 22:13:00 +01:00
• d85d2a3fe8 Move OneLineSimpleLayout into commons so all can use it. Andrew Jackson 2017-10-12 22:10:58 +01:00
• b0594ac0f4 Allow JavaDNS to be disabled and use that to avoid issues in StatisticsSelfTest. Andrew Jackson 2017-10-12 21:29:34 +01:00
• f7a5376fbe Merge branch 'master' into fix-ci-and-add-java-8 Andrew Jackson 2017-10-12 09:31:47 +01:00
• 09ee95f37b Allow results with or without DNS lookup of localhost. Andrew Jackson 2017-10-11 23:30:17 +01:00
• ae8a0ef95f update webarchive-commons dependency to incorporate fix to warc size calculation, and uncomment test that exercises the fix Noah Levitt 2017-10-11 14:54:37 -07:00
• 4e96d0c94f Merge pull request #189 from adam-miller/master Noah Levitt 2017-10-11 14:46:58 -07:00
• b702bf3d86 WARCLimitEnforcer.java - Add support for multiple warc writers. Adam Miller 2017-10-11 21:06:35 +00:00
• 46fa54da0c Set an appropriate Content Size when using non-DNS resolves. Andrew Jackson 2017-10-11 17:08:15 +01:00
• a5381440fd Use build folder for temp files as tmp gets wiped on Travis CI sometimes. Andrew Jackson 2017-10-09 22:13:53 +01:00
• 401e27fc64 Adapt test to cope with variation in how localhost gets resolved. Andrew Jackson 2017-10-09 21:50:05 +01:00
• a0a783c750 oraclejdk7 EOL'd in May 2015, not supported in Travis-CI Trusty image Jaimie Murdock 2017-10-09 16:42:32 -04:00
• 1de6f7d2a0 Update Travis CI file to suitable Java 7 + 8 variants. Andrew Jackson 2017-10-09 09:54:16 +01:00
• d4b8566fea updateRobots deem S_CONNECT_FAILED 404 ARI-5439 Barbara Miller 2017-10-06 12:55:29 -07:00
• 46e2542e41 Merge branch 'robots-allow-failed-fetch' into ait-qa Noah Levitt 2017-10-06 12:00:58 -07:00
• 84debcfe4d treat a failed fetch (e.g. socket timeout) of robots.txt the same way as http errors: full allow Noah Levitt 2017-10-06 12:00:31 -07:00
• 0ec055c211 Merge pull request #186 from nlevitt/trough-logging-tweak Barbara Miller 2017-10-04 16:32:11 -07:00
• ddc81f15a7 Merge branch 'trough-logging-tweak' into ait-qa Noah Levitt 2017-10-04 16:29:40 -07:00
• d01f2c3a30 fix copy&paste mistake Noah Levitt 2017-10-04 16:29:22 -07:00
• 4940e6f28e Merge branch 'trough-logging-tweak' into ait-qa Noah Levitt 2017-10-04 15:35:57 -07:00
• 8fea6c4783 reduce batch size to 400 and avoid ridiculously long log lines Noah Levitt 2017-10-04 15:35:16 -07:00
• 65004ea748 Merge pull request #185 from internetarchive/trough-esc-sql jkafader 2017-09-21 12:19:35 -07:00
• 6caf007b75 Merge branch 'trough-esc-sql' into ait-qa Noah Levitt 2017-09-21 12:16:58 -07:00
• 7fc1c2f25f escape strings in sql posted to trough Noah Levitt 2017-09-21 12:16:25 -07:00
• 4c438d3852 Merge pull request #180 from internetarchive/aitfive-1176 jkafader 2017-09-05 17:37:36 -07:00
• 47e8064101 Merge branch 'aitfive-1176' into ait-qa Noah Levitt 2017-06-27 16:01:22 -07:00
• 1c92be2d74 realized this stuff needs to be thread safe Noah Levitt 2017-06-27 16:01:15 -07:00
• 7bc8289637 Merge branch 'aitfive-1176' into ait-qa Noah Levitt 2017-06-27 15:51:06 -07:00
• 66ead24762 missed another comma Noah Levitt 2017-06-27 15:50:33 -07:00
• ab8a19fd53 Merge branch 'aitfive-1176' into qa Noah Levitt 2017-06-27 15:46:07 -07:00
• 698a02a5b7 missed a comma Noah Levitt 2017-06-27 15:45:55 -07:00
• a3b45aa94a Merge branch 'aitfive-1176' into ait-qa Noah Levitt 2017-06-27 15:00:00 -07:00
• 2f29949b88 include http response payload when logging errors from trough Noah Levitt 2017-06-26 17:37:39 -07:00
• c0631d387d Merge branch 'master' into ait-qa Noah Levitt 2017-06-15 17:35:54 -07:00
• 3892c17b56 ughhh appease java 8 javadoc rules Noah Levitt 2017-06-15 17:29:22 -07:00
• a1a7232530 update for new data model queued_url->uncrawled_url and some new fields Noah Levitt 2017-06-14 18:31:39 -07:00
• c1bd972ced Merge branch 'aitfive-1176' into ait-qa Noah Levitt 2017-06-08 11:39:16 -07:00
• b670c2f59a missed the maven tarball assembly file Noah Levitt 2017-06-08 11:39:08 -07:00
• 330457ce5d Merge branch 'aitfive-1176' into ait-qa Noah Levitt 2017-06-08 11:36:07 -07:00
• a1f59da000 oops, String.join is a java 8 feature Noah Levitt 2017-06-08 11:35:17 -07:00
• 68318d2573 oops, String.join is a java 8 feature Noah Levitt 2017-06-08 11:35:17 -07:00
• 87a15bddc8 Merge pull request #168 from internetarchive/gh-167 Noah Levitt 2017-06-07 18:15:19 -07:00
• c001470ad7 Merge branch 'aitfive-1176' into ait-qa Noah Levitt 2017-06-07 18:13:26 -07:00
• 66fabf790e hacked up tarball with heritrix, contrib and dependencies Noah Levitt 2017-06-07 17:45:41 -07:00
• eed0210a8f batch trough writes Noah Levitt 2017-06-07 13:28:27 -07:00
• d1452c1d21 initial version of TroughCrawlLogFeed.java Noah Levitt 2017-06-07 11:09:36 -07:00
• a4bb99e828 Merge pull request #179 from BitBaron/aitfive-1225 Noah Levitt 2017-06-02 12:01:17 -07:00