Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• cc97f01656 Merge pull request #123 from kris-sigur/HostsReport Noah Levitt 2016-01-15 10:33:07 -08:00
• 70c688a4f5 Merge pull request #135 from vonrosen/ARI-4562 Noah Levitt 2016-01-15 10:15:23 -08:00
• e92e5f2bb3 Merge branch 'master' into form-login-extra-inputs Noah Levitt 2016-01-15 09:39:24 -08:00
• 02e3a554f0 Merge pull request #142 from nlevitt/form-login-multipart Hunter 2016-01-15 08:19:23 -08:00
• c1abbfa5bd handle login forms with <input type="text"> fields in addition to username (for this to work there must be exactly one input field whose name contains the string "username") Noah Levitt 2016-01-14 18:02:54 -08:00
• 368d586eb1 fix NullPointerException when using old HtmlFormCredential login Noah Levitt 2016-01-14 16:00:08 -08:00
• 3b12a751e1 to make this branch only about multipart/form-data, undo the changes to handle more complicated forms with extra input fields; add unit test; store CrawlURI submit data in http-library agnostic way and remove httpcomponents-specific code from HTMLForm, refactor creation of post data accordingly Noah Levitt 2016-01-14 12:17:19 -08:00
• 309847cc40 Be sure to return at least one of the candidates Hunter Stern 2016-01-11 16:40:28 -08:00
• 704ec62487 Remove unneeded constant Hunter Stern 2016-01-07 15:08:02 -08:00
• a2a39fe052 Make sure to check form encoding type when addign form to curi data Hunter Stern 2016-01-07 15:03:25 -08:00
• 93bca3d55c More changes for https://webarchive.jira.com/browse/ARI-4656 Hunter Stern 2016-01-07 14:52:00 -08:00
• 4997014b34 More changes Hunter Stern 2015-12-28 10:06:53 -08:00
• ad329ceea8 Save enctype of form in submission curi Hunter Stern 2015-12-17 10:40:57 -08:00
• 34cffd512a Add ability to submit forms with enctype="multipart/form-data" Hunter Stern 2015-12-17 09:46:32 -08:00
• afa4f6535b If multiple username inputs in login form, favor the one with the word 'login' in it. Hunter Stern 2015-12-07 16:36:49 -08:00
• 5f630e4268 Make regex more inclusive for parsing out youtube videoId and add youtube watch page as relative to base Hunter Stern 2016-01-14 09:03:42 -08:00
• 7bcdf03a78 Merge branch 'master' of github.com:internetarchive/heritrix3 into ARI-4562 Hunter Stern 2016-01-14 08:48:39 -08:00
• eeeea6f0c4 Merge pull request #122 from nlevitt/form-login-checked-boxes Hunter 2016-01-14 08:00:49 -08:00
• bf2a887688 Merge pull request #138 from nlevitt/novel-quotas Adam Miller 2016-01-13 16:25:24 -08:00
• 9e4c7f0024 Merge branch 'master' into novel-quotas Noah Levitt 2016-01-13 13:44:54 -08:00
• b82896927a Merge pull request #132 from nlevitt/amqp-url-receiver-changes Adam Miller 2016-01-13 13:35:43 -08:00
• df5748d1f1 Merge pull request #136 from ukwa/amqp-declare-recieved-seeds Noah Levitt 2016-01-13 13:27:43 -08:00
• 22336fb71b fix stats in unusual case of "failed" fetch with response > 0 (only case where this can happen currently is if basic auth is configured for a url, but fails and url returns "401 Unauthorized") Noah Levitt 2016-01-12 17:34:27 -08:00
• 73be17af06 Disable SNI for a request if that request failed due to an SNI error 'javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name' Hunter Stern 2016-01-12 16:58:48 -08:00
• 3dbf754d56 Merge pull request #137 from nlevitt/seed-limits Adam Miller 2016-01-12 16:39:17 -08:00
• 65a1077425 license header Noah Levitt 2016-01-12 14:59:45 -08:00
• 32dd07375d Merge pull request #139 from nlevitt/robots-sep-sections Neil Minton 2016-01-11 18:39:25 -06:00
• 8918ce99c6 check that sourceTag of CrawlURI actually matches configured sourceTag Noah Levitt 2015-12-15 11:24:53 -08:00
• 7a8a614804 avoid exception in case applyToSubdomains is enabled and configured host is an ip address Noah Levitt 2015-12-15 11:06:33 -08:00
• a95519543d avoid exception in case applyToSubdomains is enabled and CrawlURI host is a nonstandard host like "dns:" in HostQuotaEnforcer Noah Levitt 2015-12-15 11:02:38 -08:00
• e3e6bfeb11 new HostQuotaEnforcer option applyToSubdomains, to additionally apply the quotas (separately) to each subdomain of the configured host Noah Levitt 2015-12-14 18:17:44 -08:00
• 2f82d43110 remove already-outdated stuff from javadoc Noah Levitt 2015-12-14 15:24:08 -08:00
• 6f9c235fd0 fix javadoc Noah Levitt 2015-12-14 15:23:16 -08:00
• 013029d9ad fix annotation copy-paste-o Noah Levitt 2015-12-14 15:22:51 -08:00
• 99c5af0cf2 comment out currently failing test (needs webarchive-commons pull request merged) Noah Levitt 2015-12-14 10:46:50 -08:00
• 2a56886c01 add license header Noah Levitt 2015-12-14 10:46:14 -08:00
• 8deead15ec new class to enforce quotas on a host Noah Levitt 2015-12-11 17:02:32 -08:00
• f1173789f7 fix test now that ContentDigestHistoryTest.shouldProcess() is smarter Noah Levitt 2015-12-11 16:13:17 -08:00
• c05678e296 CrawlLimitEnforcer support for maxWarcNovelBytes and maxWarcNovelUrls, with tests Noah Levitt 2015-12-11 16:08:47 -08:00
• 38931f152a handle multiple clauses for same user agent in robots.txt Noah Levitt 2015-12-11 15:42:16 -08:00
• dfc1dd940e tally warc novel content bytes and warc novel urls in CrawledBytesHistotable, with tests Noah Levitt 2015-12-11 12:49:02 -08:00
• c20a06ec27 Merge branch 'seed-limits' into novel-quotas -- so I can add tests to StatisticsSelfTest.java in this branch Noah Levitt 2015-12-11 11:35:11 -08:00
• 87f70c3e6a WARCLimitEnforcer to enforce limits (by stopping the crawl job) on any of the stats kept by WARCWriterProcessor Noah Levitt 2015-12-11 11:26:11 -08:00
• 294cfa3524 limited test of warc writer stats Noah Levitt 2015-12-11 11:22:03 -08:00
• 9ce12d2dbe add warc stats to CrawlURI.getData() dynamic attributes list after writing records; new method WARCWriterProcessor.getStats() to expose cumulative stats Noah Levitt 2015-12-11 11:21:05 -08:00
• 35a7954882 refactor FetchStats to be a subclass of CrawledBytesHistotable, to remove redundant code Noah Levitt 2015-12-11 11:05:44 -08:00
• 566c1a600d remove warning which can be misleading if crawl is configured to write only certain urls to warcs Noah Levitt 2015-12-11 11:01:26 -08:00
• cce07084e5 CrawlLimitEnforcer support for maxNovelBytes and maxNovelUrls, with tests Noah Levitt 2015-12-09 17:15:34 -08:00
• 501efe4a7b QuotaEnforcer support for quotas on novelBytes and novelUrls, with tests Noah Levitt 2015-12-09 15:54:36 -08:00
• ce31cada5f change class originally known as SeedLimitsEnforcer to SourceQuotaEnforcer; make it a Processor instead of a DecideRule (because checking quota at link scoping time doesn't work, since many urls which would go over quota can be added to the frontier); support quotas on any of the fields tracked by CrawledBytesHistotable Noah Levitt 2015-12-08 13:45:25 -08:00
• 7929e16d6a fix checkpointing problems with new statsBySource Noah Levitt 2015-12-04 21:33:40 -08:00
• de22096dde SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if source seed byte or document limit has been reached Noah Levitt 2015-12-03 17:50:46 -08:00
• 5559a5353d SourceSeedDecideRule applies the configured decision for any URI with discovered from one of a set of seeds Noah Levitt 2015-12-03 17:43:26 -08:00
• ce4a865764 add support to StatisticsTracker to keep a CrawledBytesHistotable per source tag when trackSources is enabled; integration test for this functionality Noah Levitt 2015-12-03 16:13:34 -08:00
• 0f6d385b24 Hook in submitted seeds properly. Andrew Jackson 2015-11-27 20:51:35 +00:00
• 89cd085e76 Merge pull request #133 from vonrosen/ARI-4565 Noah Levitt 2015-11-10 17:00:23 -08:00
• d3070cef54 Allow KnowledgableExtractorJS to parse out youtube watch from youtube javascript embed Hunter Stern 2015-11-06 11:39:41 -08:00
• 17f58f4bf8 Remove uneeded logger Hunter Stern 2015-11-06 10:42:55 -08:00
• 580391e0a7 Move most of addCookie functionality into AbstractCookieStore Hunter Stern 2015-11-06 10:40:01 -08:00
• 3142bbd485 Revise based on feedback in pull request from nlevitt Hunter Stern 2015-11-02 13:07:11 -08:00
• 44ac1030f2 Make sure to synchronize in addCookie now that domain count limits are imposed and modify unit test to also take into account domain limits Hunter Stern 2015-10-16 17:12:42 -07:00
• d9684e913b Restrict number of cookies per domain to 50 Hunter Stern 2015-10-16 14:05:56 -07:00
• 5ec6c738a6 Revert "Possible fix that probably isn't needed" Hunter Stern 2015-10-13 17:21:28 -07:00
• c6f019a095 Possible fix that probably isn't needed Hunter Stern 2015-10-13 14:18:06 -07:00
• e37b2b42cb avoid spurious logging Noah Levitt 2015-10-08 18:11:09 -07:00
• e4acb6c512 try very hard to start url consumer, and therefore bind the queue to the routing key, so that no messages are dropped, before crawling starts (should always work unless rabbitmq is down); some other tweaks for clarity and stability Noah Levitt 2015-10-08 17:22:44 -07:00
• 79ebb0e03c Merge pull request #129 from nlevitt/java8keytool Hunter 2015-10-06 16:09:43 -07:00
• a9c4cb42c9 Merge pull request #130 from nlevitt/stay-alive-statistics-tracker Hunter 2015-10-06 16:09:11 -07:00
• 98d41051b8 Merge pull request #131 from nlevitt/amqp-url-receiver-changes Hunter 2015-10-06 15:52:05 -07:00
• af5c6c3dde set isRunning=true so that stop() gets called to avoid leaking connections when crawls finish Noah Levitt 2015-10-06 15:50:47 -07:00
• 11d4bab1ac catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening Noah Levitt 2015-10-05 11:46:35 -07:00
• 41dac25cb6 Merge pull request #128 from ukwa/amqp-url-receiver-changes Noah Levitt 2015-09-24 16:52:04 -07:00
• b7afc627ed fix typo Noah Levitt 2015-09-24 15:31:12 -07:00
• 88c7f3c3d4 Merge pull request #124 from vonrosen/ari-4371 Noah Levitt 2015-09-24 10:26:49 -07:00
• e883977de6 Downgrade Throwable to Exception. Andrew Jackson 2015-09-23 23:14:11 +01:00
• 9aabcb3272 add license header to new class Noah Levitt 2015-09-23 12:07:38 -07:00
• 12dde2b693 load keytool utility main class dynamically, trying both the old and new (java 8) class names Noah Levitt 2015-09-23 12:06:11 -07:00
• d1ea01c75d Updated with additional optional fields and separated isConsuming from isRunning. Andrew Jackson 2015-09-19 21:39:19 +01:00
• cd3fb8d717 Configurable and 3.3 compatible AMQP receiver for #1 Andrew Jackson 2015-09-18 15:48:09 +01:00
• 490a257cfe Merge pull request #126 from caofangkun/her-001 Noah Levitt 2015-08-25 10:40:59 -07:00
• 6cf0d4a0c5 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing caofangkun 2015-08-24 17:56:56 +08:00
• f097939a1c Don't call setRealm so other realms that can be used. Hunter Stern 2015-08-14 13:01:54 -07:00
• f5a4ecbb03 Merge pull request #125 from ldko/AMQP-declarations-fix Noah Levitt 2015-08-03 14:26:38 -07:00
• c94f515956 Sets the default clientId to the same as on Receiver side Lauren Ko 2015-07-31 15:45:49 -05:00
• 592c70fa3d declare AMQP exchange in AMQPUrlReceiver to prevent exception when exchange not yet created Lauren Ko 2015-07-30 17:34:48 -05:00
• 3495851029 JavaDoc typo Kristinn Sigurðsson 2015-07-13 15:10:44 +00:00
• b15fbbd13e Enable suppression of empty hosts in report. Until now hosts that have never been crawled are included in the hosts report. This can be a very large number of hosts and is infrequently of interest. This makes it possible to exclude the empty hosts from the report. Default behavior remains to include them. The option is highlighted by including it in the relevant, commented-out section of the default profile. Kristinn Sigurðsson 2015-07-13 15:08:03 +00:00
• 0cf009eb4c Use default value in sample profile. Kristinn Sigurðsson 2015-07-13 14:58:23 +00:00
• 96b927d9fa Make sorting of host report configurable. Can now set a maximum size for sorted reports. Longer reports are unsorted. Default behavior is still to sort all reports. Default profile updated to reflect this new option in a commented-out section. Kristinn Sigurðsson 2015-07-13 14:55:26 +00:00
• bf638939b0 Remove unused "top" hosts tracking. This used to be used for an overview report that was available in the H1 UI. Had become vestigial code that served no purpose. Kristinn Sigurðsson 2015-07-13 14:32:34 +00:00
• 4e22f1f809 Allow realm to be set by server for basic auth. Hunter Stern 2015-07-07 13:06:29 -07:00
• 3c4d09f6c0 only submit checkbox and radio button form fields if they are on by default ("checked" attribute is present) Noah Levitt 2015-06-15 17:27:59 -07:00
• 7afe03540a Merge pull request #120 from nlevitt/extractorjs-twodots Hunter 2015-05-04 15:45:16 -07:00
• eb78de6963 Merge pull request #121 from nlevitt/knowledgable-extractorjs Hunter 2015-05-04 15:41:21 -07:00
• fea49f0e30 expand stubbed javadoc comment Noah Levitt 2015-05-04 15:16:11 -07:00
• 0f43a3dae6 add license header Noah Levitt 2015-05-04 15:08:52 -07:00
• e9fcb34508 new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS that has some customized behavior for specific kinds of web pages; initially, the one special behavior it has is for drupal generated pages Noah Levitt 2015-04-21 01:19:33 -07:00
• 1027f854ec for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js") Noah Levitt 2015-04-20 17:08:47 -07:00
• 0923a7ca18 Merge pull request #119 from adam-miller/HER-2082 Noah Levitt 2015-03-26 16:30:49 -07:00
• 458cbb51b0 Fix for HER-2082 https://webarchive.jira.com/browse/HER-2082 WARCWriterProcessor.java Add last hop character and link context to outlinks in metadata records Adam Miller 2015-03-26 16:19:09 -07:00