Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• af5c6c3dde set isRunning=true so that stop() gets called to avoid leaking connections when crawls finish Noah Levitt 2015-10-06 15:50:47 -07:00
• 11d4bab1ac catch exceptions and log error in StatisticsTracker.run(), to make sure statistics events don't stop happening Noah Levitt 2015-10-05 11:46:35 -07:00
• 41dac25cb6 Merge pull request #128 from ukwa/amqp-url-receiver-changes Noah Levitt 2015-09-24 16:52:04 -07:00
• b7afc627ed fix typo Noah Levitt 2015-09-24 15:31:12 -07:00
• 88c7f3c3d4 Merge pull request #124 from vonrosen/ari-4371 Noah Levitt 2015-09-24 10:26:49 -07:00
• e883977de6 Downgrade Throwable to Exception. Andrew Jackson 2015-09-23 23:14:11 +01:00
• 9aabcb3272 add license header to new class Noah Levitt 2015-09-23 12:07:38 -07:00
• 12dde2b693 load keytool utility main class dynamically, trying both the old and new (java 8) class names Noah Levitt 2015-09-23 12:06:11 -07:00
• d1ea01c75d Updated with additional optional fields and separated isConsuming from isRunning. Andrew Jackson 2015-09-19 21:39:19 +01:00
• cd3fb8d717 Configurable and 3.3 compatible AMQP receiver for #1 Andrew Jackson 2015-09-18 15:48:09 +01:00
• 490a257cfe Merge pull request #126 from caofangkun/her-001 Noah Levitt 2015-08-25 10:40:59 -07:00
• 6cf0d4a0c5 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing caofangkun 2015-08-24 17:56:56 +08:00
• f097939a1c Don't call setRealm so other realms that can be used. Hunter Stern 2015-08-14 13:01:54 -07:00
• f5a4ecbb03 Merge pull request #125 from ldko/AMQP-declarations-fix Noah Levitt 2015-08-03 14:26:38 -07:00
• c94f515956 Sets the default clientId to the same as on Receiver side Lauren Ko 2015-07-31 15:45:49 -05:00
• 592c70fa3d declare AMQP exchange in AMQPUrlReceiver to prevent exception when exchange not yet created Lauren Ko 2015-07-30 17:34:48 -05:00
• 3495851029 JavaDoc typo Kristinn Sigurðsson 2015-07-13 15:10:44 +00:00
• b15fbbd13e Enable suppression of empty hosts in report. Until now hosts that have never been crawled are included in the hosts report. This can be a very large number of hosts and is infrequently of interest. This makes it possible to exclude the empty hosts from the report. Default behavior remains to include them. The option is highlighted by including it in the relevant, commented-out section of the default profile. Kristinn Sigurðsson 2015-07-13 15:08:03 +00:00
• 0cf009eb4c Use default value in sample profile. Kristinn Sigurðsson 2015-07-13 14:58:23 +00:00
• 96b927d9fa Make sorting of host report configurable. Can now set a maximum size for sorted reports. Longer reports are unsorted. Default behavior is still to sort all reports. Default profile updated to reflect this new option in a commented-out section. Kristinn Sigurðsson 2015-07-13 14:55:26 +00:00
• bf638939b0 Remove unused "top" hosts tracking. This used to be used for an overview report that was available in the H1 UI. Had become vestigial code that served no purpose. Kristinn Sigurðsson 2015-07-13 14:32:34 +00:00
• 4e22f1f809 Allow realm to be set by server for basic auth. Hunter Stern 2015-07-07 13:06:29 -07:00
• 3c4d09f6c0 only submit checkbox and radio button form fields if they are on by default ("checked" attribute is present) Noah Levitt 2015-06-15 17:27:59 -07:00
• 7afe03540a Merge pull request #120 from nlevitt/extractorjs-twodots Hunter 2015-05-04 15:45:16 -07:00
• eb78de6963 Merge pull request #121 from nlevitt/knowledgable-extractorjs Hunter 2015-05-04 15:41:21 -07:00
• fea49f0e30 expand stubbed javadoc comment Noah Levitt 2015-05-04 15:16:11 -07:00
• 0f43a3dae6 add license header Noah Levitt 2015-05-04 15:08:52 -07:00
• e9fcb34508 new contrib module KnowledgableExtractorJS, a subclass of ExtractorJS that has some customized behavior for specific kinds of web pages; initially, the one special behavior it has is for drupal generated pages Noah Levitt 2015-04-21 01:19:33 -07:00
• 1027f854ec for ARI-4267 accept possible uris with two dots in the filename part if they end with a known good extension (e.g. "foo.min.js") Noah Levitt 2015-04-20 17:08:47 -07:00
• 0923a7ca18 Merge pull request #119 from adam-miller/HER-2082 Noah Levitt 2015-03-26 16:30:49 -07:00
• 458cbb51b0 Fix for HER-2082 https://webarchive.jira.com/browse/HER-2082 WARCWriterProcessor.java Add last hop character and link context to outlinks in metadata records Adam Miller 2015-03-26 16:19:09 -07:00
• bcae99daa8 Merge pull request #118 from kris-sigur/HER-2080 Noah Levitt 2015-03-23 11:16:36 -07:00
• f5107f60b8 Add annotation for server-not-modified duplicates. Kristinn Sigurðsson 2015-03-20 13:46:20 +00:00
• dff219e884 Harmonizing whitespace use Kristinn Sigurðsson 2015-03-13 14:32:23 +00:00
• 843f6468c4 Use digest from previous capture when encountering 304s. The previous capture may also have been a 304 but there must be an original 200 capture at the start. This assumption may fail when using an index generated by a version of Heritrix without this fix. But even then you should be no worse off than before. Kristinn Sigurðsson 2015-03-13 14:30:39 +00:00
• 3ffe5b8b4f Stop automatically writing WARC-Payload-Digest for revisit records. Refer that functionality to the relevant revisit profile classes. Kristinn Sigurðsson 2015-03-11 13:52:31 +00:00
• b5da93d04b Merge pull request #117 from nlevitt/ari-4222 vonrosen 2015-02-26 10:04:34 -08:00
• a0c3fcdae0 avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory Noah Levitt 2015-02-23 14:09:17 -08:00
• 1b6d78bf7a Merge pull request #116 from nlevitt/log4j-breaks-tests Adam Miller 2015-02-23 11:59:50 -08:00
• 8009275075 make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/ Noah Levitt 2015-02-23 11:36:29 -08:00
• 0a418a8425 Merge pull request #114 from nlevitt/webarchive-commons-issue-38 vonrosen 2015-02-12 15:33:22 -08:00
• 1bd8b141aa bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39 Noah Levitt 2015-02-11 18:53:34 -08:00
• 789bd8ac7c Merge pull request #115 from kris-sigur/report-encoding Noah Levitt 2015-02-10 23:54:21 -08:00
• 4ea7efdfc8 Set character set to UTF-8 when passing through files. Kristinn Sigurðsson 2015-02-10 16:07:37 +00:00
• 2078a9f0e2 remove TmpDirTestCase.java (moving to webarchive-commons) Noah Levitt 2015-01-06 16:08:18 -08:00
• 64e3b1b02a remove RecordingOutputStreamTest.java (moving to webarchive-commons) Noah Levitt 2015-01-06 16:01:16 -08:00
• 2c9e82a261 Merge pull request #112 from nlevitt/amqp-receiver-deadlock vonrosen 2014-11-21 10:15:53 -08:00
• 3609abe38c Merge pull request #108 from nlevitt/aitfive-203 vonrosen 2014-11-14 10:29:41 -08:00
• 54555e8566 Merge pull request #103 from adam-miller/SeedsReportRedirectFix Noah Levitt 2014-11-11 18:04:51 -08:00
• 40bc02c03f Merge pull request #110 from kris-sigur/httpclient-4.3.6 Noah Levitt 2014-11-11 17:35:26 -08:00
• 5f17707a2e somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92) Noah Levitt 2014-11-11 12:45:24 -08:00
• 0212a554e6 Upgrade to httpclient 4.3.6 Kristinn Sigurðsson 2014-11-11 14:40:33 +00:00
• 7be1fc606b make sure the kafka producer thread is not in the ToePool thread group, to stop it from being interrupted at the end of the crawl Noah Levitt 2014-11-10 16:43:00 -08:00
• 743ab3c9c0 Revert "Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)"" Noah Levitt 2014-11-10 16:44:37 -08:00
• a0f70d0865 Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)" Noah Levitt 2014-11-10 13:55:13 -08:00
• d96b1f8eca Merge pull request #100 from vonrosen/ari-4061 Noah Levitt 2014-11-08 00:05:00 -08:00
• fa0a28f78b Merge branch 'aitfive-203' of github.com:nlevitt/heritrix3 into aitfive-203 Noah Levitt 2014-11-07 15:59:24 -08:00
• 192b3f9b4b for kafka/amqp crawl log feeds, remove now redundant contentSize from extraInfo Noah Levitt 2014-11-07 15:59:14 -08:00
• 4527d5af29 Merge branch 'extra-info-size' into aitfive-203 Noah Levitt 2014-11-07 15:58:08 -08:00
• 7312ad597e so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ) Noah Levitt 2014-11-07 13:21:42 -08:00
• e2be3bbf9c new style kafka configuration properties Noah Levitt 2014-11-06 19:53:41 -08:00
• 82bef2a337 only release the lock if we actually own it (avoid java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantLock.tryRelease(ReentrantLock.java:155)) Noah Levitt 2014-11-06 19:33:35 -08:00
• a37b879a4a logging improvement Noah Levitt 2014-11-06 19:28:19 -08:00
• d2f70e5c97 switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message) Noah Levitt 2014-11-06 19:26:32 -08:00
• 23fa29d673 fix log messages to say "kafka" Noah Levitt 2014-11-05 14:03:54 -08:00
• 77e2a47c4c AITFIVE-203 kafka crawl log feed Noah Levitt 2014-11-05 12:59:10 -08:00
• 4d0eff23d2 Merge pull request #107 from vonrosen/ari-4102 Noah Levitt 2014-11-04 15:55:50 -08:00
• 9a41b31264 Handle case where form does not have an action defined. Hunter Stern 2014-11-04 11:22:38 -08:00
• f5cf30ec3e Merge pull request #102 from nlevitt/amqp-receiver-deadlock vonrosen 2014-11-03 08:11:00 -08:00
• 716e0cc7ef use "fair" ReentrantLock -- I stepped through heritrix that appeared to be deadlocked, found that the thread trying to shut down was being starved of the lock by another thread (the StarterRestarter) locking and unlocking it repeatedly, which can apparently happen because the lock was "unfair" Noah Levitt 2014-10-27 17:34:41 -07:00
• edcc7cef21 Merge pull request #106 from nlevitt/aitfive-39 jkafader 2014-10-26 19:20:38 -07:00
• 6f21ede979 seriously, fix extraInfo handling in AMQPCrawlLogFeed Noah Levitt 2014-10-26 19:17:23 -07:00
• 23a07d5cef Merge pull request #105 from nlevitt/aitfive-39 jkafader 2014-10-26 18:29:21 -07:00
• 7e4b1717d1 fix extraInfo handling in AMQPCrawlLogFeed Noah Levitt 2014-10-26 18:24:47 -07:00
• f25c171a79 Merge pull request #104 from nlevitt/aitfive-39 jkafader 2014-10-26 17:11:57 -07:00
• 13a9d3a1be change field names to match new druid config Noah Levitt 2014-10-22 15:28:40 -07:00
• fb23b40e74 CandidatesProcessor.java Keep CrawlURI outlinks intact for use in SeedRecord reporting. Adam Miller 2014-10-22 13:18:25 -07:00
• 0712cb381c Merge pull request #101 from vonrosen/ari-4086 Noah Levitt 2014-10-19 17:38:35 -07:00
• d154cfbf36 avoid deadlock in AMQPUrlReceiver hopefully Noah Levitt 2014-10-17 15:27:08 -07:00
• 8c318e53f6 Remove forcefetch for AMQP received urls so they don't get crawled twice. Hunter Stern 2014-10-16 22:02:43 +00:00
• 78e98365d6 Remove unused variable. Hunter Stern 2014-10-16 17:51:08 +00:00
• 5909042b84 Merge pull request #99 from nlevitt/aitfive-39 vonrosen 2014-10-16 17:24:37 +00:00
• a27dbd182d change DecideRuleSequence extraInfo field name 'source' to 'seed' (for reason external to heritrix) Noah Levitt 2014-10-14 15:35:10 -07:00
• 79aadf919b rename AMQPCrawlLogFeed fields to match James's names in druid (looks like there's no way to translate) Noah Levitt 2014-10-14 14:00:55 -07:00
• 3a82b37e52 Ensure Jerico extractor html doesn't call new metadata test method. Hunter Stern 2014-10-10 23:33:22 +00:00
• 3347de3d39 Make meta tag content url extraction rules more generate so as to apply to all meta tags 'content' attributes, not just specific og:video. Hunter Stern 2014-10-09 23:42:24 +00:00
• ff5c59c555 Allow extraction of content url in opengraph video meta tag - og:video Hunter Stern 2014-10-09 17:57:05 +00:00
• 5f7dc49599 Merge pull request #95 from kngenie/patch-beanbrowser-failure Noah Levitt 2014-10-03 19:21:27 -07:00
• 0f1760fbfe Merge pull request #97 from kris-sigur/HER-2074 Noah Levitt 2014-10-03 18:48:42 -07:00
• 0b16ff0fad Fix for HER-2074 https://webarchive.jira.com/browse/HER-2074 Kristinn Sigurðsson 2014-10-03 11:30:38 +00:00
• 86ebed04cc Merge pull request #96 from nlevitt/her-2070 Kristinn Sigurðsson 2014-10-03 09:33:04 +00:00
• 5cb5f05437 add field "extra info" to DecideRuleSequence log, which is a json object with entries for host, via, source and hop path Noah Levitt 2014-10-02 02:34:32 -07:00
• 743d76c966 set <encoding>UTF-8</encoding> in maven-compiler-plugin; project.build.sourceEncoding seems not to be enough, see http://stackoverflow.com/questions/8978013/error-unmappable-character-for-encoding-utf8-during-maven-compilation Noah Levitt 2014-10-02 01:48:13 -07:00
• 78e6b8b6ab replace accidental U+2212 MINUS SIGN with normal ascii 0x2d '-' to avoid compilation error "unmappable character for encoding ASCII" Noah Levitt 2014-10-01 21:31:14 -07:00
• e5e5cd87e4 configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency Noah Levitt 2014-10-01 13:38:39 -07:00
• 060054e6b3 avoid java 7 api Noah Levitt 2014-10-01 13:38:14 -07:00
• 6faa1b8c1c fix misleading variable name Noah Levitt 2014-09-30 11:43:09 -07:00
• 8d98882eb7 an optimization suggested by Gordon: organize cookies by host (domain), rather than top private domain, and supply a composite list of cookies from the host and each parent domain to the http client library Noah Levitt 2014-09-29 18:32:31 -07:00
• 5fd120dffd some javadocs and other comments, some minor refactoring, change delimiter to ';' for the sortable key Noah Levitt 2014-09-29 17:51:24 -07:00
• 87b408cc23 rename limited cookie store facade class Noah Levitt 2014-09-28 14:35:22 -07:00