Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• bcae99daa8 Merge pull request #118 from kris-sigur/HER-2080 Noah Levitt 2015-03-23 11:16:36 -07:00
• f5107f60b8 Add annotation for server-not-modified duplicates. Kristinn Sigurðsson 2015-03-20 13:46:20 +00:00
• dff219e884 Harmonizing whitespace use Kristinn Sigurðsson 2015-03-13 14:32:23 +00:00
• 843f6468c4 Use digest from previous capture when encountering 304s. The previous capture may also have been a 304 but there must be an original 200 capture at the start. This assumption may fail when using an index generated by a version of Heritrix without this fix. But even then you should be no worse off than before. Kristinn Sigurðsson 2015-03-13 14:30:39 +00:00
• 3ffe5b8b4f Stop automatically writing WARC-Payload-Digest for revisit records. Refer that functionality to the relevant revisit profile classes. Kristinn Sigurðsson 2015-03-11 13:52:31 +00:00
• b5da93d04b Merge pull request #117 from nlevitt/ari-4222 vonrosen 2015-02-26 10:04:34 -08:00
• a0c3fcdae0 avoid java.lang.NullPointerException at org.archive.modules.writer.WriterPoolProcessor.copyForwardWriteTagIfDupe(WriterPoolProcessor.java:504); this exception has not always been possible, but is possible now because IdenticalDigestDecideRule.hasIdenticalDigest(curi) can return true even if there is no curi.getFetchHistory(), since uri-agnostic dedup doesn't use fetchHistory Noah Levitt 2015-02-23 14:09:17 -08:00
• 1b6d78bf7a Merge pull request #116 from nlevitt/log4j-breaks-tests Adam Miller 2015-02-23 11:59:50 -08:00
• 8009275075 make sure log4j is configured when running unit tests, to avoid log4j error messages polluting captured output and causing tests to fail, like so: https://builds.archive.org/job/Heritrix-3/org.archive.heritrix$heritrix-commons/587/testReport/org.archive.io.arc/ARCWriterTest/testLengthTooLongCompressed/ Noah Levitt 2015-02-23 11:36:29 -08:00
• 0a418a8425 Merge pull request #114 from nlevitt/webarchive-commons-issue-38 vonrosen 2015-02-12 15:33:22 -08:00
• 1bd8b141aa bump webarchive-commons dependency to 1.1.5 to pick up https://github.com/iipc/webarchive-commons/pull/39 Noah Levitt 2015-02-11 18:53:34 -08:00
• 789bd8ac7c Merge pull request #115 from kris-sigur/report-encoding Noah Levitt 2015-02-10 23:54:21 -08:00
• 4ea7efdfc8 Set character set to UTF-8 when passing through files. Kristinn Sigurðsson 2015-02-10 16:07:37 +00:00
• 2078a9f0e2 remove TmpDirTestCase.java (moving to webarchive-commons) Noah Levitt 2015-01-06 16:08:18 -08:00
• 64e3b1b02a remove RecordingOutputStreamTest.java (moving to webarchive-commons) Noah Levitt 2015-01-06 16:01:16 -08:00
• 2c9e82a261 Merge pull request #112 from nlevitt/amqp-receiver-deadlock vonrosen 2014-11-21 10:15:53 -08:00
• 3609abe38c Merge pull request #108 from nlevitt/aitfive-203 vonrosen 2014-11-14 10:29:41 -08:00
• 54555e8566 Merge pull request #103 from adam-miller/SeedsReportRedirectFix Noah Levitt 2014-11-11 18:04:51 -08:00
• 40bc02c03f Merge pull request #110 from kris-sigur/httpclient-4.3.6 Noah Levitt 2014-11-11 17:35:26 -08:00
• 5f17707a2e somewhat ugly fix to handle exceptions from the bean browser like java.lang.RuntimeException: not implemented at org.archive.modules.fetcher.BdbCookieStore$RestrictedCollectionWrappedList.get(BdbCookieStore.java:92) Noah Levitt 2014-11-11 12:45:24 -08:00
• 0212a554e6 Upgrade to httpclient 4.3.6 Kristinn Sigurðsson 2014-11-11 14:40:33 +00:00
• 7be1fc606b make sure the kafka producer thread is not in the ToePool thread group, to stop it from being interrupted at the end of the crawl Noah Levitt 2014-11-10 16:43:00 -08:00
• 743ab3c9c0 Revert "Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)"" Noah Levitt 2014-11-10 16:44:37 -08:00
• a0f70d0865 Revert "switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message)" Noah Levitt 2014-11-10 13:55:13 -08:00
• d96b1f8eca Merge pull request #100 from vonrosen/ari-4061 Noah Levitt 2014-11-08 00:05:00 -08:00
• fa0a28f78b Merge branch 'aitfive-203' of github.com:nlevitt/heritrix3 into aitfive-203 Noah Levitt 2014-11-07 15:59:24 -08:00
• 192b3f9b4b for kafka/amqp crawl log feeds, remove now redundant contentSize from extraInfo Noah Levitt 2014-11-07 15:59:14 -08:00
• 4527d5af29 Merge branch 'extra-info-size' into aitfive-203 Noah Levitt 2014-11-07 15:58:08 -08:00
• 7312ad597e so that it can appear in the crawl log, add contentSize to CrawlURI extraInfo, for http transactions (the only case in heritrix currently where contentLength and contentSize differ) Noah Levitt 2014-11-07 13:21:42 -08:00
• e2be3bbf9c new style kafka configuration properties Noah Levitt 2014-11-06 19:53:41 -08:00
• 82bef2a337 only release the lock if we actually own it (avoid java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantLock.tryRelease(ReentrantLock.java:155)) Noah Levitt 2014-11-06 19:33:35 -08:00
• a37b879a4a logging improvement Noah Levitt 2014-11-06 19:28:19 -08:00
• d2f70e5c97 switch to org.apache.kafka.clients.producer.KafkaProducer to see if we can avoid the InterruptedException at the end of the crawl (and log warnings on failure to send message) Noah Levitt 2014-11-06 19:26:32 -08:00
• 23fa29d673 fix log messages to say "kafka" Noah Levitt 2014-11-05 14:03:54 -08:00
• 77e2a47c4c AITFIVE-203 kafka crawl log feed Noah Levitt 2014-11-05 12:59:10 -08:00
• 4d0eff23d2 Merge pull request #107 from vonrosen/ari-4102 Noah Levitt 2014-11-04 15:55:50 -08:00
• 9a41b31264 Handle case where form does not have an action defined. Hunter Stern 2014-11-04 11:22:38 -08:00
• f5cf30ec3e Merge pull request #102 from nlevitt/amqp-receiver-deadlock vonrosen 2014-11-03 08:11:00 -08:00
• 716e0cc7ef use "fair" ReentrantLock -- I stepped through heritrix that appeared to be deadlocked, found that the thread trying to shut down was being starved of the lock by another thread (the StarterRestarter) locking and unlocking it repeatedly, which can apparently happen because the lock was "unfair" Noah Levitt 2014-10-27 17:34:41 -07:00
• edcc7cef21 Merge pull request #106 from nlevitt/aitfive-39 jkafader 2014-10-26 19:20:38 -07:00
• 6f21ede979 seriously, fix extraInfo handling in AMQPCrawlLogFeed Noah Levitt 2014-10-26 19:17:23 -07:00
• 23a07d5cef Merge pull request #105 from nlevitt/aitfive-39 jkafader 2014-10-26 18:29:21 -07:00
• 7e4b1717d1 fix extraInfo handling in AMQPCrawlLogFeed Noah Levitt 2014-10-26 18:24:47 -07:00
• f25c171a79 Merge pull request #104 from nlevitt/aitfive-39 jkafader 2014-10-26 17:11:57 -07:00
• 13a9d3a1be change field names to match new druid config Noah Levitt 2014-10-22 15:28:40 -07:00
• fb23b40e74 CandidatesProcessor.java Keep CrawlURI outlinks intact for use in SeedRecord reporting. Adam Miller 2014-10-22 13:18:25 -07:00
• 0712cb381c Merge pull request #101 from vonrosen/ari-4086 Noah Levitt 2014-10-19 17:38:35 -07:00
• d154cfbf36 avoid deadlock in AMQPUrlReceiver hopefully Noah Levitt 2014-10-17 15:27:08 -07:00
• 8c318e53f6 Remove forcefetch for AMQP received urls so they don't get crawled twice. Hunter Stern 2014-10-16 22:02:43 +00:00
• 78e98365d6 Remove unused variable. Hunter Stern 2014-10-16 17:51:08 +00:00
• 5909042b84 Merge pull request #99 from nlevitt/aitfive-39 vonrosen 2014-10-16 17:24:37 +00:00
• a27dbd182d change DecideRuleSequence extraInfo field name 'source' to 'seed' (for reason external to heritrix) Noah Levitt 2014-10-14 15:35:10 -07:00
• 79aadf919b rename AMQPCrawlLogFeed fields to match James's names in druid (looks like there's no way to translate) Noah Levitt 2014-10-14 14:00:55 -07:00
• 3a82b37e52 Ensure Jerico extractor html doesn't call new metadata test method. Hunter Stern 2014-10-10 23:33:22 +00:00
• 3347de3d39 Make meta tag content url extraction rules more generate so as to apply to all meta tags 'content' attributes, not just specific og:video. Hunter Stern 2014-10-09 23:42:24 +00:00
• ff5c59c555 Allow extraction of content url in opengraph video meta tag - og:video Hunter Stern 2014-10-09 17:57:05 +00:00
• 5f7dc49599 Merge pull request #95 from kngenie/patch-beanbrowser-failure Noah Levitt 2014-10-03 19:21:27 -07:00
• 0f1760fbfe Merge pull request #97 from kris-sigur/HER-2074 Noah Levitt 2014-10-03 18:48:42 -07:00
• 0b16ff0fad Fix for HER-2074 https://webarchive.jira.com/browse/HER-2074 Kristinn Sigurðsson 2014-10-03 11:30:38 +00:00
• 86ebed04cc Merge pull request #96 from nlevitt/her-2070 Kristinn Sigurðsson 2014-10-03 09:33:04 +00:00
• 5cb5f05437 add field "extra info" to DecideRuleSequence log, which is a json object with entries for host, via, source and hop path Noah Levitt 2014-10-02 02:34:32 -07:00
• 743d76c966 set <encoding>UTF-8</encoding> in maven-compiler-plugin; project.build.sourceEncoding seems not to be enough, see http://stackoverflow.com/questions/8978013/error-unmappable-character-for-encoding-utf8-during-maven-compilation Noah Levitt 2014-10-02 01:48:13 -07:00
• 78e6b8b6ab replace accidental U+2212 MINUS SIGN with normal ascii 0x2d '-' to avoid compilation error "unmappable character for encoding ASCII" Noah Levitt 2014-10-01 21:31:14 -07:00
• e5e5cd87e4 configure maven-compiler-plugin in root pom to avoid redundancy, inconsistency Noah Levitt 2014-10-01 13:38:39 -07:00
• 060054e6b3 avoid java 7 api Noah Levitt 2014-10-01 13:38:14 -07:00
• 6faa1b8c1c fix misleading variable name Noah Levitt 2014-09-30 11:43:09 -07:00
• 8d98882eb7 an optimization suggested by Gordon: organize cookies by host (domain), rather than top private domain, and supply a composite list of cookies from the host and each parent domain to the http client library Noah Levitt 2014-09-29 18:32:31 -07:00
• 5fd120dffd some javadocs and other comments, some minor refactoring, change delimiter to ';' for the sortable key Noah Levitt 2014-09-29 17:51:24 -07:00
• 87b408cc23 rename limited cookie store facade class Noah Levitt 2014-09-28 14:35:22 -07:00
• 610a1e1e97 cookie store is called BdbCookieStore again Noah Levitt 2014-09-28 14:32:34 -07:00
• 54eb1c950f rudimentary test of cookie at url with ip address as host Noah Levitt 2014-09-28 14:30:01 -07:00
• 5c21ad1762 AbstractCookieStore.getCookieFor(CrawlURI) computes top private domain, so FetchHTTP doesn't have to know anything about that Noah Levitt 2014-09-28 14:29:04 -07:00
• 4aa85af1b0 a few tests exercising some basic domain/subdomain cookie behaviors Noah Levitt 2014-09-28 14:15:27 -07:00
• a37a8de4a2 improved implementation not using TreeSet, and some initial work on integration testing of cookie handling Noah Levitt 2014-09-28 11:59:14 -07:00
• 6588641017 restore saving and loading, checkpoints, testing Noah Levitt 2014-09-27 13:55:07 -07:00
• 111294e887 oops, commit classes for initial bdb implementation Noah Levitt 2014-09-27 12:22:50 -07:00
• 47b7caad29 initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains Noah Levitt 2014-09-26 20:56:45 -07:00
• 886a0f2136 (forgot to add SimpleHostCookieSetStore) initial proof of concept fix for HER-2070 "cookie monster" bug Noah Levitt 2014-09-26 20:05:37 -07:00
• f769e0636d initial proof of concept fix for HER-2070 "cookie monster" bug Noah Levitt 2014-09-26 20:01:52 -07:00
• 29379c7e19 Merge pull request #84 from nlevitt/iipc-webarchive-commons Kenji Nagahashi 2014-09-11 17:19:08 -07:00
• 6ada6801de Merge pull request #94 from nlevitt/ari-3724 Kenji Nagahashi 2014-09-11 17:17:55 -07:00
• 8136046442 Merge pull request #92 from nlevitt/remove-cruft Kenji Nagahashi 2014-09-11 17:13:48 -07:00
• 5821fe7b1e FIX corner-case of bean browser failing due to an exception from hashCode(). Kenji Nagahashi 2014-09-11 11:22:01 -07:00
• e2064237cf Merge pull request #87 from vonrosen/ari-3977 Noah Levitt 2014-09-10 12:42:34 -07:00
• 1d0f60a2be webarchive-commons 1.1.4 has been released, bump dependency Noah Levitt 2014-09-10 12:31:01 -07:00
• 066cd17b5e Merge pull request #93 from vonrosen/ari-4020 Noah Levitt 2014-09-08 18:03:34 -07:00
• 2af19fa03b Revise regex instead of unescaping js. Hunter Stern 2014-09-08 16:37:03 -07:00
• 98a6c08a86 remove extraneous backslashes (thanks Kris) Noah Levitt 2014-09-05 14:38:14 -07:00
• df7d3b506c do not require "+" (plus sign) before @OPERATOR_CONTACT_URL@ in user-agent -- many bots including googlebot have the +, but some don't, so having it in the default template seems sufficient Noah Levitt 2014-09-05 12:59:17 -07:00
• 035a7b0a08 Merge pull request #80 from nlevitt/ari-3617-data-remote vonrosen 2014-09-04 16:08:33 -07:00
• 2e4f513a4c Revise parsing of string with unicode quoted data Hunter Stern 2014-09-03 15:27:58 -07:00
• ea235f4466 Allow ExtractorJS to detect urls within unicode code points for " and \' Hunter Stern 2014-09-02 15:18:13 -07:00
• 935b8f9658 remove unused dependency jets3t (thanks kenji) Noah Levitt 2014-09-02 11:00:34 -07:00
• 27d632159d Add test case for ExtractorXML Hunter Stern 2014-09-01 10:13:25 -07:00
• 2705bd3179 Merge remote-tracking branch 'internetarchive/master' into ari-3977 Hunter Stern 2014-09-01 09:23:26 -07:00
• 51a1cad300 new class DecideRuleSequenceWithAMQPFeed publishes scoping decisions to amqp Noah Levitt 2014-08-29 17:08:32 -07:00
• 37442104cc move AMQP producer code to a separate class so it can be used by classes that are not heritrix processors Noah Levitt 2014-08-29 17:05:16 -07:00
• 25c3b3e680 Merge pull request #77 from vonrosen/ari-3943 Noah Levitt 2014-08-29 15:26:37 -07:00
• ec0839a6fe remove more unused classes Noah Levitt 2014-08-29 15:23:21 -07:00
• 01af800b35 Merge pull request #91 from adam-miller/FetchHTTP_Empty_Content_Length Kenji Nagahashi 2014-08-27 17:43:34 -07:00