Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 50a38e1376 Merge pull request #61 from nlevitt/her-2065 Kenji Nagahashi 2014-04-25 11:55:14 -07:00
• 381442dfb6 ARI-3765 - make AMQPUrlReceiver gracefully handle amqp server going up and down Noah Levitt 2014-04-24 21:23:03 -07:00
• 10e0f262b3 ARI-3765 - make AMQPPublishProcessor gracefully handle amqp server going up and down Noah Levitt 2014-04-24 20:31:40 -07:00
• d777c198bd HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji) Noah Levitt 2014-04-24 17:23:01 -07:00
• 3a456638ca Merge pull request #60 from nlevitt/cosmetics Adam Miller 2014-04-21 15:52:43 -07:00
• 0cd1043cb1 change memory bar color, use consistent non-capitalization on buttons Noah Levitt 2014-04-21 11:53:40 -07:00
• 1b72ce5dd8 remove extra </div> Noah Levitt 2014-04-21 11:53:05 -07:00
• cfb9474f7a cosmetic changes Noah Levitt 2014-04-21 09:06:15 -07:00
• a48a29f9bd Adding ExtractorPDFContent to find nav links from the text content of PDFs Adam Miller 2014-04-14 16:02:35 -07:00
• 67a7a4fb6c Merge pull request #59 from nlevitt/ari-3723 vonrosen 2014-04-10 20:22:15 -07:00
• e73810d0a6 fix unit test now that we accept speculative urls with query params with no value Noah Levitt 2014-04-10 17:25:27 -07:00
• 5e75ba6506 Merge pull request #57 from nlevitt/umbra Eldon 2014-04-10 15:35:48 -04:00
• 63c4264de6 Merge pull request #58 from nlevitt/ari-3723 vonrosen 2014-04-10 12:09:16 -07:00
• 339e6ec87a for ARI-3723, accept speculative urls with query params with no value Noah Levitt 2014-04-04 10:15:07 -07:00
• ac9d8e6ba7 AMQPUrlReceiver - improve handling of case where rabbitmq is unreachable at crawl startup - spawn off a thread to start up the amqp consumer to avoid the crawl launch thread getting stuck Noah Levitt 2014-04-03 11:39:36 -07:00
• 030063f7b3 Merge pull request #56 from nlevitt/fix-form-login vonrosen 2014-04-03 09:10:44 -07:00
• 36a207ce07 fix FormLoginProcessor checkpointing Noah Levitt 2014-04-02 17:33:09 -07:00
• 20350091af Merge pull request #54 from nlevitt/fix-form-login vonrosen 2014-04-01 15:30:30 -07:00
• 878fd9f7a9 oops, update test to expect post data as url-encoded query string Noah Levitt 2014-04-01 15:18:17 -07:00
• 4bd8045bfb Merge pull request #53 from nlevitt/fix-form-login vonrosen 2014-04-01 12:29:50 -07:00
• 552200b2fa fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass Noah Levitt 2014-03-31 20:38:17 -07:00
• 20a5f31e19 ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet) Noah Levitt 2014-03-31 18:57:33 -07:00
• a82886b747 restore support for sending http post data, add unit test Noah Levitt 2014-03-31 18:09:23 -07:00
• 23e8d41651 avoid NPE; root cause was assumption that LoadingCache.asMap().get() would call CacheLoader.load() but it doesnt, see https://code.google.com/p/guava-libraries/wiki/MapMakerMigration Noah Levitt 2014-03-31 13:40:29 -07:00
• cc838747af pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available Noah Levitt 2014-03-31 13:38:54 -07:00
• 26a577ff6e basic unit tests for FormLoginProcessor Noah Levitt 2014-03-31 13:38:39 -07:00
• 0d7f633fdb pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available Noah Levitt 2014-03-31 13:36:44 -07:00
• 19e8cd5b66 Merge pull request #52 from nlevitt/expression-deciderule Adam Miller 2014-03-18 14:10:36 -07:00
• ef2889cf17 Merge pull request #50 from shriphani/master Noah Levitt 2014-03-18 11:27:49 -07:00
• 74f017c53e Merge branch 'master' of https://github.com/shriphani/heritrix3 Shriphani Palakodety 2014-03-18 06:42:54 -04:00
• 84f29e4e10 build with the right key and value types Shriphani Palakodety 2014-03-16 21:00:10 -04:00
• c81eeced7a added removal listener Shriphani Palakodety 2014-03-14 16:20:50 -04:00
• 2f60b5f935 migration to latest guava complete Shriphani Palakodety 2014-03-14 15:36:22 -04:00
• 8c9892586a removed guava dependency from heritrix. use the webarchive-commons version Shriphani Palakodety 2014-03-14 15:13:34 -04:00
• 1e8e2e2016 prevent intellij from complaining Shriphani Palakodety 2014-03-13 21:50:06 -04:00
• b0ad9358e8 bump version in pom.xml Shriphani Palakodety 2014-03-13 20:02:58 -04:00
• d0e0ee6bc6 forgot to add expireAfterWrite Shriphani Palakodety 2014-03-13 15:29:37 -04:00
• bd1047d2d2 bumped guava version to 11 Shriphani Palakodety 2014-03-13 15:26:57 -04:00
• 60cc78dac8 Merge branch 'master' of https://github.com/internetarchive/heritrix3 Shriphani Palakodety 2014-03-18 06:38:12 -04:00
• 863ff90157 Implicitly add the ${} around groovyExpression. When cxml contains ${}, it fights with spring's PropertyPlaceholderConfigurer (if enabled); and since what we're interested in is an expression, not a template for producing a string, it's more appropriate to leave it out anyway. Noah Levitt 2014-03-17 18:11:47 -07:00
• 2018da4cd6 Merge pull request #51 from nlevitt/expression-deciderule Adam Miller 2014-03-17 17:49:12 -07:00
• fb039291ef fix indentation (spaces only) Noah Levitt 2014-03-16 18:14:29 -07:00
• fb2be86e43 ExpressionDecideRule - evaluate groovy template with variable "curi" bound Noah Levitt 2014-03-16 18:13:12 -07:00
• 4c302f84be build with the right key and value types Shriphani Palakodety 2014-03-16 21:00:10 -04:00
• d4d23639b6 added removal listener Shriphani Palakodety 2014-03-14 16:20:50 -04:00
• 439f78b038 migration to latest guava complete Shriphani Palakodety 2014-03-14 15:36:22 -04:00
• abd61abbf2 removed guava dependency from heritrix. use the webarchive-commons version Shriphani Palakodety 2014-03-14 15:13:34 -04:00
• ad8cfd7075 prevent intellij from complaining Shriphani Palakodety 2014-03-13 21:50:06 -04:00
• 25ce16e893 bump version in pom.xml Shriphani Palakodety 2014-03-13 20:02:58 -04:00
• ee6a3bddb3 forgot to add expireAfterWrite Shriphani Palakodety 2014-03-13 15:29:37 -04:00
• 2cac0ebf80 bumped guava version to 11 Shriphani Palakodety 2014-03-13 15:26:57 -04:00
• a36acbacac Merge pull request #49 from nlevitt/youtube-march-2014 Adam Miller 2014-03-06 16:14:12 -08:00
• 5138a01240 warn if youtube video url is missing signature parameter Noah Levitt 2014-03-06 16:08:07 -08:00
• 6e3bd2a2e8 update to handle latest changes from youtube Noah Levitt 2014-03-05 19:05:31 -08:00
• 8a2533c95c Merge pull request #48 from nlevitt/umbra vonrosen 2014-03-03 08:21:13 -08:00
• d1eeb43e40 Merge branch 'master' into umbra Noah Levitt 2014-03-01 16:10:56 -08:00
• e46c9ed759 Continue processing url after sending to amqp (fetch, etc). Solves 2 problems, 1) crawl finishing after sending off umbra seeds and before umbra has a chance to send anything back, and 2) seeds sent to umbra not being fetched because they were already seen. We may want to change this in the future as umbra gets fancier, e.g. sending back html from DOM, but for now it should work. Noah Levitt 2014-03-01 16:10:41 -08:00
• 6a92e88b1c Merge pull request #40 from internetarchive/her-2064 Noah Levitt 2014-02-28 16:48:42 -08:00
• 54f572c356 Merge pull request #47 from adam-miller/youtube-itag-priority Noah Levitt 2014-02-28 14:43:50 -08:00
• 7fa3809828 fixing formatting Adam Miller 2014-02-28 14:33:55 -08:00
• 09c113163e Adjusting youtube extraction itag priority Adam Miller 2014-02-28 14:15:49 -08:00
• 5e55921abb Merge pull request #46 from nlevitt/switch-to-ia-webarchive-commons ikreymer 2014-02-28 12:48:44 -08:00
• e3efa3a5f2 switch dependency from ia-web-commons 1.1.1-SNAPSHOT to webarchive-commons 1.1.1-IA-SNAPSHOT Noah Levitt 2014-02-28 12:46:36 -08:00
• 22cc1222b2 Merge pull request #45 from nlevitt/update-youtube-itags Adam Miller 2014-02-28 11:13:11 -08:00
• 61539ab930 update ExtractorYoutubeFormatStreamTest.testDefaultItag() Noah Levitt 2014-02-27 17:41:38 -08:00
• 8b124ecaa9 update youtube default video format priority (see https://webarchive.jira.com/browse/ARI-3730) Noah Levitt 2014-02-27 17:37:45 -08:00
• da8e284581 Merge pull request #43 from martinsbalodis/master Noah Levitt 2014-02-27 09:34:07 -08:00
• eba9a880be Merge pull request #1 from nlevitt/pull43 Mārtiņš Balodis 2014-02-27 18:42:55 +02:00
• ea19dacbd2 handle main job log handler like current launch job log handler, close on teardown Noah Levitt 2014-02-26 17:19:50 -08:00
• a297abb02d Merge pull request #44 from nlevitt/update-hc4_3_3 Eldon 2014-02-26 14:39:47 -05:00
• cf5e295d42 update httpcomponents, should address NPE we've seen https://issues.apache.org/jira/browse/HTTPCLIENT-1417 Noah Levitt 2014-02-26 11:35:26 -08:00
• 88dc01305f fix job.log file handler was left open when jobdir is removed Martins Balodis 2014-02-26 21:15:38 +02:00
• 659c5e886f Merge pull request #42 from eldondev/master Noah Levitt 2014-02-25 14:15:30 -08:00
• 06a75c6fcf Adding the queue declaration and binding to the UrlReceiver Eldon 2014-02-25 17:11:56 -05:00
• bbd818e6e4 Merge pull request #39 from nlevitt/umbra Eldon 2014-02-25 16:24:38 -05:00
• e08936d029 Merge pull request #41 from nlevitt/fix-slow-cookies Adam Miller 2014-02-21 15:51:49 -08:00
• cf775bec94 remove extraneous code related to logging removed in last commit Noah Levitt 2014-02-21 15:46:52 -08:00
• fd806a514d remove extra logging from performance test Noah Levitt 2014-02-21 13:28:13 -08:00
• 39f06a1f1c fix slowness caused by BdbCookieStore by maintaining cookie list in memory parallel to the one stored on disk; add tests for consistency and performance under load Noah Levitt 2014-02-21 13:26:58 -08:00
• d130e3fc44 call the extra info we send to umbra "clientId", kinda clarifies the purpose of it Noah Levitt 2014-02-13 19:25:09 -08:00
• e389499aa7 don't try to stop consuming amqp when crawl is shutting down, can end up reopening the connection after it's been closed Noah Levitt 2014-02-13 19:24:31 -08:00
• d6e61b83da For https://webarchive.jira.com/browse/HER-2064 Hunter Stern 2014-02-13 16:59:05 -08:00
• 06ba50c410 Reverting previous commit: https://github.com/internetarchive/heritrix3/commit/164d0fbdc5fa32cd90edd9f1b4091ed876552232 Hunter Stern 2014-02-13 16:50:21 -08:00
• 164d0fbdc5 For https://webarchive.jira.com/browse/HER-2064 Hunter Stern 2014-02-13 16:42:58 -08:00
• 72b2d58034 pause amqp consumption on crawl pause, and resume on unpause Noah Levitt 2014-02-10 21:04:19 -08:00
• c493248863 formatting change only - use 4 spaces to indent Noah Levitt 2014-02-10 20:43:50 -08:00
• eccddb3df3 use parent url metadata received via amqp in building CrawlURI Noah Levitt 2014-02-10 20:38:37 -08:00
• 8d82c440d3 send url metadata with amqp message so that umbra can send it back with child urls Noah Levitt 2014-02-10 20:36:51 -08:00
• 704f4aa443 Merge pull request #38 from nlevitt/umbra Eldon 2014-02-06 16:51:03 -05:00
• 6dbcac40d9 syncronize start() and stop() Noah Levitt 2014-02-03 20:27:51 -08:00
• 500dafd390 AMQPPublishProcessor - annotate urls sent to AMQP; set special fetch status; add default amqpUri; replace queueName setting with exchange and routingKey, which seem to be what are needed; avoid sending urls that were received from AMQP, and robots.txt urls; set content-type of amqp message to application/json, which umbra seems to need Noah Levitt 2014-02-03 18:41:33 -08:00
• af44522fdf FetchHTTPRequest - include CrawlURI.getData().get("customHttpRequestHeaders"), if any, in http request Noah Levitt 2014-02-03 18:41:23 -08:00
• ecb0eaf435 AMQPUrlReceiver - ack amqp messages, get rid of misguided thread starting infinite consumers Noah Levitt 2014-02-03 18:36:52 -08:00
• e90ba8278b Merge branch 'master' into umbra Noah Levitt 2014-02-03 18:22:25 -08:00
• 63b70b427e Merge pull request #37 from nlevitt/her-2063 vonrosen 2014-02-03 17:42:34 -08:00
• edd62f3de0 fix HER-2063 - omit port in Host request header when it is default for the scheme Noah Levitt 2014-02-03 17:18:40 -08:00
• c980b452c9 annotate CrawlURI with "receivedViaAMQP" Noah Levitt 2014-02-01 18:26:07 -08:00
• c7cf62bf80 license header Noah Levitt 2014-02-01 18:23:49 -08:00
• bf0590a083 spaces not tabs for indentation Noah Levitt 2014-02-01 18:23:05 -08:00
• c163f31851 AMQPUrlReceiver - receive urls and request headers via amqp in the format produced by https://github.com/internetarchive/umbra and add to frontier Noah Levitt 2014-01-31 20:42:27 -08:00