Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 26222ef660 Merge pull request #69 from vonrosen/ari-3853 Noah Levitt 2014-05-15 17:38:41 -07:00
• d570498fd9 New Unit tests added now. Hunter Stern 2014-05-15 17:37:26 -07:00
• 80880b4409 Unit test and other changes. Hunter Stern 2014-05-15 17:34:43 -07:00
• aff5b9ac4e for ARI-3712, add extracted links relative to both via and base, and annotate with "extractorSWFRelToVia", "extractorSWFRelToBase", or "extractorSWFRelToBoth" if resulting link is the same whether relative to base or via Noah Levitt 2014-05-15 13:31:29 -07:00
• 519717fd7b make addRelativeToBase() and addRelativeToVia() return the link added, or null; copy getAnnotations() from CrawlURI Noah Levitt 2014-05-15 13:27:50 -07:00
• f24cfff6af New youtube channel video extractor for https://webarchive.jira.com/browse/ARI-3865 Hunter Stern 2014-05-13 17:41:42 -07:00
• 963e04e50c handle exception determining whether to apply overlay Noah Levitt 2014-05-13 15:47:21 -07:00
• 3e213c82bc Merge pull request #67 from nlevitt/umbra Adam Miller 2014-05-13 14:44:09 -07:00
• 08a4aafa23 don't log severe with stack trace on normal amqp shutdown Noah Levitt 2014-05-13 14:39:35 -07:00
• 38cd14ecac Merge pull request #63 from adam-miller/ExtractorPDFContent Noah Levitt 2014-05-13 14:09:20 -07:00
• aaecd6d78c Merge branch 'master' of git://github.com/internetarchive/heritrix3 into ExtractorPDFContent Adam Miller 2014-05-13 14:04:29 -07:00
• fb42c07c92 bugfix, new test case. Adam Miller 2014-05-13 14:03:03 -07:00
• ab7a60f9e3 Merge pull request #66 from nlevitt/cosmetics Adam Miller 2014-05-13 13:31:11 -07:00
• 85e7a5ee78 Bugfix & new test cases. Adam Miller 2014-05-12 16:52:32 -07:00
• ba68649ba0 Merge remote-tracking branch 'origin/master' into cosmetics Noah Levitt 2014-05-07 10:02:06 -07:00
• b7f8a0669f Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics Noah Levitt 2014-05-07 10:02:00 -07:00
• 33e782e2fd oops, make "exit java process" button work again Noah Levitt 2014-05-07 10:01:47 -07:00
• eb32d7a8b8 Merge pull request #65 from nlevitt/umbra Adam Miller 2014-05-02 15:47:44 -07:00
• 949c9f086c shut down the starter-restarter thread at crawl finish!! Noah Levitt 2014-05-02 15:42:29 -07:00
• dd510cbf3b Merge pull request #64 from adam-miller/ViaSurtPrefixedDecideRule Noah Levitt 2014-05-01 18:14:25 -07:00
• ae878d29cf Code cleanup Adam Miller 2014-05-01 15:22:16 -07:00
• f2bebd41b0 Cleanup / Optimization Adam Miller 2014-05-01 12:36:05 -07:00
• 753404f662 Adding ViaSurtPrefixedDecideRule. Returns true if a surtform 'via' matches any surt in a specified list Adam Miller 2014-03-03 16:51:11 -08:00
• 2b1c330be6 Further cleanup Adam Miller 2014-04-30 12:18:14 -07:00
• e6d646d4b4 Cleanup / optimization Adam Miller 2014-04-30 12:14:32 -07:00
• 34a9172e69 Organize Imports Adam Miller 2014-04-29 17:25:43 -07:00
• 5181c1ef64 Cleaning up test cases and removing unused code Adam Miller 2014-04-29 17:11:18 -07:00
• 33277445f6 Merge branch 'master' of git://github.com/internetarchive/heritrix3 into ExtractorPDFContent Adam Miller 2014-04-29 17:06:55 -07:00
• 49069f2dd1 Adding comment for library dependencies Adam Miller 2014-04-29 17:06:11 -07:00
• 7e8a66e7d8 New Test cases, fixing bugs Adam Miller 2014-04-29 10:50:04 -07:00
• 747bbeba97 Merge pull request #62 from nlevitt/ari-3765 vonrosen 2014-04-28 09:39:37 -07:00
• 50a38e1376 Merge pull request #61 from nlevitt/her-2065 Kenji Nagahashi 2014-04-25 11:55:14 -07:00
• 381442dfb6 ARI-3765 - make AMQPUrlReceiver gracefully handle amqp server going up and down Noah Levitt 2014-04-24 21:23:03 -07:00
• 10e0f262b3 ARI-3765 - make AMQPPublishProcessor gracefully handle amqp server going up and down Noah Levitt 2014-04-24 20:31:40 -07:00
• d777c198bd HER-2065 - in activateInactiveQueue() close synchronized block earlier, and move updateHighestWaiting() up to keep it inside the block; now the call to deactivateQueue() is not in the synchronized block, so the deadlock should be avoided (fix suggested by kenji) Noah Levitt 2014-04-24 17:23:01 -07:00
• 3a456638ca Merge pull request #60 from nlevitt/cosmetics Adam Miller 2014-04-21 15:52:43 -07:00
• 0cd1043cb1 change memory bar color, use consistent non-capitalization on buttons Noah Levitt 2014-04-21 11:53:40 -07:00
• 1b72ce5dd8 remove extra </div> Noah Levitt 2014-04-21 11:53:05 -07:00
• cfb9474f7a cosmetic changes Noah Levitt 2014-04-21 09:06:15 -07:00
• a48a29f9bd Adding ExtractorPDFContent to find nav links from the text content of PDFs Adam Miller 2014-04-14 16:02:35 -07:00
• 67a7a4fb6c Merge pull request #59 from nlevitt/ari-3723 vonrosen 2014-04-10 20:22:15 -07:00
• e73810d0a6 fix unit test now that we accept speculative urls with query params with no value Noah Levitt 2014-04-10 17:25:27 -07:00
• 5e75ba6506 Merge pull request #57 from nlevitt/umbra Eldon 2014-04-10 15:35:48 -04:00
• 63c4264de6 Merge pull request #58 from nlevitt/ari-3723 vonrosen 2014-04-10 12:09:16 -07:00
• 339e6ec87a for ARI-3723, accept speculative urls with query params with no value Noah Levitt 2014-04-04 10:15:07 -07:00
• ac9d8e6ba7 AMQPUrlReceiver - improve handling of case where rabbitmq is unreachable at crawl startup - spawn off a thread to start up the amqp consumer to avoid the crawl launch thread getting stuck Noah Levitt 2014-04-03 11:39:36 -07:00
• 030063f7b3 Merge pull request #56 from nlevitt/fix-form-login vonrosen 2014-04-03 09:10:44 -07:00
• 36a207ce07 fix FormLoginProcessor checkpointing Noah Levitt 2014-04-02 17:33:09 -07:00
• 20350091af Merge pull request #54 from nlevitt/fix-form-login vonrosen 2014-04-01 15:30:30 -07:00
• 878fd9f7a9 oops, update test to expect post data as url-encoded query string Noah Levitt 2014-04-01 15:18:17 -07:00
• 4bd8045bfb Merge pull request #53 from nlevitt/fix-form-login vonrosen 2014-04-01 12:29:50 -07:00
• 552200b2fa fix form login processor by saving login post data as url-encoded query string, which FetchHTTP now expects; tweaks to make FormLoginSelfTest pass Noah Levitt 2014-03-31 20:38:17 -07:00
• 20a5f31e19 ExtractorHTMLForms+FormLoginProcessor full crawl job integration test (not passing yet) Noah Levitt 2014-03-31 18:57:33 -07:00
• a82886b747 restore support for sending http post data, add unit test Noah Levitt 2014-03-31 18:09:23 -07:00
• 23e8d41651 avoid NPE; root cause was assumption that LoadingCache.asMap().get() would call CacheLoader.load() but it doesnt, see https://code.google.com/p/guava-libraries/wiki/MapMakerMigration Noah Levitt 2014-03-31 13:40:29 -07:00
• cc838747af pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available Noah Levitt 2014-03-31 13:38:54 -07:00
• 26a577ff6e basic unit tests for FormLoginProcessor Noah Levitt 2014-03-31 13:38:39 -07:00
• 0d7f633fdb pull methods getRecorder() and makeCrawlURI(String) into ModuleTestBase, and move ModuleTestBase into heritrix-modules so that CrawlURI is available Noah Levitt 2014-03-31 13:36:44 -07:00
• 19e8cd5b66 Merge pull request #52 from nlevitt/expression-deciderule Adam Miller 2014-03-18 14:10:36 -07:00
• ef2889cf17 Merge pull request #50 from shriphani/master Noah Levitt 2014-03-18 11:27:49 -07:00
• 74f017c53e Merge branch 'master' of https://github.com/shriphani/heritrix3 Shriphani Palakodety 2014-03-18 06:42:54 -04:00
• 84f29e4e10 build with the right key and value types Shriphani Palakodety 2014-03-16 21:00:10 -04:00
• c81eeced7a added removal listener Shriphani Palakodety 2014-03-14 16:20:50 -04:00
• 2f60b5f935 migration to latest guava complete Shriphani Palakodety 2014-03-14 15:36:22 -04:00
• 8c9892586a removed guava dependency from heritrix. use the webarchive-commons version Shriphani Palakodety 2014-03-14 15:13:34 -04:00
• 1e8e2e2016 prevent intellij from complaining Shriphani Palakodety 2014-03-13 21:50:06 -04:00
• b0ad9358e8 bump version in pom.xml Shriphani Palakodety 2014-03-13 20:02:58 -04:00
• d0e0ee6bc6 forgot to add expireAfterWrite Shriphani Palakodety 2014-03-13 15:29:37 -04:00
• bd1047d2d2 bumped guava version to 11 Shriphani Palakodety 2014-03-13 15:26:57 -04:00
• 60cc78dac8 Merge branch 'master' of https://github.com/internetarchive/heritrix3 Shriphani Palakodety 2014-03-18 06:38:12 -04:00
• 863ff90157 Implicitly add the ${} around groovyExpression. When cxml contains ${}, it fights with spring's PropertyPlaceholderConfigurer (if enabled); and since what we're interested in is an expression, not a template for producing a string, it's more appropriate to leave it out anyway. Noah Levitt 2014-03-17 18:11:47 -07:00
• 2018da4cd6 Merge pull request #51 from nlevitt/expression-deciderule Adam Miller 2014-03-17 17:49:12 -07:00
• fb039291ef fix indentation (spaces only) Noah Levitt 2014-03-16 18:14:29 -07:00
• fb2be86e43 ExpressionDecideRule - evaluate groovy template with variable "curi" bound Noah Levitt 2014-03-16 18:13:12 -07:00
• 4c302f84be build with the right key and value types Shriphani Palakodety 2014-03-16 21:00:10 -04:00
• d4d23639b6 added removal listener Shriphani Palakodety 2014-03-14 16:20:50 -04:00
• 439f78b038 migration to latest guava complete Shriphani Palakodety 2014-03-14 15:36:22 -04:00
• abd61abbf2 removed guava dependency from heritrix. use the webarchive-commons version Shriphani Palakodety 2014-03-14 15:13:34 -04:00
• ad8cfd7075 prevent intellij from complaining Shriphani Palakodety 2014-03-13 21:50:06 -04:00
• 25ce16e893 bump version in pom.xml Shriphani Palakodety 2014-03-13 20:02:58 -04:00
• ee6a3bddb3 forgot to add expireAfterWrite Shriphani Palakodety 2014-03-13 15:29:37 -04:00
• 2cac0ebf80 bumped guava version to 11 Shriphani Palakodety 2014-03-13 15:26:57 -04:00
• a36acbacac Merge pull request #49 from nlevitt/youtube-march-2014 Adam Miller 2014-03-06 16:14:12 -08:00
• 5138a01240 warn if youtube video url is missing signature parameter Noah Levitt 2014-03-06 16:08:07 -08:00
• 6e3bd2a2e8 update to handle latest changes from youtube Noah Levitt 2014-03-05 19:05:31 -08:00
• 8a2533c95c Merge pull request #48 from nlevitt/umbra vonrosen 2014-03-03 08:21:13 -08:00
• d1eeb43e40 Merge branch 'master' into umbra Noah Levitt 2014-03-01 16:10:56 -08:00
• e46c9ed759 Continue processing url after sending to amqp (fetch, etc). Solves 2 problems, 1) crawl finishing after sending off umbra seeds and before umbra has a chance to send anything back, and 2) seeds sent to umbra not being fetched because they were already seen. We may want to change this in the future as umbra gets fancier, e.g. sending back html from DOM, but for now it should work. Noah Levitt 2014-03-01 16:10:41 -08:00
• 6a92e88b1c Merge pull request #40 from internetarchive/her-2064 Noah Levitt 2014-02-28 16:48:42 -08:00
• 54f572c356 Merge pull request #47 from adam-miller/youtube-itag-priority Noah Levitt 2014-02-28 14:43:50 -08:00
• 7fa3809828 fixing formatting Adam Miller 2014-02-28 14:33:55 -08:00
• 09c113163e Adjusting youtube extraction itag priority Adam Miller 2014-02-28 14:15:49 -08:00
• 5e55921abb Merge pull request #46 from nlevitt/switch-to-ia-webarchive-commons ikreymer 2014-02-28 12:48:44 -08:00
• e3efa3a5f2 switch dependency from ia-web-commons 1.1.1-SNAPSHOT to webarchive-commons 1.1.1-IA-SNAPSHOT Noah Levitt 2014-02-28 12:46:36 -08:00
• 22cc1222b2 Merge pull request #45 from nlevitt/update-youtube-itags Adam Miller 2014-02-28 11:13:11 -08:00
• 61539ab930 update ExtractorYoutubeFormatStreamTest.testDefaultItag() Noah Levitt 2014-02-27 17:41:38 -08:00
• 8b124ecaa9 update youtube default video format priority (see https://webarchive.jira.com/browse/ARI-3730) Noah Levitt 2014-02-27 17:37:45 -08:00
• da8e284581 Merge pull request #43 from martinsbalodis/master Noah Levitt 2014-02-27 09:34:07 -08:00
• eba9a880be Merge pull request #1 from nlevitt/pull43 Mārtiņš Balodis 2014-02-27 18:42:55 +02:00
• ea19dacbd2 handle main job log handler like current launch job log handler, close on teardown Noah Levitt 2014-02-26 17:19:50 -08:00