Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• bf248f407b FetchHTTP.java Handle response header with a null or empty Content-Length Adam Miller 2014-08-27 17:14:05 -07:00
• 152c4aeb9f use JSONObject.NULL for null values so that they appear explicitly in the json Noah Levitt 2014-08-27 15:33:06 -07:00
• 4109095286 switch to json instead of tsv, and split size/length field into two fields, contentLength (not including protocol headers) and size (includes headers) Noah Levitt 2014-08-27 15:06:05 -07:00
• 9e1ade3de6 Merge pull request #90 from kngenie/wbm-recrawl Noah Levitt 2014-08-27 10:24:37 -07:00
• f343ce2ae1 suppress JUnit 4 dependency through hbase-client. rewrite WbmPersistLoadProcessorTest for JUnit 3. (also disable a test that depends on production CDX server) Kenji Nagahashi 2014-08-26 10:56:35 -07:00
• fc38175182 FIX: WbmPersistLoadProcessor.requestHeaders are ignored. (code commented out during update for httpclient 4.3.3) Kenji Nagahashi 2014-08-25 17:05:50 -07:00
• bb511796ec Update POMs for newly imported Wayback-dedup module. lock httpclient version to 4.3.3 stated by heritrix-commons. have to exclude hadoop’s dependency on jets3t 0.9.0 as it pulls in httpcore 4.1.2. add easy mock 3.1 to heritrix-contrib. Kenji Nagahashi 2014-08-25 16:44:48 -07:00
• 651de1584d import Wayback-dedup modules from crawltools project to Heritrix3 contrib Kenji Nagahashi 2014-08-25 15:38:04 -07:00
• d88e19716b move WbmPersistLoadProcessor to final location in the tree Kenji Nagahashi 2014-08-25 15:37:22 -07:00
• 93dd896217 Merge pull request #86 from nlevitt/remove-cruft Kenji Nagahashi 2014-08-25 15:30:08 -07:00
• b6599d3b22 Merge pull request #89 from kngenie/beanbrowser-fix Kenji Nagahashi 2014-08-25 14:56:18 -07:00
• 2a38fa0b12 Don’t let exception from property getter fail entire bean-browser. Kenji Nagahashi 2014-08-25 13:36:25 -07:00
• 3450cf37e4 Update HQ modules and WBM-dedup module to match Heritrix 3.3.0. HQ modules: for method removed in Heritrix, API deprecated in httpclient WBM-dedup: fix NPE in FetchHistoryProcessor due to changed expectation. Kenji Nagahashi 2014-08-24 00:45:04 -07:00
• 6868a9ee5c Merge pull request #88 from nlevitt/crawluri-compare-bug Kenji Nagahashi 2014-08-19 17:17:03 -07:00
• 04921073ca fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering() Noah Levitt 2014-08-19 17:14:32 -07:00
• f5263f6b1d dumpPendingAtClose for AMQPCrawlLogFeed, dumps pending urls with 0 status to the amqp feed, similar to Frontier.dumpPendingAtClose for the crawl log Noah Levitt 2014-08-13 13:04:17 -07:00
• 591f0f9f0f expose forAllPendingDo() as public so it can be used elsewhere Noah Levitt 2014-08-13 13:02:28 -07:00
• f1b0633a3d remove unused Transform* classes Noah Levitt 2014-08-13 10:53:03 -07:00
• 4a6d92e344 include host in feed Noah Levitt 2014-08-12 16:46:25 -07:00
• 9a0c0656b2 move HostsReport.fixup(String) to CrawlHost.fixUpName() so it can be called elsewhere Noah Levitt 2014-08-12 16:45:53 -07:00
• ae16163aff new module AMQPCrawlLogFeed, sends crawl log lines in tab-separated format to amqp; refactor so that amqp code is shared with AMQPPublishProcessor by way of new parent abstract class AMQPProducerProcessor Noah Levitt 2014-08-12 16:23:08 -07:00
• 51081a652c make AbstractFrontier.needReenqueuing() public so it can be called elsewhere Noah Levitt 2014-08-12 16:20:25 -07:00
• 47a62f7c6f remove a couple of crufty lines of code Noah Levitt 2014-08-12 16:13:59 -07:00
• 08e24bbdff update guess at log line length based on real world data Noah Levitt 2014-08-12 16:13:11 -07:00
• 65cf7942c2 replace use of another method removed in updated guava library Noah Levitt 2014-08-11 11:23:59 -07:00
• 651d18beb1 replace use of method removed from guava library Noah Levitt 2014-08-08 17:38:51 -07:00
• 21f17a0d9f switch to mainline iipc webarchive-commons latest release Noah Levitt 2014-08-08 17:26:21 -07:00
• 7f1700400b Allow xml extractor to handle urls in CDATA. Hunter Stern 2014-08-01 17:37:31 -07:00
• 4298c3de69 Merge pull request #83 from nlevitt/revisit-redux vonrosen 2014-07-25 16:08:33 -07:00
• e5ecde6c2f oops! count novel urls/bytes for hosts report, etc Noah Levitt 2014-07-25 15:57:53 -07:00
• e7b034c60e Merge pull request #82 from kris-sigur/HER-2071 Noah Levitt 2014-07-25 12:13:02 -07:00
• 8edebc0943 Fix for HER-2071 Kristinn Sigurðsson 2014-07-25 13:32:08 +00:00
• 1c6746c6c8 Merge pull request #81 from nlevitt/hbase-cdh5 Vinay Goel 2014-07-23 13:34:54 -07:00
• 40eae4cbce Merge branch 'master' into hbase-cdh5 Noah Levitt 2014-07-23 13:33:58 -07:00
• ba48f96d93 Merge pull request #78 from nlevitt/extract-html-no-mimetype vonrosen 2014-07-22 15:41:44 -07:00
• d0ebd40578 Merge pull request #79 from nlevitt/revisit-redux Kristinn Sigurðsson 2014-07-21 08:40:30 +00:00
• 410e1709b7 ExtractorHTML when a/@href links include the attribute data-remote="true", include that info the viaContext; and in FetchHTTPRequest, when a CrawlURI has data-remote="true", add the http header "X-Requested-With: XMLHttpRequest"; data-remote is a jquery thing that means the url can be requested via ajax in a browser... some urls work wrong without the X-Requested-With header... this is not a great place for a non-generic thing like this, but where else could it go... Noah Levitt 2014-07-18 19:48:15 -07:00
• 4b75dfa16f treat content as html and extract links if it looks like html, even if mime type belies that Noah Levitt 2014-07-17 21:09:51 -07:00
• bdeaa8bf0e Remove fetch status setting in AMQP publisher. Hunter Stern 2014-07-16 15:46:53 -07:00
• c3d5996314 Force urls received from AMQP to be recrawled so custom http headers can be used to make the request for that url. Hunter Stern 2014-07-14 17:29:15 -07:00
• cb6ac8b8b5 restore writeRevisitFor* settings, but issue warning in setter and ignore the value Noah Levitt 2014-07-14 10:53:37 -07:00
• da30439d61 oops, fix 304 tests Noah Levitt 2014-07-11 12:47:15 -07:00
• 77492802b3 test 304 not modified revist profile in FetchHTTPTest Noah Levitt 2014-07-10 21:30:54 -07:00
• 2292ea9287 remove deprecated unused stuff Noah Levitt 2014-07-10 21:29:47 -07:00
• 810ad91b9d set revisit profile on 304 not modified Noah Levitt 2014-07-10 21:28:52 -07:00
• 905a98c0c2 use RevisitProfile for tallying stats Noah Levitt 2014-07-10 21:27:50 -07:00
• d9fd66878b Merge branch 'master' into revisit-redux Noah Levitt 2014-07-10 15:01:43 -07:00
• f101bba509 Merge pull request #76 from nlevitt/HER-2039 Kristinn Sigurðsson 2014-07-10 09:41:00 +00:00
• ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper Noah Levitt 2014-07-09 10:14:24 -07:00
• 7bd9f3acaf switch usage of Link to CrawlURI in heritrix-contrib Noah Levitt 2014-07-08 11:56:56 -07:00
• 6e9a0a4d05 only write WARC-Refers-To header when value is not null Noah Levitt 2014-07-07 19:21:07 -07:00
• 2854798964 new parameter keySuffix-- If not null, keySuffix is appended to the lookup key when loading and storing digest history. Thus the key looks like {digest}{keySuffix}, e.g. "sha1:22SFHXERHNFOEY6WK7YOUN4PFIPZSB4D-1193". The purpose is to support multiple namespaces in a single hbase table, to avoid proliferation of small tables. The reason we use a suffix instead of a prefix is to leave open the possibility of deduplication across these different namespaces at some point in the future. Noah Levitt 2014-07-07 19:05:39 -07:00
• bed1e58adf copy over and adjust equals(), hashCode(), compareTo() from old Link class to CrawlURI, so that tests pass Noah Levitt 2014-07-07 16:59:18 -07:00
• bfac3c8bbc ensure non-null pathFromSeed to fix some unit tests Noah Levitt 2014-07-07 12:25:27 -07:00
• 3c2e1315bb other changes required for merge from master into HER-2039 Noah Levitt 2014-07-07 11:48:36 -07:00
• f23a59a918 Merge branch 'master' into HER-2039 Noah Levitt 2014-07-07 11:45:27 -07:00
• c48de14663 Merge pull request #75 from nlevitt/uri-inherited-data-fix Adam Miller 2014-07-02 15:45:56 -07:00
• f2055ee2ae in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data from Link Noah Levitt 2014-07-02 13:17:42 -07:00
• 47e4e98608 Merge pull request #74 from vonrosen/ari-3943 Noah Levitt 2014-06-27 17:00:40 -07:00
• 5eade58de4 Fix for https://webarchive.jira.com/browse/ARI-3943 Hunter Stern 2014-06-27 16:20:42 -07:00
• 3e35c967ab upgrade hbase to cdh5 Noah Levitt 2014-06-16 18:33:25 -07:00
• 8c7b0fed92 Merge pull request #73 from kris-sigur/HER-2068 Noah Levitt 2014-06-04 13:03:04 -07:00
• ece0149339 Fix for HER-2068 Kristinn Sigurðsson 2014-06-04 14:59:13 +00:00
• fc43a29459 Merge pull request #68 from nlevitt/handle-apply-overlay-exc Roger G. Coram 2014-05-28 09:59:54 +01:00
• 12186af93d Merge pull request #72 from nlevitt/persist-annotations vonrosen 2014-05-22 16:49:37 -07:00
• 67a0c59758 add A_ANNOTATIONS to persistentKeys so that CrawlURI doesn't lose its annotations when deferred Noah Levitt 2014-05-22 16:48:01 -07:00
• f2bc21435f Merge pull request #71 from nlevitt/checkpoint-no-ui-hang vonrosen 2014-05-22 16:44:22 -07:00
• 39814d1efd avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress Noah Levitt 2014-05-22 16:40:44 -07:00
• 1c0d6c60fb Merge pull request #70 from nlevitt/ari-3712 vonrosen 2014-05-22 10:58:21 -07:00
• 26222ef660 Merge pull request #69 from vonrosen/ari-3853 Noah Levitt 2014-05-15 17:38:41 -07:00
• d570498fd9 New Unit tests added now. Hunter Stern 2014-05-15 17:37:26 -07:00
• 80880b4409 Unit test and other changes. Hunter Stern 2014-05-15 17:34:43 -07:00
• aff5b9ac4e for ARI-3712, add extracted links relative to both via and base, and annotate with "extractorSWFRelToVia", "extractorSWFRelToBase", or "extractorSWFRelToBoth" if resulting link is the same whether relative to base or via Noah Levitt 2014-05-15 13:31:29 -07:00
• 519717fd7b make addRelativeToBase() and addRelativeToVia() return the link added, or null; copy getAnnotations() from CrawlURI Noah Levitt 2014-05-15 13:27:50 -07:00
• f24cfff6af New youtube channel video extractor for https://webarchive.jira.com/browse/ARI-3865 Hunter Stern 2014-05-13 17:41:42 -07:00
• 963e04e50c handle exception determining whether to apply overlay Noah Levitt 2014-05-13 15:47:21 -07:00
• 3e213c82bc Merge pull request #67 from nlevitt/umbra Adam Miller 2014-05-13 14:44:09 -07:00
• 08a4aafa23 don't log severe with stack trace on normal amqp shutdown Noah Levitt 2014-05-13 14:39:35 -07:00
• 38cd14ecac Merge pull request #63 from adam-miller/ExtractorPDFContent Noah Levitt 2014-05-13 14:09:20 -07:00
• aaecd6d78c Merge branch 'master' of git://github.com/internetarchive/heritrix3 into ExtractorPDFContent Adam Miller 2014-05-13 14:04:29 -07:00
• fb42c07c92 bugfix, new test case. Adam Miller 2014-05-13 14:03:03 -07:00
• ab7a60f9e3 Merge pull request #66 from nlevitt/cosmetics Adam Miller 2014-05-13 13:31:11 -07:00
• 85e7a5ee78 Bugfix & new test cases. Adam Miller 2014-05-12 16:52:32 -07:00
• ba68649ba0 Merge remote-tracking branch 'origin/master' into cosmetics Noah Levitt 2014-05-07 10:02:06 -07:00
• b7f8a0669f Merge branch 'cosmetics' of github.com:nlevitt/heritrix3 into cosmetics Noah Levitt 2014-05-07 10:02:00 -07:00
• 33e782e2fd oops, make "exit java process" button work again Noah Levitt 2014-05-07 10:01:47 -07:00
• eb32d7a8b8 Merge pull request #65 from nlevitt/umbra Adam Miller 2014-05-02 15:47:44 -07:00
• 949c9f086c shut down the starter-restarter thread at crawl finish!! Noah Levitt 2014-05-02 15:42:29 -07:00
• dd510cbf3b Merge pull request #64 from adam-miller/ViaSurtPrefixedDecideRule Noah Levitt 2014-05-01 18:14:25 -07:00
• ae878d29cf Code cleanup Adam Miller 2014-05-01 15:22:16 -07:00
• f2bebd41b0 Cleanup / Optimization Adam Miller 2014-05-01 12:36:05 -07:00
• 753404f662 Adding ViaSurtPrefixedDecideRule. Returns true if a surtform 'via' matches any surt in a specified list Adam Miller 2014-03-03 16:51:11 -08:00
• 2b1c330be6 Further cleanup Adam Miller 2014-04-30 12:18:14 -07:00
• e6d646d4b4 Cleanup / optimization Adam Miller 2014-04-30 12:14:32 -07:00
• 34a9172e69 Organize Imports Adam Miller 2014-04-29 17:25:43 -07:00
• 5181c1ef64 Cleaning up test cases and removing unused code Adam Miller 2014-04-29 17:11:18 -07:00
• 33277445f6 Merge branch 'master' of git://github.com/internetarchive/heritrix3 into ExtractorPDFContent Adam Miller 2014-04-29 17:06:55 -07:00
• 49069f2dd1 Adding comment for library dependencies Adam Miller 2014-04-29 17:06:11 -07:00
• 7e8a66e7d8 New Test cases, fixing bugs Adam Miller 2014-04-29 10:50:04 -07:00
• 747bbeba97 Merge pull request #62 from nlevitt/ari-3765 vonrosen 2014-04-28 09:39:37 -07:00