Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 610a1e1e97 cookie store is called BdbCookieStore again Noah Levitt 2014-09-28 14:32:34 -07:00
• 54eb1c950f rudimentary test of cookie at url with ip address as host Noah Levitt 2014-09-28 14:30:01 -07:00
• 5c21ad1762 AbstractCookieStore.getCookieFor(CrawlURI) computes top private domain, so FetchHTTP doesn't have to know anything about that Noah Levitt 2014-09-28 14:29:04 -07:00
• 4aa85af1b0 a few tests exercising some basic domain/subdomain cookie behaviors Noah Levitt 2014-09-28 14:15:27 -07:00
• a37a8de4a2 improved implementation not using TreeSet, and some initial work on integration testing of cookie handling Noah Levitt 2014-09-28 11:59:14 -07:00
• 6588641017 restore saving and loading, checkpoints, testing Noah Levitt 2014-09-27 13:55:07 -07:00
• 111294e887 oops, commit classes for initial bdb implementation Noah Levitt 2014-09-27 12:22:50 -07:00
• 47b7caad29 initial bdb implementation; facade by topmost assigned domain, since cookies can apply to subdomains Noah Levitt 2014-09-26 20:56:45 -07:00
• 886a0f2136 (forgot to add SimpleHostCookieSetStore) initial proof of concept fix for HER-2070 "cookie monster" bug Noah Levitt 2014-09-26 20:05:37 -07:00
• f769e0636d initial proof of concept fix for HER-2070 "cookie monster" bug Noah Levitt 2014-09-26 20:01:52 -07:00
• 29379c7e19 Merge pull request #84 from nlevitt/iipc-webarchive-commons Kenji Nagahashi 2014-09-11 17:19:08 -07:00
• 6ada6801de Merge pull request #94 from nlevitt/ari-3724 Kenji Nagahashi 2014-09-11 17:17:55 -07:00
• 8136046442 Merge pull request #92 from nlevitt/remove-cruft Kenji Nagahashi 2014-09-11 17:13:48 -07:00
• 5821fe7b1e FIX corner-case of bean browser failing due to an exception from hashCode(). Kenji Nagahashi 2014-09-11 11:22:01 -07:00
• e2064237cf Merge pull request #87 from vonrosen/ari-3977 Noah Levitt 2014-09-10 12:42:34 -07:00
• 1d0f60a2be webarchive-commons 1.1.4 has been released, bump dependency Noah Levitt 2014-09-10 12:31:01 -07:00
• 066cd17b5e Merge pull request #93 from vonrosen/ari-4020 Noah Levitt 2014-09-08 18:03:34 -07:00
• 2af19fa03b Revise regex instead of unescaping js. Hunter Stern 2014-09-08 16:37:03 -07:00
• 98a6c08a86 remove extraneous backslashes (thanks Kris) Noah Levitt 2014-09-05 14:38:14 -07:00
• df7d3b506c do not require "+" (plus sign) before @OPERATOR_CONTACT_URL@ in user-agent -- many bots including googlebot have the +, but some don't, so having it in the default template seems sufficient Noah Levitt 2014-09-05 12:59:17 -07:00
• 035a7b0a08 Merge pull request #80 from nlevitt/ari-3617-data-remote vonrosen 2014-09-04 16:08:33 -07:00
• 2e4f513a4c Revise parsing of string with unicode quoted data Hunter Stern 2014-09-03 15:27:58 -07:00
• ea235f4466 Allow ExtractorJS to detect urls within unicode code points for " and \' Hunter Stern 2014-09-02 15:18:13 -07:00
• 935b8f9658 remove unused dependency jets3t (thanks kenji) Noah Levitt 2014-09-02 11:00:34 -07:00
• 27d632159d Add test case for ExtractorXML Hunter Stern 2014-09-01 10:13:25 -07:00
• 2705bd3179 Merge remote-tracking branch 'internetarchive/master' into ari-3977 Hunter Stern 2014-09-01 09:23:26 -07:00
• 51a1cad300 new class DecideRuleSequenceWithAMQPFeed publishes scoping decisions to amqp Noah Levitt 2014-08-29 17:08:32 -07:00
• 37442104cc move AMQP producer code to a separate class so it can be used by classes that are not heritrix processors Noah Levitt 2014-08-29 17:05:16 -07:00
• 25c3b3e680 Merge pull request #77 from vonrosen/ari-3943 Noah Levitt 2014-08-29 15:26:37 -07:00
• ec0839a6fe remove more unused classes Noah Levitt 2014-08-29 15:23:21 -07:00
• 01af800b35 Merge pull request #91 from adam-miller/FetchHTTP_Empty_Content_Length Kenji Nagahashi 2014-08-27 17:43:34 -07:00
• bf248f407b FetchHTTP.java Handle response header with a null or empty Content-Length Adam Miller 2014-08-27 17:14:05 -07:00
• 152c4aeb9f use JSONObject.NULL for null values so that they appear explicitly in the json Noah Levitt 2014-08-27 15:33:06 -07:00
• 4109095286 switch to json instead of tsv, and split size/length field into two fields, contentLength (not including protocol headers) and size (includes headers) Noah Levitt 2014-08-27 15:06:05 -07:00
• 9e1ade3de6 Merge pull request #90 from kngenie/wbm-recrawl Noah Levitt 2014-08-27 10:24:37 -07:00
• f343ce2ae1 suppress JUnit 4 dependency through hbase-client. rewrite WbmPersistLoadProcessorTest for JUnit 3. (also disable a test that depends on production CDX server) Kenji Nagahashi 2014-08-26 10:56:35 -07:00
• fc38175182 FIX: WbmPersistLoadProcessor.requestHeaders are ignored. (code commented out during update for httpclient 4.3.3) Kenji Nagahashi 2014-08-25 17:05:50 -07:00
• bb511796ec Update POMs for newly imported Wayback-dedup module. lock httpclient version to 4.3.3 stated by heritrix-commons. have to exclude hadoop’s dependency on jets3t 0.9.0 as it pulls in httpcore 4.1.2. add easy mock 3.1 to heritrix-contrib. Kenji Nagahashi 2014-08-25 16:44:48 -07:00
• 651de1584d import Wayback-dedup modules from crawltools project to Heritrix3 contrib Kenji Nagahashi 2014-08-25 15:38:04 -07:00
• d88e19716b move WbmPersistLoadProcessor to final location in the tree Kenji Nagahashi 2014-08-25 15:37:22 -07:00
• 93dd896217 Merge pull request #86 from nlevitt/remove-cruft Kenji Nagahashi 2014-08-25 15:30:08 -07:00
• b6599d3b22 Merge pull request #89 from kngenie/beanbrowser-fix Kenji Nagahashi 2014-08-25 14:56:18 -07:00
• 2a38fa0b12 Don’t let exception from property getter fail entire bean-browser. Kenji Nagahashi 2014-08-25 13:36:25 -07:00
• 3450cf37e4 Update HQ modules and WBM-dedup module to match Heritrix 3.3.0. HQ modules: for method removed in Heritrix, API deprecated in httpclient WBM-dedup: fix NPE in FetchHistoryProcessor due to changed expectation. Kenji Nagahashi 2014-08-24 00:45:04 -07:00
• 6868a9ee5c Merge pull request #88 from nlevitt/crawluri-compare-bug Kenji Nagahashi 2014-08-19 17:17:03 -07:00
• 04921073ca fix bug in CrawlURI.compare() discovered by Kenji, add unit test CrawlURITest.testOrdering() Noah Levitt 2014-08-19 17:14:32 -07:00
• f5263f6b1d dumpPendingAtClose for AMQPCrawlLogFeed, dumps pending urls with 0 status to the amqp feed, similar to Frontier.dumpPendingAtClose for the crawl log Noah Levitt 2014-08-13 13:04:17 -07:00
• 591f0f9f0f expose forAllPendingDo() as public so it can be used elsewhere Noah Levitt 2014-08-13 13:02:28 -07:00
• f1b0633a3d remove unused Transform* classes Noah Levitt 2014-08-13 10:53:03 -07:00
• 4a6d92e344 include host in feed Noah Levitt 2014-08-12 16:46:25 -07:00
• 9a0c0656b2 move HostsReport.fixup(String) to CrawlHost.fixUpName() so it can be called elsewhere Noah Levitt 2014-08-12 16:45:53 -07:00
• ae16163aff new module AMQPCrawlLogFeed, sends crawl log lines in tab-separated format to amqp; refactor so that amqp code is shared with AMQPPublishProcessor by way of new parent abstract class AMQPProducerProcessor Noah Levitt 2014-08-12 16:23:08 -07:00
• 51081a652c make AbstractFrontier.needReenqueuing() public so it can be called elsewhere Noah Levitt 2014-08-12 16:20:25 -07:00
• 47a62f7c6f remove a couple of crufty lines of code Noah Levitt 2014-08-12 16:13:59 -07:00
• 08e24bbdff update guess at log line length based on real world data Noah Levitt 2014-08-12 16:13:11 -07:00
• 65cf7942c2 replace use of another method removed in updated guava library Noah Levitt 2014-08-11 11:23:59 -07:00
• 651d18beb1 replace use of method removed from guava library Noah Levitt 2014-08-08 17:38:51 -07:00
• 21f17a0d9f switch to mainline iipc webarchive-commons latest release Noah Levitt 2014-08-08 17:26:21 -07:00
• 7f1700400b Allow xml extractor to handle urls in CDATA. Hunter Stern 2014-08-01 17:37:31 -07:00
• 4298c3de69 Merge pull request #83 from nlevitt/revisit-redux vonrosen 2014-07-25 16:08:33 -07:00
• e5ecde6c2f oops! count novel urls/bytes for hosts report, etc Noah Levitt 2014-07-25 15:57:53 -07:00
• e7b034c60e Merge pull request #82 from kris-sigur/HER-2071 Noah Levitt 2014-07-25 12:13:02 -07:00
• 8edebc0943 Fix for HER-2071 Kristinn Sigurðsson 2014-07-25 13:32:08 +00:00
• 1c6746c6c8 Merge pull request #81 from nlevitt/hbase-cdh5 Vinay Goel 2014-07-23 13:34:54 -07:00
• 40eae4cbce Merge branch 'master' into hbase-cdh5 Noah Levitt 2014-07-23 13:33:58 -07:00
• ba48f96d93 Merge pull request #78 from nlevitt/extract-html-no-mimetype vonrosen 2014-07-22 15:41:44 -07:00
• d0ebd40578 Merge pull request #79 from nlevitt/revisit-redux Kristinn Sigurðsson 2014-07-21 08:40:30 +00:00
• 410e1709b7 ExtractorHTML when a/@href links include the attribute data-remote="true", include that info the viaContext; and in FetchHTTPRequest, when a CrawlURI has data-remote="true", add the http header "X-Requested-With: XMLHttpRequest"; data-remote is a jquery thing that means the url can be requested via ajax in a browser... some urls work wrong without the X-Requested-With header... this is not a great place for a non-generic thing like this, but where else could it go... Noah Levitt 2014-07-18 19:48:15 -07:00
• 4b75dfa16f treat content as html and extract links if it looks like html, even if mime type belies that Noah Levitt 2014-07-17 21:09:51 -07:00
• bdeaa8bf0e Remove fetch status setting in AMQP publisher. Hunter Stern 2014-07-16 15:46:53 -07:00
• c3d5996314 Force urls received from AMQP to be recrawled so custom http headers can be used to make the request for that url. Hunter Stern 2014-07-14 17:29:15 -07:00
• cb6ac8b8b5 restore writeRevisitFor* settings, but issue warning in setter and ignore the value Noah Levitt 2014-07-14 10:53:37 -07:00
• da30439d61 oops, fix 304 tests Noah Levitt 2014-07-11 12:47:15 -07:00
• 77492802b3 test 304 not modified revist profile in FetchHTTPTest Noah Levitt 2014-07-10 21:30:54 -07:00
• 2292ea9287 remove deprecated unused stuff Noah Levitt 2014-07-10 21:29:47 -07:00
• 810ad91b9d set revisit profile on 304 not modified Noah Levitt 2014-07-10 21:28:52 -07:00
• 905a98c0c2 use RevisitProfile for tallying stats Noah Levitt 2014-07-10 21:27:50 -07:00
• d9fd66878b Merge branch 'master' into revisit-redux Noah Levitt 2014-07-10 15:01:43 -07:00
• f101bba509 Merge pull request #76 from nlevitt/HER-2039 Kristinn Sigurðsson 2014-07-10 09:41:00 +00:00
• ee52f48be9 remove LinksScoper (deprecated since 2009) and remove recently added code only used in LinksScoper Noah Levitt 2014-07-09 10:14:24 -07:00
• 7bd9f3acaf switch usage of Link to CrawlURI in heritrix-contrib Noah Levitt 2014-07-08 11:56:56 -07:00
• 6e9a0a4d05 only write WARC-Refers-To header when value is not null Noah Levitt 2014-07-07 19:21:07 -07:00
• 2854798964 new parameter keySuffix-- If not null, keySuffix is appended to the lookup key when loading and storing digest history. Thus the key looks like {digest}{keySuffix}, e.g. "sha1:22SFHXERHNFOEY6WK7YOUN4PFIPZSB4D-1193". The purpose is to support multiple namespaces in a single hbase table, to avoid proliferation of small tables. The reason we use a suffix instead of a prefix is to leave open the possibility of deduplication across these different namespaces at some point in the future. Noah Levitt 2014-07-07 19:05:39 -07:00
• bed1e58adf copy over and adjust equals(), hashCode(), compareTo() from old Link class to CrawlURI, so that tests pass Noah Levitt 2014-07-07 16:59:18 -07:00
• bfac3c8bbc ensure non-null pathFromSeed to fix some unit tests Noah Levitt 2014-07-07 12:25:27 -07:00
• 3c2e1315bb other changes required for merge from master into HER-2039 Noah Levitt 2014-07-07 11:48:36 -07:00
• f23a59a918 Merge branch 'master' into HER-2039 Noah Levitt 2014-07-07 11:45:27 -07:00
• c48de14663 Merge pull request #75 from nlevitt/uri-inherited-data-fix Adam Miller 2014-07-02 15:45:56 -07:00
• f2055ee2ae in CrawlURI.createCrawlURI(), avoid clobbering inherited data with data from Link Noah Levitt 2014-07-02 13:17:42 -07:00
• 47e4e98608 Merge pull request #74 from vonrosen/ari-3943 Noah Levitt 2014-06-27 17:00:40 -07:00
• 5eade58de4 Fix for https://webarchive.jira.com/browse/ARI-3943 Hunter Stern 2014-06-27 16:20:42 -07:00
• 3e35c967ab upgrade hbase to cdh5 Noah Levitt 2014-06-16 18:33:25 -07:00
• 8c7b0fed92 Merge pull request #73 from kris-sigur/HER-2068 Noah Levitt 2014-06-04 13:03:04 -07:00
• ece0149339 Fix for HER-2068 Kristinn Sigurðsson 2014-06-04 14:59:13 +00:00
• fc43a29459 Merge pull request #68 from nlevitt/handle-apply-overlay-exc Roger G. Coram 2014-05-28 09:59:54 +01:00
• 12186af93d Merge pull request #72 from nlevitt/persist-annotations vonrosen 2014-05-22 16:49:37 -07:00
• 67a0c59758 add A_ANNOTATIONS to persistentKeys so that CrawlURI doesn't lose its annotations when deferred Noah Levitt 2014-05-22 16:48:01 -07:00
• f2bc21435f Merge pull request #71 from nlevitt/checkpoint-no-ui-hang vonrosen 2014-05-22 16:44:22 -07:00
• 39814d1efd avoid calling CheckpointService.hasAvailableCheckpoints() when crawl not launchable and therefore the info is not needed, because it calls isRunning() which is synchronized and blocks when checkpoint is in progress Noah Levitt 2014-05-22 16:40:44 -07:00
• 1c0d6c60fb Merge pull request #70 from nlevitt/ari-3712 vonrosen 2014-05-22 10:58:21 -07:00