Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 1f700e5cac poking around at basic auth, nothing close to working yet Noah Levitt 2012-07-02 10:31:32 -07:00
• 29abbecdf3 make data key "httpResponseHeaders" a constant Noah Levitt 2012-07-02 10:30:27 -07:00
• 8abe871af6 configure logging Noah Levitt 2012-07-02 10:29:19 -07:00
• 4790be501c fix up saving of http response headers in CrawlURI Noah Levitt 2012-06-29 22:00:35 -07:00
• 5140a3f5bb basic-auth test works! Noah Levitt 2012-06-29 18:34:17 -07:00
• 373abbbe81 latest jetty 6.x; starting on basic-auth test Noah Levitt 2012-06-29 17:19:46 -07:00
• 09d1839e6c test sending cookies and ignoring cookies Noah Levitt 2012-06-29 15:42:31 -07:00
• 69d7b0a0cd make jetty server not require anything external; also make it work with maven Noah Levitt 2012-06-29 13:43:54 -07:00
• 72303cc103 make jetty server not require anything external; also make it work with maven Noah Levitt 2012-06-29 12:33:20 -07:00
• 93958abfcd these help classes for fetchhttp2 need to be in modules/ since they use stuff from modules/ Noah Levitt 2012-06-29 11:19:56 -07:00
• c7ba0675cf more license headers Noah Levitt 2012-06-29 07:40:53 -07:00
• 9ef01de7b4 twiddles Noah Levitt 2012-06-29 07:36:36 -07:00
• b6cc860438 refactor to get rid of redundancy Noah Levitt 2012-06-29 07:24:09 -07:00
• 98a4be5eb1 fix user-agent test, add more tests Noah Levitt 2012-06-29 07:05:44 -07:00
• 1a79348cf7 test a few more things Noah Levitt 2012-06-29 02:55:26 -07:00
• 1a1d1b008c allow FetchHTTP2 infrastructure to fall back on default java hostname resolution Noah Levitt 2012-06-29 02:40:34 -07:00
• 1f315a7cc7 unit tests for FetchHTTP and FetchHTTP2 Noah Levitt 2012-06-29 02:04:39 -07:00
• 2653583cdd unit tests for FetchHTTP and FetchHTTP2 Noah Levitt 2012-06-29 01:59:58 -07:00
• 58d827c85b Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-06-28 18:08:39 -07:00
• 6225f6fa0b Fix HER-2013 scheme-less relative URI not resolved correctly * LaxURI.java parseUriReference() - fix logic testing if uri string is a path to respect scheme-less uris even if they contain colons * UURIFactoryTest.java add test for scheme-less relative uri (with colons) Noah Levitt 2012-06-28 16:49:50 -07:00
• 63c6ca59c2 Merge branch 'master' into fetchhttp-hc Noah Levitt 2012-06-25 16:42:34 -07:00
• c40c24e97d Merge pull request #12 from travisfw/generics-fixes Travis Wellman 2012-06-25 11:37:18 -07:00
• 6626b44286 generics warnings fixes Travis Wellman 2012-06-21 18:23:06 -07:00
• 2b0141fd0d bdb cookie store for FetchHTTP2 Noah Levitt 2012-06-20 18:49:57 -07:00
• 49c00c0387 do not rely on ToeThread for the ServerCache, pass it to where it's needed instead Noah Levitt 2012-06-19 13:54:05 -07:00
• 56f0c92690 resolve hostnames from heritrix server cache (the bad old way) Noah Levitt 2012-06-19 13:32:34 -07:00
• 1d0beedf2b use http 1.0 by default, other request configuration stuff Noah Levitt 2012-06-19 12:09:46 -07:00
• 233b72cb34 get charset, encodings from headers Noah Levitt 2012-06-19 11:28:01 -07:00
• de152eaa9e browser-compatibility mode for cookie matching; that's what old FetchHTTP does, and also avoids sending weird Cookie2:$Version1 Noah Levitt 2012-06-18 17:49:09 -07:00
• 3c7bd8afd6 no keep-alive Noah Levitt 2012-06-18 17:06:46 -07:00
• a779e46ac7 set user-agent Noah Levitt 2012-06-18 15:55:07 -07:00
• 7422a7b992 move httpclient helper classes to commons/ Noah Levitt 2012-06-18 15:43:52 -07:00
• 7f0efbdac8 do recording through SessionOutputBuffer implementation like we have for SessionInputBuffer; refactor helper classes into their own file RecordingHttpClient.java Noah Levitt 2012-06-18 15:42:50 -07:00
• 0d172be888 not using HcInputWrapper anymore Noah Levitt 2012-06-18 12:31:52 -07:00
• 316391786e license headers Noah Levitt 2012-06-17 23:23:39 -07:00
• 6c56ce41a0 different deeper way to insert our Recorder, letting us correctly note end of http headers. is there a problem with not buffering at this level? Noah Levitt 2012-06-17 15:48:19 -07:00
• b27b3f67e5 continuing to mess with httpcomponents Noah Levitt 2012-06-15 18:29:21 -07:00
• 9a5570c6f1 continuing to mess with httpcomponents Noah Levitt 2012-06-15 16:34:55 -07:00
• 3a0f50d031 FetchHTTP2 - record, archive request as well Noah Levitt 2012-06-14 19:01:15 -07:00
• 5043dcc35b FetchHTTP2, based on httpcomponents, is working in a rudimentary way. This commit also gets rid of most egregious commons-httpclient-specific code in CrawlURI Noah Levitt 2012-06-14 18:34:43 -07:00
• 793d733f1e continuing to mess with httpcomponents Noah Levitt 2012-06-14 14:00:16 -07:00
• e5668726a1 starting to mess with httpcomponents Noah Levitt 2012-06-13 14:11:11 -07:00
• 6ab7415342 Revert "* ConfigFileEditor.java, ConfigPathEditor.java" Noah Levitt 2012-06-12 10:15:27 -07:00
• 6dae8f9a17 * ConfigFileEditor.java, ConfigPathEditor.java remove unused classes Noah Levitt 2012-06-11 17:21:23 -07:00
• e1f5bc5319 HER-1987 record charset used for link extraction in warc metadata record * WARCWriterProcessor.java writeMetadata() - write metadata entry charsetForLinkExtraction Noah Levitt 2012-06-06 12:02:41 -07:00
• 7eabc7ef73 HER-1987 record charset used for link extraction in warc metadata record * WARCWriterProcessor.java writeMetadata() - if extractors have left annotations about the encoding used to interpret content, include them in the metadata (HER-1987) Noah Levitt 2012-06-06 11:40:58 -07:00
• 99acaa9bfa Fix tests revealed to be failing * UriUtils.java isLikelyFalsePositive() - do not flag urls starting with ../ as likely false positives * UriUtilsTest.java remove test url that is now rejected Noah Levitt 2012-06-06 09:00:13 -07:00
• 342ea33be5 Move test classes into test area of source tree * LongToIntConsistentHashTest.java, UriUtilsTest.java Noah Levitt 2012-06-05 18:03:33 -07:00
• 35378df2fa Merge branch 'master' into no-package-scope Noah Levitt 2012-06-05 17:58:40 -07:00
• 3dce1e909d Change scope of most/all default-scope i.e. package-scoped class members to protected, to make heritrix more friendly to third-party extensions. (I don't think there was ever an intent to limit the scope of any of these to package-only.) Noah Levitt 2012-06-05 17:55:41 -07:00
• e965421281 * WorkQueueFrontier.java remove unused variable Noah Levitt 2012-06-05 16:26:00 -07:00
• ee1e25ca69 Merge pull request #11 from travisfw/new-reports Travis Wellman 2012-06-05 16:24:43 -07:00
• 07a2fc65ea use methods to specify which report rather than string constants Travis Wellman 2012-06-01 17:57:25 -07:00
• 493e2423f6 begin removing MultiReporter Travis Wellman 2012-06-01 17:14:41 -07:00
• 3bebf10674 * ScriptedDecideRule.java getEngine() - tiny cosmetic fixups Noah Levitt 2012-05-31 16:29:33 -07:00
• 392a4b3fd0 correct synchronization and threadlocal logic Travis Wellman 2012-05-31 16:09:42 -07:00
• 1015c5fa00 Fix HER-1973 Regression in queue budget overrides * WorkQueueFrontier.java findEligibleURI() - apply sheet overrides before refreshing queue budgets Noah Levitt 2012-05-15 17:35:49 -07:00
• 809981ef02 * PersistProcessor.java populatePersistEnvFromLog() - handle exception deserializing line Noah Levitt 2012-05-10 19:00:27 -07:00
• 440129ad23 Merge branch 'master' of git@github.com:internetarchive/heritrix3.git gojomo 2012-05-02 01:48:21 -07:00
• 9fffa8fc53 **/pom.xml bump version to 3.1.2-SNAPSHOT Noah Levitt 2012-05-01 18:17:33 -07:00
• 43d57dcaf4 * README.txt update for 3.1.1 3.1.1 Noah Levitt 2012-05-01 18:05:38 -07:00
• 79c40346fc **/pom.xml * bump version number to 3.1.1 for release Noah Levitt 2012-05-01 18:01:06 -07:00
• a7c63b8d00 * FetchHTTP.java remove redundant nonsensical comment Noah Levitt 2012-05-01 17:53:52 -07:00
• b05f64e2ab Merge branch 'master' of git@github.com:internetarchive/heritrix3.git gojomo 2012-04-25 15:09:41 -07:00
• e4f12bb20f Merge pull request #6 from ato/codemirror Gordon Mohr 2012-04-25 15:04:10 -07:00
• b80966ebd9 HER-2001: Use the CodeMirror editor for crawl config and script console Alex Osborne 2012-03-31 18:34:11 +11:00
• 42e759c01d List correct defaults for BdbModule cacheSize and cachePercentage with an explanatory comment. Noah Levitt 2012-04-25 11:10:56 -07:00
• a2880c4306 HER-1990 large robots.txt problematic * Robotstxt.java enforce max size of robots.txt Noah Levitt 2012-04-20 13:57:53 -07:00
• 52d5307a9b Merge branch 'her-1948' Noah Levitt 2012-04-20 12:33:19 -07:00
• 8d0a1678de * effective_tld_names.dat update to latest from publicsuffix.org Noah Levitt 2012-04-20 12:25:57 -07:00
• 3fed078817 Merge pull request #9 from kngenie/publicsuffixes2 Noah Levitt 2012-04-20 12:19:47 -07:00
• da28258f90 Option to set bdb cache size as an absolute number instead of a percentage (part of HER-1948) Noah Levitt 2012-04-19 18:16:53 -07:00
• 4a64c6ba59 Revert "Altered FetchStats to also keep track of the number of URIs that have" Noah Levitt 2012-04-18 15:53:49 -07:00
• 9551c774af Merge branch 'master' into http-status-deciderule Noah Levitt 2012-04-18 15:47:24 -07:00
• 8e853c5f8d HER-2005 existing content length decide rule is not flexible enough * ResponseContentLengthDecideRule.java new predicated decide rule that will ACCEPT or REJECT a uri if the content body is within a specified size range Noah Levitt 2012-04-18 13:15:15 -07:00
• 0a4d56b638 Fix for HER-2004 SEVERE error in FetchWhois on some government domains * FetchWhois.java addWhoisLinks() - handle IllegalStateException from InternetDomainName.topPrivateDomain() Noah Levitt 2012-04-18 10:37:24 -07:00
• 27d621f21a renamed PublicSuffixes2 to PublicSuffixes. deleted debug println. Kenji Nagahashi 2011-11-05 23:26:55 -07:00
• a0d52e6c17 PublicSuffixes2Test still had tests for PublicSuffix. fixed. Kenji Nagahashi 2011-10-20 19:39:36 -07:00
• b2d62a81ff trial rewrite of PublicSuffixes for small memory footprint (no significant performance gain) Kenji Nagahashi 2011-10-11 17:11:06 -07:00
• 31df3a8f03 Merge branch 'master' into speculative-links Noah Levitt 2012-04-13 18:16:08 -07:00
• 52eef84658 note about preselector.recheckScope for use of IpAddressSetDecideRule Travis Wellman 2012-04-12 17:34:21 -07:00
• 54b1e245eb gitignore for maven build artifacts Travis Wellman 2012-04-12 17:32:57 -07:00
• 563da14330 Merge pull request #7 from travisfw/IpAddressSetDecideRule Travis Wellman 2012-04-12 16:52:00 -07:00
• 57d2f42954 formatting Travis Wellman 2012-04-12 16:50:22 -07:00
• 64e9cc367b remove println, make class public Travis Wellman 2012-04-12 16:46:16 -07:00
• 2a3945ff79 Merge branch 'master' of github.com:internetarchive/heritrix3 into IpAddressSetDecideRule Travis Wellman 2012-04-02 18:17:44 -07:00
• f9f9e62f4c Reject or accept matching a set of IP addresses. Travis Wellman 2012-04-02 18:16:23 -07:00
• f19a72efaf HER-2001: Use the CodeMirror editor for crawl config and script console Alex Osborne 2012-03-31 18:34:11 +11:00
• 9626b8454a * EditRepresentation.java make edit page (for cxml mainly) prettier with big textarea (see Kenji's comment on HER-1993 dated 2012-03-27 18:03) Noah Levitt 2012-03-30 11:44:28 -07:00
• 335b32aecc * ExtractorHTML.java some fixup on RELEVANT_TAG_EXTRACTOR javadoc comment Noah Levitt 2012-03-20 15:14:14 -07:00
• 4eab6461ed Merge remote-tracking branch 'adam-miller/master' Noah Levitt 2012-03-20 15:06:58 -07:00
• a108795d50 HER-1998 - Adding comments, adjusting test case to prevent subclasses from failing new tests Adam Miller 2012-03-19 18:19:00 -07:00
• ea42c0c653 * EngineResource.java acceptRepresentation() - "I'm sure" parameter is now "im_sure" Noah Levitt 2012-03-19 17:30:18 -07:00
• 1a333fbd35 HER-1998 - ExtractorHTML modified to parse html inside conditional comments. Still ignores normal comments Adam Miller 2012-03-16 14:36:15 -07:00
• f7505e6673 More HTML cleanup. Html5 doctype declaration, get rid of <br> and xml-style "/>" tag endings, other cleanup to make it validate. Noah Levitt 2012-03-15 18:47:18 -07:00
• 52703c6eb4 * ArchiveUtils.java doubleToString() - return "NaN" for NaN (inf symbol didn't quite make sense to me) Noah Levitt 2012-03-15 18:46:01 -07:00
• 7cde3b4f17 Merge branch 'master' of github.com:internetarchive/heritrix3 into html-cleanup Noah Levitt 2012-03-15 15:59:10 -07:00
• 64f24d4c66 HER-1993: In JobResource, write error message instead of throwing RuntimeException when it failed to read job log or crawl log. Kenji Nagahashi 2012-03-09 16:48:11 -08:00
• f05287c8ca restored deleted space after header "Crawl Log" Kenji Nagahashi 2012-03-09 16:29:52 -08:00
• e7d75cbb5a * AbstractPersistProcessor.java, PersistProcessor.java, PersistStoreProcessor.java refactor some common persist processor code into new class AbstractPersistProcessor Noah Levitt 2012-03-06 16:20:58 -08:00