Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 5d8fe0443d Merge branch 'master' into HER-2031 Noah Levitt 2013-03-23 19:16:05 -07:00
• 0631123783 close recorders on failure to avoid exception from retry cycle within httpclient (restore functionality from fcece02646 to work with current httpcomponents Noah Levitt 2013-03-20 16:59:06 -07:00
• 75abd2509f bit of a hack to support shouldFetchBodyRule, given httpcomponents limitation discussed at https://issues.apache.org/jira/browse/HTTPCORE-325 Noah Levitt 2013-03-20 16:17:34 -07:00
• 956024ec77 update httpclient dependency Noah Levitt 2013-03-20 16:11:47 -07:00
• 1993c4388c Merge branch 'master' into hc43 Noah Levitt 2013-03-20 14:28:13 -07:00
• 042ed066b3 add more eclipse stuff to .gitignore Noah Levitt 2013-03-20 14:27:34 -07:00
• 8fd361e518 update source/target version to 1.6 since we use 1.6 features, specifically @Override for methods only implementing an interface (not sure why maven wasn't enforcing this); also remove maven-antrun-plugin created timestamp.txt, doesn't seem to be used anywhere Noah Levitt 2013-03-20 14:25:22 -07:00
• c509b6e552 remove more eclipse stuff, people like to manage their own Noah Levitt 2013-03-20 13:39:28 -07:00
• 654c106d19 remove eclipse stuff and add to .gitignore (except .settings which has code formatting config) Noah Levitt 2013-03-20 12:55:45 -07:00
• 8a14b40564 Merge branch 'master' into hc43 Noah Levitt 2013-03-20 12:39:58 -07:00
• 79422099b4 remove old profiling stuff Noah Levitt 2013-03-20 12:35:41 -07:00
• b4bff3ffb9 new method RecordingInputStream.readToEndOfContent(), because now we can't rely on stopping right after the headers, which is an assumption of readFullyOrUntil() Noah Levitt 2013-03-19 18:24:02 -07:00
• 9e054dd1f8 detect http message body begin in RecordingOutputStream, see https://issues.apache.org/jira/browse/HTTPCORE-325 Noah Levitt 2013-03-19 14:18:11 -07:00
• 0be1b8e318 partly done not working update to latest httpcomponents Noah Levitt 2013-03-19 13:01:35 -07:00
• 52ad4395d9 Merge branch 'master' into hc43 Noah Levitt 2013-03-19 13:00:27 -07:00
• 75bb19c4ea switch .classpath from archive-commons to ia-web-commons Noah Levitt 2013-03-19 11:25:13 -07:00
• 12da824316 Merge branch 'master' into hc43 Noah Levitt 2013-03-18 23:32:51 -07:00
• 9fa198944a depend on new "ia-web-commons" instead of "archive-commons" Noah Levitt 2013-03-18 23:32:05 -07:00
• b0c6619f80 Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2013-03-18 20:02:03 -07:00
• 0c9e2a1f4d ContentDigestHistory{Loader,Storer} - ignore uris with empty content body (deduping that is counterproductive) Noah Levitt 2013-03-18 20:01:52 -07:00
• ff36546eda fix serverless whois url example in comment Noah Levitt 2013-03-18 14:03:39 -07:00
• 2f4d437734 Beans.ftl Fixed a bug that showed objects with no properties as null instead of displaying the class name Job.ftl Fixed a bug that prevented the configuration referenced paths from displaying Script.ftl Fixed a bug that rendered extra whitespace before and after the script block Adam Miller 2013-03-15 16:06:06 -07:00
• c5c54ac138 comment-out debug output gojomo 2013-03-12 11:42:47 -07:00
• 7c8295ee82 checkpoint support gojomo 2013-03-12 11:42:29 -07:00
• a844878e7d Merge branch 'simple_logins' into HER-2031 gojomo 2013-03-12 01:16:31 -07:00
• 3f8590fe44 when found in CrawlURI, add WARC-Simple-Form-Province-Status header gojomo 2013-03-11 20:57:41 -07:00
• bf9cab8eb2 when POST/submit-data requested, alter request method accordingly gojomo 2013-03-11 20:57:09 -07:00
• bbac7d7320 per (overlay) settings, spawn high-priority POST when appropriate; annotate/add-headers to describe gojomo 2013-03-11 20:56:31 -07:00
• ed639c8342 extract HTMLForm INPUT details gojomo 2013-03-11 20:55:20 -07:00
• 94a61b4fb7 retain offsets of discovered FORMs to aid later processing gojomo 2013-03-11 20:53:41 -07:00
• 6b0abae8ee new constants, hop-type; refactoring to help support form-submissions gojomo 2013-03-11 20:52:59 -07:00
• eb0760a37e refactor for runCandidatesChain; also run outCandidates gojomo 2013-03-11 20:50:18 -07:00
• 0e3a2ec80a CrawlJobModel.java Adding check for application context Adam Miller 2013-03-04 16:34:18 -08:00
• b7b7953dc9 EngineResource.java Pulling individual values for necessary data from crawl jobs instead of building the full CrawlJobModel CrawlJobModel.java Moving some data from methods into fields so they are exposed in the XML representation EngineModel.java Removing full building of CrawlJobModel. Handling instead in EngineResource Job.ftl Updating fields that relied on CrawlJobModel methods which were moved to fields. Adam Miller 2012-09-26 16:36:09 -07:00
• 03f4f381af JobResource.java Removing temporary template loading. Finishing converting XML and HTML generation to use the same data model CrawlJobModel.java Converting to same Map based reports used by XML API EngineModel.java Converting to same Map based reports used by XML API Engine.ftl Creating template for Map based reports Job.ftl Creating template for Map based reports Adam Miller 2012-09-25 18:07:14 -07:00
• ef7fd37a0a BeanBrowseResource.java Removing obsolete method: makePresentableMap() Adam Miller 2012-09-12 14:26:31 -07:00
• d26a712211 Fixed bad merge conflict Adam Miller 2012-09-11 16:33:39 -07:00
• 60d00e3ef3 Converted data models to be usable for conversion to HTML and XML. Updated Models and freemarker templates to match XML naming Adam Miller 2012-09-10 17:03:49 -07:00
• 50e49b6399 Removed commented out code. Adam Miller 2012-09-10 17:02:27 -07:00
• 7426c78d73 Removed temporary template loading from file system. Removed unused method: getAvailableActions Adam Miller 2012-09-10 17:00:54 -07:00
• d545899fd2 Removed temporary template loading from file system Adam Miller 2012-09-10 16:59:56 -07:00
• 929f3cb489 Scripts and Beans converted to freemarker Adam Miller 2012-08-24 15:25:45 -07:00
• 783d48b49c Removing old commented out code Adam Miller 2012-07-26 16:42:56 -07:00
• a32f00c2b3 Finishing /engine and /engine/job/{job} Adam Miller 2012-07-26 16:36:23 -07:00
• e4ad60dda6 Beginning conversion to freemarker templates. /engine updated Adam Miller 2012-07-02 10:14:21 -07:00
• 01072b4b46 support mixing fetch histories from multiple sources (hbase, WBM). Kenji Nagahashi 2013-03-06 17:28:38 -08:00
• 8cbb84b8df Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2013-02-22 15:43:30 -08:00
• fb758e53e3 Avoid pulling in hadoop-core.jar and its dependencies; add plugin versions to get rid of warnings Noah Levitt 2013-02-22 15:43:15 -08:00
• 0841840baf Fix for HER-2032 - test stream for compression; add tests with and without compression Noah Levitt 2013-02-05 18:45:14 -08:00
• 60d93cbb5b Remove obsolete <scm> section that pointed to old svn repository Noah Levitt 2013-02-05 18:37:56 -08:00
• 878e3b1afe Remove spurious "throws" declaration Noah Levitt 2013-02-05 18:37:22 -08:00
• bda50a56b1 Merge pull request #19 from travisfw/bean-bindings Noah Levitt 2013-01-24 13:59:29 -08:00
• bf82ea3524 Rename to README.md so github treats it as markdown Noah Levitt 2013-01-24 13:57:02 -08:00
• c26e1c9732 Rename to README.md so github treats it as markdown Noah Levitt 2013-01-24 13:56:20 -08:00
• a99b2a26b4 Convert readme to markdown Noah Levitt 2013-01-24 13:54:37 -08:00
• 56e6a0d504 ScriptResource actually uses the BeanLookupBindings Travis Wellman 2013-01-24 10:56:16 -08:00
• 1883d81bc8 bindings on top of beans instead of vice versa Travis Wellman 2013-01-23 18:27:40 -08:00
• 4a2e7512b6 Merge branch 'master' into hc43 Noah Levitt 2013-01-12 21:22:19 -08:00
• 91aeb6d0bf disable bean lookup by default Travis Wellman 2013-01-11 12:09:54 -08:00
• 03914a246c BeanLookupBindings for simpler script access to beans Travis Wellman 2013-01-10 18:54:33 -08:00
• d27f67897e Fix for HER-2029 Checkpoints recover error on windows - replace CLibrary.INSTANCE.link() with FilesystemLinkMaker.makeHardLink() in doRecover() - contributed by Andrés Aguilar Noah Levitt 2013-01-09 18:28:05 -08:00
• 1364b0f571 updated queryURL to production WBM. Kenji Nagahashi 2013-01-09 12:44:30 -08:00
• 5af4ff657a Avoid pulling in [non-archive-overlay] commons-httpclient from archive-commons dependency Noah Levitt 2013-01-04 18:48:05 -08:00
• c904cef241 Quiet down SSL warnings from restlet server by default Noah Levitt 2013-01-04 10:56:20 -08:00
• c171a98c07 Bring up to date with latest httpcomponents after commit for https://issues.apache.org/jira/browse/HTTPCLIENT-1288 Noah Levitt 2013-01-03 19:13:53 -08:00
• 7d3fe554af Merge branch 'master' into hc43 Noah Levitt 2013-01-03 18:18:22 -08:00
• 3a6ade1e09 More stuff moved to archive-commons Noah Levitt 2013-01-03 17:44:30 -08:00
• 9d5986c078 Moving GZIPMembersInputStream from heritrix to archive-commons Noah Levitt 2013-01-03 17:16:50 -08:00
• b6133aee5f Update httpcomponents.diff Noah Levitt 2012-12-31 14:15:05 -08:00
• 04af55739d We need the bleeding edge httpcomponents (with patches too, but thats not in the pom) Noah Levitt 2012-12-31 10:33:46 -08:00
• a0b62054c9 Fixes to httpcomponents patch to make it build Noah Levitt 2012-12-31 10:27:28 -08:00
• 163871d462 Tweaks and twiddles Noah Levitt 2012-12-31 01:57:03 -08:00
• 25234898ef Somewhat redundant form auth integration test restored, passes Noah Levitt 2012-12-30 22:35:28 -08:00
• fcece02646 Close recorders on failure to avoid exception from retry cycle within httpclient- now all the FetchHTTPTests pass Noah Levitt 2012-12-30 20:45:05 -08:00
• d9311affbc Improve debug logging Noah Levitt 2012-12-30 19:54:04 -08:00
• cb725bb38e Fix socket timeout Noah Levitt 2012-12-30 19:05:55 -08:00
• dac492f2c6 DnsResolver using ServerCache (should add a test for this) Noah Levitt 2012-12-30 01:00:07 -08:00
• d1c41d624f SSL tests pass Noah Levitt 2012-12-30 00:18:44 -08:00
• 17ad459053 Fix bug in httpclient so that midfetch abort works and testShouldFetchBodyRule() passes Noah Levitt 2012-12-29 23:01:28 -08:00
• 7cb05c6961 Make sure wire logger is disabled for testMaxFetchKBSec(), otherwise it can slow things down enough that it fails Noah Levitt 2012-12-29 21:37:46 -08:00
• 2a6a2aa6d4 testHttpProxyAuth() passes Noah Levitt 2012-12-29 21:05:37 -08:00
• 6e8d4efe35 Refactor code building the request into new class FetchHTTPRequest; testHttpProxy() passes Noah Levitt 2012-12-29 20:33:47 -08:00
• 01a9c89088 testHttpBindAddress() passes Noah Levitt 2012-12-29 15:09:46 -08:00
• 959504b7f0 testAcceptCompression() passes Noah Levitt 2012-12-29 14:59:59 -08:00
• c01e50a5a5 Digest auth test passes Noah Levitt 2012-12-29 14:29:47 -08:00
• 70875f1605 Basic auth test passes Noah Levitt 2012-12-29 14:00:36 -08:00
• 34e0ba96ce Get rid of "UmmSession*Buffer* classes and use Session*BufferImpl directly Noah Levitt 2012-12-28 21:23:18 -08:00
• 6fe901d1f8 New approach to overriding SessionInputBuffer seems promising. (httpcomponents.diff is the patch to httpcomponents head that is needed for this h3 commit to work, will keep up to date with each commit) Noah Levitt 2012-12-28 21:06:15 -08:00
• 6300caf367 A little more work on the approach of wrapping the SessionInputBuffer, to remember that overriding BHttpConnectionBase.getSessionInputBuffer() makes it possible (but it doesn't work because we need to get in there and no-op the buffering since we do our own, and it eats up the stream too soon...) Noah Levitt 2012-12-28 18:46:39 -08:00
• 33a500b213 Some work on an approach to overriding SessionInputBuffer that looks like is not the way Noah Levitt 2012-12-28 15:37:58 -08:00
• c9223a9a02 More work on adapting to httpcomponents HEAD with custom patches Noah Levitt 2012-12-22 17:10:38 -08:00
• d3db71e3bb Working on adapting to httpcomponents HEAD with custom patches (hoping to get those incorporated) Noah Levitt 2012-12-21 16:42:30 -08:00
• e26ee96895 * FetchHTTPTests.java testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix should let pass through to mimic browser behavior (test fails currently) Noah Levitt 2012-12-20 11:14:16 -08:00
• c7813938e0 Merge branch 'master' into new-fetchhttp-only Noah Levitt 2012-12-20 11:04:42 -08:00
• 543e8820ef Merge branch 'master' into archive-commons-refactor Noah Levitt 2012-12-20 02:44:57 -08:00
• 0de9869c0f Tweaks to new UURIFactory Noah Levitt 2012-12-20 02:42:16 -08:00
• 9168813ed7 Fix bugs in refactoring of UURI/UsableURI Noah Levitt 2012-12-20 02:32:56 -08:00
• 3a65fa25d7 Working on refactoring some stuff into archive-commons Noah Levitt 2012-12-18 14:46:52 -08:00
• 0eac126f7d Special WARC-Profile for uri-agnostic revisit records. See HER-2022, http://tech.groups.yahoo.com/group/archive-crawler/message/7806 Noah Levitt 2012-12-13 14:11:14 -08:00
• 5586d5452d * JerichoExtractorHTMLTest.java avoid redundancy by using extractor built in setUp() makeExtractor() - call extractor.setExtractorJS(new ExtractorJS()) since we got rid of the static ExtractorJS stuff testConditionalComment1() - override to skip the test since it fails with JerichoExtractorHTML Noah Levitt 2012-12-11 16:58:48 -08:00