Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• a297abb02d Merge pull request #44 from nlevitt/update-hc4_3_3 Eldon 2014-02-26 14:39:47 -05:00
• cf5e295d42 update httpcomponents, should address NPE we've seen https://issues.apache.org/jira/browse/HTTPCLIENT-1417 Noah Levitt 2014-02-26 11:35:26 -08:00
• 88dc01305f fix job.log file handler was left open when jobdir is removed Martins Balodis 2014-02-26 21:15:38 +02:00
• 659c5e886f Merge pull request #42 from eldondev/master Noah Levitt 2014-02-25 14:15:30 -08:00
• 06a75c6fcf Adding the queue declaration and binding to the UrlReceiver Eldon 2014-02-25 17:11:56 -05:00
• bbd818e6e4 Merge pull request #39 from nlevitt/umbra Eldon 2014-02-25 16:24:38 -05:00
• e08936d029 Merge pull request #41 from nlevitt/fix-slow-cookies Adam Miller 2014-02-21 15:51:49 -08:00
• cf775bec94 remove extraneous code related to logging removed in last commit Noah Levitt 2014-02-21 15:46:52 -08:00
• fd806a514d remove extra logging from performance test Noah Levitt 2014-02-21 13:28:13 -08:00
• 39f06a1f1c fix slowness caused by BdbCookieStore by maintaining cookie list in memory parallel to the one stored on disk; add tests for consistency and performance under load Noah Levitt 2014-02-21 13:26:58 -08:00
• d130e3fc44 call the extra info we send to umbra "clientId", kinda clarifies the purpose of it Noah Levitt 2014-02-13 19:25:09 -08:00
• e389499aa7 don't try to stop consuming amqp when crawl is shutting down, can end up reopening the connection after it's been closed Noah Levitt 2014-02-13 19:24:31 -08:00
• d6e61b83da For https://webarchive.jira.com/browse/HER-2064 Hunter Stern 2014-02-13 16:59:05 -08:00
• 06ba50c410 Reverting previous commit: https://github.com/internetarchive/heritrix3/commit/164d0fbdc5fa32cd90edd9f1b4091ed876552232 Hunter Stern 2014-02-13 16:50:21 -08:00
• 164d0fbdc5 For https://webarchive.jira.com/browse/HER-2064 Hunter Stern 2014-02-13 16:42:58 -08:00
• 72b2d58034 pause amqp consumption on crawl pause, and resume on unpause Noah Levitt 2014-02-10 21:04:19 -08:00
• c493248863 formatting change only - use 4 spaces to indent Noah Levitt 2014-02-10 20:43:50 -08:00
• eccddb3df3 use parent url metadata received via amqp in building CrawlURI Noah Levitt 2014-02-10 20:38:37 -08:00
• 8d82c440d3 send url metadata with amqp message so that umbra can send it back with child urls Noah Levitt 2014-02-10 20:36:51 -08:00
• 704f4aa443 Merge pull request #38 from nlevitt/umbra Eldon 2014-02-06 16:51:03 -05:00
• 6dbcac40d9 syncronize start() and stop() Noah Levitt 2014-02-03 20:27:51 -08:00
• 500dafd390 AMQPPublishProcessor - annotate urls sent to AMQP; set special fetch status; add default amqpUri; replace queueName setting with exchange and routingKey, which seem to be what are needed; avoid sending urls that were received from AMQP, and robots.txt urls; set content-type of amqp message to application/json, which umbra seems to need Noah Levitt 2014-02-03 18:41:33 -08:00
• af44522fdf FetchHTTPRequest - include CrawlURI.getData().get("customHttpRequestHeaders"), if any, in http request Noah Levitt 2014-02-03 18:41:23 -08:00
• ecb0eaf435 AMQPUrlReceiver - ack amqp messages, get rid of misguided thread starting infinite consumers Noah Levitt 2014-02-03 18:36:52 -08:00
• e90ba8278b Merge branch 'master' into umbra Noah Levitt 2014-02-03 18:22:25 -08:00
• 63b70b427e Merge pull request #37 from nlevitt/her-2063 vonrosen 2014-02-03 17:42:34 -08:00
• edd62f3de0 fix HER-2063 - omit port in Host request header when it is default for the scheme Noah Levitt 2014-02-03 17:18:40 -08:00
• c980b452c9 annotate CrawlURI with "receivedViaAMQP" Noah Levitt 2014-02-01 18:26:07 -08:00
• c7cf62bf80 license header Noah Levitt 2014-02-01 18:23:49 -08:00
• bf0590a083 spaces not tabs for indentation Noah Levitt 2014-02-01 18:23:05 -08:00
• c163f31851 AMQPUrlReceiver - receive urls and request headers via amqp in the format produced by https://github.com/internetarchive/umbra and add to frontier Noah Levitt 2014-01-31 20:42:27 -08:00
• 5e8219b014 pull in json dependency from ia-web-commons Noah Levitt 2014-01-31 20:41:05 -08:00
• cc18521759 Merge remote-tracking branch 'internetarchive/master' into revisit-redux Kristinn Sigurðsson 2014-01-30 15:06:01 +00:00
• 701457ed4b ContentDigestHistoryLoader now sets the revisit profile on the CrawlURI after determining that it is a revisit. Kristinn Sigurðsson 2014-01-30 15:03:48 +00:00
• 2905cc556c Revisit headers now come correctly formated from the revisit profile. Kristinn Sigurðsson 2014-01-30 15:02:40 +00:00
• 685f5d5d6a Store refersToDate in String format (as intended to write in WARC). Retain long setter for convenience. Kristinn Sigurðsson 2014-01-30 15:01:41 +00:00
• 10ec391da1 Merge pull request #36 from nlevitt/bad-charset Eldon 2014-01-29 10:43:24 -08:00
• 31aea014d9 also handle case where Charset.forName() throws IllegalCharsetNameException, as opposed to UnsupportedCharsetException Noah Levitt 2014-01-28 16:36:51 -08:00
• 161b359979 Avoid the exception below by handling bad charsets in FetchHTTP. Restore annotation "unsatisfiableCharsetInHeader:...". Includes unit tests for good and bad charsets. Noah Levitt 2014-01-28 16:19:44 -08:00
• 6933232583 Merge pull request #35 from nlevitt/urls-with-spaces vonrosen 2014-01-27 18:04:00 -08:00
• b803ea0ec0 whoops! send escaped path+query on http request line; had been sending raw spaces (includes unit test) Noah Levitt 2014-01-27 16:57:37 -08:00
• 21f04cdf79 Merge pull request #34 from nlevitt/401-no-challenge Adam Miller 2014-01-27 11:52:51 -08:00
• 26a5dfebdc fix NullPointerException in case of 401 with no auth challenge (includes unit test) Noah Levitt 2014-01-27 11:47:37 -08:00
• b1a28358d7 Merge pull request #33 from eldondev/master Noah Levitt 2014-01-24 18:52:33 -08:00
• b33d3faa75 Better Log messages Eldon 2014-01-24 18:49:04 -08:00
• ab3aadf199 Pull out most umbra refs, only do http/https Eldon 2014-01-24 18:47:50 -08:00
• d837536dc6 Use toString method instead Eldon 2014-01-24 18:28:38 -08:00
• d3e96b2770 First pass at a processor to publish crawluris to AMQP channels Eldon 2014-01-24 18:11:02 -08:00
• 7353f62d84 Merge pull request #32 from nlevitt/hc43 Kenji Nagahashi 2014-01-24 16:43:06 -08:00
• 6b72ac9705 Switch to BasicHttpClientConnectionManager instead of PoolingHttpClientConnectionManager, and make the connection manager a member variable of HttpRequest. We had been seeing a strange bug where once in every few thousand urls or so, the connection manager would get gc'd while the http request was being sent, and RecordingOutputStream.flush() would get called, as in the stack trace below. The ToeThread was using it at the same time, so if conditions were right, the recorded stream would get mangled. This commit makes the connection manager a member of FetchHTTPRequest so it doesn't get gc'd prematurely. Also switches to BasicHttpClientConnectionManager, since we're not doing any pooling. Noah Levitt 2014-01-24 16:07:04 -08:00
• b1b2f2bc42 Merge pull request #31 from nlevitt/proxy-port-configurable Kenji Nagahashi 2014-01-22 16:05:44 -08:00
• c5c8788227 make http proxy port configurable in cxml, avoiding this: org.springframework.beans.NotWritablePropertyException: Invalid property 'httpProxyPort' of bean class [org.archive.modules.fetcher.FetchHTTP]: Bean property 'httpProxyPort' is not writable or has an invalid setter method. Did you mean 'httpProxyHost'? Noah Levitt 2014-01-22 16:01:21 -08:00
• 28eb9e5962 Revisit profile is now populated with original URI and date. Kristinn Sigurðsson 2014-01-22 15:00:08 +00:00
• 803e077f43 Profiles refactored, shared elements in AbstractProfile. WARC-Refers-To-Date now set as long and correctly formated in getWarcHeaders. Kristinn Sigurðsson 2014-01-22 14:59:31 +00:00
• fe5ecfaf8a Corrected javadoc Kristinn Sigurðsson 2014-01-22 09:03:32 +00:00
• 48a41d94c3 IdenticalDigestDecideRule now consults the new revisit profile mechanism on the CrawlURI to determine if it represents an identical digest revisit. Kristinn Sigurðsson 2014-01-22 08:59:01 +00:00
• 5a6318033a Ensure CrawlURI discards RevisitProfile after it has finished, in case the same CrawlURI is issued again. Kristinn Sigurðsson 2014-01-22 08:43:48 +00:00
• fafcc1e28b Merge pull request #30 from nlevitt/fix-bdb-cookie-store Kenji Nagahashi 2014-01-21 12:01:10 -08:00
• cab64eafd1 Simplified revisit writing. Uses revisit profile attached to CrawlURI. Kristinn Sigurðsson 2014-01-21 13:15:08 +00:00
• c007d20a07 Basic revisit notation framework Kristinn Sigurðsson 2014-01-21 13:13:31 +00:00
• 3ba476c39b restore useful comment Noah Levitt 2014-01-20 20:05:12 -08:00
• cba663a107 fix BdbCookieStore so that cookies get replaced correctly! and test it Noah Levitt 2014-01-20 20:02:56 -08:00
• 9a7e868022 BdbCookieStoreTest.java partly done Noah Levitt 2014-01-20 09:13:55 -08:00
• 4b8567dca2 Merge pull request #29 from kris-sigur/HER-2062 Roger G. Coram 2014-01-16 02:48:27 -08:00
• e36b9e74e4 remove debug logging Noah Levitt 2014-01-15 13:41:36 -08:00
• df72bc8afc Minor tidying up Kristinn Sigurðsson 2014-01-15 12:51:35 +00:00
• bc77e82da5 Fix for WorkQueueFrontier.deleteURIs mishandles deletions from retired queues (HER-2062) Kristinn Sigurðsson 2014-01-15 12:49:41 +00:00
• 07293fbfb2 Merge pull request #28 from nlevitt/hc43 Kenji Nagahashi 2014-01-14 14:58:36 -08:00
• 420452195c pull in stock commons-httpclient dependency through ia-web-commons instead of specifying it explicitly Noah Levitt 2014-01-14 14:52:24 -08:00
• 5930561042 fix contrib modules that had been using CrawlURI.getHttpMethod() Noah Levitt 2014-01-14 14:27:34 -08:00
• c50393fe53 remove some cruft from testing Noah Levitt 2014-01-13 21:29:23 -08:00
• 9f88da1735 add javadoc explaining newfound absence of getHttpMethod() Noah Levitt 2014-01-13 20:51:08 -08:00
• 2f137dab50 remove obsolete httpcomponents.diff Noah Levitt 2014-01-13 12:38:42 -08:00
• a86efb419f bump httpcomponents httpclient dependency to latest 4.3.1 Noah Levitt 2014-01-12 13:24:48 -08:00
• 16fa2b25f1 selftest-crawler-beans.cxml - replace obsolete BdbCookieStorage with BdbCookieStore Noah Levitt 2014-01-12 13:11:25 -08:00
• 41ffae253e mark beginning of content at beginning of test payload, now that Recorder tries to auto-detect Noah Levitt 2014-01-12 12:42:46 -08:00
• 3a429ffa44 Merge branch 'master' into hc43 Noah Levitt 2014-01-10 17:52:03 -08:00
• 6bdfec7e7f Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2014-01-10 16:22:51 -08:00
• 749bd6a214 bump version to 3.3.0-SNAPSHOT after 3.2.0 release Noah Levitt 2014-01-10 16:22:43 -08:00
• 78a6bea665 Merge pull request #27 from nlevitt/release-3.2.0 Noah Levitt 2014-01-10 14:16:07 -08:00
• 563c5b034d rename dist/README.md to dist/README.txt so that maven bundles it in the distribution Noah Levitt 2014-01-10 14:13:32 -08:00
• 484b3574d7 Merge pull request #26 from nlevitt/release-3.2.0 3.2.0 Noah Levitt 2014-01-10 13:17:46 -08:00
• 0467304f76 update readme for 3.2.0 release Noah Levitt 2014-01-10 13:15:48 -08:00
• 752e288ef8 Merge pull request #25 from nlevitt/release-3.2.0 Eldon 2014-01-10 12:47:58 -08:00
• 32c083f720 bump version number to 3.2.0 for release Noah Levitt 2014-01-10 12:36:45 -08:00
• 9ac0ee057f Merge pull request #24 from nlevitt/url-agnostic-standard Adam Miller 2014-01-08 11:38:51 -08:00
• 84e78db40a for url-agnostic dedup, follow "Proposal for Standardizing the Recording of Arbitrary Duplicates in WARC Files" https://docs.google.com/document/d/1QyQBA7Ykgxie75V8Jziz_O7hbhwf7PF6_u9O6w6zgp0 Noah Levitt 2014-01-07 14:40:14 -08:00
• 506e68316a Merge pull request #23 from nlevitt/her-1979 ikreymer 2013-12-18 01:12:16 -08:00
• 7dd67bd876 fix HER-1979 so heritrix can run on windows xp Noah Levitt 2013-12-17 19:25:12 -08:00
• 2388a35a06 useProjectArtifact=false to avoid [WARNING] Cannot include project artifact: ... it doesn't have an associated file or directory Noah Levitt 2013-12-15 11:54:36 -08:00
• 0ea10665f1 abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner Noah Levitt 2013-12-15 11:53:18 -08:00
• 6f171b575a fix report links Noah Levitt 2013-12-11 21:14:04 -08:00
• 3b06c88537 test fix (in ia-web-commons) for HER-2059 Noah Levitt 2013-12-06 19:22:16 -08:00
• 3c8a0a389f moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons Noah Levitt 2013-12-05 17:48:57 -08:00
• 98d0901531 moving org.archive.net.PublicSuffixes to ia-web-commons Noah Levitt 2013-12-05 16:13:30 -08:00
• 552794dc89 don't let the exception from commit df2a313797 derail the whole crawl job Noah Levitt 2013-11-26 16:08:38 -08:00
• 1163847644 try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted Noah Levitt 2013-11-21 18:51:16 -08:00
• 8e1125d062 bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson) Noah Levitt 2013-11-20 09:18:31 -08:00
• d93b5449be new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc Noah Levitt 2013-11-19 11:05:14 -08:00
• 7089caf3c4 remove unused engine.css Noah Levitt 2013-11-15 19:11:19 -08:00