Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 5e8219b014 pull in json dependency from ia-web-commons Noah Levitt 2014-01-31 20:41:05 -08:00
• cc18521759 Merge remote-tracking branch 'internetarchive/master' into revisit-redux Kristinn Sigurðsson 2014-01-30 15:06:01 +00:00
• 701457ed4b ContentDigestHistoryLoader now sets the revisit profile on the CrawlURI after determining that it is a revisit. Kristinn Sigurðsson 2014-01-30 15:03:48 +00:00
• 2905cc556c Revisit headers now come correctly formated from the revisit profile. Kristinn Sigurðsson 2014-01-30 15:02:40 +00:00
• 685f5d5d6a Store refersToDate in String format (as intended to write in WARC). Retain long setter for convenience. Kristinn Sigurðsson 2014-01-30 15:01:41 +00:00
• 10ec391da1 Merge pull request #36 from nlevitt/bad-charset Eldon 2014-01-29 10:43:24 -08:00
• 31aea014d9 also handle case where Charset.forName() throws IllegalCharsetNameException, as opposed to UnsupportedCharsetException Noah Levitt 2014-01-28 16:36:51 -08:00
• 161b359979 Avoid the exception below by handling bad charsets in FetchHTTP. Restore annotation "unsatisfiableCharsetInHeader:...". Includes unit tests for good and bad charsets. Noah Levitt 2014-01-28 16:19:44 -08:00
• 6933232583 Merge pull request #35 from nlevitt/urls-with-spaces vonrosen 2014-01-27 18:04:00 -08:00
• b803ea0ec0 whoops! send escaped path+query on http request line; had been sending raw spaces (includes unit test) Noah Levitt 2014-01-27 16:57:37 -08:00
• 21f04cdf79 Merge pull request #34 from nlevitt/401-no-challenge Adam Miller 2014-01-27 11:52:51 -08:00
• 26a5dfebdc fix NullPointerException in case of 401 with no auth challenge (includes unit test) Noah Levitt 2014-01-27 11:47:37 -08:00
• b1a28358d7 Merge pull request #33 from eldondev/master Noah Levitt 2014-01-24 18:52:33 -08:00
• b33d3faa75 Better Log messages Eldon 2014-01-24 18:49:04 -08:00
• ab3aadf199 Pull out most umbra refs, only do http/https Eldon 2014-01-24 18:47:50 -08:00
• d837536dc6 Use toString method instead Eldon 2014-01-24 18:28:38 -08:00
• d3e96b2770 First pass at a processor to publish crawluris to AMQP channels Eldon 2014-01-24 18:11:02 -08:00
• 7353f62d84 Merge pull request #32 from nlevitt/hc43 Kenji Nagahashi 2014-01-24 16:43:06 -08:00
• 6b72ac9705 Switch to BasicHttpClientConnectionManager instead of PoolingHttpClientConnectionManager, and make the connection manager a member variable of HttpRequest. We had been seeing a strange bug where once in every few thousand urls or so, the connection manager would get gc'd while the http request was being sent, and RecordingOutputStream.flush() would get called, as in the stack trace below. The ToeThread was using it at the same time, so if conditions were right, the recorded stream would get mangled. This commit makes the connection manager a member of FetchHTTPRequest so it doesn't get gc'd prematurely. Also switches to BasicHttpClientConnectionManager, since we're not doing any pooling. Noah Levitt 2014-01-24 16:07:04 -08:00
• b1b2f2bc42 Merge pull request #31 from nlevitt/proxy-port-configurable Kenji Nagahashi 2014-01-22 16:05:44 -08:00
• c5c8788227 make http proxy port configurable in cxml, avoiding this: org.springframework.beans.NotWritablePropertyException: Invalid property 'httpProxyPort' of bean class [org.archive.modules.fetcher.FetchHTTP]: Bean property 'httpProxyPort' is not writable or has an invalid setter method. Did you mean 'httpProxyHost'? Noah Levitt 2014-01-22 16:01:21 -08:00
• 28eb9e5962 Revisit profile is now populated with original URI and date. Kristinn Sigurðsson 2014-01-22 15:00:08 +00:00
• 803e077f43 Profiles refactored, shared elements in AbstractProfile. WARC-Refers-To-Date now set as long and correctly formated in getWarcHeaders. Kristinn Sigurðsson 2014-01-22 14:59:31 +00:00
• fe5ecfaf8a Corrected javadoc Kristinn Sigurðsson 2014-01-22 09:03:32 +00:00
• 48a41d94c3 IdenticalDigestDecideRule now consults the new revisit profile mechanism on the CrawlURI to determine if it represents an identical digest revisit. Kristinn Sigurðsson 2014-01-22 08:59:01 +00:00
• 5a6318033a Ensure CrawlURI discards RevisitProfile after it has finished, in case the same CrawlURI is issued again. Kristinn Sigurðsson 2014-01-22 08:43:48 +00:00
• fafcc1e28b Merge pull request #30 from nlevitt/fix-bdb-cookie-store Kenji Nagahashi 2014-01-21 12:01:10 -08:00
• cab64eafd1 Simplified revisit writing. Uses revisit profile attached to CrawlURI. Kristinn Sigurðsson 2014-01-21 13:15:08 +00:00
• c007d20a07 Basic revisit notation framework Kristinn Sigurðsson 2014-01-21 13:13:31 +00:00
• 3ba476c39b restore useful comment Noah Levitt 2014-01-20 20:05:12 -08:00
• cba663a107 fix BdbCookieStore so that cookies get replaced correctly! and test it Noah Levitt 2014-01-20 20:02:56 -08:00
• 9a7e868022 BdbCookieStoreTest.java partly done Noah Levitt 2014-01-20 09:13:55 -08:00
• 4b8567dca2 Merge pull request #29 from kris-sigur/HER-2062 Roger G. Coram 2014-01-16 02:48:27 -08:00
• e36b9e74e4 remove debug logging Noah Levitt 2014-01-15 13:41:36 -08:00
• df72bc8afc Minor tidying up Kristinn Sigurðsson 2014-01-15 12:51:35 +00:00
• bc77e82da5 Fix for WorkQueueFrontier.deleteURIs mishandles deletions from retired queues (HER-2062) Kristinn Sigurðsson 2014-01-15 12:49:41 +00:00
• 07293fbfb2 Merge pull request #28 from nlevitt/hc43 Kenji Nagahashi 2014-01-14 14:58:36 -08:00
• 420452195c pull in stock commons-httpclient dependency through ia-web-commons instead of specifying it explicitly Noah Levitt 2014-01-14 14:52:24 -08:00
• 5930561042 fix contrib modules that had been using CrawlURI.getHttpMethod() Noah Levitt 2014-01-14 14:27:34 -08:00
• c50393fe53 remove some cruft from testing Noah Levitt 2014-01-13 21:29:23 -08:00
• 9f88da1735 add javadoc explaining newfound absence of getHttpMethod() Noah Levitt 2014-01-13 20:51:08 -08:00
• 2f137dab50 remove obsolete httpcomponents.diff Noah Levitt 2014-01-13 12:38:42 -08:00
• a86efb419f bump httpcomponents httpclient dependency to latest 4.3.1 Noah Levitt 2014-01-12 13:24:48 -08:00
• 16fa2b25f1 selftest-crawler-beans.cxml - replace obsolete BdbCookieStorage with BdbCookieStore Noah Levitt 2014-01-12 13:11:25 -08:00
• 41ffae253e mark beginning of content at beginning of test payload, now that Recorder tries to auto-detect Noah Levitt 2014-01-12 12:42:46 -08:00
• 3a429ffa44 Merge branch 'master' into hc43 Noah Levitt 2014-01-10 17:52:03 -08:00
• 6bdfec7e7f Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2014-01-10 16:22:51 -08:00
• 749bd6a214 bump version to 3.3.0-SNAPSHOT after 3.2.0 release Noah Levitt 2014-01-10 16:22:43 -08:00
• 78a6bea665 Merge pull request #27 from nlevitt/release-3.2.0 Noah Levitt 2014-01-10 14:16:07 -08:00
• 563c5b034d rename dist/README.md to dist/README.txt so that maven bundles it in the distribution Noah Levitt 2014-01-10 14:13:32 -08:00
• 484b3574d7 Merge pull request #26 from nlevitt/release-3.2.0 3.2.0 Noah Levitt 2014-01-10 13:17:46 -08:00
• 0467304f76 update readme for 3.2.0 release Noah Levitt 2014-01-10 13:15:48 -08:00
• 752e288ef8 Merge pull request #25 from nlevitt/release-3.2.0 Eldon 2014-01-10 12:47:58 -08:00
• 32c083f720 bump version number to 3.2.0 for release Noah Levitt 2014-01-10 12:36:45 -08:00
• 9ac0ee057f Merge pull request #24 from nlevitt/url-agnostic-standard Adam Miller 2014-01-08 11:38:51 -08:00
• 84e78db40a for url-agnostic dedup, follow "Proposal for Standardizing the Recording of Arbitrary Duplicates in WARC Files" https://docs.google.com/document/d/1QyQBA7Ykgxie75V8Jziz_O7hbhwf7PF6_u9O6w6zgp0 Noah Levitt 2014-01-07 14:40:14 -08:00
• 506e68316a Merge pull request #23 from nlevitt/her-1979 ikreymer 2013-12-18 01:12:16 -08:00
• 7dd67bd876 fix HER-1979 so heritrix can run on windows xp Noah Levitt 2013-12-17 19:25:12 -08:00
• 2388a35a06 useProjectArtifact=false to avoid [WARNING] Cannot include project artifact: ... it doesn't have an associated file or directory Noah Levitt 2013-12-15 11:54:36 -08:00
• 0ea10665f1 abbreviate maven repository clauses, reduce redundancy, and let default settings prevail, such as updatePolicy=daily, pointed out by Erik Hetzner Noah Levitt 2013-12-15 11:53:18 -08:00
• 6f171b575a fix report links Noah Levitt 2013-12-11 21:14:04 -08:00
• 3b06c88537 test fix (in ia-web-commons) for HER-2059 Noah Levitt 2013-12-06 19:22:16 -08:00
• 3c8a0a389f moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons Noah Levitt 2013-12-05 17:48:57 -08:00
• 98d0901531 moving org.archive.net.PublicSuffixes to ia-web-commons Noah Levitt 2013-12-05 16:13:30 -08:00
• 552794dc89 don't let the exception from commit df2a313797 derail the whole crawl job Noah Levitt 2013-11-26 16:08:38 -08:00
• 1163847644 try to avoid nearly endless loop of "java.lang.IllegalStateException: Can't open a cursor Database was closed." when startup fails and dumpPendingToLog is attempted Noah Levitt 2013-11-21 18:51:16 -08:00
• 8e1125d062 bump version to 3.2.0-SNAPSHOT since there are some incompatible api changes since 3.1.1 (requested by kris sigurðsson) Noah Levitt 2013-11-20 09:18:31 -08:00
• d93b5449be new CrawlURI annotation "duplicate:digest" to simplify handling dups in stats, etc Noah Levitt 2013-11-19 11:05:14 -08:00
• 7089caf3c4 remove unused engine.css Noah Levitt 2013-11-15 19:11:19 -08:00
• 119e565acc Adam's responsive UI Noah Levitt 2013-11-15 18:59:14 -08:00
• 2b87219496 oops, avoid npe Noah Levitt 2013-11-15 17:59:34 -08:00
• df2a313797 throw exception when host cache returns 0.0.0.0 for any host (trying to track down a strange problem) Noah Levitt 2013-11-15 17:20:55 -08:00
• e2f4b7287f Merge branch 'master' of github.com:internetarchive/heritrix3 Ilya Kreymer 2013-11-01 15:44:12 -07:00
• 8633ed0ec3 removed ThreadLocalHttpConnectionManager -- it has been moved to ia-web-commons Ilya Kreymer 2013-11-01 15:43:46 -07:00
• b137022b79 Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.) Noah Levitt 2013-10-28 17:35:55 -07:00
• 38a12b2008 avoid NPE resuming from checkpoint if "stats" is {} in warcWriter checkpoint json Noah Levitt 2013-10-11 14:44:34 -07:00
• a1f84d3cc4 HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility Noah Levitt 2013-10-10 17:57:18 -07:00
• 87e8e6b31d WbmPersistLoadProcessor: add simple performance metric. drop confusing metric failedCount, add missedCount and errorCount. Kenji Nagahashi 2013-10-03 12:18:35 -07:00
• aa9208f5e6 updated WbmPersistLoadProcessor for cookie-based authentication. allows turning gzip compression on/off. main for quick test. Kenji Nagahashi 2013-10-01 12:10:25 -07:00
• df03249a65 WbmPersistLoadProcessor: make buildURL() more flexible. now uses template string with placeholders. no hard-coded parameters. Kenji Nagahashi 2013-09-27 11:58:41 -07:00
• 4d559ec81d WbmPersistLoadProcessor.java: fixed indentation. Kenji Nagahashi 2013-09-27 10:21:51 -07:00
• 7ad90b3126 Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2013-09-12 12:04:32 -07:00
• 57311db117 use libs version ${project.version} Noah Levitt 2013-09-10 15:24:58 -07:00
• 140ba3e89e tweak io block size in FileUtils.appendTo(), referencing performance metrics documented in gnu coreutils (used e.g. for /bin/cat) Noah Levitt 2013-09-08 12:41:48 -07:00
• 254811ff7f commented-out default for new setting forgetAllButLatest on CheckpointService Noah Levitt 2013-09-08 12:39:54 -07:00
• f096609a50 avoid infinite loop spitting out stack traces on BeansException on startup Noah Levitt 2013-09-08 00:11:54 -07:00
• a00141eef7 avoid deadlock in case of BeansException at crawl startup Noah Levitt 2013-09-08 00:10:27 -07:00
• 071cbe99cd include urlsWritten in checkpoint, and be backward compatible with old checkpoints Noah Levitt 2013-09-07 23:40:17 -07:00
• c49b465d82 custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test Noah Levitt 2013-09-07 19:40:11 -07:00
• 52d72e661d part of HER-2048 - WARCWriterProcessor stats resume from checkpoint Noah Levitt 2013-09-07 11:35:03 -07:00
• e0f78976d7 part of HER-2048 - restore BdbServerCache from checkpoint Noah Levitt 2013-09-07 10:53:22 -07:00
• fd71833050 avoid NPE when attempting to terminate already-finished job Noah Levitt 2013-09-07 09:57:33 -07:00
• d01bc4ed55 when resuming from checkpoint, make next checkpoint serial number follow serial number of resumed checkpoint Noah Levitt 2013-09-07 09:57:07 -07:00
• d61171f076 CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size Noah Levitt 2013-09-07 09:55:58 -07:00
• b670a3c2a7 Merge branch 'master' into always-resumable Noah Levitt 2013-09-06 23:32:32 -07:00
• 25d4de186f don't log warning in normal case Noah Levitt 2013-09-06 22:04:56 -07:00
• cc0b63066b Merge branch 'master' into always-resumable Noah Levitt 2013-09-03 18:38:55 -07:00
• 34cb015ecf adjust PublicSuffixesTest for updated public suffixes list Noah Levitt 2013-09-03 18:38:44 -07:00
• 59596c7043 prevent redundant checkpoints when crawl is stopping Noah Levitt 2013-09-03 14:24:41 -07:00
• 0ab8eee24e forgetting old checkpoints - roll up previous checkpointed CrawlerJournals (e.g. frontier.recover.gz) into new checkpoint Noah Levitt 2013-08-29 10:56:12 -07:00