Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 119e565acc Adam's responsive UI Noah Levitt 2013-11-15 18:59:14 -08:00
• 2b87219496 oops, avoid npe Noah Levitt 2013-11-15 17:59:34 -08:00
• df2a313797 throw exception when host cache returns 0.0.0.0 for any host (trying to track down a strange problem) Noah Levitt 2013-11-15 17:20:55 -08:00
• e2f4b7287f Merge branch 'master' of github.com:internetarchive/heritrix3 Ilya Kreymer 2013-11-01 15:44:12 -07:00
• 8633ed0ec3 removed ThreadLocalHttpConnectionManager -- it has been moved to ia-web-commons Ilya Kreymer 2013-11-01 15:43:46 -07:00
• b137022b79 Include in checkpoint data a list of ".open" WARCs and their current raw on-disk length, so the exact extent of pre-checkpoint content can be deduced/reconstructed. Also when flushing or closing writers, block to wait on any that remain active. (In context of heritrix crawl this is not normally necessary because of the dispositionInProgressLock managed by the frontier, but seems like a good idea in case WriterPool is used elsewhere.) Noah Levitt 2013-10-28 17:35:55 -07:00
• 38a12b2008 avoid NPE resuming from checkpoint if "stats" is {} in warcWriter checkpoint json Noah Levitt 2013-10-11 14:44:34 -07:00
• a1f84d3cc4 HER-2056 new option startNewFilesOnCheckpoint on writers, true by default for backward compatibility Noah Levitt 2013-10-10 17:57:18 -07:00
• 87e8e6b31d WbmPersistLoadProcessor: add simple performance metric. drop confusing metric failedCount, add missedCount and errorCount. Kenji Nagahashi 2013-10-03 12:18:35 -07:00
• aa9208f5e6 updated WbmPersistLoadProcessor for cookie-based authentication. allows turning gzip compression on/off. main for quick test. Kenji Nagahashi 2013-10-01 12:10:25 -07:00
• df03249a65 WbmPersistLoadProcessor: make buildURL() more flexible. now uses template string with placeholders. no hard-coded parameters. Kenji Nagahashi 2013-09-27 11:58:41 -07:00
• 4d559ec81d WbmPersistLoadProcessor.java: fixed indentation. Kenji Nagahashi 2013-09-27 10:21:51 -07:00
• 7ad90b3126 Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2013-09-12 12:04:32 -07:00
• 57311db117 use libs version ${project.version} Noah Levitt 2013-09-10 15:24:58 -07:00
• 140ba3e89e tweak io block size in FileUtils.appendTo(), referencing performance metrics documented in gnu coreutils (used e.g. for /bin/cat) Noah Levitt 2013-09-08 12:41:48 -07:00
• 254811ff7f commented-out default for new setting forgetAllButLatest on CheckpointService Noah Levitt 2013-09-08 12:39:54 -07:00
• f096609a50 avoid infinite loop spitting out stack traces on BeansException on startup Noah Levitt 2013-09-08 00:11:54 -07:00
• a00141eef7 avoid deadlock in case of BeansException at crawl startup Noah Levitt 2013-09-08 00:10:27 -07:00
• 071cbe99cd include urlsWritten in checkpoint, and be backward compatible with old checkpoints Noah Levitt 2013-09-07 23:40:17 -07:00
• c49b465d82 custom kryo serialization for Inet4Address in CrawlHost since default doesn't work right, and junit test Noah Levitt 2013-09-07 19:40:11 -07:00
• 52d72e661d part of HER-2048 - WARCWriterProcessor stats resume from checkpoint Noah Levitt 2013-09-07 11:35:03 -07:00
• e0f78976d7 part of HER-2048 - restore BdbServerCache from checkpoint Noah Levitt 2013-09-07 10:53:22 -07:00
• fd71833050 avoid NPE when attempting to terminate already-finished job Noah Levitt 2013-09-07 09:57:33 -07:00
• d01bc4ed55 when resuming from checkpoint, make next checkpoint serial number follow serial number of resumed checkpoint Noah Levitt 2013-09-07 09:57:07 -07:00
• d61171f076 CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size Noah Levitt 2013-09-07 09:55:58 -07:00
• b670a3c2a7 Merge branch 'master' into always-resumable Noah Levitt 2013-09-06 23:32:32 -07:00
• 25d4de186f don't log warning in normal case Noah Levitt 2013-09-06 22:04:56 -07:00
• cc0b63066b Merge branch 'master' into always-resumable Noah Levitt 2013-09-03 18:38:55 -07:00
• 34cb015ecf adjust PublicSuffixesTest for updated public suffixes list Noah Levitt 2013-09-03 18:38:44 -07:00
• 59596c7043 prevent redundant checkpoints when crawl is stopping Noah Levitt 2013-09-03 14:24:41 -07:00
• 0ab8eee24e forgetting old checkpoints - roll up previous checkpointed CrawlerJournals (e.g. frontier.recover.gz) into new checkpoint Noah Levitt 2013-08-29 10:56:12 -07:00
• 73f6b20e48 Merge branch 'master' into always-resumable Noah Levitt 2013-08-28 18:31:42 -07:00
• 981cf5bc9d update public suffixes list Noah Levitt 2013-08-28 18:31:28 -07:00
• 1315717002 Merge branch 'master' into always-resumable Noah Levitt 2013-08-28 18:14:03 -07:00
• 418c7138a9 handle case where there is no whois server for a domain Noah Levitt 2013-08-28 18:13:53 -07:00
• e8c5083d1a handle case where there is no whois server for a domain Noah Levitt 2013-08-28 18:13:33 -07:00
• 43bb42d346 BdbModule#doCheckpoint - support forgetting all but latest checkpoint Noah Levitt 2013-08-28 11:22:37 -07:00
• 8e7c26bc8f fix setForgetAllButLatest Noah Levitt 2013-08-28 10:58:32 -07:00
• 2fe7d8cd45 some progress on forgetting old checkpoints when checkpointing anew Noah Levitt 2013-08-27 18:39:25 -07:00
• 77a12e63b8 make special "host" "whois:" appear correctly in hosts report Noah Levitt 2013-08-27 17:39:07 -07:00
• 4abfcb01b0 HER-1895 support HTTP Refresh header Noah Levitt 2013-08-26 18:33:02 -07:00
• d1bb1a4637 further improve performance of BdbUriUniqFilter.forgetAllSchemeAuthorityMatching() by skipping conversion of key data to long and comparing bytes directly Noah Levitt 2013-08-21 11:44:46 -07:00
• a06c141740 rename BdbUriUniqFilter.forgetSchemeHost() to forgetAllSchemeAuthorityMatching(), improve performance Noah Levitt 2013-08-21 11:31:05 -07:00
• a6958951df Reset Recorder state when uri processing is finished (had been seeing cases where a url's entry in crawl.log would sometimes have nonzero size value if the url was erroring out and RecordingOutputStream.open() was never called) Noah Levitt 2013-08-12 19:28:28 -07:00
• 6d451d8285 Merge branch 'master' into hc43 Noah Levitt 2013-08-09 19:15:14 -07:00
• 68adaa70fd need to close the recorder here so it can calculate the content length properly Noah Levitt 2013-08-09 19:14:58 -07:00
• a2977490c4 update for httpclient 4.3-beta3-SNAPSHOT Noah Levitt 2013-08-09 18:33:15 -07:00
• 317ff7eb65 Merge branch 'master' into hc43 Noah Levitt 2013-08-09 18:17:24 -07:00
• 705a375daf uses of UriUtils.isLikelyUri() in Extractor{HTML,SWF,XML} with UriUtils.isVeryLikelyUri() to reap the benefits of HER-1523 improvements (should address archive-it issue ARI-3492) Noah Levitt 2013-08-09 18:17:11 -07:00
• 7a1f998542 bump httpclient dependency Noah Levitt 2013-08-09 17:48:48 -07:00
• 805331fcbe fix off-by-one mistake with integer arithmetic so RecordingInputStreamTest.testReadFullyOrUntil() test passes (how does this pass on master?) Noah Levitt 2013-08-09 15:39:42 -07:00
• 750a688670 Merge branch 'master' into hc43 Noah Levitt 2013-08-08 21:14:11 -07:00
• 8c64cba9ae TransclusionDecideRule.java - consider form submission "S" hop as navlink like "L", not transcluded Noah Levitt 2013-08-05 16:05:02 -07:00
• f4f3b9be38 add logging to BdbUriUniqFilter.forgetSchemeHost(), clean up some javadocs Noah Levitt 2013-07-26 17:50:36 -07:00
• b9726c8d4f new method BdbUriUniqFilter.forgetSchemeHost(String schemeHost), and unit test Noah Levitt 2013-07-25 18:51:26 -07:00
• b6b3ae6970 make log level check for debug/performance logging consistent to avoid unnecessary operations Noah Levitt 2013-07-25 17:28:16 -07:00
• 72094213a4 MirrorWriterProcessor.java - change "path" setting to a ConfigPath and set default value "${launchId}/mirror" Noah Levitt 2013-07-22 17:54:06 -07:00
• d37b43e4f3 Merge branch 'master' of github.com:internetarchive/heritrix3 Ilya Kreymer 2013-07-18 21:01:10 -07:00
• b60c43fb81 ARCRecord: Add error offset to position when adjusting for bad header lines Ilya Kreymer 2013-07-18 21:00:01 -07:00
• 452304ef12 profile-crawler-beans.cxml - include commented out default value for WARCWriterProcessor "template" setting - see http://tech.groups.yahoo.com/group/archive-crawler/message/8182 Noah Levitt 2013-07-18 18:38:33 -07:00
• 0c555b4b62 Merge branch 'master' of github.com:internetarchive/heritrix3 Ilya Kreymer 2013-07-18 17:11:08 -07:00
• 1b67f5ce6b ArcRecord: Silently skip any header fields before a status line in arc record. This allows the arc reader to be fault-tolerant of 'bad' arcs that have extra headers inserted before the status line (but are otherwise valid) Ilya Kreymer 2013-07-18 17:08:38 -07:00
• dd135c9383 Making youtube extractor log messages less noisy. Adam Miller 2013-06-25 11:33:24 -07:00
• 3916175e4d FIX: Add ArchiveRecordHeader.getContentLength() to specifically return the warc/arc record content length, not including headers Ilya Kreymer 2013-06-21 14:56:31 -07:00
• b85828135b Convert MatchesStatusCodeDecideRule and NotMatchesStatusCodeDecideRule to use CrawlURI.getFetchStatus() to avoid null status codes and broaden use. Added test cases for MatchesStatusCodeDecideRule and NotMatchesStatusCodeDecideRule Fix comments for youtube extractor. Adam Miller 2013-06-21 11:55:57 -07:00
• 1e3e66ac4c SingleHBaseTable: FIX error message typo, print friendlier message for TableNotFoundException. Kenji Nagahashi 2013-06-16 08:23:12 -07:00
• b88242ee16 bean browser: handle array of non-Object element type properly. (just avoiding throwing exception - further change would be necessary for better rendering.) Kenji Nagahashi 2013-06-16 08:21:53 -07:00
• ad70b3abd3 moved hbaes de-duplication code to heritrix-contrib. chnaged groupId to address duplicate groupId warning. Kenji Nagahashi 2013-06-13 20:54:24 -07:00
• 030006ddd5 drop debug="true" from log4j.xml in contrib Kenji Nagahashi 2013-06-13 18:45:55 -07:00
• 9bf098fd44 add contrib/target to .gitignore Kenji Nagahashi 2013-06-13 18:43:35 -07:00
• b2a0495a08 merged Kenji's change to hbase de-duplication module: - SingleHBaseTable using single HTable instances (no HTablePool) with backing-off upon communication errors. - FetchHistoryHelper for allowing multiple crawl history sources. - row key backward compatibility mode. Kenji Nagahashi 2013-06-13 17:13:03 -07:00
• 646133ce4e pom.xml Adding heritrix-engine dependency Adam Miller 2013-06-12 16:43:28 -07:00
• 1743817e1e make code formatting consistent with main heritrix proj by changing spaces to tabs; also remove unused imports from HBase.java, no other substantive changes Noah Levitt 2013-06-11 15:55:35 -07:00
• b7720ed254 Merge branch 'master' of https://github.com/internetarchive/heritrix3 Adam Miller 2013-06-11 15:42:22 -07:00
• ed6cc473f8 ExtractorYoutubeFormatStream.java Adjusting json extraction regex to account for semi-colons within the content Adam Miller 2013-06-11 15:40:07 -07:00
• 9ef2fb0f7f Merge branch 'topic/scripting-xml-fix' Kenji Nagahashi 2013-06-10 15:36:13 -07:00
• 358afd4486 revised scripting console xml response fix based on comments. removed scripting console actions from ScriptModel into ScriptingConsole class (resurrection of ScriptExec). added basic ScriptingConsoleTest. Kenji Nagahashi 2013-06-10 14:17:27 -07:00
• f2984881f6 address HER-2042 - new annotation duplicate:uriAgnosticDigest for all url agnostic duplicates, whether written to warcs or not; stats consult this annotation so that duplicate/novel numbers are correct Noah Levitt 2013-06-07 17:17:24 -07:00
• f478798935 implement itagPriority and extractLimit parameters Noah Levitt 2013-06-06 16:15:13 -07:00
• 593cae448e test crawl reveals generate_204 urls apparently not needed Noah Levitt 2013-06-06 14:41:26 -07:00
• 9a9e49d92d only extract urls that look like youtube watch pages Noah Levitt 2013-06-06 14:28:53 -07:00
• 9828310f60 make ExtractorYoutubeFormatStream extend Extractor instead of ContentExtractor because we don't really want the extra stuff that ContentExtractor does Noah Levitt 2013-06-06 14:13:26 -07:00
• a7dda8b785 ExtractorYoutubeFormatStream.java Parse parameters instead of attempting to regex desired values Add priority itag list and extract limit Adam Miller 2013-06-05 16:39:10 -07:00
• 11f8fd51c6 extractIndependently=true so that earlier ExtractorHTML extraction doesn't stop us from extracting here; also use java.net.URLDecoder Noah Levitt 2013-06-05 13:25:25 -07:00
• feec423b52 import Adam's ExtractorYoutubeFormatStream from its former home https://github.com/adam-miller/ExtractorYoutubeFormatStream Noah Levitt 2013-06-05 13:16:55 -07:00
• 0508abb9ad new contrib subproject for extra generally useful modules not part of the main distribution Noah Levitt 2013-06-05 13:12:36 -07:00
• ffd248f780 HER-2040 ExtractorJS shouldExtract when content-type is application/json Noah Levitt 2013-06-03 11:23:16 -07:00
• 3a54600a79 ADD unit test for XmlMarshaller Kenji Nagahashi 2013-05-29 12:03:49 -07:00
• c53b821c85 FIX: 73f58dc56a broke XML variant of scripting console response, and will produce incompatible structure (extra nesting with scriptExec element). moved ScriptExecution code to ScriptModel, which is now a bean, not Map. XmlMarshaller is extended to support simple JavaBeans to XML marshalling, borrowing annotation classes from javax.xml. Kenji Nagahashi 2013-05-28 13:28:37 -07:00
• 7f2aacba2c BeanLookupBindings: check appCtx != null explicitly, rather than using assert. Kenji Nagahashi 2013-05-28 13:20:23 -07:00
• 73888819dc fix bug where scripting console no longer retains your script when you hit execute Noah Levitt 2013-05-15 14:23:53 -07:00
• 858523db94 HER-2039: Initial work. Eliminated Link class and dealt with resultant errors. Not tested any further than that. Kristinn Sigurðsson 2013-05-15 13:36:01 +00:00
• d4c5bd6e98 avoid npe by setting default <input> type "text" when unspecified Noah Levitt 2013-05-14 18:17:09 -07:00
• d972731958 ExtractorHTMLForms.java - handle a corner case extracting attribute values Noah Levitt 2013-05-14 16:23:41 -07:00
• 7bd010e3c7 HTMLForm.java - default html <input> type is text, so consider <input> with no type as candidateUsernameInput Noah Levitt 2013-05-13 22:53:37 -07:00
• bea5c6f267 followup on HER-2031 ExtractorHTMLForms.java - improve regexes looking for form and input attributes, mainly to avoid maxing cpu for long periods of time for certain input Noah Levitt 2013-05-13 22:37:02 -07:00
• 5b4d8c7f0d use intended File.separator instead of File.pathSeparator Noah Levitt 2013-05-13 22:06:09 -07:00
• ec824e9724 avoid need for getRawData() method by introducing Link.hasData() Noah Levitt 2013-05-13 17:43:45 -07:00
• 7c5b2ceb00 Added a data map to Link. Becomes the CrawlURI's data map when Link is promoted to CrawlURI Kristinn Sigurðsson 2013-05-13 15:18:02 +00:00
• 3841d096d5 removed commented-out old code. Kenji Nagahashi 2013-05-11 15:42:53 -07:00