Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#781/merge

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• c2b82efb89 Fixes from trial guide testing * EngineApplication.java better handle job directories with spaces in them gojomo 2009-06-09 08:22:21 +00:00
• 3862c361a4 Fixes from trial guide testing * defaults.xml remove error-causing redundant BdbCookieStorage declaration * TextUtils.java exception- and deprecation-free UTF8 URL escaping * CrawlJob.java, JobResource.java ensure job log shows INFO-level items better handle job directories with spaces in them gojomo 2009-06-08 23:30:55 +00:00
• 02b5ca5f1d [HER-1601] Restletify: per-job operator arbitrary script execution * ScriptResource.java textarea offering execution of any script supported by available JSR-223 script engines gojomo 2009-06-05 02:59:28 +00:00
• 28ed31c498 [HER-1604] Springify/restletify: offer web interface to view/edit beans in running crawl * BeanBrowseResource.java offers browsing (and limited editting of simple values) of beans in built crawl * JobRelatedResource.java, ReportGenResource.java refactor for reuse by other job sub-resources * JobResource.java link to job/beans and job/script * DescriptorUpdater.java helper for marking-up PropertyDescriptors to aid UI bean access * CrawlJob.java don't auto-instantiate context on request * EngineApplication.java add uri routes * FetchHTTP.java, BdbCookieStorage.java, LongWrapper.java WorkQueueFrontier.java, BdbUriUniqFilter.java improve display when browsed (esp. before launch) gojomo 2009-06-05 02:51:11 +00:00
• 771d23240c refactor sheets to better enable mid-crawl changes * Sheet.java make non-abstract; allow for name that might not be beanName * SheetAssociation.java define targets as list of names, rather than Sheets themselves * SheetForDecideRuled.java -> DecideRuledSheetAssociation.java; SheetForSurtPrefixes.java -> SurtPrefixesSheetAssociation.java make association separate from Sheet -- so same Sheet can be associated more than one place, more than one way * SheetOverlaysManager.java match changes to Sheet and SheetAssociations offer convenience methods for post-construction addition of Sheets, associations use internal name-to-Sheet mapping, allowing for mid-crawl Sheet additions not in bean factory * Precedence1SelfTest.java, Precedence4SelfTest.java update configs for Sheet changes gojomo 2009-06-05 02:12:32 +00:00
• d336ea760f quiet excessive logging gojomo 2009-06-05 02:00:07 +00:00
• 92f5ca2a45 remove unused imports/code gojomo 2009-06-05 01:12:51 +00:00
• 02dd47b8dc [HER-1505] Expand BeanshellProcessor/BeanshellDecideRule for every JVM-friendly scripting language (Javascript/Jython/Groovy/JRuby) * BeanShellProcessor.java -> ScriptedProcessor.java; BeanShellDecideRule.java -> ScriptedDecideRule.java update to use any present JSR-223 scripting engine (requested by free-text name) update to allow script to come from inline string or file (just as with seeds) * modules/pom.xml include groovy and jsr-223-capable JARs * .classpath add new JARs to classpath * BeanShellProcessorTest.java -> ScriptedProcessorTest.java rename (empty) test class gojomo 2009-06-05 01:03:10 +00:00
• 7ca3fbff72 * heritrix.cmd remove JMX-related paths; streamline comments; link to Heritrix3-on-Windows wiki page gojomo 2009-05-28 00:05:44 +00:00
• 6a5b5e4639 Fix HER-1509 changes carried forward from h2 * DefaultProcessorURI.java add constructor that supports setting via * CustomSWFTags.java, CrawlUriSWFAction.java look for links in a couple more places * ExtractorSWF.java ExtractorTagParser.parsePlaceObject2() - read 32 bit clip event flags when flash version >= 6 * ExtractorSWFTest.java xestHer1509() - test this fix (rename to enable the test) szznax 2009-05-23 00:17:41 +00:00
• b125b05deb minor improvement * TestUtils.java added tmpDir() w/deleteOnExit() as a convenience method for classes not needing to implement TmpDirTestCase szznax 2009-05-21 23:46:34 +00:00
• 8c0b539331 minor improvement * CrawlURI.java removed static MAX_OUTLINKS in favor of ExtractorParameters.getMaxOutlinks() szznax 2009-05-21 23:39:50 +00:00
• 78a38d6fc1 minor improvement * Extractor.java added instance initializer for DEFAULT_PARAMETERS, and set @Autowired(required=false) eliminates the need for test classes to set them explicitly. * ExtractorCSSTest.java remove explicit setting of default parameters * ExtractorHTMLTest.java remove explicit setting of default parameters * JerichoExtractorHTMLTest.java remove explicit setting of default parameters szznax 2009-05-21 23:34:18 +00:00
• ca8a02bb17 [HER-1637] upgrade JE jar to 3.3.82 * commons/pom.xml, .classpath update refs to 3.3.82 gojomo 2009-05-21 05:46:27 +00:00
• 5746e422df [HER-1598] Restletify: enable report generating, viewing, refreshing * ReportGenResource.java restlet to generate-report-to-file and redirect to view file * JobResource.java links to reports from appropriate places for built job clarify profiles, simplify into-job-dir path-construction * CrawlJob.java remove leading '/' from job-relative paths * EngineApplication.java install ReportGenResource at appropriate in-job path gojomo 2009-05-18 21:23:31 +00:00
• 895e822a81 [HER-1598] Restletify: enable report generating, viewing, refreshing * StatisticsTracker.java fortify for reporting even pre-start/post-stop add new report types, new groupings: ALL_, LIVE_, END_ dump END_ reports at stop() dump report by short (class) name gojomo 2009-05-18 21:20:55 +00:00
• a504d518ba [HER-1598] Restletify: enable report generating, viewing, refreshing * CrawlController.java fortify for report-generation even pre-start/post-stop gojomo 2009-05-18 21:18:03 +00:00
• 7e4245bfee [HER-1598] Restletify: enable report generating, viewing, refreshing * FrontierSummaryReport.java, FrontierNonemptyReport.java, ToeThreadsReport.java Report classes for usual frontier/toethread situations gojomo 2009-05-18 21:15:26 +00:00
• 25b5da8c5b pull Lifecycle interface up to interface gojomo 2009-05-18 21:07:28 +00:00
• 35fbf91626 added joda-time dependency, used in szznax 2009-05-18 17:45:31 +00:00
• 3f6d60a054 copy to h3 Fix for [HER-1588] Arc2Warc transformed WARC has incorrect date in warcinfo record * Arc2Warc.java correct WARC-Date format, and use Joda-time szznax 2009-05-15 18:19:45 +00:00
• 8b59ef1086 Restore directory-browsing broken by password Guard * Heritrix.java reorder restlet setup so Context is passed through Guard to EngineApplication gojomo 2009-05-15 02:02:04 +00:00
• 81a9a9f0f3 Fix unit test * BadURIsStopPageParsingSelfTest.java update expected URIs to match H2 branch / fixed code gojomo 2009-05-15 00:27:27 +00:00
• a84e5b5dca copied to h3 from h2 commits Misc: maintain object identity of peek()ed objects * StoredQueueTest.java test that consecutive peek()s return exactly identical object * StoredQueue.java cache head object object for peek consistency gojomo 2009-05-15 00:08:29 +00:00
• c891a0b278 copy to h3 szznax 2009-05-14 23:55:45 +00:00
• 5a4e8c63d3 copy to h3 Fix for [HER-1533] robots.txt fetch referer header is via url which is confusing * FetchHTTP.java configureMethod() don't set referer header if this is a prerequisite gojomo 2009-05-14 22:35:47 +00:00
• bb6d1c9c41 copy to h3 Fix for [HER-1570] update extensions regexps in MatchesFilePatternDecideRule * MatchesFilePatternDecideRule.java gojomo 2009-05-14 22:32:47 +00:00
• dccf2becae copy to h3 Fix for [HER-1612] lock contention in BeanShellDecideRule/BeanShellProcessor occurs even if isolate-threads=true * BeanShellDecideRule.java (getInterpreter) add synchronization (decisionFor) remove method synchronization; sync on interpreter/getInterpreter makes it a superfluous bottleneck * BeanShellProcessor.java (getInterpreter) add synchronization (innerProcess) remove method synchronization; sync on interpreter/getInterpreter makes it a superfluous bottleneck gojomo 2009-05-14 22:27:10 +00:00
• dc117fb958 * commons/pom.xml use latest 3.3.75 bdb je jar gojomo 2009-05-14 22:26:21 +00:00
• 58c9e5f0b1 copy to h3 Fix for [HER-1567] extractor for Wap 2.0 pages -- reported by Christian Dehning * ExtractorHTML.java shouldExtract() - accept various Wap mime types gojomo 2009-05-14 22:23:17 +00:00
• db794496fb copy to h3 Fix for [HER-1550] set-source-tag option nonfunctional * CrawlURI.java ensure 'heritable' key is itself heritable; also change heritable collection to a set gojomo 2009-05-14 22:20:07 +00:00
• 58d2a2ec31 copy to h3 Fix for [HER-1552] Redirects not reported in seeds-report * ExtractorHTTP.java restore expected convention of http-header context ending in ':' gojomo 2009-05-14 22:10:06 +00:00
• a1e9d064da copy to h3 Fix for [HER-1520] Windows mapped-files cause intermittent ReplayCharSequenceTest.testReplayCharSequenceByteToStringMulti failures, risk other problems * GenericReplayCharSequence.java use more diverse filenames, so subsequent decodes don't run into lingering open mapped sections of previous decodes do delete via new FileUtils.deleteSoonerOrLater utility * FileUtils.java (deleteSoonerOrLater) try immediate delete, but if fails, add to a list of files to be retried later gojomo 2009-05-14 21:31:45 +00:00
• 9155e5f4f9 copying noah's work to h3 Fix for [HER-1506] ByteReplayCharSequence ignores specified/non-default character encodings, reported by Jean-No?\195?\171l Rivasseau * RecordingOutputStream.java use ByteReplayCharSequence only for iso8859-1 * ReplayCharSequenceTest.java new tests for iso8859-1, windows-1252 and ascii gojomo 2009-05-14 21:19:24 +00:00
• ad33d302ff copying noah's work to h3 [HER-1506] correct javadoc on use of these classes * GenericReplayCharSequence.java * Latin1ByteReplayCharSequence.java gojomo 2009-05-14 21:12:34 +00:00
• 5baac7cc45 copying noah's work to h3 [HER-1506] rename classes to reflect true roles * MultiByteReplayCharSequence.java -> GenericReplayCharSequence.java * ByteReplayCharSequence.java -> Latin1ByteReplayCharSequence.java * RecordingOutputStream.java update references gojomo 2009-05-14 21:10:44 +00:00
• 37ea37b883 Fix for [HER-1564] Update 'public suffix list' (effective_tld_names.dat) * effective_tld_names.dat update to latest version from publisuffix.org (Mozilla) -- downloaded 2009-05-14 gojomo 2009-05-14 20:56:51 +00:00
• dae03cb1de port to h3 noah's fix Fix for [HER-1533] bottleneck in StatisticsTracker.saveSourceStats * StatisticsTrackerImpl.java saveSourceStats() -- call put() on the CachedBdbMap only when entry is added, not on every change gojomo 2009-05-14 20:51:46 +00:00
• 8c620ae846 Fix for [HER-1632] Update Jericho HTML Parser to 2.6 * commons/pom.xml update requested versions to 2.6.1 gojomo 2009-05-14 20:47:29 +00:00
• e9622784f6 Apply to h3 Fix for [HER-1225] HtmlFormCredential.java may create non-wellformed URLs * HtmlFormCredential.java only insert '&' if previous query-string was non-empty gojomo 2009-05-14 20:32:42 +00:00
• 9994edb170 HER-565 / improve output if problem recurs gojomo 2009-05-14 20:29:01 +00:00
• badbac399f apply to h3 Fix for [HER-1521] Make the UncompressedARCReader and CompressedARCReader in the ARCReaderFactory as public or protected * ARCReaderFactory.java, WARCReaderFactory.java make compressed/uncompressed implementation inner classes public to enable direct reuse gojomo 2009-05-14 20:24:56 +00:00
• 8ff73f8a97 apply to h3 Fix for [HER-1558] ${HOSTNAME} in arc suffix is only replaced completely * WriterPoolProcessor.java replace ${HOSTNAME} in suffix, rather than test for exactly ${HOSTNAME} gojomo 2009-05-14 20:17:38 +00:00
• b2e1fe93fd apply to h3 Fix for [#HER-1554] META http-equiv refresh content containing only a number misinterpreted as a URI - IA Webteam JIRA: * ExtractorHTML.java don't use full 'content' as META refresh URL in absence of '=' gojomo 2009-05-14 20:14:03 +00:00
• 7480d2933b Cleanup for h3 of [HER-1045] [contrib] make it easier to set FetchHTTP local bind address, use rotation of alternate local addresses * FetchHTTP.java advertise key for CrawlURI-based bindAddress override as static constant HTTP_BIND_ADDRESS * CoreAttributeConstants.java remove unused A_HTTP_BIND_ADDRESS gojomo 2009-05-13 23:49:36 +00:00
• a89cf707ce apply to h3 Fix for [HER-1499] crawl.log has no annotation indicating that WARCWriterProcessor wrote a revisit record * WARCWriterProcessor.java add annotations "warcRevisit:digest" or "warcRevisit:notModified" when revisit record written gojomo 2009-05-13 23:37:55 +00:00
• 9558d94b5d apply to h3 Fix for [HER-1532] robots.txt with html <BR> markup causing NumberFormatException * RobotsTxtTest.java test based on ecnountered robots.txt with HTML markup but clear intent * RobotsTxt.java strip HTML markup from each line before parsing strip expected numbers in Crawl-Delay to first run of digits-and-decimal-points catch any NumberFormatException in Crawl-Delay number and ignore, continuing other parsing gojomo 2009-05-13 23:34:29 +00:00
• 58486c5f18 (copying from h2 to h3) Fix for [HER-1277] speculative javascript link caused 404s for "http%3A" and "www.example.com" like strings * ExtractorJS.java perform additional decoding or prepending of "http:" for speculative candidate URIs that appear to need it * ArchiveUtils.java add utility set/method with list of all TLDs gojomo 2009-05-13 02:14:15 +00:00
• ab6942a60d Fix for [HER-1291] accurate scoping not possible at log-recovery time; URIs may be recrawled against expectations * FrontierJournal.java refactor to always operate on CrawlURI instances (with hops-path and via info), and to write longer info on success/failure lines gojomo 2009-05-13 01:43:41 +00:00
• a530af6e5c Improvement for [HER-1] Support 'Crawl-Delay' and 'Allow' robots.txt directives * AbstractFrontier.java change boolean 'respect-crawl-delay' to integer 'respect-crawl-delay-up-to-secs' Fix for [HER-1516] in datamodel.Robotstxt, Integer.parseInt generates NumberFormatException on floating point Crawl-delay strings * RobotsDirectives.java, RobotsTxt.java, RobotsExclusionPolicy.java parse, store, return crawlDelay as float * RobotsTxtTest.java compare value as float gojomo 2009-05-13 01:14:58 +00:00
• 2c7c07ba64 apply to h3 noah's fix [HER-1510] meta refresh url from a seed considered as a seed * LinksScoper.java (considerAsSeed) return true any seed with hop type Hop.REFER, which includes meta refreshes; also simplify the code a bit gojomo 2009-05-12 22:29:54 +00:00
• 2dec5dfd1a Fix for [HER-1268] ExtractorHTML not able to extract some links (due to them violating RFC2396) * ExtractorHTMLTest.java, UURIFactoryTest.java test that relative-URIs with late-position colons aren't interpreted as absolute URIs with long, illegal schemes * UURIFactory.java update RFC2396REGEX to require legal scheme * LaxURI.java allow illegal candidate scheme to just be interpreted as something else gojomo 2009-05-12 22:22:11 +00:00
• 252cf63863 Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme * ExtractorHTMLTest.java test for maintaining 'https' when speculative, leading-domain URI is on same host * ExtractorJS.java (speculativeFixup) maintain origin URI scheme if apparent target host is same gojomo 2009-05-12 22:09:22 +00:00
• 2a73339211 Forward port for [HER-1525] Bump WARC version to match latest (0.18, June-2008) ISO draft * WARCConstants.java, WARCWriter.java, WARCWriterProcessor.java remove/update references to pre-0.18 versions or old class names gojomo 2009-05-12 19:51:16 +00:00
• 2c11eb1dc2 Remove cruft comments gojomo 2009-05-12 19:50:05 +00:00
• fb97744dfd Fix for [HER-1495] ARCRecord throws null pointer when trying to generate a RecoverableIOException * ARCRecord.java guard against bull statusBytes in exception message gojomo 2009-05-12 19:43:20 +00:00
• 8de6b2a9c5 Fix for [HER-1502] selftests break if 'localhost' does not resolve to 127.0.0.1 * (all) change 'localhost' to '127.0.0.1' gojomo 2009-05-12 18:40:18 +00:00
• 3f7d85d7aa Fix for [HER-1528] NumberFormatException in UURIFactory.checkPort() * UURIFactoryTest.java test that port-looking-part that's out of integer bounds throws URIException not NumberFormatException * UURIFactory.java catch NumberFormatException; allow to surface as URIException gojomo 2009-05-12 18:21:17 +00:00
• 4082195690 Fix for [HER-1529] AggressiveExtractorHTML reports back as ExtractorHTML2 in processor reports * AggressiveExtractorHTML.java correct self-reported name gojomo 2009-05-12 18:04:14 +00:00
• c8a5446695 Test robustness * SelfTestBase.java don't allow negative random passwords, in case leading '-' is causing problems * Heritrix.java echo arguments when showing usage info -- to understand what went wrong gojomo 2009-05-12 02:10:37 +00:00
• daf5ed7590 Update pom.xml * 4 more naming improvements ignore test cruft update SCM url gojomo 2009-05-12 00:17:01 +00:00
• e047bf68e0 copy 'springy' branch to heritrix3 gojomo 2009-05-11 22:56:36 +00:00