Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 2f9a8b860d * GZIPMembersInputStream ensure mark()ing more frequently, when reset to before current position won't be needed * ArchiveReaderFactory avoid misinterpreting a "c:" local path as a URI scheme gojomo 2011-02-26 00:59:10 +00:00
• fe5363feb0 * GzipMembersInputStreamTest test with 6 small members gojomo 2011-02-24 22:13:46 +00:00
• 9fef4f1f26 * ArchiveReader better logging of caught/recovery-attempted IOException * GZIPMemberInputStream handling of previously-unrecognized case in pre-JDK6u23: attempt at read() exhausts inflater with no new bytes available (previous nonzero-length read had not indicated inflater finished) gojomo 2011-02-24 19:22:50 +00:00
• bb07080e02 [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling * ArchiveReaderFactoryTest test cases for generic reader-get methods -- one of which had obsolete STREAM_ALL=-1 problem gojomo 2011-02-23 01:09:05 +00:00
• c6a94c2264 [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling * ArchiveReaderFactory discard obsolete -1 flag value creating problems with usual uses gojomo 2011-02-23 00:24:47 +00:00
• f03b1ac689 * URIAuthorityBasedQueueAssignmentPolicy.java rename "whois" queue to "whois..." to avoid conflicting with queue for theoretical hostname "whois" nlevitt 2011-02-19 02:10:39 +00:00
• b9c408cb07 Change form of generic serverless whois url to e.g. whois:archive.org * FetchWhois.java * UURIFactory.java nlevitt 2011-02-19 02:00:18 +00:00
• a6f3b7db25 * ARCWriterTest test inspired by Erik Hetzner's HER-1865 test code gojomo 2011-02-19 00:35:09 +00:00
• 368ff65286 * FetchWhois.java add license preamble nlevitt 2011-02-17 22:52:37 +00:00
• 4991b295bc HER-1645 whois fetcher * URIAuthorityBasedQueueAssignmentPolicy.java whois urls all go in special "whois" queue * PreconditionEnforcer.java, CrawlURI.java move markPrerequisite() method from PreconditionEnforcer to CrawlURI * profile-crawler-beans.cxml commented-out whois fetcher clause * FetchStatusCodes.java new status codes for whois * ServerCache.java getHostFor() - return "whois:" for whois urls, following dns: convention * UURIFactory.java ugly hack for whois urls * SchemeNotInSetDecideRule.java add whois to list of supported schemes * WriterPoolProcessor.java shouldWrite() - we should write successful whois fetches * WARCWriterProcessor.java write whois records * FetchWhois.java the fetcher nlevitt 2011-02-17 22:33:28 +00:00
• 885a25c90c Robustify based on JUnit issues on Windows * ArchiveReader improve warning message * WriterPoolMember attempt pre-delete of target filename, warn on failure, for better odds on some filesystems * ARCWriterTest, WARCWriterTest ensure timely close of readers/writers in case filenames are reused restore originally-intended unique target names gojomo 2011-02-17 01:21:48 +00:00
• 9d588b9978 implement Closeable gojomo 2011-02-17 01:17:35 +00:00
• b86f0afc83 * TopNSet resolve strange type complaint in my Windows Eclipse JDK6u24 setup gojomo 2011-02-17 01:14:32 +00:00
• 61dc661f47 * TopNSet resolve strange type complaint in my Windows Eclipse JDK6u24 setup gojomo 2011-02-17 01:13:37 +00:00
• 548114dbde [HER-1864] cookies are being set and sent back on TLDs/public-suffixes (like '.com') (sometimes called 'supercookies') * CookieSpecBase use Guava InternetDomainName public-suffixes to both (1) prevent storing cookie on public-suffix; (2) prevent looking-up cookies for public-suffixes * BdbCookieStorage correct checkpoint/relaunch behavior: don't reuse persisted cookies on normal relaunches; do reuse on resume-from-checkpoint gojomo 2011-02-16 01:18:57 +00:00
• 5de965590c Update Guava library to r08 * .classpath, commons/pom.xml update reference * ObjectIdentityBdbManualCache use new maximumSize() option on dirtyItems map gojomo 2011-02-16 01:15:13 +00:00
• 9b78770027 [HER-1865] JDK6u23 breaks GzippedInputStream & W/ARCReaders with different GZIP handling * GZIPMembersInputStream new workaround class that offers choice of pre/post JDK6u23 behavior and new accessors for finding member boundary offsets * GZIPMembersInputStreamTest test for above * ArchiveUtils, ARCWriter move static compress utilities to ArchiveUtils * ArchiveReader, (W)ARCReaderFactory, (W)ARCWriterTest adapt to use GZIPMembersInputStream * GzippedInputStream(Test) deleted gojomo 2011-02-16 01:12:25 +00:00
• 9e53f8be7c [HER-1867] make H3 ssl adhoc self-signed certificate valid for longer (appears to expire after 90 days) * Heritrix pass '-validity 3650' to KeyTool gojomo 2011-02-14 22:07:10 +00:00
• 9468cb0c1f Report total bytes, as well as breakdown, in xml version of job page, to match html. * CrawlJob.java sizeTotalsReportData() - add "total" (bytes) and "total-count" (urls) to map * CrawledBytesHistotable.java new method getTotalUrls() to complement getTotalBytes() nlevitt 2011-02-09 01:17:26 +00:00
• f58fdcdba4 * Heritrix use heritrix.home, if available, to find default logging.properties (to let those who launch Heritrix from elsewhere save themselves from HttpClient's copious logging) gojomo 2011-02-05 03:26:02 +00:00
• 244c1cf563 Update to latest restlet 1.1.10. It includes Daniel Truemper's fix for the restlet bug he discovered while using heritrix, which I ran into. http://tech.dir.groups.yahoo.com/group/archive-crawler/message/6338 http://restlet.tigris.org/issues/show_bug.cgi?id=1018 http://restlet.tigris.org/source/browse/restlet?view=rev&revision=6135 * .classpath * engine/pom.xml nlevitt 2011-02-05 00:36:11 +00:00
• 40f7debeaa [HER-1846] IOException due to disk full situation can cause frontier lock-up * WARCWriterProcessor move checkSize into block protected by catch/finally cleanup of errors * WARCWriterProcessorTest test to simulate record-write-failure and verify does not cause pool lockup * WARCWriterTest, WriterPoolMember comment and warnings cleanup gojomo 2011-01-28 03:02:17 +00:00
• a6fd015f6b * commons/pom.xml fix inconsistent line endings (remove ^M's) - a trivial change to poke continuum which is truncating this file nlevitt 2011-01-28 02:04:55 +00:00
• c1a99a8e35 [HER-1861] Add 'first-named' robots policy, which considers rules for more than one user-agent, in order * Robotstxt, RobotstxtTest treat wildcardRules specially, so that it is possible to ask for only explicitly-named matches * MostFavoredRobotsPolicy improve comment, change masquerade default to true * FirstNamedRobotsPolicy, FirstNamedRobotsPolicyTest policy to consider alternate user-agents in order, using the first that has any explicit (non-wildcard) rules declared (or failing that, the original default user-agent) gojomo 2011-01-28 00:48:25 +00:00
• d22ab1cb6d * PreconditionEnforcer prefer 'protected' over 'private' gojomo 2011-01-28 00:22:02 +00:00
• c2027b7a30 * SimpleSelfTest more useful assertEquals gojomo 2011-01-26 03:11:17 +00:00
• a3d4dc5ac2 [HER-1834] log-bloat, memory problems when hopsPath grows endlessly with no max-hops limit set * TooManyHopsDecideRule remove inadvertent reversal of sense of comparison, making default scope reject almost everything (and breaking most self-tests) gojomo 2011-01-26 03:08:32 +00:00
• 8212d1c0dd [HER-1860] H3: Remove dependence on GC/finalization/PhantomReference magic in used ObjectIdentityCache implementation * StatisticsTracker move small stat maps to in-memory ConcurrentMaps update processedSeedRecords for new ObjectIdentityCache shape move hostsDistribution/hostsBytes/hostsLastFinished tracking to CrawlHosts/hostsCache * CrawlSummaryReport gets hosts count from hostsCache * SeedRecord implement IdentityCacheable * FetchStats remember last-success times gojomo 2011-01-20 23:36:50 +00:00
• 6bdc7642b4 * CrawlController ensure final stat snapshot before stop gojomo 2011-01-20 23:29:59 +00:00
• 0b577c1245 [HER-1860] H3: Remove dependence on GC/finalization/PhantomReference magic in used ObjectIdentityCache implementation * IdentityCacheable new interface required of objects stored in ObjectIdentityCaches * IdentityCacheableWrapper wrapper for storing arbitrary objects in ObjectIdentityCaches * ObjectIdentityCache keys now always Strings values now always IdentityCacheables new dirtyKey() method to ensure a key is persisted * ObjectIdentityBdbCache, ObjectIdentityMemCache update for new ObjectIdentityCache shape; still using legacy GC magic * ObjectIdentityBdbManualCache alternate implementation relying on dirtying to ensure persistence use Guava library MapMaker for soft memMap; capped dirtyMap * BdbModule use ObjectIdentityBdbManualCache by default up expectedConcurrency default to 64 * BdbModuleTest, ObjectIdentityBdbCacheTest, ObjectIdentityBdbManualCacheTest update, add tests * Frontier FrontierGroup as IdentityCacheable * AbstractFrontier, WorkQueueFrontier, BdbFrontier touchup queue/group accessors make queue instances dirty whenever mutated * WorkQueue, CrawlHost, CrawlServer IdentityCacheable support; appropriate makeDirty()s * ServerCache, DefaultServerCache update for new ObjectIdentityCache shape gojomo 2011-01-20 23:28:38 +00:00
• d2f28f051c [HER-1834] log-bloat, memory problems when hopsPath grows endlessly with no max-hops limit set * CrawlURI add getHopCount() for total (including tallied overflow) hop count * TooManyHopsDecideRule base decision on getHopCount() gojomo 2011-01-19 07:37:43 +00:00
• c782714b55 [HER-1834] log-bloat, memory problems when hopsPath grows endlessly with no max-hops limit set * BdbMultipleWorkQueuesTest use setOrdinal rather than now-gone constructor gojomo 2011-01-19 04:16:18 +00:00
• 6795b3bfba [HER-1834] log-bloat, memory problems when hopsPath grows endlessly with no max-hops limit set * CrawlURI MAX_HOPS_DISPLAYED constant, 50 build new CrawlURI pathFromSeed value via extendHopsPath, which shows only MAX_HOPS_DISPLAYED hops, and precedes string with int value of hops unshown * CrawlURITest test for extendHopsPath gojomo 2011-01-19 02:30:42 +00:00
• f0f8aeb9b9 * WARCWriter remove that pesky reference to a 'modules' class (usu. available in dev setup, but not in ordered formal builds) gojomo 2011-01-19 01:25:42 +00:00
• 57edfde618 [HER-1859] ExtractorJS improper handling of Javascript string literal escapes * ExtractorJS apply JS literal string deescaping before interpretation as link * ExtractorJSTest 3 basic tests, including one for this issue * Link provide logging and fallback for case where via may be null; risk identified in creation of test cases could present again if a '.js' file is ever provided as a (via-less) seed gojomo 2011-01-18 23:34:35 +00:00
• 0af0d3cc6a * WARCWriterProcessor.java addStats() - use putIfAbsent() to avoid race conditions; move creation of stats map to constructor to avoid another race condition * WARCWriter.java switch back to thread-unsafe data structures for stats since object is used in one thread at a time according to javadoc nlevitt 2011-01-18 04:02:42 +00:00
• 823aaf0154 Add Javadoc class comment notes about (potentially) left-open-by-design ReplayCharSequence instances gojomo 2011-01-18 01:32:26 +00:00
• 799e0d9ff8 Remove errant import that eclipse added to satisfy a javadoc reference, but that breaks build nlevitt 2011-01-17 23:06:11 +00:00
• cf597b2796 Fix fix on commented out scope.logToFile - default value is false nlevitt 2011-01-17 22:53:43 +00:00
• c966301100 Fix commented out scope.logToFile nlevitt 2011-01-17 22:51:33 +00:00
• 69429f2d4e Missed one file r7051 commit * ExtractorJS.java do not close ReplayCharSequence nlevitt 2011-01-17 21:27:39 +00:00
• 50bb39a839 * WARCWriterProcessor.java initialize urlsWritten (fix npe, oops) nlevitt 2011-01-15 19:46:42 +00:00
• 72932b8ca8 Rename and comment on WARCWriter stats bits to clarify their use as temporary accumulators on behalf of WARCWriterProcessor. nlevitt 2011-01-15 04:17:49 +00:00
• 9a34201c8e Make warc writer stats thread-safe. nlevitt 2011-01-15 04:01:04 +00:00
• 6de68e3e36 Let Recorder remember its replayCharSequence, and ToeThread close it when uri processing is finished. * Recorder.java getReplayCharSequence() - remember ReplayCharSequence, obtain new one if none remembered or remembered has been closed endReplays() - close replayCharSequence * ToeThread.java run() - call Recorder.endReplays() when finished with uri * ReplayCharSequence.java isOpen() - return false if close() has been called * GenericReplayCharSequence.java, InMemoryReplayCharSequence.java implement isOpen() * KeyWordProcessor.java, ExtractorHTML.java, HTTPContentDigest.java, ExtractorCSS.java, ExtractorXML.java do not close ReplayCharSequence nlevitt 2011-01-15 03:13:27 +00:00
• ded64befb5 W/ARCWriter-related refactoring * WriterPoolSettingsData useful defaults for frequentFlushes and writeBufferSize gojomo 2011-01-15 02:43:08 +00:00
• cbd65d8350 [HER-1848] Add option to disable link-extraction from 404 error pages * AbstractFrontier add new/renamed ExtractorParameters methods gojomo 2011-01-15 01:32:54 +00:00
• 6f095b409c [HER-1848] Add option to disable link-extraction from 404 error pages * FetchStatusCodes add 404 constant * ExtractorParameters rename isIndependentExtractor (which seems to describe the parameters object) to getExtractIndependently (the paramters' imperative to extractors) add getExtract404s * Extractor add default ExtractorParameters impl remove redundant shouldProcess * ContentExtractor consult getExtract404s, if false skip extraction of 404s gojomo 2011-01-15 01:21:09 +00:00
• cd398d4c46 W/ARCWriter-related refactoring * Arc2Warc, Warc2Arc adapt to new constructors, settings-object * MiserOutputStream stream to monitor position, optionally suppress flushes * WriterPoolMember use MiserOutputStream rather than special file-access to find compressed offsets use shared settings object rather than copying multiple values * WriterPoolSettings add settings for frequentFlushes and writeBufferSize for IO optimization * ARCReader, ARCWriter, ARCWriterPool adapt to new constructors, settings-object * WriterPoolSettingsData impl of WriterPoolSettings for testing and adhoc use * WARCWriter adapt to new constructors, settings-object remove checkSize rollover to new file (done elsewhere) remove extraneous id-generator code/static-method * WARCWriterPool adapt to new constructors, settings-object * WARCWriterPoolSettings, WARCWriterPoolSettingsData add recordIDGenerator setting, impl class for testing/adhoc use * WARCWriterTest adapt to new constructors, settings-object * Generator, GeneratorFactory removed in favor of RecordIDGenerator interface * RecordIDGenerator common interface for classes that can provide WARC record IDs * UUIDGenerator, UUIDGeneratorTest adapt to derive from RecordIDGenerator avoid throwing exceptions * ARCWriterPoolTest, ARCWriterTest adapt to new constructors, settings-object * WARCWriterProcessor do checkSize file-rollover here, so related records don't span WARCs calculate processor's totalBytesWritten via position offset changes (as before stats-additions) delegate record-ID generation * WriterPoolProcessor add frequentFlushes and writeBufferSize from new WARCWriterPoolSettings interface gojomo 2011-01-14 23:26:31 +00:00
• c5717f80d4 * DisposableStoredSortedMap better class comment, error handling gojomo 2011-01-14 22:09:21 +00:00
• df1d064598 * StoredQueue.java protect isEmpty() like size() for access after backing BDB closed gojomo 2011-01-14 22:07:01 +00:00
• 402b9a45a6 * RobotstxtTest add test case for robots.txt with extraneous leading whitespace and missing blank lines gojomo 2011-01-12 23:45:49 +00:00
• 898689ab38 * WriterPoolMember.java restore void return value of write(*) methods * WriterPoolMember.java do not expect return from write(*) methods nlevitt 2011-01-12 08:38:56 +00:00
• fd212669af [HER-1856] remove redundant profile-crawler-beans.cxml nlevitt 2011-01-12 01:11:36 +00:00
• ef9d2e8c0e [HER-1789] stats from WARCWriterProcessor * WriterPoolMember.java write(*), copyFrom() - return number of (uncompressed) bytes written, for convenience * WARCWriter.java tally by record type: number of records written, content bytes, total bytes, and (possibly compressed) size on disk * WARCWriterProcessor.java keep totals and write report to processors-report.txt nlevitt 2011-01-12 00:54:01 +00:00
• 5fe28f2ecc [HER-1858] H3: frontier futureUris (rescheduling) unreliable for otherwise empty frontier/queue * WorkQueue.java add noteExhausted() for emptied queues * WorkQueueFrontier.java add synchronization around checkFutures() activity use noteExhausted when queue is totally empty gojomo 2011-01-12 00:21:40 +00:00
• 9e75ff0019 * GenerationFileHandler disable flush-minimizing optimization until its effect on watching logs in web UI can be minimized gojomo 2011-01-12 00:18:28 +00:00
• 463d295c24 * SurtPrefixedDecideRule.java buildSurtPrefixSet() - close the reader obtained from surtsSource when we're finished with it nlevitt 2011-01-11 20:46:15 +00:00
• 36515bdbf6 Part of HER-1857. Rename textSource->surtsSource, and make old surtsSourceFile a facade for surtsSource. nlevitt 2011-01-11 20:42:05 +00:00
• 64a25031e4 * engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml synch with engine/src/main/resources/org/archive/crawler/restlet/profile-crawler-beans.cxml nlevitt 2011-01-09 21:33:37 +00:00
• 4ee1e09fd3 * SurtPrefixedDecideRule.java new property textSource, an org.archive.io.ReadSource, allowing specification inline in cxml as well as in external file * profile-crawler-beans.cxml commented out clauses demonstrating use of textSource nlevitt 2011-01-09 21:10:37 +00:00
• 8b510facc5 * CrawlerLoggerModule.java setupSimpleLog() - use existing utility method for timestamp format nlevitt 2011-01-07 23:34:08 +00:00
• 14f4d01be6 * Scoper.java use new logToFile paradigm a la DecideRuleSequence nlevitt 2011-01-07 02:38:11 +00:00
• dd439dac3b Fixes and improvements to to DecideRuleSequence/logToFile. * DecideRuleSequence.java - use Spring Lifecycle start() to initialize logToFile - inject logger module as SimpleFileLoggerProvider, since we're in heritrix-modules and don't have access to CrawlerLoggerModule from heritrix-engine * CrawlerLoggerModule.java - implement SimpleFileLoggerProvider - setupSimpleLog() - use 'T' instead of '+' between date and time in timestamp * SimpleFileLoggerProvider.java new interface with one method, setupSimpleLog() * Scoper.java do not call scope.start()/scope.stop(), these are handled using Spring Lifecycle now * DecideRule.java remove unused, unneeded start()/stop() nlevitt 2011-01-07 02:09:31 +00:00
• bdd7f7e1ab Fix exceptions at launch, post-crawl-finish * KryoBinding.java always default to registrationOptional, for now * StatisticsTracker.java explicitly set valueClass of sourceDistribution to ConcurrentHashMap * StoredQueue.java answer 0 for size() when backing DB closed gojomo 2011-01-07 01:24:38 +00:00
• 93cf655bed Add logToFile support to DecideRuleSequence. File is logs/{spring-bean-id}.log, format: [timestamp] [decisive-rule-num] [decisive-rule-class] [decision] [uri] * Scoper.java start(),stop() - call scope.start()/scope.stop() * CrawlerLoggerModule.java setupSimpleLog() - new logToFile paradigm * DecideRule.java start(),stop() - noop * DecideRuleSequence.java logToFile support * profile-crawler-beans.cxml <!-- <property name="logToFile" /> --> nlevitt 2011-01-06 18:39:19 +00:00
• 641f193e17 * ExtractorXML.java shouldExtract(CrawlUri) - Avoid java.lang.IndexOutOfBoundsException checking for xml preamble in brief/empty url content. nlevitt 2011-01-03 22:41:00 +00:00
• ea49e55c89 [HER-1850] IllegalStateException from WorkQueueFrontier.wakeQueues in BlockIterator * WorkQueueFrontier.java double in-memory snoozedClassQueues size protect all adds/removes to snoozedOverflow in synchronized blocks gojomo 2010-12-09 22:28:20 +00:00
• c1b8138d4d [HER-1762] H3 performance tuning Charset lookup sync de-bottlenecking * org/apache/commons/httpclient/EncodingUtil.java use explicit Charset instances or cached to minimize lookups * LaxURLCodec, BdbMultipleWorkQueues use explicit Charset instances gojomo 2010-12-06 19:47:49 +00:00
• 7678326121 remove obsolete renamed classes gojomo 2010-12-06 19:43:16 +00:00
• 35e015698c [HER-1847] DocumentLengthT(h)resholdDecideRule spelling, naming * (Not)ExceedsDocumentLengthT(h)resholdDecideRule improve comments, rename to: ResourceLongerThanDecideRule ResourceNoLongerThanDecideRule gojomo 2010-12-06 18:39:19 +00:00
• 2546bf5a01 correct misspelling of 'Threshold' gojomo 2010-12-06 18:13:39 +00:00
• f9a59f56a0 * .classpath reference to guava library gojomo 2010-12-03 01:30:28 +00:00
• e3181ffd37 * StoredQueue.java simplify to reduce synchronization intricacy; possibly reduce contention gojomo 2010-12-03 01:29:49 +00:00
• 85e4123b3c * TopNSet refactor to narrow synchronized areas gojomo 2010-12-03 01:03:41 +00:00
• a5256f38ee * commons/pom.xml dependency on guava java libraries gojomo 2010-12-03 00:59:32 +00:00
• 0500cdc8c1 * FetchStatusDecideRule.java eliminate unused gojomo 2010-12-03 00:40:09 +00:00
• 72682be382 * org.apache.commons.httpclient.util.EncodingUtil added in prep for Chaset synchronization bottleneck workarounds gojomo 2010-11-23 02:00:38 +00:00
• 1c8f9374f6 [HER-1845] recovery process generates duplicated F+ entries in new recovery log * AbstractFrontier remove redundant logging gojomo 2010-11-23 01:25:17 +00:00
• e90eff9c84 [HER-1843] Possible int overflow in CheckpointService.java * CheckpointService Made checkpointIntervalMinutes setter accept a long instead of an int (was missed when the class variable was made a long). While it was non-harmful the way it was, this is more in line with Java conventions. kristinn_sig 2010-11-18 08:53:56 +00:00
• 675d71d557 [HER-1843] Possible int overflow in CheckpointService.java * CheckpointService Conversion from mintues to milliseconds is now done using all longs, preventing an int overflow. Removed errant semi-colon. kristinn_sig 2010-11-18 00:46:06 +00:00
• ba4b01261e [HER-1838] H3: job-not-found 404 not very clear * EngineApplication better message for simple 404 (contributed by Matt Warhaftig) gojomo 2010-11-16 23:17:37 +00:00
• 04656f56c8 More synchronization-bottleneck-minimizing * GenerationFileHandler override flush to only flush 1/100th of the time gojomo 2010-11-16 02:03:55 +00:00
• 600a4ca326 Fix selftest failures where empty 'profile' directory may not exist (eg git) * SelfTestBase ...and make a tmp empty directory instead gojomo 2010-11-16 01:01:31 +00:00
• c170390c0d Fix selftest failures where empty 'profile' directory may not exist (eg git) * SelfTestBase only copy profile directory if it exists gojomo 2010-11-16 00:58:13 +00:00
• cb7f96682b Remove synchronization on frontier methods; rely on safe queues, synchronization-per-WorkQueue instead * AbstractFrontier remove inbound/outbound queues; simplify managerThread perform findEligible/schedule/receive/finish immediately * WorkQueueFrontier eliminate holdQueues setting synchronize sendToQueue on target queue make findEligibleUri reentrant; rely on [Blocking|Stored]Queue thread-safety make arriving at front of ready trigger for 'active'/budget-session simplify inactiveQueues management simplify waking from overflow * WorkQueue redefine 'active' as in-budget-session rename 'held' as 'managed' synchronize major methods, relying on allQueues cache that operations on same intended queue use identical instance split over-budget test to isOverSessionBudget and isOverTotalBudget have toString show classKey * BdbFrontier checkpoint fixes: remember nextOrdinal, rotate frontier-recover log, reset queues on recovery consistencyCheck method for probing state during stress tests gojomo 2010-11-15 22:56:03 +00:00
• 881c829fe5 Checkpoint fixes/improvements * BdbModule store lengths in JDB manifest; limit extension of last JDB on recover * CrawlerJournal rotate frontier-recover on checkpoints like other logs gojomo 2010-11-15 22:37:58 +00:00
• c8074bdbf3 Improve ability to force-terminate crawls * RecordingInputStream.java check for interrupt on each socket-timeout * CrawlController.java on second requestCrawlStop, interrupt threads via ToePool.cleanup * ToePool.java adjust for new earlier/repeated cleanup * ToeThread.java better warnings/recovery on forced-interrupts gojomo 2010-11-15 22:26:36 +00:00
• 9fb8374b67 * ObjectIdentityBdbCache comment-out debug output gojomo 2010-11-15 21:05:29 +00:00
• f2edeea28b * StoredQueue.java added debugging info and possible fix for occasionally-seen peek-timing issue gojomo 2010-11-15 21:03:15 +00:00
• d4f5004b9c Lessen synchronization bottleneck on crawl.log * UriProcessingFormatter.java allow pre-caching of a specific LogRecord's formatted version, outside the synchronized publish() * GenerationFileHandler.java force a format before publish(), to benefit above (a small hit in other cases where it's redundant) gojomo 2010-11-15 21:01:37 +00:00
• 1501e1ddd1 * ScriptResource.java flush rather than close, avoiding console dump gojomo 2010-11-15 20:59:45 +00:00
• 6a2716ce00 Reduce synchronization bottlenecks * FrontierJournal, CrawlerJournal do barest minimum inside 'synchronized' method gojomo 2010-11-12 19:37:52 +00:00
• 6f67f9f2ba * StoredQueue improved debugging output for seldom-seen internal inconsistency gojomo 2010-11-12 19:35:35 +00:00
• f44610d79a Remove synchronization bottlenecks * HttpState remove not-strictly-necessary 'synchronized' guards gojomo 2010-11-12 19:34:47 +00:00
• ea797a99f5 remove unnecessary SuppressWarnings gojomo 2010-11-11 06:00:54 +00:00
• 301b943f03 Consolidate ModuleAttributeConstants into CoreAttributeConstants * (many) distinction between these two constant-collecting classes was fuzzy (to the point that they already referred to each other), and they were both in same subproject/package already as well; so, merged constants into the larger, older class gojomo 2010-11-09 23:47:37 +00:00
• 6452013af7 Frontier throughput experiments & fixed, continued * WorkQueueFrontier (findEligibleURI) avoid outbound.capacity-sensitive activation, which under a race created by recent changes led to infinite recursion here * AbstractFrontier (next) when nothing is immediately ready, try adding one-at-a-time to outbound, rather than fillOutbound() gojomo 2010-11-09 23:32:12 +00:00
• ad57749dea * WorkQueueFrontier.java debugging output for reported findEligibleURI deep-recursion gojomo 2010-11-08 23:45:19 +00:00
• b5eb4f3c1e * AbstractFrontier reduce size of inbound now that many threads will opportunistically clear it gojomo 2010-11-08 23:44:43 +00:00