Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_amqp_consumer_leak

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

collectd-socket-stats-writer

fix-invoker-command-fallback

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

paged-log-viewer-search

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1/merge

#10

#10/merge

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11/merge

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12/merge

#120

#121

#122

#123

#124

#125

#126

#127

#127/merge

#128

#129

#13

#13/merge

#130

#131

#132

#133

#134

#134/merge

#135

#136

#137

#138

#139

#14

#14/merge

#140

#140/merge

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15/merge

#150

#151

#152

#153

#153/merge

#154

#159

#16

#16/merge

#161

#163

#164

#165

#166

#168

#169

#17

#17/merge

#171

#174

#174/merge

#178

#179

#18

#180

#183

#183/merge

#184

#184/merge

#185

#186

#187

#188

#188/merge

#189

#19

#19/merge

#190

#192

#194

#196

#197

#2

#2/merge

#20

#20/merge

#205

#207

#209

#21

#21/merge

#210

#213

#215

#216

#217

#218

#219

#22

#22/merge

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292/merge

#294

#295

#296

#298

#3

#3/merge

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388/merge

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4/merge

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5/merge

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55/merge

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6/merge

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7/merge

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761/merge

#762

#763

#764

#765

#766

#767

#77

#770

#771

#772

#774

#775

#776

#777

#778

#779

#78

#780

#781

#781/merge

#782

#783

#784

#785

#786

#787

#788

#79

#8

#8/merge

#80

#81

#82

#83

#84

#85

#85/merge

#86

#87

#88

#89

#9

#9/merge

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98/merge

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.17.0

3.17.1

3.18.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 65f962dc25 whitespace corrections szznax 2010-03-08 18:07:30 +00:00
• 65d552ce5c fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses * Arc2Warc.java when HTTP response, write response record rather than resource enable reconstruction of ARC from WARC via namedFields * ARCRecord.java added headerString and accessor to save verbatim ARC header string szznax 2010-03-08 17:43:42 +00:00
• 3e5cf29315 nit correction in javadoc szznax 2010-03-08 17:35:47 +00:00
• 86268def97 * LongToIntConsistentHash.java Skip unnecessary conversion from CharSequence to String nlevitt 2010-02-25 18:19:46 +00:00
• e9df65d446 fix for HER-1744 Incorrect comparison of Strings in ProcessorChain.process() with "!=" operator * ProcessorChain.java use equals() to compare string values rather than objects (with "!=") szznax 2010-02-16 19:13:43 +00:00
• ed3d64d91e part of fix for HER-1736 long pause at startup reading large per-job job.log(s) * CrawlJob.java szznax 2010-02-16 19:03:59 +00:00
• acfe95fea3 naming/branching/visibility review for HER-1736 long pause at startup reading large per-job job.log(s) * CrawlJob.java added accessor, renamed a bit, unified 2 branches * JobResource.java use accessor gojomo 2010-02-13 00:47:41 +00:00
• e1308e80f5 reverting last checkin, also testing passwordless checkin with Karmic Eclipse 3.5.1 SVNKit 1.6.7 update szznax 2010-02-13 00:21:22 +00:00
• 6c13f42242 testing Karmic Eclipse 3.5.1 (w/updated SVNKit 1.6.7) szznax 2010-02-13 00:18:10 +00:00
• 2d441d611e part of fix for HER-1736 long pause at startup reading large per-job job.log(s) * CrawlJob.java added boolean partialJobLogScan, and print "at least X launches" when true * JobResource.java print "at least X launches" when partialJobLogScan szznax 2010-02-12 23:21:59 +00:00
• 18d459f6b9 part of fix for HER-1736 long pause at startup reading large per-job job.log(s) * CrawlJob.java check for excessively long job log, scan near end of file for lastLaunch inform user OK to move big job log aside szznax 2010-02-12 18:54:57 +00:00
• 4097e29d50 part of fix for HER-1736 long pause at startup reading large per-job job.log(s) * CrawlJob.java return early from scanJobLog() when job log > 1GB szznax 2010-02-09 19:45:23 +00:00
• a53d9ea733 per Gordon's suggestions; renamed "Engine.css" to "engine.css" and changed flashACK class background-color to palegreen szznax 2010-01-26 20:54:16 +00:00
• 0b5e5b30e3 removed references to unused repos, and updated servlet-api to use newer jetty referenced repo szznax 2010-01-26 20:31:36 +00:00
• 1a95e530e8 Clean-up where assumption of serializability no longer appropriate gojomo 2010-01-26 01:25:47 +00:00
• 248f720cab Make Recorder facility usable independent of special crawler threads * Recorder.java store reusable per-thread Recorder instance in Recorder static thread-local * ToeThread.java setup reusable instance into Recorder * RecorderMarker.java deleted -- unnecessary gojomo 2010-01-26 01:21:59 +00:00
• f35da10154 * EnhDirectoryResource.java improve error for POST to file when EditRepresentation-wrapping (via '?format=textedit') hasn't occurred gojomo 2010-01-26 00:18:23 +00:00
• 760c40c440 * EngineResource.java use platform-specific path-separator in partial-path-prompt gojomo 2010-01-26 00:16:52 +00:00
• 3dff26e277 attempt to remedy Continuum build problem with SVN. "Server certificate verification failed: issuer is not trusted (https://archive-crawler.svn.sourceforge.net)" szznax 2010-01-12 16:45:20 +00:00
• 5abacafd43 fix for [HER-1729] add (deprecated) style code from Engine class to Flash class * Heritrix.java added CLAP protocol to access representations via classloader * EngineApplication.java added staticDir resource to serve static files under "/engine/static" also added org.archive.crawler.restlet package to hold static files * EngineResource.java added <link> to static CSS resource * Engine.css style for flashACK/NACK/ADVISORY divs szznax 2009-12-29 23:21:10 +00:00
• b9ab0b296d [HER-1731] Unable to set HTTPProxyPort on org.archive.modules.fetcher.FetchHttp * FetchHTTP.java make getter/setter types agree gojomo 2009-12-28 01:56:32 +00:00
• 0cb627fff8 tried to incorporate gojomo's suggestions of 2009-11-02: * Engine.java removed ERROR and WARNING strings, rely on Flash class log jobsDir check failure as SEVERE and return early collapse error-handling in considerAsJobPath() and log SEVERE renamed considerAsJobDirectory() to addJobDirectory() remove catch-and-rethrow from leaveJobPathFile() when writing, instead log SEVERE renamed leaveJobPathFile() to writeJobPathFile() added boolean "userRequest" arg to addJobDirectory(), and moved call to writeJobPathFile() inside of addJobDirectory() changed void considerAsJobPath(File) to String getJobPathFromFile and removed call to addJobDirectory make seperate calls to addJobDirectory() in findJobConfigs() * EngineResource.java restructured "create" logic for clarity eliminate style code, rely on Flash class remove limit on "addpath" input, change to "createpath" removed all-caps and bangs ("!") from error messages removed platform-specific path separators ("/") call Engine.addJobDirectory() with userRequest=true in when form action=add szznax 2009-12-17 01:54:21 +00:00
• bf2eaef6e2 Part of fix for [HER-1725] on engine page, 'create new job' form gives NPE * Engine.java made protected getProfileCxmlResource() for clear testability * EngineTest.java added testGetProfileCxmlResource() szznax 2009-12-11 00:47:06 +00:00
• 56d6e2ee26 Change XmlMarshaller to static util method api, and add javadocs to XmlMarshaller. * XmlMarshaller.java change to static api, and add javadocs to XmlMarshaller * *Resource.java use changed XmlMarshaller api nlevitt 2009-12-10 07:06:17 +00:00
• 7cf44fdf4d * JobResource.java Remove try/catch from acceptRepresentation() - comment didn't match code and I don't think I intended to commit it (oops). nlevitt 2009-12-10 06:08:15 +00:00
• b0e140708a Rename and document methods formerly known as presentablify*() * BeanBrowseResource.java, ScriptResource.java, EngineResource.java, JobResource.java rename presentablify() -> makePresentableMap(), and add javadocs * JobRelatedResource.java rename presentablifyNestedNames() -> addPresentableNestedNames() and presentablifyObject() -> makePresentableMapFor(), and add javadocs nlevitt 2009-12-10 03:38:25 +00:00
• 0616ddfb07 part of fix for [HER-1725] create new job gives NPE * relocated (non-dist) profile-crawler-beans.cxml from engine/src/main/java to engine/src/main/resource szznax 2009-12-10 00:05:51 +00:00
• 9f12f5777a * ToePool.java shortReportLineTo(): get data from shortReportMap() to avoid duplicate code nlevitt 2009-12-09 03:12:48 +00:00
• df6e31697d Rename short report methods: - singleLineReport() -> shortReportLine() - singleLineReportTo() -> shortReportLineTo() - singleLineReportData() -> shortReportMap() nlevitt 2009-12-09 02:53:02 +00:00
• 3b75b28168 removed extraneous reference to jsp-2.0. dbernstein 2009-12-07 23:51:14 +00:00
• f8cc922cb3 [HER-1724] revive missing newline at end of crawl-report.txt nlevitt 2009-12-07 22:54:03 +00:00
• cb7df416d4 post-3.0.0 release build * **/pom.xml bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release) gojomo 2009-12-05 23:48:38 +00:00
• 5661658b1b prep for 3.0.0 release * **/pom.xml bump version-id to "3.0.0" * README.txt refer to exactly 3.0.0 release notes 3.0.0 gojomo 2009-12-05 09:28:19 +00:00
• 7d0061884c Prevent timing NPEs on CrawlLimitEnforcer-triggered crawl end * AbstractFrontier.java close recovery log in stop() (comes later, usu. from managerThread) rather than terminate() (earlier, may occur in other threads like progress-stats-snapshot thread) gojomo 2009-12-05 00:55:55 +00:00
• 21d4bce819 correct typo in built-in profile templates * org/archive/crawler/restlet/profile-crawler-beans.cxml, org/archive/crawler/migrate/migrate-template-crawler-beans.cxml Percentage -> Percent gojomo 2009-12-05 00:16:55 +00:00
• ac255d0ebe Prep for 3.0.0 * README.txt updated, pointers to wiki references gojomo 2009-12-05 00:12:14 +00:00
• 4da5095212 Keep html spartan for now. * EditRepresentation.java write() remove background color and font size styling on textarea nlevitt 2009-12-04 23:30:24 +00:00
• e85972848d Format, indent xml. See discussion on [HER-1603]. * XmlMarshaller.java (constructor) enable xml formatting/indentation nlevitt 2009-12-04 23:16:35 +00:00
• 29e8b05308 Improvement to [HER-1603?focusedCommentId=22557] fix * BaseResource.java getPreferredVariant(): always prefer HTML if anything in the header indicates it's acceptable (even "*/*" with low q-preference) nlevitt 2009-12-04 23:09:24 +00:00
• c3539d3fdc Fix for [HER-1603?focusedCommentId=22557] browsers based on WebKit (Safari, iPhone Safari, Chrome) use a screwy 'Accept' header that if taken literally means they prefer 'application/xml' content. Thus trying to view web UI in these browsers results in seeing the XML version, without element markup * BaseResource.java override getPreferredVariant() to bump up preference level of text/html for any client can accept it * EngineResource.java, JobResource.java inherit from BaseResource nlevitt 2009-12-04 22:56:25 +00:00
• 04012279fc Misc logging-cleanup/refactoring * CrawlerJournal.java improve exception logging combine writeLine() methods to use varargs * FrontierJournal.java improve exception logging * ArchiveUtils.java replace ignored count parameter with meaingful boolean one * CrawlURI.java use updated ArchiveUtils method gojomo 2009-12-04 22:23:48 +00:00
• d423af6838 changed cachePercentage to cachePercent szznax 2009-12-04 22:09:49 +00:00
• 4bb5dca139 * pom.xml Configure custom @contributor javadoc tag. nlevitt 2009-12-04 06:00:36 +00:00
• 67a25f2dce * CheckpointService.java getAvailableCheckpointDirectories() - avoid NullPointerException when checkpoints dir does not exist nlevitt 2009-12-04 00:52:55 +00:00
• a49159a2cc Use specific includes to define contents of source distribution, to avoid packaging whatever junk might be lying around your source checkout. * dist/src/main/assembly/src.xml nlevitt 2009-12-03 22:42:12 +00:00
• e1bbbf5a59 [HER-1704] WorkQueueFrontier.snoozedClassQueues not bounded in size, not reported in useful order * WorkQueueFrontier.java add snoozedOverflow map of (wakeTime)->(DelayedWorkQueue) for snoozes over a capped size maintain in parallel to snoozedClassQueues DelayQueue, maintain atomic size count in standard report, presort snoozedClassQueues before displaying * DelayedWorkQueue.java move to top-level so serialization doesn't pull out whole frontier * BdbFrontier.java initialize snoozedOverflow * BdbMultipleWorkQueues.java improve error message seen on problematic checkpoint-recovery gojomo 2009-12-03 19:37:27 +00:00
• 095f2fdffb Make Maven build more robust * engine/pom.xml, .classpath remove apparently-no-longer-needed tomcat library reference, avoiding Maven problems with m2eclipse 0.9.8.200905041414 in eclipse-galileo on ubuntu gojomo 2009-12-03 19:02:13 +00:00
• 950e68091c UI touchup * EngineResource.java give add-job options their own header, reorder/reword for clarity * JobResource.java return to always showing 'checkpoint' button, disable if necessary, so main control buttons aren't appearing/disappearing or moving gojomo 2009-12-03 18:28:21 +00:00
• 757dbf3f19 Output/UI touchup * CheckpointService.java list checkpoint directories in reverse-last-modified order disable dubugging output to System.err gojomo 2009-12-03 18:24:17 +00:00
• dd4a8d4705 [HER-1718] misleading "writer subdirectory" shown in webui * WriterPoolProcessor.java change description of WriterPoolProcessor.directory ConfigPath to not imply as strongly that is where writing much be occurring gojomo 2009-12-03 08:41:08 +00:00
• f3f7a98240 [HER-1708] h3 - crawl summary report byte totals off somehow * CrawlSummaryReport.java relabel, reorder, regroup displayed values for simplicity/consistency gojomo 2009-12-03 03:12:38 +00:00
• 19b9833fb4 Fix (harmless?) stack trace from closing output too early. It looked like this: ========== 2009-12-03 01:36:16.726 SEVERE thread-10 com.noelios.restlet.http.HttpServerConverter.commit() An exception occured writing the response entity java.io.IOException: Stream closed at sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:26) [...] 2009-12-03 01:36:16.733::WARN: Committed before 500 An exception occured writing the response entity 2009-12-03 01:36:16.735 WARNING thread-10 com.noelios.restlet.http.HttpServerHelper.handle() Error while handling an HTTP server call: ========== * JobResource.java writeHtml(Writer) - do not close writer nlevitt 2009-12-03 01:40:42 +00:00
• 6edbc5181e [HER-1721] pressing build immediately after adding or creating a new job gives IOE * JobResource.java presentablify(), writeHtml() - make sure crawl log exists before attempting to read and serve it nlevitt 2009-12-03 01:26:26 +00:00
• e15f49dd60 [HER-1708] h3 - crawl summary report byte totals off somehow * CrawlController.java send event of crawl-end before appCtx.stop() * StatisticsTracker.java force final progressStatisticsEvent and dumpReports at crawlEnded() before stop() add force-dump of reports (so dumpReports never reuses prior reports) * CrawledBytesHistotable.java add getTotalBytes(), a total of just the byte-counts (getTotal() includes URI counts) * CrawlStatSnapshot.java use getTotalBytes() gojomo 2009-12-02 06:25:23 +00:00
• e3fc474ca4 Return http 404 when appropriate. * BeanBrowseResource.java * JobResource.java nlevitt 2009-12-02 03:24:11 +00:00
• ed2061560e Improve Maven build robustness * UURIFactoryTest.java specify stirng literal with escape-encoding, rather than as raw UTF-8 * **/pom.xml include 'project.build.sourceEncoding' property to suppress platform-encoding warnings (unfortunately, neither this nor similar 'maven.compile.encoding' property actually casues maven build to interpret Java source as UTF-8) gojomo 2009-12-02 01:34:40 +00:00
• aa3bfc7f4c Remove unused jasper dependency. * .classpath * dist/pom.xml nlevitt 2009-11-26 01:43:15 +00:00
• 75fa939995 Update restlet to latest 1.1.6 * .classpath * engine/pom.xml nlevitt 2009-11-25 01:49:17 +00:00
• 8ba641ed4b Part of [HER-1603] Restletify: xml representations of all resources that also offer html * XmlMarshaller.java new class to write xml based on nested map structure * BeanBrowseResource.java ScriptResource.java EngineResource.java JobResource.java JobRelatedResource.java support xml representation * WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java ProcessorChain.java CrawlURI.java Reporter.java offer same data that is returned in the single line reports in a map suitable for use with XmlMarshaller * FileUtils.java new method tryToCanonicalize(File) nlevitt 2009-11-25 01:43:57 +00:00
• a1022c8246 [HER-1711] Some Internationalized Domain Name seeds ignored by Heritrix * EnhDirectoryResource.java ensure files shown for edit or written-back to disk handled as UTF-8 * ConfigFile.java read file as UTF-8 gojomo 2009-11-25 01:14:59 +00:00
• 79fd64a2b5 Post-RC1-build * **/pom.xml revert version name to SNAPSHOT gojomo 2009-11-23 10:23:28 +00:00
• 0c7a74bd35 RC1 * **/pom.xml rename version to '3.0.0-RC1' for distinguished 'RC1' identified build (instead of SNAPSHOT/timestamp) gojomo 2009-11-23 09:57:42 +00:00
• 5d843d46c1 * CachedBdbMap.java remove unnecessary/build-breaking ServerCache import gojomo 2009-11-23 09:36:56 +00:00
• faa8e899db * CachedBdbMap.java remove unnecessary/build-breaking CrawlController import gojomo 2009-11-23 09:26:08 +00:00
• 18e165f21e [HER-1714] fixed-interval rescheduling * AbstractFrontier.java, WorkQueueFrontier.java, BdbFrontier.java add 'futureUris' map of CrawlURIs to be reenqueued at specific future dates the usual findEligibleUri also checks for newly-eligible rescheduled URIs the frontier is no longer empty unless the future count is also 0 * CrawlURI.java add rescheduleTime property * ReschedulingProcessor.java optional post-processor to set a CrawlURI's reschedulingTime * CrawlStatSnapshot.java, CrawlJob.java add futureUriCount to stats/uriTotalsReport gojomo 2009-11-23 08:47:14 +00:00
• 3c9df9a8c6 BDBJE updates, refactoring * BdbFrontier.java use BdbModule.openDatabase() gojomo 2009-11-23 08:24:54 +00:00
• 5b6b2fe954 BDBJE updates, refactoring * BdbModule.java (openDatabase) all databases now 'managed' (getStoredMap) all reuse of prior data as for checkpoints * (others) match above changes gojomo 2009-11-23 08:18:41 +00:00
• 4127286a1c Checkpoint cleanup * Checkpoint.java calculate shortName correctly * CrawlerLoggerModule.java use full checkpoint-name for log-rotation-suffix gojomo 2009-11-23 08:10:07 +00:00
• d93129115e BDBJE updates, refactoring * CachedBdbMap.java update for BDB-JE 4.0.71 exception changes gojomo 2009-11-23 08:03:30 +00:00
• eb9bf5a8e2 BDBJE updates, refactoring * commons/pom.xml, .classpath update to BDB-JE 4.0.71 * BdbModule.java (getStoredQueue) added (getStoredMap) made more general (openDatabase) all databases now 'managed' * TempStoredSortedMap.java -> DisposableStoredSortedMap.java renamed * StatisticsTracker.java, **Report.java use new name * StoredQueue.java, StoredQueueTest.java move to org.archive.bdb package * BdbMultipleWorkQueues.java, PrefixFinderTest.java update for BDB-JE 4 changes * BdbFrontier.java use getStoredQueue drop unneeded custom serialization support * WorkQueueFrontier.java drop unneeded custom serialization support gojomo 2009-11-23 04:41:32 +00:00
• 8251620481 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * profile-crawler-beans.cxml*2, migrate-template-crawler-beans.cxml add CheckpointService as recommended bean gojomo 2009-11-20 18:35:13 +00:00
• f9d56e3f6b move json dependency to commons from engine gojomo 2009-11-20 02:03:27 +00:00
• 25a568fff7 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * JobResource.java suppress checkpoint-selector after terminate * Checkpoint.java adapt to work without direct reference to CheckpointService * CheckpointService.java configure Checkpoint without making it dependent on CheckpointService gojomo 2009-11-20 01:24:57 +00:00
• daab317c94 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls processor-related bean checkpoint support: * Checkpoint.java move to commons for proper dependency graph * (others) recognize above gojomo 2009-11-20 01:10:32 +00:00
• dfacba51e5 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls UI support * JobResource.java enable 'checkpoint' button as appropriate display active recovery-checkpoint or select bos of recovery-options when available * CrawlJob.java (getCheckpointService) accessor gojomo 2009-11-20 01:02:14 +00:00
• 50ee78f6d2 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * BdbFrontier.java implement Checkpointable; save state via JSON or BDB; restore queues-of-queues on recovery * WorkQueueFrontier.java (deactivateQueue) (retireQueue) protected * BdbUriUniqFilter.java implement Checkpointable; save state via JSON or BDB; restore * WorkQueue.java make peekItem transient -- should not be held through checkpoint make retired protected * Frontier.java, AbstractFrontier.java remove/update obsolete code gojomo 2009-11-20 00:59:00 +00:00
• be17d5ed33 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * CrawlController.java remove older checkpoint-functionality passthrough implement Checkpointable so behavior can vary during recovery only trigger annnounce of seeds on 'cold' (non-recovery) start gojomo 2009-11-20 00:51:59 +00:00
• 6481081697 * DefaultWriterPoolSettings.java remove obsolete serialization/checkpoint support gojomo 2009-11-20 00:50:17 +00:00
• ac6b58cb0e [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * Checkpoint.java (generateFrom) version more suitable for unit testing gojomo 2009-11-20 00:49:31 +00:00
• fbd3af35ac [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * PersistLogProcessor.java rotate logs on checkpoint * CrawlerJournal.java, FrontierJournal.java rotate for checkpoint support without full Checkpointable gojomo 2009-11-20 00:47:54 +00:00
• 8591002573 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls * BdbModule.java implement Checkpointable; save/restore consistent sets of jdb log files * BdbModuleTest.java testDoCheckpoint gojomo 2009-11-20 00:45:07 +00:00
• f01ad9ada8 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls stats & logging checkpoint support * StatisticsTracker.java implement Checkpointable; save small state in JSON, recycle BDB data when recovering * CrawlerLoggerModule.java implement Checkpointable; rotate logs on checkpoint * GenerationFileHandler.java avoid clobbering leftover logs from earlier futures via moveAside * CrawlStatSnapshot.java (sameProgressAs) test for lack-of-progress * ObjectIdentityMemCache.java, TopNSet.java offer deeper access to assist checkpoint/resume gojomo 2009-11-19 23:18:15 +00:00
• b63c2bdc77 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls processor-related bean checkpoint support: * ToeThread.java use getBeanName gojomo 2009-11-19 23:12:47 +00:00
• 124d1524d9 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls processor-related bean checkpoint support: * Processor.java implement Checkpointable and BeanNameAware; methods to support JSON state-saving * DispositionChain.java implement Checkpointable and during-checkpoint disposition-lockout for state stability gojomo 2009-11-19 23:10:13 +00:00
• 2bb65e9200 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls processor-related bean checkpoint support: * Processor.java implement Checkpointable and BeanNameAware; methods to support JSON state-saving * DispositionChain.java implement Checkpointable and during-checkpoint disposition-lockout for state stability * **Extractor**.java pull common reporting & checkpointing up to Extractor.java * WriterPoolProcessor.java implement Checkpointable; freeze and roll writer-pool during checkpoint * PersistLogProcessor.java roll log on checkpoint * TextSeedModule.java, SeedModule.java eliminate obsolete (and probably unnecessary) checkpoint support * (others) adapt to above changes gojomo 2009-11-19 22:52:39 +00:00
• bdf4437ca2 [HER-1546] Springify(5): Update checkpointing to work smoothly with spring-configured crawls core checkpointing system classes: * Checkpointable.java basic interface for checkpoint aware/capable beans * Checkpoint.java one Checkpoint, either in progress or recovery -- a name + storage directory * CheckpointService.java helper bean to enable, trigger checkpoints and recovery * Checkpointer.java(*2), CheckpointInputStream.java, CheckpointRecovery.java DefaultCheckpointRecovery.java, RecoverAction.java (deleted) no longer used gojomo 2009-11-19 22:45:07 +00:00
• e42e2ed7f6 Cleanup/consolidation/expansion of *Utils classes * IoUtils.java remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils * ArchiveUtils.java, FileUtils.java receive relocated methods; eliminated deprecated or unused methods * JSONUtils.java new class to collect common JSON actions * Iteratorable.java wrap Iterator as Iterable for foreach usage * (many) update to use alternate utils methods gojomo 2009-11-19 22:39:53 +00:00
• 300ce15a06 * EditRepresentation.java add filename to TITLE to help identify browser tabs gojomo 2009-11-19 21:58:12 +00:00
• b97319fb07 reformat gojomo 2009-11-19 21:57:05 +00:00
• 2cb91e33f5 Fix or suppress warnings gojomo 2009-11-19 21:55:43 +00:00
• f9e1191144 Fix build - cleanup leak of incomplete code to prior commit gojomo 2009-11-18 20:53:41 +00:00
• be81d3733a [HER-1710] on problem loading persist log, close reader nlevitt 2009-11-18 03:02:28 +00:00
• 56055d8236 [HER-1709] h3 - dumpPendingAtClose doesn't work * CrawlController.java avoid unnecessarily early/redundant close of logs files (and checkpointer cleanup) * WorkQueueFrontier.java add stop() that does close() * BdbMultipleWorkQueues.java add cursor.close() * AbstractFrontier.java, BdbFrontier.java use closeQuietly gojomo 2009-11-18 00:29:37 +00:00
• 148c7332c4 Merge ServerCache+ServerCacheUtil abstract superclass ServerCache * ServerCache.java make abstract, implement getHostFor(UURI) and getServerFor(UURI) * DefaultServerCache.java extend ServerCache, remove getHostFor(UURI) and getServerFor(UURI) * ServerCacheUtil.java removed * other classes change usage of ServerCacheUtil to ServerCache nlevitt 2009-11-17 06:10:42 +00:00
• e57ad68230 * AbstractFrontier.java implement ApplicationListener like it seems to want to, being that it has a method onApplicationEvent() nlevitt 2009-11-17 04:53:11 +00:00
• f9843974bc Update commons-net to 2.0 to incorporate fix to this bug - http://issues.apache.org/jira/browse/NET-36 - we saw the problem manifest in an archive-it crawl. * ClientFTP.java override buggy org.apache.commons.net.ftp.FTP.getReplyStrings() with fixed version, see https://issues.apache.org/jira/browse/NET-257 * .classpath, commons/pom.xml update dependency to 2.0 jar nlevitt 2009-11-17 02:49:57 +00:00
• 92f3bfe0ec check for exactly 2 args, tightened up SAXException handling by making sourceOrderXmlDom a protected field, cleaned up unused import and whitespace. szznax 2009-11-16 23:45:04 +00:00
• 9059395617 Make Bloom size configurable without system properties * BloomUriUniqFilter.java make bloom size configurable via bean properties change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps) * BloomUriUniqFilterTest.java update test for property-set sizing * BloomFilter64bit.java improve comment gojomo 2009-11-16 22:42:18 +00:00
• 69bedb8240 a bit more error-handling for input args szznax 2009-11-16 19:31:30 +00:00
• 24463da8bc Fix for [HER-1689] build, launch buttons remain enabled after problem building * CrawlJob.java renamed isContainerOk() to hasApplicationContext() increased granularity of isLaunchable() to offer launch button * JobResource.java more coarse check for hasApplicationContext() to disable build button szznax 2009-11-13 23:23:52 +00:00