Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#750

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 14b8649aa9 minor cleanup/clarification on review szznax 2010-04-26 21:49:41 +00:00
• 27cca4da60 Refinement of HER-1737 excessive job.log WARNINGs for misreported encoding (FLV reported as text/html;charset=UTF-8 * ReplayCharSequence.java, GenericReplayCharSequence.java, InMemoryReplayCharSequence.java store actual first exception for reference * ExtractorHTML.java report exception in nonfatal-log, rather than WARNING (which was still polluting job log more than desired -- but should still be logged to allow future investigation of sites where encoding issues may exist) gojomo 2010-04-24 02:22:24 +00:00
• 9fc5e42cfe Fix failing test * InMemoryReplayCharSequence.java ensure redecode after problem restarts at right position gojomo 2010-04-23 22:57:50 +00:00
• 792d8e528b HER-1766 The use of the MatchesFilePatternDecideRule with the CUSTOM preset causes a fatal error (infinite loop) * MatchesFilePatternDecideRule.java fix getRegex() to super.getRegex() gojomo 2010-04-23 22:16:25 +00:00
• 2855572278 ? gojomo 2010-04-23 04:20:12 +00:00
• a7caceb8ad HER-1737 excessive job.log WARNINGs for misreported encoding (FLV reported as text/html;charset=UTF-8 * ReplayCharSequence.java new interface method: getDecoderExceptionCount(), reporting >0 if any decoding problems were encountered and silently replaced * GenericReplayCharSequence.java count decoding problems, and only log as FINE rather than WARNING * InMemoryReplayCharSequence.java try decoding with reporting; if any problems note at least one then decode without reporting errors * ExtractorHTML.java after using ReplayCharSequence, log WARNING if nonzero decoding errors gojomo 2010-04-23 04:11:53 +00:00
• 67bf73026f remove obsolete/moved example properties gojomo 2010-04-23 04:05:31 +00:00
• cb5ec19b6c HER-1765 quiet excessive INFO level logging * GenericReplayCharSequence.java * Scoper.java * FetchFTP.java * FetchHTTP.java demote a bunch of INFO logging to FINE so that be default, console isn't filled with output for every URI handled by a busy crawl gojomo 2010-04-23 00:44:25 +00:00
• 3f0059c8a8 * BeanBrowseResource.java remove extraneous close() that seemed to be causing (harmless but distracting) console error stacks after web hits gojomo 2010-04-23 00:41:25 +00:00
• 9fd648f3ef HER-1467 Allow periodic checkpointing and checkpoint-copy-logs to be set/unset during crawl * CheckpointService.java reorganize so that checkpoint timedTask is reset whenever interval is reset on a running crawl * CheckpointSuccessEvent.java, CrawlJob.java report successes as new kind of ApplicationEvent; propagagte to job log * BdbModule.java remove obsolete unconsulted checkpointCopyLogs setting gojomo 2010-04-23 00:15:44 +00:00
• ca32d0d448 HER-1692 Cannot create profile with same name as job * JobResource.java if destination of copy-job fails, show flash error gojomo 2010-04-22 01:59:41 +00:00
• c1207582c4 HER-1728 Heritrix could be enhanced to parse flashvars parameter to correctly extract the links * ExtractorHTMLTest.java tests for embed/@flshvars, param[@name=flashvars]/@value likely-URI discovery * ExtractorHTML.java move likely-URI testing to UriUtils add FLASHVARS support * JerichoExtractorHTML.java use UriUtils ikely-URI testing add FLASHVARS support gojomo 2010-04-22 01:27:24 +00:00
• f933dc922d HER-1763 unify/improve/document likely-URI string tests * UriUtils.java move likely-URI heuristics here for better consistency, reuse, testing, improvement * UriUtilsTest.java testing for UriUtils * ExtractorJS.java move likely-URI support to UriUtils; use new test * ExtractoSWF.java use new UriUtils likely-URI testing * ExtractorHTML.java move likely-URI testing to UriUtils, use new test * JerichoExtractorHTML.java use new UriUtils likely-URI testing gojomo 2010-04-22 01:08:14 +00:00
• 72fb1c55bd Move TLD list to resource file for easier drop-in updating * ArchiveUtils.java read TLDs from tlds-alpha-by-domain.txt * ArchiveUtilsTest.java verify TLDs read * tlds-alpha-by-domain.txt from http://data.iana.org/TLD/tlds-alpha-by-domain.txt gojomo 2010-04-22 01:05:23 +00:00
• f018a2e6f1 * StatisticsTracker.java correct outdated comment gojomo 2010-04-13 22:47:19 +00:00
• affa3b672c HER-1620 Tolerate trailing '*' in robots disallow (& allow?) rules * Robotstxt.java trim single trailing '*' if present (assume unintentional or redundant) gojomo 2010-04-13 22:46:23 +00:00
• 9e12e4abe5 test/deprecation cleanup * CachedBdbMap.java @deprecated * BdbModule.java note (unused) 'CBM' option is deprecated * ObjectIdentityBdbCache.java tighten logging to FINE * ObjectIdentityBdbCacheTest.java fix test that was only trying to log, rather than fail, problems gojomo 2010-04-13 02:15:53 +00:00
• a82838cd27 HER-1756 improved crawl status reporting: more definitive start-after-setup, conclusion-after-logging of crawling * AbstractFrontier.java call new finalTasks just before announcing FINISH state * WorkQueueFrontier.java eliminate closeQueue * BdbFrontier.java move dumpAllPendingToLog from closeQueue (triggered by CrawlController FINISHED) to finalTasks (before Frontier FINISH) replace closeQueue with close() override gojomo 2010-04-13 02:10:56 +00:00
• 8be7be2983 fix race causing intermittent test failures on MacOS * CrawlJob.java synchronize methods which mutate or change behavior based on field 'ac'; previously, some self-tests were exitting/failing immediately because they were not seeing a non-null 'ac' value. gojomo 2010-04-13 00:55:09 +00:00
• 1698412130 HER-1450 Reading at EOF using read(byte[]) causes position pointer to decrement. * ARCWriterTest.java one more test to ensure the problem with read(void) was exercised/fixed gojomo 2010-04-09 21:25:26 +00:00
• cc6fb841d9 part of fix for [HER-1450] Reading at EOF using read(byte[]) causes position pointer to decrement. * ArchiveRecord.java do not decrement position after EOR, port from H1 * ARCWriterTest.java added testArchiveRecordEORConsistent(), testArchiveRecordMarkSupport() port from H1 szznax 2010-04-09 18:25:16 +00:00
• 647a0b3edf HER-1754 treat robots.txt non-response as 404 (optionally?) * CrawlServer.java enable a specific kind of connect-lost (no response given) to be counted as 'deemed 404' on robots retries over a minimum * FetchStatusCodes.java add a synthetic code: -404, S_DEEMED_NOT_FOUND * FetchHTTP.java better distinguish CONNECT_FAILED and CONNECT_LOST * DispositionProcessor.java treat CONNECT_LOST same as _FAILED; simplify a test gojomo 2010-04-02 03:55:12 +00:00
• 693f050fb7 successfully adding external job dir is ACK flash kind szznax 2010-03-29 15:56:09 +00:00
• fd5644f109 HER-1627 followup: show error rather than match faulReadSize * PersistProcessor.java rename setupEnvironment to setupCopyEnvironment to make limited purpose of env clear; catch and rethrow IllegalArgumentException with hint that second-open may be cause * PersistLoadProcessor.java catch and display IllegalArgumentException same as other failure-to-open SEVERE but proceedable exceptions * PrecedenceLoader.java update method names gojomo 2010-03-26 01:02:48 +00:00
• 37c32b4e5c Maven cleanup to enable warningless build on fresh machine with Maven2.2.0: * engine/pom.xml restore a reference to a (new version of a) necessary servlet library * pom.xml remove deprecated {pom.artifactId}; add needed javadoc module version decl * .classpath update ref to currently-used servlet jar gojomo 2010-03-23 22:57:27 +00:00
• 9f77412581 part of fix for HER-1627 incorrect source tag in crawl.log (of persist-loaded crawl) * EnhancedEnvironment.java in getClassCatalog(), when read-only env, insist on read-only catalog * PersistLoadProcessor.java remove existing curi members from prior to prevent clobbering source * PersistProcessor.java set historyDbConfig read-only, set je.log.faultReadSize for compatability with BDBModule.setup() szznax 2010-03-23 20:52:40 +00:00
• 20b8a7bc90 mention url in question in debug level logging nlevitt 2010-03-19 18:39:14 +00:00
• af85fed7f4 part of fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses * Arc2Warc.java reverted writing ARC namedFields, deferred for further analysis as part of HER-1750 szznax 2010-03-12 23:44:07 +00:00
• cc8ea6494a changed namedField "ARC-Compressed-Offset" to "ARC-Gzip-Offset" for even more clarity szznax 2010-03-11 15:59:10 +00:00
• 4802e5093e part of fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses * Arc2Warc.java changed namedField "ARC-Offset" to "ARC-Compressed-Offset" for clarity szznax 2010-03-11 01:45:49 +00:00
• 8a567c9bc0 part of fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses * Arc2Warc.java removed unnecessary URISyntaxException import moved ARC named fields outside of check for IP in header field szznax 2010-03-10 00:33:38 +00:00
• 65f962dc25 whitespace corrections szznax 2010-03-08 18:07:30 +00:00
• 65d552ce5c fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses * Arc2Warc.java when HTTP response, write response record rather than resource enable reconstruction of ARC from WARC via namedFields * ARCRecord.java added headerString and accessor to save verbatim ARC header string szznax 2010-03-08 17:43:42 +00:00
• 3e5cf29315 nit correction in javadoc szznax 2010-03-08 17:35:47 +00:00
• 86268def97 * LongToIntConsistentHash.java Skip unnecessary conversion from CharSequence to String nlevitt 2010-02-25 18:19:46 +00:00
• e9df65d446 fix for HER-1744 Incorrect comparison of Strings in ProcessorChain.process() with "!=" operator * ProcessorChain.java use equals() to compare string values rather than objects (with "!=") szznax 2010-02-16 19:13:43 +00:00
• ed3d64d91e part of fix for HER-1736 long pause at startup reading large per-job job.log(s) * CrawlJob.java szznax 2010-02-16 19:03:59 +00:00
• acfe95fea3 naming/branching/visibility review for HER-1736 long pause at startup reading large per-job job.log(s) * CrawlJob.java added accessor, renamed a bit, unified 2 branches * JobResource.java use accessor gojomo 2010-02-13 00:47:41 +00:00
• e1308e80f5 reverting last checkin, also testing passwordless checkin with Karmic Eclipse 3.5.1 SVNKit 1.6.7 update szznax 2010-02-13 00:21:22 +00:00
• 6c13f42242 testing Karmic Eclipse 3.5.1 (w/updated SVNKit 1.6.7) szznax 2010-02-13 00:18:10 +00:00
• 2d441d611e part of fix for HER-1736 long pause at startup reading large per-job job.log(s) * CrawlJob.java added boolean partialJobLogScan, and print "at least X launches" when true * JobResource.java print "at least X launches" when partialJobLogScan szznax 2010-02-12 23:21:59 +00:00
• 18d459f6b9 part of fix for HER-1736 long pause at startup reading large per-job job.log(s) * CrawlJob.java check for excessively long job log, scan near end of file for lastLaunch inform user OK to move big job log aside szznax 2010-02-12 18:54:57 +00:00
• 4097e29d50 part of fix for HER-1736 long pause at startup reading large per-job job.log(s) * CrawlJob.java return early from scanJobLog() when job log > 1GB szznax 2010-02-09 19:45:23 +00:00
• a53d9ea733 per Gordon's suggestions; renamed "Engine.css" to "engine.css" and changed flashACK class background-color to palegreen szznax 2010-01-26 20:54:16 +00:00
• 0b5e5b30e3 removed references to unused repos, and updated servlet-api to use newer jetty referenced repo szznax 2010-01-26 20:31:36 +00:00
• 1a95e530e8 Clean-up where assumption of serializability no longer appropriate gojomo 2010-01-26 01:25:47 +00:00
• 248f720cab Make Recorder facility usable independent of special crawler threads * Recorder.java store reusable per-thread Recorder instance in Recorder static thread-local * ToeThread.java setup reusable instance into Recorder * RecorderMarker.java deleted -- unnecessary gojomo 2010-01-26 01:21:59 +00:00
• f35da10154 * EnhDirectoryResource.java improve error for POST to file when EditRepresentation-wrapping (via '?format=textedit') hasn't occurred gojomo 2010-01-26 00:18:23 +00:00
• 760c40c440 * EngineResource.java use platform-specific path-separator in partial-path-prompt gojomo 2010-01-26 00:16:52 +00:00
• 3dff26e277 attempt to remedy Continuum build problem with SVN. "Server certificate verification failed: issuer is not trusted (https://archive-crawler.svn.sourceforge.net)" szznax 2010-01-12 16:45:20 +00:00
• 5abacafd43 fix for [HER-1729] add (deprecated) style code from Engine class to Flash class * Heritrix.java added CLAP protocol to access representations via classloader * EngineApplication.java added staticDir resource to serve static files under "/engine/static" also added org.archive.crawler.restlet package to hold static files * EngineResource.java added <link> to static CSS resource * Engine.css style for flashACK/NACK/ADVISORY divs szznax 2009-12-29 23:21:10 +00:00
• b9ab0b296d [HER-1731] Unable to set HTTPProxyPort on org.archive.modules.fetcher.FetchHttp * FetchHTTP.java make getter/setter types agree gojomo 2009-12-28 01:56:32 +00:00
• 0cb627fff8 tried to incorporate gojomo's suggestions of 2009-11-02: * Engine.java removed ERROR and WARNING strings, rely on Flash class log jobsDir check failure as SEVERE and return early collapse error-handling in considerAsJobPath() and log SEVERE renamed considerAsJobDirectory() to addJobDirectory() remove catch-and-rethrow from leaveJobPathFile() when writing, instead log SEVERE renamed leaveJobPathFile() to writeJobPathFile() added boolean "userRequest" arg to addJobDirectory(), and moved call to writeJobPathFile() inside of addJobDirectory() changed void considerAsJobPath(File) to String getJobPathFromFile and removed call to addJobDirectory make seperate calls to addJobDirectory() in findJobConfigs() * EngineResource.java restructured "create" logic for clarity eliminate style code, rely on Flash class remove limit on "addpath" input, change to "createpath" removed all-caps and bangs ("!") from error messages removed platform-specific path separators ("/") call Engine.addJobDirectory() with userRequest=true in when form action=add szznax 2009-12-17 01:54:21 +00:00
• bf2eaef6e2 Part of fix for [HER-1725] on engine page, 'create new job' form gives NPE * Engine.java made protected getProfileCxmlResource() for clear testability * EngineTest.java added testGetProfileCxmlResource() szznax 2009-12-11 00:47:06 +00:00
• 56d6e2ee26 Change XmlMarshaller to static util method api, and add javadocs to XmlMarshaller. * XmlMarshaller.java change to static api, and add javadocs to XmlMarshaller * *Resource.java use changed XmlMarshaller api nlevitt 2009-12-10 07:06:17 +00:00
• 7cf44fdf4d * JobResource.java Remove try/catch from acceptRepresentation() - comment didn't match code and I don't think I intended to commit it (oops). nlevitt 2009-12-10 06:08:15 +00:00
• b0e140708a Rename and document methods formerly known as presentablify*() * BeanBrowseResource.java, ScriptResource.java, EngineResource.java, JobResource.java rename presentablify() -> makePresentableMap(), and add javadocs * JobRelatedResource.java rename presentablifyNestedNames() -> addPresentableNestedNames() and presentablifyObject() -> makePresentableMapFor(), and add javadocs nlevitt 2009-12-10 03:38:25 +00:00
• 0616ddfb07 part of fix for [HER-1725] create new job gives NPE * relocated (non-dist) profile-crawler-beans.cxml from engine/src/main/java to engine/src/main/resource szznax 2009-12-10 00:05:51 +00:00
• 9f12f5777a * ToePool.java shortReportLineTo(): get data from shortReportMap() to avoid duplicate code nlevitt 2009-12-09 03:12:48 +00:00
• df6e31697d Rename short report methods: - singleLineReport() -> shortReportLine() - singleLineReportTo() -> shortReportLineTo() - singleLineReportData() -> shortReportMap() nlevitt 2009-12-09 02:53:02 +00:00
• 3b75b28168 removed extraneous reference to jsp-2.0. dbernstein 2009-12-07 23:51:14 +00:00
• f8cc922cb3 [HER-1724] revive missing newline at end of crawl-report.txt nlevitt 2009-12-07 22:54:03 +00:00
• cb7df416d4 post-3.0.0 release build * **/pom.xml bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release) gojomo 2009-12-05 23:48:38 +00:00
• 5661658b1b prep for 3.0.0 release * **/pom.xml bump version-id to "3.0.0" * README.txt refer to exactly 3.0.0 release notes 3.0.0 gojomo 2009-12-05 09:28:19 +00:00
• 7d0061884c Prevent timing NPEs on CrawlLimitEnforcer-triggered crawl end * AbstractFrontier.java close recovery log in stop() (comes later, usu. from managerThread) rather than terminate() (earlier, may occur in other threads like progress-stats-snapshot thread) gojomo 2009-12-05 00:55:55 +00:00
• 21d4bce819 correct typo in built-in profile templates * org/archive/crawler/restlet/profile-crawler-beans.cxml, org/archive/crawler/migrate/migrate-template-crawler-beans.cxml Percentage -> Percent gojomo 2009-12-05 00:16:55 +00:00
• ac255d0ebe Prep for 3.0.0 * README.txt updated, pointers to wiki references gojomo 2009-12-05 00:12:14 +00:00
• 4da5095212 Keep html spartan for now. * EditRepresentation.java write() remove background color and font size styling on textarea nlevitt 2009-12-04 23:30:24 +00:00
• e85972848d Format, indent xml. See discussion on [HER-1603]. * XmlMarshaller.java (constructor) enable xml formatting/indentation nlevitt 2009-12-04 23:16:35 +00:00
• 29e8b05308 Improvement to [HER-1603?focusedCommentId=22557] fix * BaseResource.java getPreferredVariant(): always prefer HTML if anything in the header indicates it's acceptable (even "*/*" with low q-preference) nlevitt 2009-12-04 23:09:24 +00:00
• c3539d3fdc Fix for [HER-1603?focusedCommentId=22557] browsers based on WebKit (Safari, iPhone Safari, Chrome) use a screwy 'Accept' header that if taken literally means they prefer 'application/xml' content. Thus trying to view web UI in these browsers results in seeing the XML version, without element markup * BaseResource.java override getPreferredVariant() to bump up preference level of text/html for any client can accept it * EngineResource.java, JobResource.java inherit from BaseResource nlevitt 2009-12-04 22:56:25 +00:00
• 04012279fc Misc logging-cleanup/refactoring * CrawlerJournal.java improve exception logging combine writeLine() methods to use varargs * FrontierJournal.java improve exception logging * ArchiveUtils.java replace ignored count parameter with meaingful boolean one * CrawlURI.java use updated ArchiveUtils method gojomo 2009-12-04 22:23:48 +00:00
• d423af6838 changed cachePercentage to cachePercent szznax 2009-12-04 22:09:49 +00:00
• 4bb5dca139 * pom.xml Configure custom @contributor javadoc tag. nlevitt 2009-12-04 06:00:36 +00:00
• 67a25f2dce * CheckpointService.java getAvailableCheckpointDirectories() - avoid NullPointerException when checkpoints dir does not exist nlevitt 2009-12-04 00:52:55 +00:00
• a49159a2cc Use specific includes to define contents of source distribution, to avoid packaging whatever junk might be lying around your source checkout. * dist/src/main/assembly/src.xml nlevitt 2009-12-03 22:42:12 +00:00
• e1bbbf5a59 [HER-1704] WorkQueueFrontier.snoozedClassQueues not bounded in size, not reported in useful order * WorkQueueFrontier.java add snoozedOverflow map of (wakeTime)->(DelayedWorkQueue) for snoozes over a capped size maintain in parallel to snoozedClassQueues DelayQueue, maintain atomic size count in standard report, presort snoozedClassQueues before displaying * DelayedWorkQueue.java move to top-level so serialization doesn't pull out whole frontier * BdbFrontier.java initialize snoozedOverflow * BdbMultipleWorkQueues.java improve error message seen on problematic checkpoint-recovery gojomo 2009-12-03 19:37:27 +00:00
• 095f2fdffb Make Maven build more robust * engine/pom.xml, .classpath remove apparently-no-longer-needed tomcat library reference, avoiding Maven problems with m2eclipse 0.9.8.200905041414 in eclipse-galileo on ubuntu gojomo 2009-12-03 19:02:13 +00:00
• 950e68091c UI touchup * EngineResource.java give add-job options their own header, reorder/reword for clarity * JobResource.java return to always showing 'checkpoint' button, disable if necessary, so main control buttons aren't appearing/disappearing or moving gojomo 2009-12-03 18:28:21 +00:00
• 757dbf3f19 Output/UI touchup * CheckpointService.java list checkpoint directories in reverse-last-modified order disable dubugging output to System.err gojomo 2009-12-03 18:24:17 +00:00
• dd4a8d4705 [HER-1718] misleading "writer subdirectory" shown in webui * WriterPoolProcessor.java change description of WriterPoolProcessor.directory ConfigPath to not imply as strongly that is where writing much be occurring gojomo 2009-12-03 08:41:08 +00:00
• f3f7a98240 [HER-1708] h3 - crawl summary report byte totals off somehow * CrawlSummaryReport.java relabel, reorder, regroup displayed values for simplicity/consistency gojomo 2009-12-03 03:12:38 +00:00
• 19b9833fb4 Fix (harmless?) stack trace from closing output too early. It looked like this: ========== 2009-12-03 01:36:16.726 SEVERE thread-10 com.noelios.restlet.http.HttpServerConverter.commit() An exception occured writing the response entity java.io.IOException: Stream closed at sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:26) [...] 2009-12-03 01:36:16.733::WARN: Committed before 500 An exception occured writing the response entity 2009-12-03 01:36:16.735 WARNING thread-10 com.noelios.restlet.http.HttpServerHelper.handle() Error while handling an HTTP server call: ========== * JobResource.java writeHtml(Writer) - do not close writer nlevitt 2009-12-03 01:40:42 +00:00
• 6edbc5181e [HER-1721] pressing build immediately after adding or creating a new job gives IOE * JobResource.java presentablify(), writeHtml() - make sure crawl log exists before attempting to read and serve it nlevitt 2009-12-03 01:26:26 +00:00
• e15f49dd60 [HER-1708] h3 - crawl summary report byte totals off somehow * CrawlController.java send event of crawl-end before appCtx.stop() * StatisticsTracker.java force final progressStatisticsEvent and dumpReports at crawlEnded() before stop() add force-dump of reports (so dumpReports never reuses prior reports) * CrawledBytesHistotable.java add getTotalBytes(), a total of just the byte-counts (getTotal() includes URI counts) * CrawlStatSnapshot.java use getTotalBytes() gojomo 2009-12-02 06:25:23 +00:00
• e3fc474ca4 Return http 404 when appropriate. * BeanBrowseResource.java * JobResource.java nlevitt 2009-12-02 03:24:11 +00:00
• ed2061560e Improve Maven build robustness * UURIFactoryTest.java specify stirng literal with escape-encoding, rather than as raw UTF-8 * **/pom.xml include 'project.build.sourceEncoding' property to suppress platform-encoding warnings (unfortunately, neither this nor similar 'maven.compile.encoding' property actually casues maven build to interpret Java source as UTF-8) gojomo 2009-12-02 01:34:40 +00:00
• aa3bfc7f4c Remove unused jasper dependency. * .classpath * dist/pom.xml nlevitt 2009-11-26 01:43:15 +00:00
• 75fa939995 Update restlet to latest 1.1.6 * .classpath * engine/pom.xml nlevitt 2009-11-25 01:49:17 +00:00
• 8ba641ed4b Part of [HER-1603] Restletify: xml representations of all resources that also offer html * XmlMarshaller.java new class to write xml based on nested map structure * BeanBrowseResource.java ScriptResource.java EngineResource.java JobResource.java JobRelatedResource.java support xml representation * WorkQueueFrontier.java PrecedenceProvider.java WorkQueue.java ToeThread.java ToePool.java CrawlJob.java Engine.java Frontier.java CrawlController.java StatisticsTracker.java CrawledBytesHistotable.java FetchStats.java ProcessorChain.java CrawlURI.java Reporter.java offer same data that is returned in the single line reports in a map suitable for use with XmlMarshaller * FileUtils.java new method tryToCanonicalize(File) nlevitt 2009-11-25 01:43:57 +00:00
• a1022c8246 [HER-1711] Some Internationalized Domain Name seeds ignored by Heritrix * EnhDirectoryResource.java ensure files shown for edit or written-back to disk handled as UTF-8 * ConfigFile.java read file as UTF-8 gojomo 2009-11-25 01:14:59 +00:00
• 79fd64a2b5 Post-RC1-build * **/pom.xml revert version name to SNAPSHOT gojomo 2009-11-23 10:23:28 +00:00
• 0c7a74bd35 RC1 * **/pom.xml rename version to '3.0.0-RC1' for distinguished 'RC1' identified build (instead of SNAPSHOT/timestamp) gojomo 2009-11-23 09:57:42 +00:00
• 5d843d46c1 * CachedBdbMap.java remove unnecessary/build-breaking ServerCache import gojomo 2009-11-23 09:36:56 +00:00
• faa8e899db * CachedBdbMap.java remove unnecessary/build-breaking CrawlController import gojomo 2009-11-23 09:26:08 +00:00
• 18e165f21e [HER-1714] fixed-interval rescheduling * AbstractFrontier.java, WorkQueueFrontier.java, BdbFrontier.java add 'futureUris' map of CrawlURIs to be reenqueued at specific future dates the usual findEligibleUri also checks for newly-eligible rescheduled URIs the frontier is no longer empty unless the future count is also 0 * CrawlURI.java add rescheduleTime property * ReschedulingProcessor.java optional post-processor to set a CrawlURI's reschedulingTime * CrawlStatSnapshot.java, CrawlJob.java add futureUriCount to stats/uriTotalsReport gojomo 2009-11-23 08:47:14 +00:00
• 3c9df9a8c6 BDBJE updates, refactoring * BdbFrontier.java use BdbModule.openDatabase() gojomo 2009-11-23 08:24:54 +00:00
• 5b6b2fe954 BDBJE updates, refactoring * BdbModule.java (openDatabase) all databases now 'managed' (getStoredMap) all reuse of prior data as for checkpoints * (others) match above changes gojomo 2009-11-23 08:18:41 +00:00
• 4127286a1c Checkpoint cleanup * Checkpoint.java calculate shortName correctly * CrawlerLoggerModule.java use full checkpoint-name for log-rotation-suffix gojomo 2009-11-23 08:10:07 +00:00
• d93129115e BDBJE updates, refactoring * CachedBdbMap.java update for BDB-JE 4.0.71 exception changes gojomo 2009-11-23 08:03:30 +00:00