lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-25 07:56:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
365 Commits 38 Branches 32 Tags
cc8ea6494a37ecf4a91492f6c7e8a1ef2d00ffa6
Commit Graph
365 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
szznax cc8ea6494a changed namedField "ARC-Compressed-Offset" to "ARC-Gzip-Offset" for even more clarity 2010-03-11 15:59:10 +00:00
szznax 4802e5093e part of fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses 
* Arc2Warc.java
    changed namedField "ARC-Offset" to "ARC-Compressed-Offset" for clarity
 2010-03-11 01:45:49 +00:00
szznax 8a567c9bc0 part of fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses 
* Arc2Warc.java
    removed unnecessary URISyntaxException import
    moved ARC named fields outside of check for IP in header field
 2010-03-10 00:33:38 +00:00
szznax 65f962dc25 whitespace corrections 2010-03-08 18:07:30 +00:00
szznax 65d552ce5c fix for HER-1748 Arc2Warc should generally write WARC "response" records for ARC HTTP responses 
* Arc2Warc.java
    when HTTP response, write response record rather than resource
    enable reconstruction of ARC from WARC via namedFields
* ARCRecord.java
    added headerString and accessor to save verbatim ARC header string
 2010-03-08 17:43:42 +00:00
szznax 3e5cf29315 nit correction in javadoc 2010-03-08 17:35:47 +00:00
nlevitt 86268def97 * LongToIntConsistentHash.java 
Skip unnecessary conversion from CharSequence to String
 2010-02-25 18:19:46 +00:00
szznax e9df65d446 fix for HER-1744 Incorrect comparison of Strings in ProcessorChain.process() with "!=" operator 
* ProcessorChain.java
    use equals() to compare string values rather than objects (with "!=")
 2010-02-16 19:13:43 +00:00
szznax ed3d64d91e part of fix for HER-1736 long pause at startup reading large per-job job.log(s) 
* CrawlJob.java
 2010-02-16 19:03:59 +00:00
gojomo acfe95fea3 naming/branching/visibility review for HER-1736 long pause at startup reading large per-job job.log(s) 
* CrawlJob.java
    added accessor, renamed a bit, unified 2 branches
* JobResource.java
    use accessor
 2010-02-13 00:47:41 +00:00
szznax e1308e80f5 reverting last checkin, also testing passwordless checkin with Karmic Eclipse 3.5.1 SVNKit 1.6.7 update 2010-02-13 00:21:22 +00:00
szznax 6c13f42242 testing Karmic Eclipse 3.5.1 (w/updated SVNKit 1.6.7) 2010-02-13 00:18:10 +00:00
szznax 2d441d611e part of fix for HER-1736 long pause at startup reading large per-job job.log(s) 
* CrawlJob.java
    added boolean partialJobLogScan, and print "at least X launches" when true
* JobResource.java
    print "at least X launches" when partialJobLogScan
 2010-02-12 23:21:59 +00:00
szznax 18d459f6b9 part of fix for HER-1736 long pause at startup reading large per-job job.log(s) 
* CrawlJob.java
    check for excessively long job log, scan near end of file for lastLaunch
    inform user OK to move big job log aside
 2010-02-12 18:54:57 +00:00
szznax 4097e29d50 part of fix for HER-1736 long pause at startup reading large per-job job.log(s) 
* CrawlJob.java
    return early from scanJobLog() when job log > 1GB
 2010-02-09 19:45:23 +00:00
szznax a53d9ea733 per Gordon's suggestions; renamed "Engine.css" to "engine.css" and changed flashACK class background-color to palegreen 2010-01-26 20:54:16 +00:00
szznax 0b5e5b30e3 removed references to unused repos, and updated servlet-api to use newer jetty referenced repo 2010-01-26 20:31:36 +00:00
gojomo 1a95e530e8 Clean-up where assumption of serializability no longer appropriate 2010-01-26 01:25:47 +00:00
gojomo 248f720cab Make Recorder facility usable independent of special crawler threads 
* Recorder.java
    store reusable per-thread Recorder instance in Recorder static thread-local
* ToeThread.java
    setup reusable instance into Recorder
* RecorderMarker.java
    deleted -- unnecessary
 2010-01-26 01:21:59 +00:00
gojomo f35da10154 * EnhDirectoryResource.java 
improve error for POST to file when EditRepresentation-wrapping (via '?format=textedit') hasn't occurred
 2010-01-26 00:18:23 +00:00
gojomo 760c40c440 * EngineResource.java 
use platform-specific path-separator in partial-path-prompt
 2010-01-26 00:16:52 +00:00
szznax 3dff26e277 attempt to remedy Continuum build problem with SVN. "Server certificate verification failed: issuer is not trusted (https://archive-crawler.svn.sourceforge.net)" 2010-01-12 16:45:20 +00:00
szznax 5abacafd43 fix for [HER-1729] add (deprecated) style code from Engine class to Flash class 
* Heritrix.java
  added CLAP protocol to access representations via classloader
* EngineApplication.java
  added staticDir resource to serve static files under "/engine/static"
  also added org.archive.crawler.restlet package to hold static files
* EngineResource.java
  added <link> to static CSS resource
* Engine.css
  style for flashACK/NACK/ADVISORY divs
 2009-12-29 23:21:10 +00:00
gojomo b9ab0b296d [HER-1731] Unable to set HTTPProxyPort on org.archive.modules.fetcher.FetchHttp 
* FetchHTTP.java
    make getter/setter types agree
 2009-12-28 01:56:32 +00:00
szznax 0cb627fff8 tried to incorporate gojomo's suggestions of 2009-11-02: 
* Engine.java
  removed ERROR and WARNING strings, rely on Flash class
  log jobsDir check failure as SEVERE and return early
  collapse error-handling in considerAsJobPath() and log SEVERE
  renamed considerAsJobDirectory() to addJobDirectory()
  remove catch-and-rethrow from leaveJobPathFile() when writing,
    instead log SEVERE
  renamed leaveJobPathFile() to writeJobPathFile()
  added boolean "userRequest" arg to addJobDirectory(), and moved
    call to writeJobPathFile() inside of addJobDirectory()
  changed void considerAsJobPath(File) to String getJobPathFromFile
    and removed call to addJobDirectory
  make seperate calls to addJobDirectory() in findJobConfigs()
* EngineResource.java
  restructured "create" logic for clarity
  eliminate style code, rely on Flash class
  remove limit on "addpath" input, change to "createpath"
  removed all-caps and bangs ("!") from error messages
  removed platform-specific path separators ("/")
  call Engine.addJobDirectory() with userRequest=true in when
    form action=add
 2009-12-17 01:54:21 +00:00
szznax bf2eaef6e2 Part of fix for [HER-1725] on engine page, 'create new job' form gives NPE 
* Engine.java
  made protected getProfileCxmlResource() for clear testability
* EngineTest.java
  added testGetProfileCxmlResource()
 2009-12-11 00:47:06 +00:00
nlevitt 56d6e2ee26 Change XmlMarshaller to static util method api, and add javadocs to XmlMarshaller. 
* XmlMarshaller.java
    change to static api, and add javadocs to XmlMarshaller
* *Resource.java
    use changed XmlMarshaller api
 2009-12-10 07:06:17 +00:00
nlevitt 7cf44fdf4d * JobResource.java 
Remove try/catch from acceptRepresentation() - comment didn't match code
    and I don't think I intended to commit it (oops).
 2009-12-10 06:08:15 +00:00
nlevitt b0e140708a Rename and document methods formerly known as presentablify*() 
* BeanBrowseResource.java, ScriptResource.java, EngineResource.java, 
  JobResource.java
    rename presentablify() -> makePresentableMap(), and add javadocs
* JobRelatedResource.java
    rename presentablifyNestedNames() -> addPresentableNestedNames()
    and presentablifyObject() -> makePresentableMapFor(), and add javadocs
 2009-12-10 03:38:25 +00:00
szznax 0616ddfb07 part of fix for [HER-1725] create new job gives NPE 
* relocated (non-dist) profile-crawler-beans.cxml
  from engine/src/main/java
  to engine/src/main/resource
 2009-12-10 00:05:51 +00:00
nlevitt 9f12f5777a * ToePool.java 
shortReportLineTo(): get data from shortReportMap() to avoid duplicate code
 2009-12-09 03:12:48 +00:00
nlevitt df6e31697d Rename short report methods: 
- singleLineReport() -> shortReportLine()
 - singleLineReportTo() -> shortReportLineTo()
 - singleLineReportData() -> shortReportMap()
 2009-12-09 02:53:02 +00:00
dbernstein 3b75b28168 removed extraneous reference to jsp-2.0. 2009-12-07 23:51:14 +00:00
nlevitt f8cc922cb3 [HER-1724] revive missing newline at end of crawl-report.txt 2009-12-07 22:54:03 +00:00
gojomo cb7df416d4 post-3.0.0 release build 
* **/pom.xml
    bump version identifiers to "3.1.1-SNAPSHOT" (odd minor number meaning dev version, odd patch number indicating expected next patch number of 3.0.x release if before a 3.2.0 release)
 2009-12-05 23:48:38 +00:00
gojomo 5661658b1b prep for 3.0.0 release 
* **/pom.xml
    bump version-id to "3.0.0"
* README.txt
    refer to exactly 3.0.0 release notes
3.0.0 		2009-12-05 09:28:19 +00:00
gojomo 7d0061884c Prevent timing NPEs on CrawlLimitEnforcer-triggered crawl end 
* AbstractFrontier.java
    close recovery log in stop() (comes later, usu. from managerThread) rather than terminate() (earlier, may occur in other threads like progress-stats-snapshot thread)
 2009-12-05 00:55:55 +00:00
gojomo 21d4bce819 correct typo in built-in profile templates 
* org/archive/crawler/restlet/profile-crawler-beans.cxml, org/archive/crawler/migrate/migrate-template-crawler-beans.cxml
    Percentage -> Percent
 2009-12-05 00:16:55 +00:00
gojomo ac255d0ebe Prep for 3.0.0 
* README.txt
    updated, pointers to wiki references
 2009-12-05 00:12:14 +00:00
nlevitt 4da5095212 Keep html spartan for now. 
* EditRepresentation.java 
    write() remove background color and font size styling on textarea
 2009-12-04 23:30:24 +00:00
nlevitt e85972848d Format, indent xml. See discussion on [HER-1603]. 
* XmlMarshaller.java
    (constructor) enable xml formatting/indentation
 2009-12-04 23:16:35 +00:00
nlevitt 29e8b05308 Improvement to [HER-1603?focusedCommentId=22557] fix 
* BaseResource.java
    getPreferredVariant(): always prefer HTML if anything in the header
    indicates it's acceptable (even "*/*" with low q-preference)
 2009-12-04 23:09:24 +00:00
nlevitt c3539d3fdc Fix for [HER-1603?focusedCommentId=22557] browsers based on WebKit (Safari, 
iPhone Safari, Chrome) use a screwy 'Accept' header that if taken literally
means they prefer 'application/xml' content. Thus trying to view web UI in
these browsers results in seeing the XML version, without element markup
* BaseResource.java
    override getPreferredVariant() to bump up preference level of text/html for
    any client can accept it
* EngineResource.java, JobResource.java
    inherit from BaseResource
 2009-12-04 22:56:25 +00:00
gojomo 04012279fc Misc logging-cleanup/refactoring 
* CrawlerJournal.java
    improve exception logging
    combine writeLine() methods to use varargs
* FrontierJournal.java
    improve exception logging
* ArchiveUtils.java
    replace ignored  count parameter with meaingful boolean one
* CrawlURI.java
    use updated ArchiveUtils method
 2009-12-04 22:23:48 +00:00
szznax d423af6838 changed cachePercentage to cachePercent 2009-12-04 22:09:49 +00:00
nlevitt 4bb5dca139 * pom.xml 
Configure custom @contributor javadoc tag.
 2009-12-04 06:00:36 +00:00
nlevitt 67a25f2dce * CheckpointService.java 
getAvailableCheckpointDirectories() - avoid NullPointerException when
    checkpoints dir does not exist
 2009-12-04 00:52:55 +00:00
nlevitt a49159a2cc Use specific includes to define contents of source distribution, to avoid 
packaging whatever junk might be lying around your source checkout.
* dist/src/main/assembly/src.xml
 2009-12-03 22:42:12 +00:00
gojomo e1bbbf5a59 [HER-1704] WorkQueueFrontier.snoozedClassQueues not bounded in size, not reported in useful order 
* WorkQueueFrontier.java
    add snoozedOverflow map of (wakeTime)->(DelayedWorkQueue) for snoozes over a capped size
    maintain in parallel to snoozedClassQueues DelayQueue, maintain atomic size count
    in standard report, presort snoozedClassQueues before displaying
* DelayedWorkQueue.java
    move to top-level so serialization doesn't pull out whole frontier
* BdbFrontier.java
    initialize snoozedOverflow
* BdbMultipleWorkQueues.java
    improve error message seen on problematic checkpoint-recovery
 2009-12-03 19:37:27 +00:00
gojomo 095f2fdffb Make Maven build more robust 
* engine/pom.xml, .classpath
    remove apparently-no-longer-needed tomcat library reference, avoiding Maven problems with m2eclipse 0.9.8.200905041414 in eclipse-galileo on ubuntu
 2009-12-03 19:02:13 +00:00