gojomo 0400d81ee5 support for... ...

[HER-1660] allow adding seeds, recovery-journal-like URI data at any point in crawl via 'action directory' 
[HER-769] Support case where millions of seeds. 
* AbstractFrontier.java
    move sourceTagSeeds setting to SeedModule
    trigger and schedule seeds via SeedModule's announcements and SeedListener protocols
* StatisticsTracker.java
    implement SeedListener, use processedSeedRecords as list of all seeds
* ProcessorURI.java, DefaultProcessorURI.java
    (getURI) convenience accessor for plain URI string
* SurtPrefixedDecideRule.java
    use seeds as prefixes via SeedListener notifications, rather than separate scan of seeds source text
* SeedListener.java
    simplify to notifications of new seed, and nonseed text mixed with seeds (as with SURT directives)
* SeedModule.java
    methods for SeedListener interface, announceSeeds, actOn(File)
* TextSeedModule.java
    eliminate interators; add methods for announcing seeds and scanning seed text sources (including arbitrary Files)
* PrefixSet.java, SurtPrefixSet.java
    derive from ConcurrentSkipListSet; eliminate now-unneeded extra synchronization and clone()s 
    offer considerAsDirective() for direct application of directive Strings

2009-08-06 21:24:04 +00:00

commons

support for...

2009-08-06 21:24:04 +00:00

dist

[HER-1642] fix HOWTO-launch document

2009-07-13 23:06:23 +00:00

engine

support for...

2009-08-06 21:24:04 +00:00

modules

support for...

2009-08-06 21:24:04 +00:00

.classpath

[HER-1505] Expand BeanshellProcessor/BeanshellDecideRule for every JVM-friendly scripting language (Javascript/Jython/Groovy/JRuby)

2009-06-05 01:03:10 +00:00

.project

Fix for [HER-1524] speculativeFixup in ExtractorJS should maintain URL scheme

2009-05-12 22:09:22 +00:00

pom.xml

* **/pom.xml

2009-06-11 09:09:43 +00:00

S

Description

No description provided

heritrixjavawarcwebcrawling

15 MiB

Languages

Java 93.3%

HTML 2.6%

Rich Text Format 1.1%

JavaScript 1%

FreeMarker 0.8%

Other 1%