lee/heritrix3
1
0
Fork 0
mirror of https://github.com/internetarchive/heritrix3.git synced 2026-08-02 04:41:11 +00:00
Code Issues Packages Projects Releases Wiki Activity
415 Commits 42 Branches 31 Tags
6be649b286d7b3e2a1193cbe808f8a46fa1ae4c4
Commit Graph
21 Commits
Author SHA1 Message Date
gojomo 6be649b286 [HER-1783] BloomFilter64bit bit-length bug prevents full bitfield from being used; premature saturation 
* BloomFilter64bit.java
    include the split-to-subarrays (for larger bitfields) and round-up-to-power-of-2 (for performance) options previously in largely-redundant classes
    fit a number of problems with int/long overflow and bitwise ops
    add methods for reporting/testing
* BloomFilter.java
    add methods for reporting/testing
* BloomFilterTest.java, BloomFilter64bitTest.java
    more extensive tests, including two lengthy tests of default/oversized blooms usually disabled by renaming
* BloomFilter32bit.java, BloomFilter32bitSplit.java, BloomFilter32bp2.java, BloomFilter32bp2Split.java
    deleted as buggy or redundant
* BenchmarkBlooms.java
    move to test source dir
* BloomUriUniqFilter.java
    change to accept filter instance (rather than parameters) for added configuration flexibility
    fix comments
* BloomUriUniqFilterTest.java
    supply filter not paramters
 2010-06-15 21:30:34 +00:00
gojomo 5f83fe75a2 HER-1010 crawl favicon.ico 
* (self tests)
    correct expectations to now include /favicon.ico
 2010-06-03 00:31:32 +00:00
gojomo b639873dae Fix broken test 
* Engine.java
    tolerate (with return-false-forfailure rather than NPE) non-job directories
* EngineTest.java
    use empty temp job directory rather than (possibly full-of-junk) general junit-tests directory
 2010-04-27 00:44:36 +00:00
gojomo 1a95e530e8 Clean-up where assumption of serializability no longer appropriate 2010-01-26 01:25:47 +00:00
szznax bf2eaef6e2 Part of fix for [HER-1725] on engine page, 'create new job' form gives NPE 
* Engine.java
  made protected getProfileCxmlResource() for clear testability
* EngineTest.java
  added testGetProfileCxmlResource()
 2009-12-11 00:47:06 +00:00
gojomo e42e2ed7f6 Cleanup/consolidation/expansion of *Utils classes 
* IoUtils.java
    remove class that confusingly-overlaps with commons IOUtils; move remaining methods to ArchiveUtils or archive's FileUtils
* ArchiveUtils.java, FileUtils.java
    receive relocated methods; eliminated deprecated or unused methods
* JSONUtils.java
    new class to collect common JSON actions
* Iteratorable.java
    wrap Iterator as Iterable for foreach usage
* (many)
    update to use alternate utils methods
 2009-11-19 22:39:53 +00:00
gojomo 9059395617 Make Bloom size configurable without system properties 
* BloomUriUniqFilter.java
    make bloom size configurable via bean properties
    change implementing class to BloomFilter64bit (given growing prevalance of 64bit JVMs and larger heaps)
* BloomUriUniqFilterTest.java
    update test for property-set sizing
* BloomFilter64bit.java
    improve comment
 2009-11-16 22:42:18 +00:00
gojomo 0497efbb32 [HER-1561] update license to Apache License 2.0 
* (many)
    update license boilerplate
* (few)
    delete no-longer used classes
 2009-11-10 21:03:27 +00:00
gojomo dbc1c1b411 * KeyWordProcessor.java 
update shouldProcess() to be based on via CrawlURI's type
 2009-10-06 02:28:47 +00:00
gojomo 4ce8aef47a Remove obsolete test class 2009-10-06 01:40:22 +00:00
gojomo 047a157fe8 [HER-1605] Refactor parameterizable policies out of frontier 
* selftest-crawler.beans.cxml, FormAuthSelfTest.java, Precedence[134]SelfTest.java
    update self-tests for new chains refactoring
 2009-10-05 23:44:30 +00:00
gojomo 16538f175c Cleanup, refactoring 
* (many)
   update license notice to Apache where appropriate
   eliminate ProcessorURI, DefaultProcessorURI in favor of CrawlURI (now in modules package)
   other comment/warning/unused code cleanup
 2009-10-05 22:49:19 +00:00
gojomo 4d06b239ee Push CrawlURI and related constants down to modules subproject/package 
(many)
 2009-08-06 20:22:00 +00:00
gojomo 4251e5311f * SelfTestBase.java 
report full path of unclosed ARC for manual checking -- truly unclosed or timing issue?
 2009-07-08 01:21:49 +00:00
gojomo 4583ff7306 Rename Rrc2617Credential to HttpAuthenticationCredential 
* (multiple)
    use more-familiar name for HTTP authentication
 2009-07-06 22:09:52 +00:00
gojomo 771d23240c refactor sheets to better enable mid-crawl changes 
* Sheet.java
    make non-abstract; allow for name that might not be beanName
* SheetAssociation.java
    define targets as list of names, rather than Sheets themselves
* SheetForDecideRuled.java -> DecideRuledSheetAssociation.java; SheetForSurtPrefixes.java -> SurtPrefixesSheetAssociation.java
    make association separate from Sheet -- so same Sheet can be associated more than one place, more than one way
* SheetOverlaysManager.java
    match changes to Sheet and SheetAssociations
    offer convenience methods for post-construction addition of Sheets, associations
    use internal name-to-Sheet mapping, allowing for mid-crawl Sheet additions not in bean factory
* Precedence1SelfTest.java, Precedence4SelfTest.java
    update configs for Sheet changes
 2009-06-05 02:12:32 +00:00
gojomo 81a9a9f0f3 Fix unit test 
* BadURIsStopPageParsingSelfTest.java
    update expected URIs to match H2 branch / fixed code
 2009-05-15 00:27:27 +00:00
gojomo 2c11eb1dc2 Remove cruft comments 2009-05-12 19:50:05 +00:00
gojomo 8de6b2a9c5 Fix for [HER-1502] selftests break if 'localhost' does not resolve to 127.0.0.1 
* (all)
    change 'localhost' to '127.0.0.1'
 2009-05-12 18:40:18 +00:00
gojomo c8a5446695 Test robustness 
* SelfTestBase.java
    don't allow negative random passwords, in case leading '-' is causing problems
* Heritrix.java
    echo arguments when showing usage info -- to understand what went wrong
 2009-05-12 02:10:37 +00:00
gojomo e047bf68e0 copy 'springy' branch to heritrix3 2009-05-11 22:56:36 +00:00