Noah Levitt 161b359979 Avoid the exception below by handling bad charsets in FetchHTTP. Restore annotation "unsatisfiableCharsetInHeader:...". Includes unit tests for good and bad charsets. ...

SEVERE thread-5973 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.nio.charset.UnsupportedCharsetException: binary occured when trying to process 'http://example.com/' at step ABOUT_TO_BEGIN_PROCESSOR in
java.nio.charset.UnsupportedCharsetException: binary
        at java.nio.charset.Charset.forName(Charset.java:543)
        at org.apache.http.entity.ContentType.<init>(ContentType.java:111)
        at org.apache.http.entity.ContentType.create(ContentType.java:210)
        at org.apache.http.entity.ContentType.get(ContentType.java:258)
        at org.apache.http.entity.ContentType.getOrDefault(ContentType.java:277)
        at org.archive.modules.fetcher.FetchHTTP.setCharacterEncoding(FetchHTTP.java:580)
        at org.archive.modules.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:693)
        at org.archive.modules.Processor.innerProcessResult(Processor.java:175)
        at org.archive.modules.Processor.process(Processor.java:142)
        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)
        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:148)

2014-01-28 16:19:44 -08:00

..

main

Avoid the exception below by handling bad charsets in FetchHTTP. Restore annotation "unsatisfiableCharsetInHeader:...". Includes unit tests for good and bad charsets.

2014-01-28 16:19:44 -08:00

test/java/org/archive/modules

Avoid the exception below by handling bad charsets in FetchHTTP. Restore annotation "unsatisfiableCharsetInHeader:...". Includes unit tests for good and bad charsets.

2014-01-28 16:19:44 -08:00