SEVERE thread-5973 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.nio.charset.UnsupportedCharsetException: binary occured when trying to process 'http://example.com/' at step ABOUT_TO_BEGIN_PROCESSOR in
java.nio.charset.UnsupportedCharsetException: binary
at java.nio.charset.Charset.forName(Charset.java:543)
at org.apache.http.entity.ContentType.<init>(ContentType.java:111)
at org.apache.http.entity.ContentType.create(ContentType.java:210)
at org.apache.http.entity.ContentType.get(ContentType.java:258)
at org.apache.http.entity.ContentType.getOrDefault(ContentType.java:277)
at org.archive.modules.fetcher.FetchHTTP.setCharacterEncoding(FetchHTTP.java:580)
at org.archive.modules.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:693)
at org.archive.modules.Processor.innerProcessResult(Processor.java:175)
at org.archive.modules.Processor.process(Processor.java:142)
at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)
at org.archive.crawler.framework.ToeThread.run(ToeThread.java:148)
PoolingHttpClientConnectionManager, and make the connection manager a member
variable of HttpRequest. We had been seeing a strange bug where once in
every few thousand urls or so, the connection manager would get gc'd
while the http request was being sent, and RecordingOutputStream.flush()
would get called, as in the stack trace below. The ToeThread was using
it at the same time, so if conditions were right, the recorded stream
would get mangled. This commit makes the connection manager a member of
FetchHTTPRequest so it doesn't get gc'd prematurely. Also switches to
BasicHttpClientConnectionManager, since we're not doing any pooling.
it.unimi.dsi.fastutil.io.FastBufferedOutputStream.dumpBuffer(boolean) line: 133
it.unimi.dsi.fastutil.io.FastBufferedOutputStream.flush() line: 166
org.archive.io.RecordingOutputStream.flush() line: 469
org.apache.http.impl.io.SessionOutputBufferImpl.flushStream() line: 128
org.apache.http.impl.io.SessionOutputBufferImpl.flush() line: 143
org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection(org.apache.http.impl.BHttpConnectionBase).close() line: 314
org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection.close() line: 520
org.apache.http.impl.conn.CPoolEntry.closeConnection() line: 69
org.apache.http.impl.conn.CPoolEntry.close() line: 95
org.apache.http.impl.conn.CPool(org.apache.http.pool.AbstractConnPool<T,C,E>).shutdown() line: 130
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.shutdown() line: 347
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.finalize() line: 168
java.lang.ref.Finalizer.invokeFinalizeMethod(java.lang.Object) line: not available [native method]
Noah Levitt