Skip to content

eth-deposit service does not reconnect to Parity node on failure #23

@bakhtin

Description

@bakhtin

eth-deposit service does not attempt to reconnect to Parity node if just a single request failed. It throws an exception and just 'hangs':

[22.05.2019 22:29:34,611] eth [INFO ] [eth-deposit:web3j:th-1:id-19] Ethereum chain handler for block 7807067
May 22, 2019 10:30:02 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
[22.05.2019 22:30:12,430] eth [ERROR] [eth-deposit:web3j:th-1:id-19] OnError called
java.net.SocketTimeoutException: timeout
	at okio.Okio$4.newTimeoutException(Okio.java:230)
	at okio.AsyncTimeout.exit(AsyncTimeout.java:285)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211)
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
	at okhttp3.RealCall.execute(RealCall.java:69)
	at org.web3j.protocol.http.HttpService.performIO(HttpService.java:151)
	at org.web3j.protocol.Service.send(Service.java:34)
	at org.web3j.protocol.core.Request.send(Request.java:72)
	at com.d3.eth.sidechain.EthChainListener$getBlockObservable$1$2.apply(EthChainListener.kt:48)
	at com.d3.eth.sidechain.EthChainListener$getBlockObservable$1$2.apply(EthChainListener.kt:22)
	at io.reactivex.internal.operators.observable.ObservableMap$MapObserver.onNext(ObservableMap.java:57)
	at io.reactivex.internal.operators.observable.ObservableFilter$FilterObserver.onNext(ObservableFilter.java:52)
	at io.reactivex.internal.operators.observable.ObservableFromPublisher$PublisherSubscriber.onNext(ObservableFromPublisher.java:56)
	at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.tryEmitScalar(FlowableFlatMap.java:234)
	at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:152)
	at io.reactivex.internal.operators.flowable.FlowableCreate$BufferAsyncEmitter.drain(FlowableCreate.java:547)
	at io.reactivex.internal.operators.flowable.FlowableCreate$BufferAsyncEmitter.onNext(FlowableCreate.java:471)
	at org.web3j.protocol.core.filters.BlockFilter.process(BlockFilter.java:32)
	at org.web3j.protocol.core.filters.Filter.pollFilter(Filter.java:129)
	at org.web3j.protocol.core.filters.Filter.lambda$run$0(Filter.java:83)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketInputStream.read(SocketInputStream.java:204)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at okio.Okio$2.read(Okio.java:139)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
	... 41 more

The service should be able to handle situations like this and try to reconnect.
Also, take a look at the log format in the exception. It differs from other logs:

Exception log format (bad):
May 22, 2019 10:30:02 PM okhttp3.internal.platform.Platform log
Other logs format (good):
[22.05.2019 22:29:34,611] eth [INFO ] [eth-deposit:web3j:th-1:id-19] Ethereum chain handler for block 7807067

UPD1: Hit us once again. Here is the link to the logs

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions