Handle TransferEncodingError and ClientConnectorError as graceful network disconnects#2
Open
livepeer-tessa wants to merge 1 commit intomainfrom
Open
Conversation
…work disconnects When an orchestrator truncates a trickle transfer mid-stream (HTTP 400 + incomplete transfer encoding), aiohttp raises ClientPayloadError (with TransferEncodingError as a subclass). Previously this propagated as an application error in both ChannelReader and JSONLReader, causing noisy ERROR-level logs and unclean session teardown. Changes: - channel_reader.py: Catch aiohttp.ClientPayloadError in both ChannelReader and JSONLReader before the generic Exception handler. Log at WARNING and return cleanly instead of wrapping as LivepeerGatewayError. This stops the error from bubbling up to livepeer_app.py's control channel handler. - trickle_publisher.py: Demote ClientConnectorError in _run_delete from ERROR to DEBUG. When the orchestrator is already down, connection-refused errors during trickle DELETE teardown are expected and not actionable. Fixes: daydreamlive/scope#805 Related: daydreamlive/scope#771 (similar but for EOFError on clean disconnect) Signed-off-by: livepeer-tessa <livepeer-tessa@users.noreply.github.com>
livepeer-tessa
pushed a commit
to daydreamlive/scope
that referenced
this pull request
Apr 2, 2026
…input loop When an orchestrator truncates the trickle connection mid-stream, aiohttp raises ClientPayloadError (subclass TransferEncodingError). Previously this was caught by the broad 'except Exception' handler and logged at ERROR level, causing noisy logs and unclean teardown. - Catch aiohttp.ClientPayloadError before the generic handler; log at WARNING and let the finally block run the normal media_output.close() path - Suppress ClientConnectorError during media_output.close() (logged at DEBUG) when the orchestrator is already unreachable at teardown time The deeper fix (in livepeer-python-gateway channel_reader.py / trickle_publisher.py) ensures the control channel subscription also terminates cleanly without raising. See: livepeer/livepeer-python-gateway#2 Fixes: #805 Related: #771 Signed-off-by: livepeer-robot <robot@livepeer.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes daydreamlive/scope#805 — media input loop and control channel fail with
TransferEncodingError: 400when an orchestrator truncates the trickle connection mid-stream, then cascades intoClientConnectorErroron teardown.Root Cause
When an orchestrator goes down or is restarted during an active session, aiohttp raises
ClientPayloadError(specificallyTransferEncodingError) on the open HTTP response. This is a network-level disconnect, not an application error — but it was being caught by the genericexcept Exceptionhandler and re-raised asLivepeerGatewayError, propagating as an ERROR up tolivepeer_app.py.Subsequently, teardown attempts to DELETE/close the trickle channels, but the orchestrator is already unreachable — resulting in
ClientConnectorErrorlogs also at ERROR level.Changes
channel_reader.pyaiohttp.ClientPayloadErrorhandler in bothChannelReaderandJSONLReaderbefore the genericexcept ExceptionclauseWARNINGlevel andreturncleanly — stops iteration without raising, so callers see a clean EOF rather than an exceptiontrickle_publisher.pyClientConnectorErrorin_run_deletefromERRORtoDEBUGBehaviour Before/After
Before:
After:
Worker shuts down cleanly instead of propagating errors.
Related
EOFErroron clean disconnect