Skip to content

Don't report cancelled remote transfers as errors - #30652

Open
fmeum wants to merge 3 commits into
bazelbuild:masterfrom
fmeum:rxfutures-undeliverable-cancellation
Open

Don't report cancelled remote transfers as errors#30652
fmeum wants to merge 3 commits into
bazelbuild:masterfrom
fmeum:rxfutures-undeliverable-cancellation

Conversation

@fmeum

@fmeum fmeum commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Description

When a bulk transfer to or from the remote cache is cancelled, every one of its in-flight transfers fails with a CancellationException. mergeBulkTransfer subscribes to all transfers of a bulk operation at once via flatMapSingle, which terminates on the first error and hands every subsequent one to RxJava's global error handler.

RemoteModule installs a global error handler that reports those as error events, so a single cancelled bulk transfer printed a java.util.concurrent.CancellationException: Task was cancelled stack trace for every transfer but the first.

A transfer is only ever cancelled as part of tearing down the operation it belongs to, so toTransferResult now maps CancellationException to the same result as InterruptedException instead of propagating it downstream.

This PR also replaces two isDisposed()-then-onError() sequences in RxFutures and AsyncTaskCache with tryOnError. Those checks aren't atomic with the delivery: the emitter can be disposed in between, in which case onError hands the error to the global error handler as well.

Motivation

Users see spurious ERROR: java.util.concurrent.CancellationException: Task was cancelled stack traces, one per concurrently cancelled transfer, whenever a build that is uploading or downloading blobs is interrupted or an action fails while a bulk transfer is in flight.

This also shows up as a flake in RemoteExecutionServiceTest.uploadInputsIfNotPresent_interrupted_requestCancelled, which fails whenever RxNoGlobalErrorsRule observes the leaked exception. That test failed in roughly 2-5% of runs before this change and passed 100 out of 100 times after it.

Build API Changes

No

Checklist

  • I have added tests for the new use cases (if any).
  • I have updated the documentation (if applicable).

The existing uploadInputsIfNotPresent_interrupted_requestCancelled covers this, but only probabilistically: the number of transfers that are cancelled concurrently determines whether an error is leaked, and the tryOnError changes fix races that can't be triggered deterministically.

Release Notes

RELNOTES: None

fmeum added 2 commits August 10, 2026 14:29
When a bulk transfer to or from the remote cache is cancelled, e.g.
because the action owning it is interrupted, every one of its in-flight
transfers fails with a `CancellationException`. `mergeBulkTransfer`
subscribes to all of them at once via `flatMapSingle`, which terminates
on the first error and hands all subsequent ones to RxJava's global
error handler. `RemoteModule` installs a handler that reports those as
error events, so a single cancelled bulk transfer printed a
`java.util.concurrent.CancellationException: Task was cancelled` stack
trace for every transfer but the first.

A transfer is only ever cancelled as part of tearing down the operation
it belongs to, so `toTransferResult` now maps `CancellationException` to
the same result as `InterruptedException` instead of propagating it.

Also replaces two `isDisposed()`-then-`onError()` sequences with
`tryOnError`: the emitter can be disposed in between, in which case
`onError` hands the error to the global error handler as well.
@fmeum
fmeum marked this pull request as ready for review August 10, 2026 12:47
@fmeum
fmeum requested a review from a team as a code owner August 10, 2026 12:47
@fmeum
fmeum requested review from coeuvre and removed request for a team August 10, 2026 12:47
@github-actions github-actions Bot added team-Remote-Exec Issues and PRs for the Execution (Remote) team awaiting-review PR is awaiting review from an assigned reviewer labels Aug 10, 2026
@fmeum

fmeum commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator Author

@bazel-io fork 9.3.0

@coeuvre coeuvre added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally team-Remote-Exec Issues and PRs for the Execution (Remote) team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants