[9.3.0] remote: add gRPC download idle timeout - #30667
Draft
sluongng wants to merge 2 commits into
Draft
Conversation
Remote cache users currently have one `--remote_timeout` value that applies across remote gRPC calls. That is a poor fit because fast control-plane RPCs such as action cache lookups benefit from short deadlines, while ByteStream reads and writes may need much longer deadlines for large blobs. Setting the flag too low can make uploads or downloads fail consistently; setting it too high makes stale requests wait too long before Bazel retries. This follows the direction discussed in bazelbuild#26741. This PR keeps the default user experience unchanged while moving remote gRPC timeout handling onto service config, then adds an opt-in advanced override. It is structured as two commits: - `remote: drive gRPC deadlines from service config` generates the same timeout policy Bazel previously applied from `--remote_timeout`, installs it on remote gRPC channels, and removes per-stub deadline plumbing. - `remote: accept gRPC service config files` adds `--remote_grpc_service_config` for a user-owned JSON file. The initial supported schema is intentionally restricted to `methodConfig`, `name`, and `timeout`; unsupported fields such as retry, hedging, load balancing, and health checking are rejected so Bazel can expand support deliberately later. Closes bazelbuild#29912. PiperOrigin-RevId: 952089183 Change-Id: I0101d0cafb9e7593a16c4106372d288c8ef0237c (cherry picked from commit 3510ade)
The gRPC service-config support from bazelbuild#29912 is now on master. It lets users lengthen the whole-call deadline for `ByteStream.Read` so large downloads can keep running while they make progress. A longer deadline also lets a stalled stream wait just as long, because a total RPC deadline cannot distinguish progress from inactivity. This adds `--remote_grpc_download_idle_timeout` as an independent inactivity limit for `ByteStream.Read`. It defaults to 60 seconds to match the existing HTTP remote-cache default; setting it to `0` disables it. When a read goes idle, Bazel cancels it and translates the timeout-owned `CANCELLED` closure to `DEADLINE_EXCEEDED` so the normal cache retrier can resume from the received offset. Other RPCs, including `ByteStream.Write`, `Execute`, and `WaitExecution`, are unaffected. Timeout tracking uses a monotonic deadline and at most one scheduled task per call. This avoids retaining a canceled task for every response and prevents a timer from canceling a read that has since made progress. Normal gRPC forwarding stays in `finally` blocks so timeout bookkeeping failures cannot suppress messages or callbacks. Tests cover: - the 60-second option default and exclusion of `ByteStream.Write`; - cancellation status and single-task scheduling as reads make progress; - request and response forwarding when timeout bookkeeping fails; and - an in-process `ByteStream` flow that sends a prefix, stalls, fires the timeout deterministically, and verifies that the retry resumes from the next offset. The interceptor currently applies only to `ByteStream.Read`. The download-specific option name leaves room to cover future server-streaming download RPCs such as [`SplitChunks`](bazelbuild/remote-apis#377) without implying that every remote stream should share this timeout. Closes bazelbuild#29916. PiperOrigin-RevId: 956359263 Change-Id: Id8ae7c6dbb172f1e7e799536193458d9cbfb8c0c (cherry picked from commit 6e27dd1)
sluongng
force-pushed
the
sluongng/cherrypick-29916-9.3.0
branch
from
August 11, 2026 12:46
96bd345 to
32ea41e
Compare
iancha1992
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gRPC service-config support from #29912 is now on master. It lets users lengthen the whole-call deadline for
ByteStream.Readso large downloads can keep running while they make progress. A longer deadline also lets a stalled stream wait just as long, because a total RPC deadline cannot distinguish progress from inactivity.This adds
--remote_grpc_download_idle_timeoutas an independent inactivity limit forByteStream.Read. It defaults to 60 seconds to match the existing HTTP remote-cache default; setting it to0disables it. When a read goes idle, Bazel cancels it and translates the timeout-ownedCANCELLEDclosure toDEADLINE_EXCEEDEDso the normal cache retrier can resume from the received offset. Other RPCs, includingByteStream.Write,Execute, andWaitExecution, are unaffected.Timeout tracking uses a monotonic deadline and at most one scheduled task per call. This avoids retaining a canceled task for every response and prevents a timer from canceling a read that has since made progress. Normal gRPC forwarding stays in
finallyblocks so timeout bookkeeping failures cannot suppress messages or callbacks.Tests cover:
ByteStream.Write;ByteStreamflow that sends a prefix, stalls, fires the timeout deterministically, and verifies that the retry resumes from the next offset.The interceptor currently applies only to
ByteStream.Read. The download-specific option name leaves room to cover future server-streaming download RPCs such asSplitChunkswithout implying that every remote stream should share this timeout.Cherry-pick of #29916. Stacked on #30666, which backports #29912.