fix: stream /metrics response to avoid 32 MiB Workers RPC return limit - #45
Open
ahmedbadr-workday wants to merge 1 commit into
Open
fix: stream /metrics response to avoid 32 MiB Workers RPC return limit#45ahmedbadr-workday wants to merge 1 commit into
ahmedbadr-workday wants to merge 1 commit into
Conversation
The aggregated Prometheus payload returned from MetricCoordinator.export() can exceed the 32 MiB Workers RPC single-return-value limit on large deployments (many zones/hosts with EXCLUDE_HOST=false), causing GET /metrics to fail with HTTP 500 on every scrape. Return the payload as a ReadableStream<Uint8Array> instead of a string; a streamed body is exempt from the 32 MiB single-return cap. A new textToStream() helper encodes the text and emits ~1 MiB chunks split on newline boundaries so no chunk splits a multi-byte UTF-8 code point and we never hold a second full-size copy in memory. The worker returns the stream directly via new Response(). Output is byte-identical; only the transport changes. Adds unit tests for textToStream (round-trip, multi-chunk, multi-byte safety).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On accounts with a large metric surface (many zones/hosts,
EXCLUDE_HOST=false),GET /metricsintermittently returns HTTP 500 (Failed to collect metrics). The failure originates in the RPC return fromMetricCoordinator.export(): the serialized Prometheus payload exceeds the 32 MiB Workers RPC single-return-value limit (https://developers.cloudflare.com/workers/platform/limits/). Once the aggregate crosses ~32 MiB, every scrape 500s.Fix
Return the payload from
MetricCoordinator.export()as aReadableStream<Uint8Array>instead of astring— a streamed body is not subject to the 32 MiB single-return cap. A smalltextToStream()helper encodes the text and emits ~1 MiB chunks split on newline boundaries, so no chunk splits a multi-byte UTF-8 code point and we neverhold a second full-size copy of the payload in memory.
worker.tsxreturnsnew Response(stream, …)directly.Output is byte-identical to the previous string response; only the transport changes. No config or API-surface changes.
Changes
src/lib/prometheus.ts: addtextToStream()src/durable-objects/MetricCoordinator.ts:export()returns a UTF-8 byte streamsrc/worker.tsx: stream the responsesrc/lib/prometheus.test.ts: unit tests (round-trip, multi-chunk, multi-byte safety)Verification
bun run check(biome) cleanbun run typecheckcleanbun run test— newtextToStreamtests pass