Skip to content

fix: Telegram persistence queue backpressures the engine and can force stream watchdog timeouts#873

Closed
sam-saffron-jarvis wants to merge 1 commit into
SamSaffron:mainfrom
sam-saffron-jarvis:feat/codereview-e18eecb9
Closed

fix: Telegram persistence queue backpressures the engine and can force stream watchdog timeouts#873
sam-saffron-jarvis wants to merge 1 commit into
SamSaffron:mainfrom
sam-saffron-jarvis:feat/codereview-e18eecb9

Conversation

@sam-saffron-jarvis

Copy link
Copy Markdown
Contributor

What changed

  • Made the Telegram callback persistence queue fail fast instead of blocking the engine callback path when its 128-slot buffer is saturated.
  • Marked the queue as degraded on saturation/cancellation, stopped draining the remaining buffered callback writes in that degraded mode, and reconciled persistence with a final ReplaceMessages snapshot.
  • Moved Telegram turn metrics/context-estimate persistence out of the callback queue and into a bounded end-of-run persistence step so queue pressure comes primarily from message writes, not extra metadata writes.
  • Added queue tests covering the new non-blocking full-queue behavior, canceled-context behavior, and close semantics.

Why this is high-value

The old code did a blocking channel send from synchronous engine callbacks. Under store contention or tool-heavy turns, once the queue filled, callback execution blocked, stream event consumption stopped, and the Telegram watchdog could cancel healthy runs as if they had gone idle.

This change removes that engine-path backpressure. When the queue cannot accept more work, Telegram now degrades to a bounded, best-effort path: the stream continues, the queue is marked for reconciliation, and the final transcript is repaired with a snapshot instead of stalling the live response.

Validation

  • Verified the issue in current code: telegramStoreOpQueue.enqueue previously did a plain blocking q.ops <- storeOp into a fixed-size channel from synchronous engine callbacks.
  • gofmt -w internal/serve/telegram.go internal/serve/telegram_queue_test.go
  • go build ./...
  • go test ./...
  • git diff --check

@SamSaffron

Copy link
Copy Markdown
Owner

Implemented in local copy: Telegram callback persistence queue now fails fast/degrades under saturation or cancellation, stops draining buffered callback writes when degraded, reconciles via final ReplaceMessages snapshot, and persists metrics/context estimate at end of run. Verified with go build ./... and targeted tests.

@SamSaffron SamSaffron closed this Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants