Skip to content

test: harden TestE2E_PushDLQLifecycle against CI load#185

Open
7schmiede wants to merge 1 commit into
mainfrom
test/harden-dlq-lifecycle-flake
Open

test: harden TestE2E_PushDLQLifecycle against CI load#185
7schmiede wants to merge 1 commit into
mainfrom
test/harden-dlq-lifecycle-flake

Conversation

@7schmiede

Copy link
Copy Markdown
Member

Problem

TestE2E_PushDLQLifecycle intermittently fails on loaded CI runners (ubuntu-latest) with:

requeued message not re-delivered within 10s (deliveryCount=3, before=3)

Root cause is a TOCTOU race in the test harness, not a too-short deadline. The re-delivery check snapshotted beforeRequeueDeliveries := deliveryCount.Load() after pd2.Start(), so under load pd2 could re-deliver the requeued message (n=3 β†’ success β†’ count becomes 3) before the snapshot was taken. The busy-poll then waited forever for count > 3, which explains the before=3, deliveryCount=3 timeout. Simply raising the deadline would not have fixed this.

Change

  • Replace the racing sleep-poll with a deterministic signal: the target handler does a non-blocking send on a buffered channel on the successful (n>=3) delivery β€” which is exactly the re-delivery of the requeued message. The test waits on that channel with a 30s safety bound instead of racing a fixed wall-clock budget.
  • Remove the racy beforeRequeueDeliveries snapshot.
  • Bump the downstream queue-empty deadline 5s β†’ 15s so the flake isn't relocated one assertion down.

The assertion stays meaningful β€” the requeued message must be re-delivered, or the test fails.

Scope

Test-harness robustness only. No dispatcher behavior change.

Verification

  • go vet ./internal/e2e/ β€” clean
  • go test ./internal/e2e/ -run TestE2E_PushDLQLifecycle -count=5 β€” passes all 5

πŸ€– Generated with Claude Code

The re-delivery check snapshotted deliveryCount *after* pd2.Start(),
so under load pd2 could re-deliver the requeued message before the
snapshot was taken, leaving the busy-poll waiting forever (observed
"deliveryCount=3, before=3" timeout on loaded runners).

Replace the racing sleep-poll with a deterministic signal: the target
handler sends on a buffered channel on the successful (n>=3) delivery,
which is exactly the re-delivery of the requeued message. Wait on that
channel with a 30s safety bound instead of a fixed wall-clock budget.
Also bump the downstream queue-empty deadline 5s -> 15s so the flake
is not relocated there.

Test-harness robustness only; no dispatcher behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant