Skip to content

fix(firmware): cancel hard-timeout token before throwing TimeoutException (fixes flaky test + ordering race) - #358

Merged
tylerkron merged 1 commit into
mainfrom
fix/wifibridge-hardtimeout-cancel-race
Jul 19, 2026
Merged

fix(firmware): cancel hard-timeout token before throwing TimeoutException (fixes flaky test + ordering race)#358
tylerkron merged 1 commit into
mainfrom
fix/wifibridge-hardtimeout-cancel-race

Conversation

@tylerkron

Copy link
Copy Markdown
Contributor

Summary

WifiBridgeActivator.RunWithHardTimeoutAsync had a subtle ordering gap that was intermittently failing CI (the RunWithHardTimeoutAsync_HardTimeoutElapses_StopsWorkerBeforeLateStep test) and, in production, could let an abandoned worker run one extra state-changing step after a timeout.

Root cause

Two independent timers of the same duration:

  1. Task.Delay(hardTimeoutMs) in the WhenAny race — decides when to throw TimeoutException.
  2. hardTimeoutCts's own timer — decides when the worker's linked token is cancelled.

Because they're separate timers, the Task.Delay can win a hair before hardTimeoutCts fires. In that window TimeoutException is thrown while the worker's linkedCts.Token is still un-cancelled — so a worker whose blocking Open() returns late sees an un-cancelled token and can perform a further step, defeating the guarantee #326 finding #2 added ("stop before performing a further state-changing step"). Under CI load the ordering varies, so the test flaked (observed failing on unrelated PRs #356 and #357).

Fix

The abandon path now calls hardTimeoutCts.Cancel() up front (idempotent if the timer already fired), so the worker's linked token is deterministically cancelled before TimeoutException propagates. No reliance on the two timers racing in a lucky order. Placement is before the existing abandon/observe-fault/dispose logic; disposal ownership and the caller-cancellation preference are unchanged.

Testing

  • dotnet test1635 passed / 0 failed / 2 skipped (net9.0 + net10.0); WifiBridgeActivator suite 17/17.
  • The previously-flaky RunWithHardTimeoutAsync_HardTimeoutElapses_StopsWorkerBeforeLateStep — which is the correct regression guard for this race — passed 30/30 consecutive runs after the fix.
  • No bench validation: this is firmware-activation timeout concurrency ordering, not device-facing behavior.

Not merging — for review.

🤖 Generated with Claude Code

…tion (fixes flaky test + ordering race)

RunWithHardTimeoutAsync decided to time out via Task.Delay(hardTimeoutMs) winning the WhenAny race,
but the worker's cancellation came from a SEPARATE hardTimeoutCts timer of the same duration. The
two timers can fire in either order, so the delay could win a hair before hardTimeoutCts, letting a
late-returning worker observe an as-yet-uncancelled token and run one further state-changing step
after the caller already received a TimeoutException — the exact guarantee #326 finding #2 added.

Now the abandon path calls hardTimeoutCts.Cancel() up front (idempotent), so the worker's linked
token is deterministically cancelled before TimeoutException propagates. This also fixes the
intermittent CI failures of RunWithHardTimeoutAsync_HardTimeoutElapses_StopsWorkerBeforeLateStep,
which caught this race under load (observed failing on unrelated PRs #356 and #357).

- Existing test is the regression guard (passed 30/30 locally after the fix); full suite 1635 pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tylerkron
tylerkron requested a review from a team as a code owner July 19, 2026 03:52
@tylerkron

Copy link
Copy Markdown
Contributor Author

/agentic_review

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Cancel hard-timeout CTS before throwing TimeoutException in WifiBridgeActivator

🐞 Bug fix 🕐 10-20 Minutes

Grey Divider

AI Description

• Cancel the hard-timeout CancellationTokenSource before propagating TimeoutException.
• Remove a race where the worker could observe an uncancelled token after timeout.
• Stabilize a flaky CI regression test by making timeout ordering deterministic.
Diagram

graph TD
  A["ActivateAsync / DeactivateAsync"] --> B["RunWithHardTimeoutAsync"] --> D["WhenAny race"] --> H["Cancel hard-timeout CTS + throw TimeoutException"]
  D --> E["Task.Delay(hardTimeoutMs)"]
  B --> C["workerTask (LongRunning)"] --> G["linkedCts.Token"]
  B --> F["hardTimeoutCts (timer)"] --> G
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Unify to a single timeout mechanism (CTS-driven)
  • ➕ Eliminates duplicated timers entirely (no Delay-vs-CTS ordering concerns).
  • ➕ Timeout detection and worker cancellation come from the same source by design.
  • ➖ More refactor risk in subtle abandonment/cleanup logic.
  • ➖ Harder to keep the existing ‘prefer caller cancellation’ semantics perfectly unchanged.
2. Use the CTS as the WhenAny signal (wait on token)
  • ➕ Keep hardTimeoutCts as the source of truth; WhenAny waits for its cancellation rather than a separate Delay(hardTimeoutMs).
  • ➕ Preserves existing token wiring while making ordering deterministic.
  • ➖ Slightly more complex to express cleanly (e.g., token-based task completion patterns).
  • ➖ Still requires careful handling of caller cancellation vs timeout precedence.

Recommendation: The current approach (explicitly calling hardTimeoutCts.Cancel() on the abandon/timeout path) is the lowest-risk fix: it preserves the existing control flow, disposal ownership, and the caller-cancellation precedence while deterministically closing the ordering gap. Consider a follow-up refactor to unify timeout signaling (CTS-driven) if this pattern appears elsewhere or grows more complex.

Files changed (1) +10 / -0

Bug fix (1) +10 / -0
WifiBridgeActivator.csCancel hard-timeout CTS before abandoning worker and throwing TimeoutException +10/-0

Cancel hard-timeout CTS before abandoning worker and throwing TimeoutException

• In RunWithHardTimeoutAsync, the timeout/abandonment path now explicitly cancels hardTimeoutCts before continuing with observe-fault/dispose logic and throwing TimeoutException. This removes a race where Task.Delay could win slightly before the CTS timer fired, allowing the worker to briefly observe an uncancelled linked token after the caller already timed out.

src/Daqifi.Core/Firmware/WifiBridgeActivator.cs

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Grey Divider

Qodo Logo

@tylerkron

Copy link
Copy Markdown
Contributor Author

Qodo review clean (Bugs 0), CI green, full suite 1635 pass, flaky test 30/30 after the fix. Ready for review.

@qodo-code-review

Copy link
Copy Markdown

review clean (Bugs 0), CI green, full suite 1635 pass, flaky test 30/30 after the fix. Ready for review.

The PR looks ready for review. The fix explicitly cancels hardTimeoutCts before throwing, closing the race where a late worker could perform another state-changing step. The clean bug scan, green CI, 1,635 passing tests, and 30/30 flaky-test runs provide strong validation.

@tylerkron
tylerkron merged commit 2ba972e into main Jul 19, 2026
1 check passed
@tylerkron
tylerkron deleted the fix/wifibridge-hardtimeout-cancel-race branch July 19, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant