ci: capture BDD-target stderr to diagnose and stop the mTLS flake#576
Conversation
The BDD target is spawned with stderr=PIPE but nothing reads it — only stdout is teed by the reader thread. The target's error handler writes every error event to stderr, so under a retrying TLS/mTLS handshake failure (the ~1ms service loop retries thousands of times in the 10s oracle wait) the undrained pipe fills; on Windows the next write blocks the target inside Service, stalling it for the rest of the scenario and producing the intermittent "oracle received 0 of N messages" flake. The discarded stderr also meant we couldn't tell a real TLS failure from an environmental one. - _start_stderr_reader drains the target's stderr into a 16KB sliding buffer (mirrors _start_stdout_reader, shares its idempotency guard). Draining removes the pipe-full stall. - after_step dumps both stdout (GUEST:) and stderr (ERR:) on failure, so the client-side cause (handshake timeout / cert rejected / connection refused / fatal exit) is visible in the behave log. - scripts/repro-mtls-flake.ps1 loops the @Mtls scenario on a native Windows box (optional -Stress load), saving failing-run logs, to confirm the hypothesis before any library/timeout change. Test-side only — no production/library change. DEVLOG entry added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 23 minutes and 27 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
☀️ Quality Summary 🚦 build-linux-gcc: 100% successful (✔️ 1521 passed) Created by Quality Monitor v1.14.0 (#f3859fd). More details are shown in the GitHub Checks Result. |
…#577) The script used `taskkill /F /IM otelcol-contrib.exe 2>$null` to clear stale oracles, but with $ErrorActionPreference='Stop' (and PowerShell 5.1's handling of native-command stderr) the "process not found" message taskkill emits when nothing is running becomes a terminating error. On a clean box — no leftover otelcol — the startup cleanup throws before the oracle is ever started, which surfaced as "not finding the oracle". Replace both taskkill calls with a PowerShell-native, non-throwing kill: Get-Process otelcol-contrib -ErrorAction SilentlyContinue | Stop-Process -Force. Verified: 60/60 @Mtls iterations pass under -Stress on a native Windows box, confirming the stderr-drain fix (PR #576) holds under load. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Why
The
bdd-windows-otelmTLS scenario intermittently fails withoracle received 0 of 1 messages within 10 seconds. The signature doesn't fit a slow handshake: the service thread retriesSolidSyslog_Servicein a ~1 ms loop, so within the 10 s wait the sender retries thousands of times — staying at 0 of 1 means the target stalled or died, not "was slow."Root-cause hypothesis: the target is spawned with
stderr=subprocess.PIPEbut nothing reads it (only stdout is teed by the reader thread). The target's error handler writes every error event to stderr. Under a transient TLS/mTLS handshake failure that keeps retrying, the undrained pipe fills (~4–64 KB on Windows) and the next write blocks the target insideService→ stall →0 of N. It's intermittent because it only bites if the handshake hasn't succeeded before the pipe fills, and mTLS is the chattiest handshake. The discarded stderr is also why we currently can't tell a real TLS bug from an environmental flake.What
Test-harness only — no production/library change:
_start_stderr_readerdrains the target's stderr into a 16 KB sliding buffer (mirrors_start_stdout_reader, shares its idempotency guard). Draining removes the pipe-full stall.after_stepnow dumps both stdout (GUEST:) and stderr (ERR:) on failure, so the client-side cause (handshake timeout / cert rejected / connection refused / fatal_Exit) is visible in the behave log.scripts/repro-mtls-flake.ps1loops the@mtlsscenario on a native Windows box (optional-Stressload), saving failing-run logs — to confirm the hypothesis before deciding whether any library/timeout change (e.g. the 200 ms connect timeout, error-spew throttling) is warranted.Next
Run the repro script on the native Windows box; let the captured
ERR:output decide whether any production change is actually needed. DEVLOG entry included.🤖 Generated with Claude Code