[Bug] CI: Linux test shards intermittently hang ~15 minutes and are killed, leaving an orphan bun process

[lidge-jun]

Client or integration

GitHub Actions — Cross-platform CI (ci.yml), Linux test N/4 shards

Area

Service lifecycle

Summary

Linux test shards intermittently stop producing output mid-run and are killed at the 15-minute job timeout. The run reports cancelled, not failure, so it reads as infrastructure noise — but it is reproducible enough across unrelated branches to be worth tracking rather than rerunning past.

Filing this because I hit it five times today while doing maintainer triage and kept treating it as flake. That was the wrong call: the hang is consistent in shape, and cancelled runs are easy to rerun into green without anyone noticing the underlying problem.

Observed pattern

Every occurrence looks the same:

1. the shard runs normally for ~30-90 seconds and passes hundreds of tests;
2. output stops abruptly right after a test that starts a proxy server;
3. nothing further is logged for ~14 minutes;
4. the runner cancels at the 15-minute timeout;
5. cleanup reports Terminate orphan process: pid (NNNN) (bun).

Across four runs on four unrelated branches, the affected shard varies, which argues against a single bad test:

Run	Branch	Shard	Result
31263738953	codex/260808-1185-windows-shard-assertion	test 4/4	cancelled (twice, incl. rerun)
31255199569	fix/windows-powershell-popup	test 2/4	cancelled, green on retry
31258815611	codex/260808-1195-unbound-quota-unknown	test 3/4	cancelled, green on retry
31152916419	agent/test-windows-ci-shard-command	test 3/4	failure — EEXIST: epoll_ctl in a Bun WriteStream

It also occurred on dev itself with no PR involved (runs 31259450263, 31259447622), so it is not contributor-branch specific.

The one non-cancelled instance is the useful clue

Run 31152916419 did not hang — it crashed outright while loading tests/autostart-health.test.ts:

# Unhandled error between tests
error: EEXIST: file already exists, epoll_ctl
      at new WriteStream (internal:fs/streams:244:58)
# collateral:
error: Cannot call describe() after the test run has completed

EEXIST from epoll_ctl means a file descriptor was registered with the event loop twice. That is the same class of resource-lifecycle problem a hang would produce if the registration deadlocked instead of erroring, which suggests the hang and the crash are one bug with two outcomes rather than two problems.

Last log line before the stall

In 31263738953 (test 4/4), the final output before 14 minutes of silence:

[web-search-loop] done — 1 search, 2 iterations, 13ms
[web-search-loop] cancelled — 1 real searches, 0 placeholders, 13ms
(pass) routed Claude requests give OpenAI sidecars main auth without leaking it to the routed provider

That test is in tests/claude-messages-endpoint.test.ts, and the stall is after it passes — so the suspect is teardown or the next file's setup, not the assertion. In the other runs the stall also follows a test that had just started a proxy listener.

Not reproducible locally

bun test tests/claude-messages-endpoint.test.ts → 38 pass / 0 fail in 2.7s on macOS. Full bun run test → 10009 pass / 0 fail. So this needs the sharded Linux runner environment; a single-file local run will not show it.

Why this matters beyond the noise

cancelled is not failure, and the natural response is to rerun. That works — three of the four went green on retry. The risk is that a genuine hang introduced by a future change is indistinguishable from this, and gets rerun into green by whoever is on triage that day.

Suggested starting points

• The orphan is always bun, and the stall always follows a proxy-server test, so an un-awaited server close or an fd registered twice across test-file boundaries is the first place to look.
• --isolate is already used; whether shard boundaries share any process-global listener state is worth checking.
• Raising the timeout would hide it. A shard that stops emitting for 14 minutes has already failed.

Version

dev at 14e948525.

Operating system

ubuntu-latest GitHub-hosted runners.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}

[lidge-jun]

Two more occurrences today, and one of them is the first case where two shards hung in the same run.

Run 31249781400 on PR #1256:

test 4/4  cancelled  18:44:24Z → 18:59:39Z   (15m15s)
test 3/4  cancelled  18:42:18Z → 18:57:33Z   (15m15s)

Both hit the 15-minute job timeout to the second, which is the same signature as every prior instance. Also #1301's run 31263738953, test 4/4, 15:28:02Z → 15:43:17Z — likewise 15m15s.

Updated tally: eight occurrences across six branches plus dev itself, with the affected shard varying between 2/4, 3/4 and 4/4. That two shards can hang in a single run makes a single bad test even less likely as the explanation.

The 15m15s figure is worth noting on its own: the job timeout is 15 minutes, so these are not tests that run long and finish late. The shard stops producing output and the runner kills it at the deadline — consistent with the orphaned bun process reported in cleanup rather than with slow work.

Practical effect on triage today: I approved and reran a batch of contributor PRs, and this is now the main source of non-green results among them. Six came back green; #1256 needed a rerun purely for this. Anyone doing PR triage will keep hitting it, and the risk stays what it was — a genuine hang from a real change is currently indistinguishable from this one, and reruns make both go away.

[lidge-jun]

A repeat at the same file, which narrows this considerably.

Run 31274685166 on PR #1301, test 3/4, 19:32:43Z → 19:47:59Z (15m16s). The last output before the silence:

##[group]tests/cli-native-profile.test.ts:
...
(pass) ocx account main > a stalled heartbeat is aborted before lease expiry, kills login, and cancels staging
##[endgroup]

##[group]tests/cli-restart-health.test.ts:
killed 1 dangling process          ← 19:33:18Z
                                    ← 14 minutes of nothing
##[error]The operation was canceled. ← 19:47:55Z
Terminate orphan process: pid (2482) (bun)

That is the second time the stall lands at tests/cli-restart-health.test.ts, immediately after cli-native-profile and immediately after that file logs killed 1 dangling process. The first was run 31152916419 (test 3/4, PR #1185's original head), which showed the identical sequence — including the same killed 1 dangling process line as the last thing emitted.

Earlier occurrences stalled after other proxy-starting tests, which is why I described the shard as varying. But the file repeating twice with the same preceding line is a stronger signal than the shard number varying, and it suggests a more specific place to look than "some proxy test":

• cli-native-profile spawns login children and cancels staging; cli-restart-health then kills a dangling process at startup.
• The orphan reported in cleanup is always bun.
• EEXIST: epoll_ctl in run 31152916419 was a descriptor registered twice — plausibly the same lifecycle fault when it errors rather than deadlocks.

Suggested next step for whoever picks this up: run bun test --isolate tests/cli-native-profile.test.ts tests/cli-restart-health.test.ts in that order under Linux, repeatedly, and watch whether a child from the former survives into the latter's dangling-process kill.

Tally is now nine occurrences. Not rerunning #1301 past it — I would rather this one stay red as evidence than be retried into green.

[lidge-jun]

Retracting the file-pair narrowing in my previous comment. It was wrong.

I claimed run 31152916419 showed the same cli-native-profile → cli-restart-health → killed 1 dangling process sequence as 31274685166. It does not. Checking its log again:

• 31152916419 hit EEXIST: ... epoll_ctl while loading tests/autostart-health.test.ts at 07:05:53, continued running, and finished as 1 fail / 2 errors across 2143 tests in 85 seconds. It contains no killed 1 dangling process line and was never cancelled. It is a failure, not a hang.
• 31274685166 (test(ci): bind the Windows shard assertion to an executable command (#1185) #1301) reports the same Bun epoll_ctl error while loading tests/api-storage.test.ts, then reaches the CLI pair and stalls there.
• My earlier hang capture (31263738953, also test(ci): bind the Windows shard assertion to an executable command (#1185) #1301) stalled after tests/claude-messages-endpoint.test.ts.

So three runs, three different first-affected files. The pair I pointed at is one observation, not a repeated signature, and I built a hypothesis on a comparison I had not actually made.

What survives, stated at the strength the evidence supports:

• a shared symptom — Bun EEXIST on epoll_ctl, a descriptor registered with the event loop twice — appearing across unrelated files;
• two distinct outcomes from it: the process errors and continues (a normal test failure), or it deadlocks and the runner kills it at 15 minutes with an orphan bun;
• no file-level culprit identified.

That is weaker than what I posted, and the earlier comment should be read as retracted rather than refined. My "the shard varies, so it is not one bad test" framing was also imprecise for a different reason — sharding distributes files differently per run, so a varying shard number never argued against a single file.

Apologies for the noise; better to correct it here than leave a wrong lead in the issue for whoever picks it up.

[lidge-jun]

Correcting my retraction — I named the wrong file for one run while retracting a claim about files, and overstated causation while correcting an overstatement.

Two different questions, which I had collapsed into one column. Where the EEXIST first appears is not where the output stops:

Run	First logged EEXIST under	Last output before silence	Outcome
31152916419	tests/autostart-health.test.ts	— (no silence)	kept running, finished 1 fail / 2 errors in 85s
31263738953	tests/baseten-provider.test.ts	tests/claude-messages-endpoint.test.ts	hung, cancelled at 15m
31274685166	tests/api-storage.test.ts	tests/cli-restart-health.test.ts	hung, cancelled at 15m

My retraction above said claude-messages-endpoint was the first affected file in 31263738953. It is the last file before the silence; the first EEXIST in that run is under baseten-provider. Neither column repeats across the three runs.

And the causal claim was still too strong. I wrote that the EEXIST has "two outcomes — error and continue, or deadlock and get killed". The logs show co-occurrence, not causation: the error appears in all three runs, one of which completed normally as a failure. Whether it causes the hang, shares a cause with it, or is incidental is not established.

The honest residue, third attempt:

• Bun EEXIST on epoll_ctl — a descriptor registered with the event loop twice — appears in all three runs, under a different file each time.
• Two of the three then stop emitting and are killed at the 15-minute job timeout, leaving an orphan bun.
• No file-level culprit, and no established causal link between the two facts.

Three passes to state three sentences accurately. Leaving the whole chain visible rather than editing it down, since the corrections are the useful part of the record.