Native-profile lock deadlock: all native requests 503 with zero profiles registered, and account main recover cannot clear it

[MarcusNeufeldt]

Client or integration

Codex CLI

Area

Authentication and account pool

Summary

The native-main-profile lock can enter a state where it is never released, and every native-routed request then fails with 503 server_is_overloaded / "Native Codex main profile is switching; retry this request".

The lock is held even when no native profiles are registered at all (profiles: [], activeProfileId: null, and an empty <CODEX_HOME>/.opencodex-native-main-profiles/ directory). The subsystem gates all traffic for a feature that has nothing configured in it.

The documented recovery command, ocx account main recover, cannot clear it: that command is a client of the proxy's management API, and the management API refuses while the lock is held. Stopping the proxy releases the lock at the filesystem level, but then every management command fails with Proxy is not running. There is no out-of-band break-glass path, so the state is unrecoverable through supported commands.

I expected either that the lock would not be taken when no profiles exist, or that ocx account main recover would be able to clear it.

Secondary usability problem: returning code: server_is_overloaded makes the Codex CLI render "Selected model is at capacity. Please try a different model." That points users at OpenAI quota exhaustion. It took a fair amount of digging to establish the request never left the machine — durationMs of 27-45ms and usageStatus: "unreported" were the giveaways.

Reproduction

Not consistently reproducible on demand; it appeared spontaneously and then recurred in bursts. What I observed:

1. ocx start with codexAccountMode: pool, a single Codex account (main, the Codex App login), and no native main profiles registered.
2. Send any request through the proxy to a native model, e.g. POST http://127.0.0.1:10100/v1/responses with {"model":"gpt-5.4-mini", ...}.
3. Observe 503 returned in single-digit milliseconds with Native Codex main profile is switching.
4. Attempt the documented recovery — it fails on the same lock:

$ ocx account main doctor --json
Error: The native-profile lock is unavailable.

$ ocx account main recover --json
Error: The native-profile lock is unavailable.

$ ocx account main list --json
{"effectiveCodexHome":"<CODEX_HOME>","activeProfileId":null,"profiles":[]}

5. ocx restart clears the currently active burst but does not prevent recurrence.

Observed burst pattern over ~3 hours, across a proxy restart. Note the bimodal durations (0-5s vs 211-268s), which look like lock contention rather than a crash:

start     end       dur    gap-since-prev-start
06:42:15  06:46:43  268s   -
06:49:04  06:49:07    3s   7m
06:59:20  07:00:37   77s   10m
07:07:48  07:07:51    3s   8m
07:12:55  07:14:22   88s   5m
07:16:34  07:16:52   18s   4m
07:32:43  07:32:58   15s   16m
07:43:55  07:43:58    3s   11m
08:22:51  08:22:54    3s   39m
08:33:04  08:33:09    5s   10m
08:36:08  08:36:38   30s   3m
08:41:16  08:41:16    0s   5m
   --- ocx restart here (08:43) ---
09:38:06  09:41:36  211s   57m
09:44:31  09:44:31    0s   6m

One correlation worth checking: a codex-quota-cache.json write landed at 09:38:19, 13 seconds after that burst began, which suggests the quota-refresh path takes the same lock.

Workaround that resolved it for me: ocx provider account-mode direct. With a single Codex account, pool mode adds nothing, and direct mode keeps the profile-switching layer out of the request path. Verified working end-to-end afterwards; no recurrence since.

Note the catch-22 in applying it: setting account-mode requires the proxy to be running, while clearing the lock requires it to be stopped. The window to apply the workaround is whatever clean interval happens to occur.

Suggested fixes

1. Do not acquire the native-profile lock when no profiles are registered. If profiles is empty and activeProfileId is null, the subsystem should be inert.
2. Give recover an out-of-band path that does not depend on the proxy's management API, e.g. an offline --force operating on the lock file directly while the proxy is stopped. A recovery tool gated behind the thing it recovers cannot function.
3. Add a lock lease/timeout so a holder that never completes releases automatically.
4. Use a distinct error code instead of server_is_overloaded, or name opencodex in the message, so users are not sent to investigate OpenAI quota.
5. Surface this in ocx doctor, which reported OAuth health: ok while all traffic was failing.

Candidate source locations from code search: src/codex/native-profile-manager.ts, src/codex/native-profile-store.ts, src/codex/native-main-owner.ts, src/codex/native-main-claim.ts, src/codex/native-profile-startup.ts, src/server/management-api.ts, src/cli/doctor.ts.

Possibly a regression from the native-main-login work in #656.

Version

2.10.1 (also inspected dev at v2.10.2; no native-profile changes in that release, so this should still reproduce)

Operating system

Windows 11 Pro 10.0.26200

Provider and model

openai (native Codex) / gpt-5.4-mini, gpt-5.6-luna, gpt-5.6-terra — all native-family models failed identically

Logs or error output

# usage.jsonl record for a failing request
{
  "provider": "openai",
  "model": "gpt-5.4-mini",
  "admissionKind": "loopback",
  "inboundProtocol": "responses",
  "status": 503,
  "durationMs": 32,
  "usageStatus": "unreported",
  "errorCode": "server_is_overloaded",
  "upstreamError": "Native Codex main profile is switching; retry this request",
  "routeDecision": { "routeKind": "native" }
}

# reproduced directly against the proxy
POST http://127.0.0.1:10100/v1/responses
-> HTTP 503 in 0.0068s
{"error":{"message":"Native Codex main profile is switching; retry this request",
          "type":"server_error","code":"server_is_overloaded"}}

# recovery blocked by the same lock
$ ocx account main doctor --json
Error: The native-profile lock is unavailable.

$ ocx account main recover --json
Error: The native-profile lock is unavailable.

# nothing registered in the subsystem holding the lock
$ ocx account main list --json
{"effectiveCodexHome":"<CODEX_HOME>","activeProfileId":null,"profiles":[]}

# lock file present and actively touched
<CODEX_HOME>/.opencodex-native-profile.lock.sqlite   8192 bytes
<CODEX_HOME>/.opencodex-native-main-profiles/        (empty)

Screenshots and supporting files

No response

Redacted configuration

{
  "providers": {
    "openai": {
      "adapter": "openai-responses",
      "baseUrl": "https://chatgpt.com/backend-api/codex",
      "codexAccountMode": "pool",
      "authMode": "forward"
    }
  },
  "defaultProvider": "openrouter",
  "openaiProviderTierVersion": 2,
  "activeCodexAccountId": "__main__"
}

Account pool state at the time: a single Codex account (main, Codex App login). auto-switch: on (threshold 80%) with actual weekly usage around 47%, so the threshold was never approached.

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[github-actions]

_{Automated translation bookkeeping — detected language: English.}

[Ingwannu]

Confirmed as a real local admission/locking bug on current dev (bbd82e7).

The lock file existing on disk is expected and is not itself proof that a transaction is held. The actionable code path is the startup/stage-sweep path: startNativeMainStartupLifecycle runs manager.sweepStages() under the native-profile SQLite transaction even when there is no stage registry and no staging tree. A transient lock or Windows ACL failure then makes runOwnedStageSweep fail, leaves the native-main gate in stage-cleanup-required/recovery state, and native requests surface the misleading 503. doctor and recover subsequently contend on the same profile transaction, which creates the reported catch-22.

I am preparing a focused fix that will:

• add a lock-free no-artifact fast path when neither a stage registry nor a staging tree exists;
• preserve the current fail-closed locked sweep whenever any stage artifact exists or the path cannot be proven absent;
• keep owner/claim protection for the physical native-main credential; this will not weaken cross-process credential ownership;
• add regression coverage proving a zero-profile/zero-stage installation cannot be fenced by an unavailable stage-sweep lock, while real or unreadable stage state still fails closed;
• improve the gate diagnostic so a local profile-maintenance fence is not presented as upstream model capacity.

I will link the dev-targeted PR here after the focused tests, typecheck, and privacy scan pass.

[Ingwannu]

I opened draft PR #1124 against dev with the absence-only fast path described above.

The change skips the native-profile stage transaction only when both the stage registry and this instance's staging tree are proven absent. Any present artifact or observation error still enters the existing locked, fail-closed cleanup path. It also makes the 503 identify local OpenCodex native-profile maintenance instead of looking like upstream capacity pressure.

Focused tests pass 51/51, along with typecheck and the privacy scan. The full suite passed 9,417 tests and hit one load-sensitive failure in the Live/Realtime sideband test; that exact case then passed seven consecutive isolated/focused executions. I documented that result in the PR rather than presenting the run as fully green.

Because this touches the native-profile/authentication boundary, I requested review from @lidge-jun and @Wibias. I am leaving this issue open and the PR in draft until review and CI complete.

[BeniSisaki]

I hit this exact symptom on 2.10.2 (Windows 11, pool mode, zero profiles) and instrumented the proxy to find the concrete mechanism — it may be the missing root cause here: the native-main owner acquisition fails permanently closed because the two startup icacls calls (ACL hardening) exhaust the shared 5s OPENCODEX_ACL_TIMEOUT_MS budget, and the non-busy error branch in drive() publishes terminal unavailable with no retry.

Captured timings on my host: first icacls exit 0 after ~2.0s, second needed ~11.2s wall time in the daemon's startup window (same harden completes in ~230ms in a standalone bun process), so the shared deadline is always exhausted → gate blocked owner-unavailable → all native requests 503 server_is_overloaded → Codex renders "Selected model is at capacity". 37/37 requests failed while /healthz and ocx status were green.

Workaround that fully recovered the proxy for me: setx OPENCODEX_ACL_TIMEOUT_MS 60000 + ocx restart — doctor went healthy and a real gpt-5.6-luna request through the proxy returned 200. Full evidence, reproduction, and fix suggestions in #1156.

[Wibias]

Closing as fixed on dev.

#1134 fixes the zero-profile startup case by skipping the native-profile stage sweep only when both the stage registry and staging tree are provably absent.

Existing fail-closed behavior is preserved when stage state exists or cannot be verified, and the misleading upstream-capacity-style 503 was also corrected.

Related Windows ACL issues such as #1156 remain separate and open.