[Bug]: Service restart rewrites the Codex catalog without handling stale app-servers, so routed models stay missing from the App picker

[lidge-jun]

Client or integration

Codex App

Area

Catalog / models

Summary

After the proxy rewrites the Codex model catalog on startup, routed models stay missing from the Codex App model picker until the long-lived Codex app-server process is killed by hand. The catalog file on disk is already correct at that point, so the state looks healthy from every disk-level check while the picker still shows only native models.

The stale-app-server problem itself is known and already handled for the CLI sync path (#476, #857). The gap is that the server startup path performs the same catalog/cache write without the accompanying app-server handling:

• src/cli/index.ts (sync, sync-cache) calls invalidateCodexModelsCache() and then afterCatalogWriteHandleAppServers({ restart }), which warns about stale app-servers or SIGTERMs them.
• src/server/index.ts:313 calls the same invalidateCodexModelsCache() on boot with no equivalent follow-up. afterCatalogWriteHandleAppServers is never referenced outside src/cli/index.ts.

So ocx service repair / a service restart rewrites models_cache.json and leaves an already-running app-server holding the previous in-memory list, with no warning emitted. The user sees a correct catalog on disk and an incorrect picker, and nothing in the output suggests why.

This is most visible in an SSH remote workspace, where the app-server is long-lived and boots independently of the app, but it is not SSH-specific.

Note that collectCodexAppServerCatalogState() already computes exactly the signal needed here (stale when an app-server start time predates the catalog mtime). It just is not consulted on the server startup path.

Reproduction

1. Install and run opencodex on a host with at least one routed provider configured (Anthropic OAuth in my case) and confirm the routed models are enabled.
2. Open that host as a workspace in the Codex App so a long-lived codex ... app-server --listen unix:// process is running.
3. On the host, change something that causes a catalog rewrite and restart the service: ocx service repair.
4. Confirm the on-disk state is correct:
   • ~/.codex/opencodex-catalog.json lists the routed models with visibility: "list"
   • ~/.codex/models_cache.json has the same entries
   • curl "http://127.0.0.1:10100/v1/models?client_version=<codex version>" returns them
   • codex exec --model <provider>/<routed-model> "..." answers normally
5. Open the model picker in the Codex App.

Actual: only the native gpt-* models are listed. The routed entries are absent even though every check in step 4 passes.

Expected: either the routed models appear, or the service restart emits the same stale-app-server warning the CLI sync path already produces (formatStaleCodexAppServerWarning).

Workaround that fixes it immediately:

pkill -f "app-server --listen unix://"
rm -f ~/.codex/app-server-control/app-server-control.sock

After the app-server restarts, the picker shows the routed models.

One secondary detail from the same session: the app-server left a stale app-server-control.sock behind, and the replacement process logged Error: app-server control socket is already in use on startup. It kept running, but the removal above was needed before the restart came up clean.

Version

2.10.0

Operating system

macOS 15 (Apple Silicon, Darwin 24.6.0)

Provider and model

anthropic / claude-opus-5, claude-fable-5 (routed as anthropic/claude-opus-5, anthropic/claude-fable-5)

Logs or error output

Service log after the restart, showing the catalog write with no stale-app-server warning:

🚀 opencodex proxy running on http://localhost:10100
[opencodex] Codex runtime: /Users/[USER]/.local/bin/codex (version=0.146.0, source=configured)
Pointed Codex's built-in openai provider at the opencodex proxy (openai_base_url).
  Codex model catalog: /Users/[USER]/.codex/opencodex-catalog.json
[opencodex] catalog sync: routed model fetch returned empty; preserving 11 existing routed entries on disk.

Catalog and cache agree on disk while the picker disagrees:

$ python3 - <<'PY'
import json
d=json.load(open("/Users/[USER]/.codex/opencodex-catalog.json"))
for m in d["models"]: print(m["slug"], m.get("visibility"))
PY
gpt-5.6-sol list
gpt-5.6-terra list
gpt-5.6-luna list
gpt-5.5 list
gpt-5.4 list
gpt-5.4-mini list
gpt-5.3-codex-spark list
anthropic/claude-fable-5 list
anthropic/claude-opus-5 list

Routing works end to end the whole time:

$ codex exec --model anthropic/claude-fable-5 --skip-git-repo-check "reply with exactly: FABLE_OK"
FABLE_OK

Stale app-server startup error mentioned above:

Error: app-server control socket is already in use at /Users/[USER]/.codex/app-server-control/app-server-control.sock

Screenshots and supporting files

Model picker after the catalog rewrite: only 5.5, 5.6 Sol, 5.6 Terra, 5.6 Luna, 5.4 Mini, 5.4, 5.3 Codex Spark are offered; no anthropic/* entries.

Redacted configuration

{
  "port": 10100,
  "providers": {
    "openai": {
      "adapter": "openai-responses",
      "baseUrl": "https://chatgpt.com/backend-api/codex",
      "authMode": "forward",
      "codexAccountMode": "pool"
    },
    "anthropic": {
      "adapter": "anthropic",
      "baseUrl": "https://api.anthropic.com",
      "authMode": "oauth",
      "defaultModel": "claude-sonnet-5"
    }
  },
  "defaultProvider": "openai",
  "codexAutoStart": true,
  "websockets": false,
  "disabledModels": ["anthropic/claude-sonnet-5", "..."]
}

(Most Anthropic models are intentionally in disabledModels; claude-opus-5 and claude-fable-5 are the two deliberately left enabled, and those are the two missing from the picker.)

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}

[lidge-jun]

Follow-up investigation with a controlled two-host comparison. This narrows the report above considerably, and also corrects one thing I got wrong.

The asymmetry that identifies the cause. The same Codex App, from the same client machine, opens SSH workspaces on two remote macOS hosts. Host B shows every routed model; host A shows only native gpt-*. Both run opencodex 2.10.0, SSH-session codex 0.146.0, identical openai_base_url / model_catalog_json injection, and both have the routed entries present with visibility: "list" in the catalog and in models_cache.json.

The only discriminator is the app-server start time relative to the catalog mtime:

host A (broken)   app-server --listen unix://   started 15:37:42
                  opencodex-catalog.json          mtime 15:42:09   <- catalog NEWER

host B (working)  app-server --listen unix://   started 14:09:10
                  opencodex-catalog.json          mtime 11:48:19   <- app-server NEWER

Host B's app-server booted after the last catalog write, so it read the finished file. Host A's catalog was rewritten 4m27s after its app-server booted, so the app-server serves an in-memory list that no longer exists on disk. Every check we ran reads the file; the picker renders memory.

Two things that are NOT the cause, both ruled out by the comparison:

• The app-server-control.sock collision I mentioned in the report — host B logs the identical control socket is already in use error and its picker is fine.
• The client_version: "0.0.0" / fetched_at: 2000-01-01 stamp in models_cache.json — that is the deliberate sentinel written by invalidateCodexModelsCache(), and both hosts carry it.

Why a proxy restart triggers it. The startup log order is literal:

🚀 opencodex proxy running on http://localhost:10100      <- src/server/index.ts:1100
   + 2 models appended to Codex catalog (...)             <- src/codex/sync.ts:93

ocx start awaits syncModelsToCodex(port) at src/cli/index.ts:319, well after startServer(). So a listening socket is not a signal that the catalog has settled, and every proxy start rewrites the catalog (host B's log shows + 29 / + 32 on its own restarts — this is normal, not something unusual about host A).

Why SSH is where it surfaces. Locally the app spawns and supervises the app-server, so quitting and reopening the app restarts it. Over SSH the app nohups codex ... app-server --listen unix:// on the remote host, detached (ppid 1), with an explicit pkill ... desktop-ssh-websocket-v0.sock guard — it is built to outlive a single connection, and reconnecting the workspace reuses it. So on a remote host the app-server routinely outlives both the app session and the proxy restart, and no user-visible action restarts it. The local fix ("quit and reopen") does nothing. That matches #851 being reported specifically against SSH workspaces.

The defect itself is surface-independent, though: any long-lived app-server predating a proxy restart yields a stale picker.

Correction to the original report. I initially suspected multiAgentMode being absent from the remote config, since the two hosts genuinely differed there. That was wrong: isEligibleV2SubagentEntry() (src/codex/catalog/sync.ts:70) allows an absent multi_agent_version, and the routed entries already had visibility: "list" before that config was touched. Setting it changed the catalog bytes without being load-bearing — the app-server restart that followed is what fixed the picker.

Suggested fix, unchanged in substance but better scoped. collectCodexAppServerCatalogState() already returns stale for exactly host A's numbers. Consult it on the startup path after a catalog/cache write and emit the existing formatStaleCodexAppServerWarning. Warning rather than terminating: a proxy start happens on login, on service repair, and on update, and SIGTERM-ing an app-server at those moments could kill an in-flight remote turn. Termination should stay opt-in via --restart-codex.

Worth considering as a second surface: ocx status was fully green on host A (including Catalog clamp: inactive) while the picker was wrong. It could report the same stale state.

Related: #476 (detection built), #857 (staleness classifier), #851 (SSH workspaces, closed on template grounds rather than resolved). Distinct from #241, where the catalog is correct and Desktop omits routed models anyway — that looks like a different, upstream failure and I would not conflate the two.

[lidge-jun]

Fixed on dev by PR #1072, merge commit 2a0cdfc. Startup catalog writes now detect and warn about stale app-servers instead of silently leaving routed models missing from the App picker (the restart path was deliberately not auto-triggered at boot — it SIGTERMs active turns; the warning names the stale processes). Ships with the next release train.