VPN compose pins a camofox image that does not exist (ghcr.io/nimblecoai/camofox:latest) — never published, not an org-transfer break

[NimbleCoAI]

Summary

generateVpnCompose() defaults the camofox sidecar image to ghcr.io/nimblecoai/camofox:latest, which does not exist and never has. Every VPN-enabled agent therefore gets a docker-compose.yml pinned to an image that cannot be pulled — silently, at compose-generation time, with no error until docker compose up fails.

lib/services/harness-compose.ts:257:

camofox:
    image: ${camofoxImage || 'ghcr.io/nimblecoai/camofox:latest'}

This is not a pin broken by the NimbleCoAI → NimbleCoOrg org transfer. It was unpullable from the start — there is no camofox package under either namespace, at any point.

Verification

Two independent methods, both with controls.

1. Authenticated package listing (sees private packages, so it distinguishes absent from private):

namespace	kind	container packages
NimbleCoOrg	Organization	hermes-agent-mt (public) — no camofox
nimblecoai	User, not an org	hermes-agent, hermes-agent-mt (both public) — no camofox

2. Anonymous manifest probe against ghcr.io/v2/.../manifests/latest, validated against a known-public positive control and a known-absent negative control:

--- CONTROL (must be HTTP 200) ---
  nimblecoorg/hermes-agent-mt:latest -> HTTP 200  digest=sha256:4283f81ebb5b...
--- NEGATIVE CONTROL (must be denied) ---
  nimblecoorg/definitely-not-a-real-image-xyz:latest -> TOKEN DENIED
--- THE DEFAULT HARDCODED AT harness-compose.ts:257 ---
  nimblecoai/camofox:latest -> TOKEN DENIED
--- other plausible spellings ---
  nimblecoorg/camofox:latest         -> TOKEN DENIED
  nimblecoorg/camofox-browser:latest -> TOKEN DENIED
  nimblecoai/camofox-browser:latest  -> TOKEN DENIED
--- UPSTREAM jo-inc ---
  jo-inc/camofox-browser:latest -> HTTP 200  digest=sha256:64b30ffdbbc4ae0e28200a66dfbd6f55ac4188229eb34ef769afcf7be40faa6e
  jo-inc/camofox-browser:1.13.0 -> HTTP 200  digest=sha256:64b30ffdbbc4ae0e28200a66dfbd6f55ac4188229eb34ef769afcf7be40faa6e

Note GHCR's token endpoint returns 403 DENIED for absent and private repos alike, which is why method 1 is the load-bearing one for "absent".

Correction to an earlier internal note

A prior investigation concluded that no camofox image is published anywhere, including the upstream jo-inc namespace. That last part is wrong. ghcr.io/jo-inc/camofox-browser is a real, public, anonymously-pullable package with tags latest and 1.13.0 (same digest today). This materially changes the options below — we do not have to publish anything to have a working pin.

What the real deployment does instead

The camofox that actually works on the Mini is built locally and unprefixed: image camofox-browser, a local docker build of ~/Documents/GitHub/camofox-browser (upstream github.com/jo-inc/camofox-browser, third-party, read-only to us), started ad-hoc via docker run — not managed by any compose project (Config.Labels = {}, absent from docker compose ls).

So the working path and the code-generated path have never been the same thing. The generated sidecar (camofox-<agentName>) is a different container from the one in service.

(Provenance: this is documented from live docker inspect in nimbleco-memory specs/2026-06-25-camofox-profile-persistence.md. I could not re-confirm it by SSH in this session — the connection hung — so treat the local-build detail as corroborated-by-that-record rather than re-verified today. The registry findings above are all first-hand.)

Blast radius

• camofoxImage is never supplied by any production caller. The only call site that builds ComposeOptions — app/api/harnesses/[id]/settings/route.ts:693-701 — passes vpnEnabled, imageOrBuild, defaultImage, vncBindHost, controlBindHost, memory, cpus. No camofoxImage. Every non-test use hits the ghost default; there is no override exposed in settings or the UI.
• Reachable purely by flipping the VPN toggle in harness settings, which regenerates and overwrites docker-compose.yml (route.ts:702).
• The failure surfaces only at docker compose up, as a pull error on a sidecar — far from the toggle that caused it.

Why this can't be fixed by editing line 257 alone

Three test assertions currently pin the ghost string:

• lib/services/harness-compose.test.ts:142 — expect(result).toContain('image: ghcr.io/nimblecoai/camofox:latest')
• lib/services/__tests__/compose-image.test.ts:31 and :57 — pass it in as the camofoxImage fixture, :61 asserts it survives untouched

A previous agent correctly declined to change the code, because a naive edit yields green tests and a compose file still pinned to a nonexistent image. Any fix has to move the tests too, and should assert pullability semantics, not a literal string.

Options

1. Point the default at the real upstream image — ghcr.io/jo-inc/camofox-browser:1.13.0 (pin the version or the digest sha256:64b30ffd..., not latest). Cheapest correct fix; verified pullable today. Caveat: third-party image we don't control, so pin by digest and decide a bump policy.
2. Publish our own to ghcr.io/nimblecoorg/camofox-browser (a rebuild/mirror of the local docker build) and default to that. Most control, matches what actually runs on the Mini, costs a build+push pipeline.
3. Make the VPN path fail loudly instead of silently pinning a ghost. Require camofoxImage when vpnEnabled is true and throw otherwise, or surface it as a required field in harness settings. Good as a guardrail alongside 1 or 2 — the real defect is that an unpullable default was allowed to be silent.

My recommendation: 1 + 3 — digest-pin the upstream image so the default is true, and make an unset/unresolvable camofox image a hard error at generation time so this class of bug can't recur.

Related side bug (same block, flagged not fixed)

The sidecar mounts ${agentDataDir}/.camofox:/data but never sets CAMOFOX_PROFILE_DIR=/data. Env vars emitted at harness-compose.ts:263-271 are CAMOFOX_PORT, ENABLE_VNC, VNC_BIND, VNC_RESOLUTION, BROWSER_IDLE_TIMEOUT_MS only. So even with a pullable image the profile volume likely doesn't persist. Previously noted in the memory spec above; recording here so it isn't lost.

---

No code changed. Filed for a human decision on which option to take.