refactor(saas): take two serialized apiserver round trips off the hosted create path

[stubbi]

Thinking Path

• Mitos boots Firecracker microVMs and forks them via copy-on-write snapshots, exposed through CRDs; the hosted gateway turns a public create into a Sandbox CR and a warm-pool claim.
• The hosted time-to-interactive is ~308 ms P50 (bench/results/2026-07-10-tti-hosted.md), and the measured gap to the fastest peers is control-plane cost, not engine cost: prod claim stage logs (24h, n=33 Ready claims) put the controller reconcile at 137 ms P50, and the gateway adds serialized apiserver round trips around it.
• Two of those round trips are avoidable without touching any correctness gate: the readiness watch was established only AFTER the Sandbox Create, which forced an authoritative re-read to close the flip-before-watch window, and the typo fast-fail pool pre-check re-read the same stable hosted pool on every create.
• The engine share is being removed separately by prepare-time restore (docs/superpowers/plans/2026-07-10-prepare-time-restore.md); the create-path round trips are the complementary lever and this serves the ROADMAP performance workstream.
• This pull request establishes the ready watch on the client-generated name BEFORE the Create (the create event arrives on the stream, so the re-read disappears and the race window no longer exists), and caches POSITIVE pool pre-check results for 30s (absence is never cached).
• The benefit is two fewer serialized apiserver round trips on every hosted create, with the flip-before-watch race closed structurally instead of patched over.

Linked Issues or Issue Description

Related: #871 (the TTI gap tracking issue names control-plane cost as the remaining lever). The reservation-protocol follow-up for the remaining serialized write is filed as #893.

What Changed

• internal/saas/controlplane/readywatch.go: split watchReady into establishSandboxWatch plus watchWait; watchReady keeps its post-create shape (establish, authoritative Get, wait) for waits whose object predates the watch, such as fork.
• internal/saas/controlplane/forward.go: create() establishes the watch before k.c.Create and waits on it directly; any establish failure falls back to the exact previous behavior (post-create watch, then poll). A stream that closes mid-wait still fails open to the ticker with the remaining deadline budget.
• internal/saas/controlplane/forward.go and controlplane.go: positive pool-existence results are cached per namespace/pool for poolCheckTTL (30s); absence is never cached, so a typo still 404s instantly and authoritatively every time.
• Tests: TestWatchDrivenCreateNeedsNoSandboxGet pins the round-trip count (zero Sandbox Gets on the watch-driven create), TestReadyFlipDuringCreateWriteIsStillSeen pins the race the removed Get used to guard, and poolcheck_test.go pins cache hit, no-negative-caching, and TTL expiry.

Verification

• go test ./internal/saas/... -race (all packages pass)
• make test-unit
• go build ./...
• golangci-lint run --timeout=5m ./internal/saas/... AND GOOS=linux golangci-lint run --timeout=5m ./internal/saas/... (both clean)
• TDD: both behavior tests were watched failing first (the no-Get test failed with 1 Get; the cache test failed with 2 Gets)
• No public latency number is claimed here; the effect on hosted TTI gets measured with bench/tti-latency.py after the next roll, per the no-unverified-claims rule.

Risks

• A pool DELETED within the 30s TTL window: a create naming it proceeds past the fast-fail and lands on the controller's bounded grace (fix(controller): sandbox with a nonexistent pool fails terminally after a bounded grace period #637), returning the ready-timeout envelope instead of an instant 404. That is exactly the behavior of the direct (non-gateway) path today; pools that never existed are unaffected (absence is never cached).
• Watch establishment on a not-yet-existing name relies on standard apiserver watch semantics (events begin at creation); on ANY establish failure the code falls back to the previous post-create order, so the failure mode is the status quo.
• Slow etcd or apiserver: one fewer read per create strictly reduces load; the watch lifetime is unchanged (bounded by the create request).

Model Used

• Claude Fable 5 (Anthropic), model id claude-fable-5, extended thinking enabled.

Checklist

• PR title is a conventional commit (feat, fix, docs, ci, chore, refactor, test)
• Thinking Path traces from project context to this change
• Model Used is filled in (with version and capability details)
• Tests added for behavior changes, in the same commit (TDD)
• Docs updated in the same PR (no user-facing docs describe the pre-check internals; API behavior envelopes are unchanged)
• Threat-model delta (docs/threat-model.md) included if the security surface moved (it did not: same client, same verbs, same RBAC, org namespacing keys the cache)
• Benchmark run (bench/) included if the hot path was touched (no number is claimed; prod TTI re-measurement follows the roll, noted in Verification)
• No em or en dashes introduced anywhere
• Secret values never logged, in errors, in condition messages, or on host paths
• No internal/instance-local references (only public #NNN / mitos-run/mitos URLs)
• Every commit carries a Signed-off-by trailer (git commit -s)

🤖 Generated with Claude Code

Summary by CodeRabbit

• Performance
  • Reduced repeated pool existence checks during sandbox creation by caching confirmed pool availability for a short TTL.
  • Never caches unknown/nonexistent pools; cached confirmations expire and are revalidated.
• Reliability
  • Improved sandbox readiness handling by preferring watch-driven event tracking when available, with safe fallback polling.
  • Ensures readiness changes occurring during creation are reflected promptly.
• Tests
  • Added/expanded coverage for pool pre-check caching (including namespace scoping and cache expiry) and watch-driven readiness behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 79b65288-53ba-4175-8689-d89163cf6b4a

📥 Commits

Reviewing files that changed from the base of the PR and between 5b1ed08 and fbf9772.

📒 Files selected for processing (1)

• internal/saas/controlplane/poolcheck_test.go

---

📝 Walkthrough

Walkthrough

Sandbox creation now caches positive pool checks for 30 seconds and establishes readiness watches before creating sandboxes. Readiness waiting falls back to polling when needed, with tests covering cache behavior and watch-driven readiness.

Changes

Sandbox create flow

Layer / File(s)	Summary
Pool pre-check cache internal/saas/controlplane/controlplane.go, internal/saas/controlplane/forward.go, internal/saas/controlplane/poolcheck_test.go	Pool existence checks are cached positively by namespace and name, expire after poolCheckTTL, and continue rechecking missing pools.
Readiness watch setup internal/saas/controlplane/readywatch.go	Watch creation is extracted into establishSandboxWatch, while initial sandbox state and phase tracking are prepared before watchWait.
Watch-first create integration internal/saas/controlplane/forward.go, internal/saas/controlplane/readywatch_test.go	The create flow establishes a readiness watch before Create, waits for terminal outcomes, and falls back to polling when watches fail or close early. Tests cover zero sandbox reads and readiness changes during creation.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant CreateFlow
  participant KubernetesWatch
  participant KubernetesAPI
  Client->>CreateFlow: sandbox.create
  CreateFlow->>KubernetesAPI: Check SandboxPool
  CreateFlow->>KubernetesWatch: establishSandboxWatch
  CreateFlow->>KubernetesAPI: Create Sandbox
  KubernetesAPI-->>KubernetesWatch: Sandbox readiness event
  KubernetesWatch-->>CreateFlow: Terminal readiness outcome
  CreateFlow-->>Client: Forward response

Loading

Possibly related PRs

• mitos-run/mitos#646: Also changes the SandboxPool existence check in the sandbox create flow.
• mitos-run/mitos#734: Also implements watch-based sandbox readiness and polling fallback.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is conventional, concise, and accurately summarizes the main hosted create-path performance change.
Description check	✅ Passed	The description matches the required template with a thinking path, linked issue, changes, verification, risks, model, and checklist.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/gateway-create-roundtrips

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/saas/controlplane/poolcheck_test.go`:
- Around line 53-124: Add a regression test alongside
TestRepeatCreateSkipsThePoolPreCheckRoundTrip that creates the same pool name in
two different namespaces or organizations, verifies each namespace performs its
own positive pre-check, and confirms the cache entry from the first namespace is
not reused by the second. Reuse existing helpers such as poolGetCountingClient
and createOnce, and assert the total SandboxPool Get count reflects two isolated
cache keys.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: afe09c5e-7050-44e7-8985-a42eb134548e

📥 Commits

Reviewing files that changed from the base of the PR and between 0d1e881 and 5b1ed08.

📒 Files selected for processing (5)

• internal/saas/controlplane/controlplane.go
• internal/saas/controlplane/forward.go
• internal/saas/controlplane/poolcheck_test.go
• internal/saas/controlplane/readywatch.go
• internal/saas/controlplane/readywatch_test.go