feat(saas): pre-claimed checkout, the gateway hands out an already-activated sandbox

[stubbi]

Thinking Path

• Mitos boots Firecracker microVMs and forks them via copy-on-write snapshots; the hosted gateway turns a public create into a Sandbox CR and a warm-pool claim.
• The measured hosted TTI budget (bench/results/2026-07-10-tti-hosted.md, plus prod claim stage logs) shows the remaining create cost is the claim round trip itself: CR write, watch wake-ups, reconcile, activate RPC, status writes. The peers ahead of us on the ComputeSDK table do not pay any of this per request; their sequential number is a warm-pool checkout (their burst ratios prove it).
• Prepare-time restore (refactor(husk): restore and resume the guest while dormant, not while a tenant waits #892) removes the engine share, and refactor(saas): take two serialized apiserver round trips off the hosted create path #895 removed two gateway round trips; the claim round trip itself is the last structural chunk.
• This pull request gives mitos the same checkout shape the fast peers have: the gateway keeps a small buffer of REAL, already-activated, org-less Sandboxes (mitos.run/buffered label, no org labels), and an eligible create pays ONE resourceVersion-guarded label patch that atomically drops the buffered label and stamps the org labels: replica mutual exclusion and claim-time billing attribution in the same write.
• The controller propagates the org to the backing husk pod (the TRUSTED billing label) on the reconcile that very patch triggers, filling only an EMPTY pod label so control-plane-derived attribution is never overridden.
• The benefit is a create hot path of gateway front + one patch for eligible creates, with every miss (feature off, ineligible body, empty buffer, lost race) falling back to the classic path unchanged. Default OFF.

Linked Issues or Issue Description

Related: #871 (the TTI gap is control-plane cost). Design spec: docs/superpowers/specs/2026-07-11-preclaimed-checkout-design.md (decision record, failure behavior, per-org-namespace migration note); implementation plan: docs/superpowers/plans/2026-07-11-preclaimed-checkout.md. Both are in this PR.

What Changed

• internal/saas/controlplane/checkout.go (new): the checkout buffer. Eligibility gate (checkout-enabled pool, no env, secrets, workspace, fan-out, or TTL), claim (pop + one optimistic-lock label patch + cached 201), refill to a floor through the normal create machinery with capped backoff on failure, adopt-on-restart by LIST (the CRs are the source of truth), staleness reap at maxAge, and a 15s reconcile loop.
• internal/saas/controlplane/forward.go: create() consults the buffer after pool resolution; createSandboxAndAwait extracted so the refill shares the watch-before-create machinery from refactor(saas): take two serialized apiserver round trips off the hosted create path #895.
• internal/saas/controlplane/controlplane.go: WithCheckout option; the buffer only constructs in single-tenant namespace mode (the design leans on the shared namespace; the spec records the per-org-namespace migration story).
• internal/controller/huskdrain.go + sandboxclaim_controller.go: propagateOrgToHuskPod fills an EMPTY husk pod org label from a Ready claim's org label during the Ready-claim reconcile; never rewrites an existing one (threat-model invariant).
• cmd/gateway/main.go: --checkout-pools/floor/cap/max-age flags (env-overridable), startup wiring, and a LOUD warning when checkout pools are set without single-tenant mode (the buffer stays off by design there).
• deploy/charts/mitos: gateway.checkout values + strict schema entries + conditional args; gateway ClusterRole gains patch on sandboxes (the one hot-path write; still no pod writes, no Secret writes).
• docs/threat-model.md: a full row for the org-less buffered Ready state (unreachable pre-checkout because getOwned matches no org; no tenant data in a buffered VM by the eligibility gate; token custody duration bounded by maxAge; exclusive handout via the RV-guarded patch; billing attribution at claim, propagated fill-only).

Verification

• TDD throughout: every behavior test was watched failing first (eligibility, served-from-buffer with zero hot-path Creates, cross-replica exclusivity via the RV-guarded patch, empty-buffer fallback, refill floor/cap, backoff after failure, adopt, prune-failed, reap-over-age, org propagation incl. the never-override case)
• go test ./internal/saas/... -race
• eval $(setup-envtest use 1.31 -p env) && go test ./internal/controller/ (full envtest suite)
• make test-unit and go build ./...
• golangci-lint run AND GOOS=linux golangci-lint run over the changed packages (repo-wide darwin run flags pre-existing internal/fork findings from main, untouched here)
• helm template renders with checkout off (default) and on (--set gateway.checkout.pools={python}); values.schema.json accepts the new keys
• No public latency number is claimed. Prod acceptance after the roll: bench/tti-latency.py before/after (the 2026-07-11 pre-change baseline is recorded: P50 327.2 ms, create median 189.3, first exec 136.8, N=20, 20/20), plus one burst-shaped run to measure the buffer-miss fallback honestly, results committed to bench/results/.

Risks

• Billing gap: a buffered sandbox bills nobody until claimed; deliberate, bounded platform cost (floor 2 / cap 4 / maxAge 10m per enabled pool) and stated in the threat model. A claimed sandbox's pod label lands via the controller reconcile the checkout patch triggers; usage windows are 60s so attribution is not materially delayed.
• Capacity: each buffered sandbox holds a warm-pool slot and its working set (~72 MiB measured on the lazy path). floor 2 on the single prod KVM node is about one extra dormant pod equivalent.
• A husk dying while buffered: pruned at the next 15s pass; a pop racing the death surfaces to the tenant like any post-create death today. No per-pop health probe by design (it would re-serialize a round trip).
• Gateway crash or restart: CRs are the truth; replicas re-adopt by LIST, nothing outlives maxAge. Slow etcd: one patch inherits it; the classic fallback inherits it worse. Capacity exhaustion: refill fails like any create and backs off; the hot path falls back classic.
• Default OFF; enabling requires naming pools in gateway.checkout.pools AND single-tenant namespace mode.

Model Used

• Claude Fable 5 (Anthropic), model id claude-fable-5, extended thinking enabled.

Checklist

• PR title is a conventional commit (feat, fix, docs, ci, chore, refactor, test)
• Thinking Path traces from project context to this change
• Model Used is filled in (with version and capability details)
• Tests added for behavior changes, in the same commit (TDD)
• Docs updated in the same PR (design spec + implementation plan + chart values comments)
• Threat-model delta (docs/threat-model.md) included if the security surface moved (full new row)
• Benchmark run (bench/) included if the hot path was touched (feature is default OFF so this PR changes no hot path by itself; the pre-change baseline is recorded above and the post-roll before/after lands in bench/results/ with the enabling roll)
• No em or en dashes introduced anywhere
• Secret values never logged, in errors, in condition messages, or on host paths
• No internal/instance-local references (only public #NNN / mitos-run/mitos URLs)
• Every commit carries a Signed-off-by trailer (git commit -s)

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added an optional pre-claimed checkout buffer for eligible requests, configurable by checkout pools with buffer floor/cap and max age.
  • Updated gateway startup and Helm values to enable the feature when configured (default off) and enforce required single-tenant namespace constraints.
  • Added control-plane logic to serve from the buffer, refill toward the floor, and recover after gateway restarts.
• Bug Fixes
  • Improved org attribution by propagating the claim’s org label onto the backing husk pod when needed.
• Documentation
  • Added/updated design, plan, and threat-model documentation for pre-claimed checkout.
• Tests
  • Added unit tests covering buffer eligibility/claiming and org propagation behavior.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds an optional pre-claimed checkout buffer for eligible sandbox creates, including control-plane reconciliation, atomic claims, gateway and Helm configuration, RBAC patch access, and husk pod organization-label propagation.

Changes

Pre-claimed checkout

Layer / File(s)	Summary
Checkout contract and control-plane integration internal/saas/controlplane/..., docs/superpowers/...	Defines checkout configuration, single-tenant gating, shared sandbox creation, lifecycle behavior, and verification requirements.
Buffer lifecycle and reconciliation internal/saas/controlplane/checkout.go, internal/saas/controlplane/checkout_test.go	Maintains per-pool Ready sandbox caches, performs atomic claims, refills toward configured limits, adopts existing entries, backs off failures, and removes failed or aged resources.
Buffered create and fallback internal/saas/controlplane/forward.go, internal/saas/controlplane/checkout_test.go	Uses eligible buffered sandboxes before classic creation and validates exclusivity, cached responses, and fallback behavior.
Husk pod org attribution internal/controller/huskdrain.go, internal/controller/sandboxclaim_controller.go, internal/controller/husk_orgpropagation_test.go	Copies claim organization labels to unlabeled husk pods during lost-pod reconciliation without overwriting existing labels.
Gateway and Helm wiring cmd/gateway/main.go, deploy/charts/mitos/*, docs/threat-model.md	Adds checkout flags, startup behavior, Helm values and arguments, sandbox patch RBAC, and threat-model documentation.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Gateway
  participant K8sControlPlane
  participant CheckoutBuffer
  participant KubernetesAPI
  participant SandboxController
  Gateway->>K8sControlPlane: submit eligible sandbox.create
  K8sControlPlane->>CheckoutBuffer: claim pool entry
  CheckoutBuffer->>KubernetesAPI: resourceVersion-guarded label patch
  KubernetesAPI-->>CheckoutBuffer: claimed sandbox metadata
  CheckoutBuffer-->>Gateway: cached endpoint and token
  CheckoutBuffer->>K8sControlPlane: refill below floor
  K8sControlPlane->>SandboxController: create buffered sandbox
  SandboxController-->>CheckoutBuffer: Ready sandbox status and token Secret

Loading

Possibly related PRs

• mitos-run/mitos#646: Changes the same hosted control-plane forward.go create path with an unknown-pool pre-check.
• mitos-run/mitos#734: Changes readiness-waiting behavior in the same sandbox-create forwarding path.
• mitos-run/mitos#739: Changes single-tenant create validation in the same forwarding handler.

Suggested labels: security

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is conventional, concise, and accurately describes the main feature change.
Description check	✅ Passed	The PR description includes the required sections and covers the change, rationale, verification, risks, and model details.
Docstring Coverage	✅ Passed	Docstring coverage is 88.46% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/gateway-preclaimed-checkout

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/gateway/main.go`:
- Around line 103-134: Add focused unit tests for splitNonEmpty, envInt, and
envDuration. Cover comma-separated values with whitespace and empty entries,
valid and malformed integer environment values, and valid, empty, and invalid
duration values, asserting invalid or unset inputs return the supplied defaults;
isolate environment changes per test.

In `@docs/superpowers/specs/2026-07-11-preclaimed-checkout-design.md`:
- Around line 88-104: Update the Checkout and Janitor data-flow descriptions to
match the controller-owned design: remove the gateway’s async husk pod org-label
patch and refill kick from Checkout step 4, and remove Janitor’s pod re-patching
responsibility. State that the controller reconcile, triggered by the CR patch
watch event, propagates the pod label, while refill runs only on its periodic
ticker; preserve that the gateway requires no pod RBAC.

In `@internal/saas/controlplane/checkout.go`:
- Around line 313-354: In refillOne, wrap the error returned by
createSandboxAndAwait with contextual information instead of returning it
directly; update that branch to use fmt.Errorf with a descriptive message and %w
while preserving the original error.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: bfb79fb2-3488-4ddb-9f90-92f417ba7008

📥 Commits

Reviewing files that changed from the base of the PR and between ab66a9b and 4d6db64.

📒 Files selected for processing (15)

• cmd/gateway/main.go
• deploy/charts/mitos/templates/gateway-rbac.yaml
• deploy/charts/mitos/templates/gateway.yaml
• deploy/charts/mitos/values.schema.json
• deploy/charts/mitos/values.yaml
• docs/superpowers/plans/2026-07-11-preclaimed-checkout.md
• docs/superpowers/specs/2026-07-11-preclaimed-checkout-design.md
• docs/threat-model.md
• internal/controller/husk_orgpropagation_test.go
• internal/controller/huskdrain.go
• internal/controller/sandboxclaim_controller.go
• internal/saas/controlplane/checkout.go
• internal/saas/controlplane/checkout_test.go
• internal/saas/controlplane/controlplane.go
• internal/saas/controlplane/forward.go

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/gateway/checkout_flags_test.go`:
- Around line 53-56: Make the unset-environment assertion in the envInt test
deterministic by explicitly clearing TEST_CHECKOUT_UNSET_KEY before calling
envInt, using the test’s cleanup-aware environment helper if available; then
retain the assertion that envInt returns 5.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 1ab83927-bfc9-48c7-8ad4-38d70e6a03e9

📥 Commits

Reviewing files that changed from the base of the PR and between 4d6db64 and b7787e0.

📒 Files selected for processing (3)

• cmd/gateway/checkout_flags_test.go
• docs/superpowers/specs/2026-07-11-preclaimed-checkout-design.md
• internal/saas/controlplane/checkout.go

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Make the “unset” environment assertion deterministic.

envInt reads the process environment, so an externally defined TEST_CHECKOUT_UNSET_KEY can make this test fail. Clear the key explicitly before asserting the default.

Proposed fix

+	t.Setenv("TEST_CHECKOUT_UNSET_KEY", "")
 	if got := envInt("TEST_CHECKOUT_UNSET_KEY", 5); got != 5 {

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	if got := envInt("TEST_CHECKOUT_UNSET_KEY", 5); got != 5 {
	t.Fatalf("envInt(unset) = %d, want 5", got)
	}
	}
	t.Setenv("TEST_CHECKOUT_UNSET_KEY", "")
	if got := envInt("TEST_CHECKOUT_UNSET_KEY", 5); got != 5 {
	t.Fatalf("envInt(unset) = %d, want 5", got)
	}
	}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/gateway/checkout_flags_test.go` around lines 53 - 56, Make the
unset-environment assertion in the envInt test deterministic by explicitly
clearing TEST_CHECKOUT_UNSET_KEY before calling envInt, using the test’s
cleanup-aware environment helper if available; then retain the assertion that
envInt returns 5.