fix(control-plane): close tenant-isolation, webhook-secret, and overlap bugs before first deploy

[JSONbored]

Summary

Closes #9143

control-plane/ (the hosted-ORB tenant-provisioning substrate) is not currently deployed and cannot be — there is no deploy workflow, and wrangler.jsonc still has placeholder REPLACE_WITH_REAL_* values. Every defect below is latent today; a first deploy would ship all eight intact. This PR fixes the three deploy blockers fully, and fixes or explicitly scopes down the remaining five.

Fixed fully

1. Cross-tenant Neon-branch collision (deploy blocker). branchNameFor (neon-database-driver.ts) now appends a hash suffix unconditionally (not only past the 63-char truncation threshold, Neon branchNameFor's 63-char truncation has no collision guard — two long, prefix-similar tenant names could resolve to the same branch #8026), and hashes the raw, pre-sanitization ${product}:${tenant.name} identity instead of the already-lowercased/punctuation-collapsed string. Two tenant names that sanitize identically ("Acme"/"acme", "acme.corp"/"acme-corp"/"acme corp") used to derive the identical branch name and share one Neon branch/database/role/live password; they now always diverge. Regression test: neon-database-driver.test.ts's "sanitize-colliding tenant names ... never share a branch (#9143 isolation)".

2. Cross-tenant webhook-routing-index deletion (deploy blocker). tenant-registry.ts's upsert no longer unconditionally deletes the old installation:<id> index entry when a tenant's orbInstallationId changes/clears — it now re-reads the index and only deletes it if it still points at the upserting tenant's own primary key (a compare-and-delete). Before this fix, a stale tenant (still carrying an old installationId from creation) being re-created or torn down long after a different tenant legitimately reclaimed that installation ID would silently delete the current claimant's live routing pointer, leaving it active but permanently unreachable by webhook with no way back (create 409s; orbInstallationId is settable only at create). Added PATCH /v1/tenants/:name/orb-installation as the manual recovery path for a pointer already lost to the pre-fix bug. Regression test: tenant-registry.test.ts's "an unrelated tenant's installation-index entry SURVIVES a stale record's later re-creation/teardown upsert (#9143)".

3. Hosted tenant containers could never verify a routed webhook (deploy blocker). createTenantContainer never injected ORB_GITHUB_WEBHOOK_SECRET, so every correctly-signed GitHub delivery to a hosted ORB tenant would 401 forever, and fetchBrokeredStoredSecret (Deliver secrets/env vars into a running hosted tenant container (OrbTenantContainer/AmsTenantContainer) #8202) had full test coverage but zero production call sites. Fixed by bundling a freshly generated per-ORB-tenant webhook secret into the same broker enrollment/bootstrap exchange secret-driver.ts already mints for the tenant's database credential (one bootstrap token, one exchange — not a second plumbing path), and adding the missing call site: src/orb/hosted-webhook-secret.ts (new) resolves it via fetchBrokeredStoredSecret when LOOPOVER_TENANT_SECRET_TOKEN is set and no direct ORB_GITHUB_WEBHOOK_SECRET is configured, wired into handleOrbWebhook. Memoized per-container (successful resolution cached forever; a failed exchange is not cached, so a transient broker outage self-heals on the next delivery). Zero behavior change for normal self-host/cloud (direct env value short-circuits with no network call).

Fixed, scoped down where noted

4. Env vars lost on container restart. envVars are now persisted in the container DO's own durable storage (worker.ts's ProvisionedContainer) and reloaded into this.envVars in the constructor via ctx.blockConcurrencyWhile — the vendored @cloudflare/containers SDK never itself persists a start({envVars}) call's values, so an implicit restart (e.g. containerFetch's auto-wake-on-request path) used to silently boot with an empty environment.

5. POST /v1/tenants/rollout was a silent no-op. All four of its promised effects were broken (only wrote the registry field; createTenantContainer early-returns once provisioned so it never reached the env-var injection site; provisionTenant constructs a fresh Tenant that drops the pin anyway on next provision; nothing ever reads LOOPOVER_PINNED_VERSION back out). Per the issue's own framing ("Rollback is the control you least want to discover is fake"), this now returns an explicit 501 not_implemented rather than accepting a request that changes nothing observable. Tenant.pinnedVersion / PINNED_VERSION_ENV_VAR are left in place for whenever a real rollout mechanism is built.

6. Unauthenticated webhook route had no body-size limit. orb-webhook-router.ts now has the same content-length precheck + streaming-limit enforcement as its declared twin src/orb/webhook.ts (fixed there by fix(github): handleOrbRelay skips the pre-read Content-Length rejection that handleGitHubWebhook has #8888) — it used to buffer an unbounded body twice (clone + .text()) before ever checking the signature.

7. Guaranteed overlapping AMS-wake ticks. ams-wake.ts now claims a tenant's nextDueAt before starting its poll (not after it resolves), caps the per-tenant poll timeout well under the 5-minute cron interval (10min → 2min), and bounds how many due tenants one tick wakes (DEFAULT_MAX_TENANTS_PER_TICK = 20, deferring the rest to the next tick). Regression test proves an overlapping tick no longer re-wakes a tenant whose prior-tick poll is still in flight.

8. KV eventual consistency makes every claim check advisory. Implemented the issue's stated minimum: DELETE /v1/tenants/:name now refuses a tenant still in the "provisioning" state (409) instead of racing the in-flight provisionTenant call. The full fix — moving the tenant index into a Durable Object or D1 so check-and-claim is one serialized transaction — is a materially larger migration and is deferred, left as an explicit follow-up.

Explicitly out of scope for this PR

• The missing control-plane deploy workflow and real wrangler.jsonc values (REPLACE_WITH_REAL_NEON_PROJECT_ID, REPLACE_WITH_REAL_KV_NAMESPACE_ID) — this PR does not make the workspace deployable, so per the issue these remain outstanding for whichever change does.
• The full DO/D1 migration for defect 8 (see above).

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes — it is a single issue's eight defects, all confined to control-plane/ plus the src/orb/ webhook-secret call site that issue control-plane: eight pre-deploy defects, headlined by two tenants sharing one Neon database (and one customer's teardown deleting another's) #9143 explicitly requires.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• Closes control-plane: eight pre-deploy defects, headlined by two tenants sharing one Neon database (and one customer's teardown deleting another's) #9143 (currently open, maintainer-owned issue).

Validation

• git diff --check
• npm run typecheck (root) — clean
• control-plane's own gate: npm run build, npm run cf:typecheck, and node --test --experimental-strip-types "test/**/*.test.ts" — 227/227 passing
• control-plane's own coverage (node scripts/control-plane-coverage.mjs): every file touched in this PR is at 100% line+branch coverage (verified per-file with a scoped c8 run); the one remaining project-level gap (settlement-backend-driver.ts) is pre-existing and untouched by this PR
• Root npm run test:changed (vitest, diff-scoped against origin/main): 121 files / 1551 tests passing, including the new test/unit/orb-hosted-webhook-secret.test.ts (100% coverage) and updated test/integration/orb-webhook.test.ts
• npm run docs:drift-check, npm run manifest:drift-check, npm run coverage-boltons:check — all pass
• Full root npm run test:ci / npm audit --audit-level=moderate — not run locally (this PR's scope is large enough that the full monorepo gate is left to CI, per direction from the repo owner); every check above that overlaps with test:ci (typecheck, control-plane's own build/test/coverage, drift checks) was run and is green
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries — see per-defect regression tests called out above
• npm run ui:openapi:check / ui:lint / ui:typecheck / ui:build / test:workers / build:mcp / test:mcp-pack — not applicable, no apps/loopover-ui, worker-test, or MCP-package changes in this PR

If any required check was skipped, explain why:

• Full npm run test:ci and npm audit were not run locally for this PR at the repo owner's direction (large multi-defect PR; CI will run the full gate). Every check materially relevant to the actual diff (control-plane's own build/typecheck/tests/coverage, root typecheck, diff-scoped vitest, and the drift checks) was run locally and is green.

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth/CORS/GitHub-App/Cloudflare/session-adjacent changes (the webhook-secret brokering, the body-size limit on the unauthenticated router) include negative-path tests (wrong signature, unreachable broker, malformed bundle, oversized body with and without a Content-Length header).
• API/MCP behavior is updated and tested where needed (new PATCH /v1/tenants/:name/orb-installation route, POST /v1/tenants/rollout behavior change — both fully covered by new tests). No OpenAPI regen needed: control-plane is a separate Hono service with no OpenAPI generation pipeline.
• UI changes — not applicable, no UI in this PR.
• UI Evidence — not applicable, no visible UI change.
• Public docs/changelogs — not applicable; CHANGELOG.md is untouched, as required.

UI Evidence

Not applicable — no visible UI/frontend change in this PR.

Notes

• This PR intentionally does not add the control-plane deploy workflow or replace the wrangler.jsonc placeholders — the workspace remains non-deployable after this PR, and the issue's own framing only asks for that "as part of whichever change first makes this deployable."
• Defect 8's full fix (Durable Object/D1-backed serialized tenant-index claim) is deferred; only the issue's stated minimum (refuse DELETE on "provisioning") is implemented here.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.69%. Comparing base (fbd1ed1) to head (e56c66c).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9170      +/-   ##
==========================================
- Coverage   93.90%   92.69%   -1.21%     
==========================================
  Files         817      818       +1     
  Lines       81039    81254     +215     
  Branches    24611    24636      +25     
==========================================
- Hits        76096    75322     -774     
- Misses       3568     4848    +1280     
+ Partials     1375     1084     -291     

Flag	Coverage Δ
backend	93.49% <100.00%> (-1.69%)	⬇️
control-plane	99.86% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
control-plane/src/ams-wake.ts	100.00% <100.00%> (ø)
control-plane/src/container-driver.ts	100.00% <100.00%> (ø)
control-plane/src/http-app.ts	100.00% <100.00%> (ø)
control-plane/src/neon-database-driver.ts	100.00% <100.00%> (ø)
control-plane/src/orb-webhook-router.ts	100.00% <100.00%> (ø)
control-plane/src/secret-driver.ts	100.00% <100.00%> (ø)
control-plane/src/tenant-registry.ts	100.00% <100.00%> (ø)
src/orb/hosted-webhook-secret.ts	100.00% <100.00%> (ø)
src/orb/webhook.ts	100.00% <100.00%> (ø)

... and 3 files with indirect coverage changes