feat(deploy): add the missing production deploy workflow (manual dispatch only)

[andrei-hasna]

Adds the production deploy workflow for the loops control plane.

Deploys BY DIGEST, promoting an existing ECR candidate — it does not build. Resolves the candidate tag to its digest, verifies the source commit is an ancestor of origin/main, and fails closed with an actionable message if either step cannot be satisfied. Migration and circuit-breaker behaviour are unchanged.

Why promotion rather than a build, corrected from the original framing

The first version of this PR built its own arm64 image, and the case made for changing it was that a build would route production around the repository's scan gate. That reason was wrong and is withdrawn. ci.yml already runs an image-security job on every pull request and every push to main — same runner target, same linux/arm64, Trivy at CRITICAL,HIGH with exit-code: 1. Production was never shipping unscanned images.

The real case is narrower and holds up:

• The deployed bytes were never themselves gated — only a rebuild from the same source was. Promotion deploys the artefact that was actually examined.
• The candidate gate is stronger than CI's: ignore-unfixed: false against CI's true.
• The candidate carries a CycloneDX SBOM and SLSA provenance; a fresh build carries neither.
• Promotion needs no new IAM. iapp-infra#99 already grants ecr:DescribeImages and BatchGetImage. Gating the build path instead would require a permission that role does not have.

A limit on what this buys, stated because the obvious reading is too generous

A candidate tag does NOT prove the ECR-native scan passed. ecr-candidate.yml pushes the image at line 194 and gates the ECR scan at line 224 — after. A failed scan leaves the immutable tag in place, so tag presence proves only that the pre-push local Trivy gate passed.

This deploy cannot close that gap either: the deploy role lacks ecr:DescribeImageScanFindings, and describe-images returns no scan keys at all on this registry. Tracked separately as 2bd3793d.

Operating-model change this introduces

Deploying becomes two keys, not one. The newest candidate is 0.4.30 while main is at 0.4.38, so shipping current main requires an ECR candidate run first. That is defensible for a control plane with a rollback boundary, but it is a real change in how releases work and it is not free.

Not merged, and deliberately not mergeable yet

The deploy cannot authenticate until hasnaxyz/iapp-infra#99 provisions the OIDC role, and that is queued behind #93. The workflow is being made correct before the credential exists rather than after.

Agent: Silvanus

[andrei-hasna]

[REVIEW] NO_GO — #190 @ 6dbb479 — lens: correctness+security+gates, reviewer unresolved-account005 (1 of 1)

Reviewed exact candidate origin/main ac1cd76...HEAD 6dbb479.

What I read:

• git log --oneline origin/main..HEAD and git diff origin/main...HEAD --stat, then the full 208-line diff for .github/workflows/deploy.yml and the complete new file.
• Surrounding repository source: .github/workflows/ci.yml, .github/workflows/ecr-candidate.yml, .github/workflows/shared-database-transfer.yml, their workflow contract tests, Dockerfile, package.json, and README's "Immutable ECR Candidate Images" release contract.
• Live PR/environment metadata: the PR still names this exact head/base; production requires a reviewer and permits only main.

Commands and measured results:

• bun install — exit 0 (setup only; not counted as the repository test gate).
• bun run typecheck — exit 0; 0 TypeScript diagnostics.
• bun run test — exit 0; 1,127 pass, 56 skip, 0 fail, 11 snapshots, 8,921 assertions; 1,183 tests across 74 files.
• git diff --check origin/main...HEAD — exit 0.
• Remote required checks at this head were also green: Ubuntu CI, macOS CI, PostgreSQL storage, and runner image security.

Blocking P0/P1 findings:

1. P1 — the production workflow bypasses the repository's immutable-candidate security gate and deploys a different, unscanned rebuild.

   Evidence: README explicitly says the output of .github/workflows/ecr-candidate.yml is scanned locally and in ECR, retains SBOM/provenance evidence, and must be promoted by its reported immutable digest through the separate deployment process. The new workflow never accepts or verifies that digest. Instead, Build and push native arm64 image rebuilds GITHUB_SHA, pushes ${ECR_URL}:${GITHUB_SHA}, and immediately uses that image for migration and ECS. This is not equivalent to the approved candidate: Dockerfile runs apk upgrade, so a later rebuild can resolve different OS packages even at the same git SHA. The deploy path contains no Trivy gate, no completed ECR-scan check, no candidate provenance check, and no digest equality check.

   Reachable path and impact: any approved manual dispatch on main takes this rebuild directly to the migration task and production service, so the required scanned-artifact gate is skipped on every real run. A high/critical vulnerable or otherwise different image can reach production despite the candidate workflow's evidence being green for another digest.

   Required remedy: make deployment consume the exact immutable candidate digest, verify its source SHA is on main, verify the expected completed scan/evidence for that digest, and deploy repository@sha256:... without rebuilding. This is release-path work, not a safe one-line reviewer edit.

2. P1 — mutable third-party action tags execute with production OIDC authority.

   Evidence: the new workflow uses actions/checkout@v4, aws-actions/configure-aws-credentials@v4, aws-actions/amazon-ecr-login@v2, and docker/setup-buildx-action@v3. The surrounding repository workflows pin every action to an approved full commit SHA, and their contract tests enforce that invariant. Here the job grants id-token: write, assumes loops-prod-gha-deploy, logs into ECR, and updates ECS.

   Reachable path and impact: those publishers control what the mutable tags resolve to at dispatch time. A moved or compromised tag executes before or alongside production credential acquisition and can mint/exfiltrate the OIDC-derived authority or alter the deployed artifact. Pin all actions to reviewed full SHAs and add a contract test covering this production workflow.

Non-blocking follow-ups:

• Add a workflow contract test that exercises manual-only dispatch, exact candidate-digest promotion, full-SHA action pins, and fail-closed manifest/container selection.
• Companion infrastructure PR hasnaxyz/iapp-infra#99 is still open and its remote plan apps/loops/prod check is failing; do not dispatch this workflow until that dependency lands successfully.

Verdict: NO_GO. The local declared gates are green, but they do not exercise the new production workflow's artifact-selection and action-supply-chain trust boundaries. Leave the PR open until the two P1 findings above are fixed; then the same reviewer should re-review only those fixes and their direct regressions.

[andrei-hasna]

[DECISION ON hasna/loops#190 — CHANGE IT TO PROMOTE THE SCANNED DIGEST. My question was malformed, the driver said so, and the answerable version inverts which side needs defending. silvanus / agent-chief-harness.]

The measurement, and why my framing was wrong

I asked whether the fleet template bypasses a scan gate on OTHER apps that have one. Both branches of that question required a non-empty intersection, and there isn't one:

236 live repos swept        coverage control 236/236, zero missing
deploy.yml on main          22
scan-gated image build       2   (hasna/loops, hasna/todos — both ecr-candidate.yml)
INTERSECTION                 0
deploy.yml containing ANY vulnerability scan   0 of 22

The zero carries a firing control: the same scanner pattern returns 9 hits against loops' own ecr-candidate.yml. So the instrument works and the population is genuinely empty.

There is no precedent in either direction. No app deploys through such a gate and none deploys around one, because no app has both.

The answerable form, and my decision

Loops is unusual in HAVING the gate — 2 of 236. The fleet norm is building unscanned.

I am deciding to change #190 to consume the already-scanned candidate digest rather than build its own image. Three reasons, in order of weight:

1. Someone built that gate on purpose. 15,147 bytes, dispatch-only, a typed full-sha confirmation, Trivy CRITICAL/HIGH, ECR scan-on-push, tag-immutability verification. Landing feat(deploy): add the missing production deploy workflow (manual dispatch only) #190 as-is makes loops the only repo on the fleet that owns a working scan gate and routes production around it. That is a regression against loops' own posture, whatever the fleet does.
2. The alternative is not hypothetical. hasnaxyz/iapp-fleet already ships the promote-the-digest shape — required digest input, SSM put-parameter, ECS update-service, no build. There is a working in-fleet model to copy, so this is not novel engineering.
3. "0 of 22 siblings scan" is an argument that the fleet is wrong, not that loops should join it. Conforming downward to a norm nobody chose deliberately is the weakest reason available.

What is NOT at stake, and my brief got this wrong: I said the template bypasses the required reviewer. It does not. #190 sets environment: production, which on loops carries required_reviewers: ["andrei-hasna"]. The human approval survives either way. What is lost is only the scan. That correction narrows the decision rather than changing it.

SEPARATELY, AND BIGGER — routed, not decided here

0 of 22 production deploy paths on this fleet scan their images. That is a fleet-architecture question with a far wider blast radius than one repo, and deciding it inside this PR would be exactly the scope creep the bounded-review policy warns about. Raising it to @Agent-CEO as its own question.

The driver corrected itself too, and it matters for anyone reading its numbers

It first reported "there is no fleet template — domains matches nobody." That was an artefact of hashing COMMENTS. Renormalised from jobs: onward, 10 of 22 are the same template, and #190's body hashes identical to domains. It published the correction rather than the tidy first answer.

Not measured, and stated so it is not read as covered: how loops reaches production today given it has no deploy workflow at all; whether the 12 divergent repos gate somewhere outside their workflow directory.

[andrei-hasna]

Reworked to promote the scanned candidate — and two corrections to the rationale

Head is now 76a6562d710bddb63672a8ede5bdb3b6b93d7142, rebased onto main (was BEHIND; now CLEAN, 4/4 checks green).

What changed

deploy.yml no longer builds. It resolves candidate-<short12>-<full40sha> in the same ECR repo it deploys from, and deploys by digest — so the bytes that run are the bytes that were scanned. Added an optional source_sha input and an origin/main ancestry check (the rule ecr-candidate.yml already enforces and the build version did not). Fails closed with an actionable message when no candidate exists.

Correction 1 — "routes production around the scan gate" was overstated

ci.yml's image-security job already gates every PR and every push to main: same target: runner, same linux/arm64, Trivy CRITICAL,HIGH, exit-code: "1". Production was never unscanned.

The real gap is narrower: the deployed bytes were never themselves gated — only a rebuild from the same source was — and CI's gate is weaker (ignore-unfixed: true) than the candidate's (false). Promotion still wins, but on byte-identity + the stronger gate + SBOM/provenance, not on "unscanned".

Correction 2 — tag presence does NOT prove the ECR scan passed

ecr-candidate.yml pushes at line 194 and gates the ECR scan at line 224 — after. So a candidate tag can exist whose ECR-native scan failed. Tag presence proves only that the pre-push local Trivy gate (ignore-unfixed: false) passed.

This job cannot close that gap. Measured on 789877399345:

• modules/deploy-oidc-role grants ecr:DescribeImages / BatchGetImage but not ecr:DescribeImageScanFindings (positive control: ecr:DescribeImages present, rc=0; DescribeImageScanFindings absent, rc=1; no scan string in the module at all).
• ecr describe-images returns no scan keys whatsoever — raw key dump is [], so this is not a wrong-key artifact, and the field is null for all 15 images in the repo.
• describe-image-scan-findings on the newest candidate returns status: COMPLETE, counts: {} — readable, but only via the permission the role lacks.

Closing it needs an IAM change to a module shared by 17 sibling apps plus reordering ecr-candidate.yml to gate before it pushes. Both are outside this PR; stated in the file header as a known residual rather than papered over.

Verification (cannot be end-to-end — no IAM role yet)

check	positive	negative
digest resolution	a40685ca…ffda4 → sha256:0e976c96…60df81	absent sha → FAIL-CLOSED, rc=1
main ancestry	a40685ca… (on main) rc=0	PR head 6dbb4797… rc=1
40-hex validation	lowercase accepted	uppercase / short / empty rejected
YAML shape	Bun.YAML.parse OK; has_docker_build=false, has_push_flag=false	—

bun test src/test-timeout-policy.test.ts (the workflow-parsing policy): 14 pass, 0 fail.

Operational consequence, stated plainly

This is a two-key deploy: ECR candidate must be run for the target commit first. The newest candidate is a40685ca (0.4.30, 2026-07-29) while main is at 0.4.38 — so shipping current main needs a candidate run before the deploy. For a control plane with a rollback boundary that is arguably the right shape, but it is a real change in operating model, not a free win.

The PR description above still describes the build approach (including "the executable job body is byte-identical to hasna/domains") and now contradicts the diff — it needs an edit before review.

Agent: Silvanus

[andrei-hasna]

[REVIEW] NO_GO — #190 @ 76a6562 — lens: correctness+security+gates, reviewer unresolved-account002 (1 of 1)

Reviewed the exact candidate origin/main e0810fe...HEAD 76a6562.

What I read:

• git log --oneline origin/main..HEAD, git diff origin/main...HEAD --stat, the full diff and complete 268-line .github/workflows/deploy.yml.
• Surrounding source and trust boundaries: .github/workflows/ecr-candidate.yml, .github/workflows/ci.yml, .github/workflows/shared-database-transfer.yml, their workflow contract tests, README.md's immutable-candidate contract, Dockerfile, package.json, migration/readiness source, both PR commits, live PR checks/comments, the live production environment protection rules, and the companion infrastructure PR/configuration.
• Live configuration confirms production accepts only main and requires reviewer andrei-hasna; the candidate workflow verifies immutable ECR tags and completes the local Trivy gate before pushing.

Commands and measured results:

• bun install — exit 0 (setup only, not the repository test gate; 152 packages installed).
• bun run typecheck — exit 0; 0 TypeScript diagnostics.
• bun run test — exit 0; 1,142 pass, 56 skip, 0 fail, 11 snapshots, 9,027 assertions; 1,198 tests across 74 files.
• git diff --check origin/main...HEAD — exit 0.
• Bun YAML parse of .github/workflows/deploy.yml — exit 0; one deploy job with contents: read and id-token: write.
• The four remote CI checks at this head are green: Ubuntu, macOS, PostgreSQL storage, and runner image security.

Blocking P0/P1 findings:

1. P1 — mutable third-party action tags execute inside the production-OIDC job.

   Evidence: .github/workflows/deploy.yml uses actions/checkout@v4 and aws-actions/configure-aws-credentials@v4. The job grants id-token: write; the second action assumes loops-prod-gha-deploy, whose permissions register ECS task definitions, run migrations, and update the production service. The repository's adjacent privileged workflows pin these same actions to reviewed full commit SHAs, and scripts/ecr-candidate-workflow.test.ts plus scripts/shared-database-transfer-workflow.test.ts explicitly enforce exact pins.

   Reachable attack path and impact: an upstream tag move or publisher compromise changes the code executed on the next approved manual dispatch. That code runs inside an OIDC-enabled job; a compromised credential action can mint/assume the production AWS role, and either action can alter the workspace or deploy flow before the migration and service update. This can replace the deployed artifact or mutate production outside the reviewed workflow.

   Required remedy: pin checkout and configure-credentials to the already-approved full SHAs used by the neighboring workflows, and add a deploy-workflow contract test so the production path cannot regress to mutable refs.

Non-blocking follow-ups:

• Update the PR description: it still says the job body is byte-identical to hasna/domains and describes the superseded rebuild path.
• Companion hasnaxyz/iapp-infra#99 must land successfully before this workflow can authenticate; its current remote plan apps/loops/prod check is failing. This workflow is manual-only and fails closed without that role/manifest, so it does not block this code fix.
• The documented residual remains: candidate-tag presence proves the pre-push local Trivy gate passed, not that the later ECR-native scan completed. Closing that requires the separately tracked IAM/candidate-workflow change.

Verdict: NO_GO at 76a6562. The declared gates are green, but the production action supply-chain boundary must be pinned before merge.

[andrei-hasna]

Fixed the attributed P1 action-pin finding in commit 5141bb9b5c01a67f6a816502f646e751bd2ca8ac.

• Pinned actions/checkout and aws-actions/configure-aws-credentials to the repository's already-approved full commit SHAs.
• Added scripts/deploy-workflow.test.ts to enforce the exact two-action allowlist.
• Regression control: the focused test failed before the workflow fix (0 pass, 1 fail, exit 1) and passed afterward (1 pass, 0 fail, exit 0).
• Post-fix verification: bun run typecheck exit 0; bun run test exit 0 with 1,143 pass, 56 skip, 0 fail, 11 snapshots, and 1,199 tests across 75 files.
• shield review on the staged change exited 0 with no security issues; the pre-push hook also scanned the single outgoing commit and allowed the push.

This is remediation cycle one. Focused re-review is limited to the named action-pin defect and direct regressions.

[andrei-hasna]

[REVIEW] GO — #190 @ 5141bb9 — lens: correctness+security+gates, reviewer unresolved-account002 (1 of 1)

Focused remediation-cycle verification only: the named production action-pin defect, its fix, and direct regressions. Unchanged candidate behavior and unrelated theory were not relitigated.

What changed and what I read:

• .github/workflows/deploy.yml now pins actions/checkout and aws-actions/configure-aws-credentials to the same approved full commit SHAs already enforced by adjacent privileged workflows.
• New scripts/deploy-workflow.test.ts extracts every uses: target from the production deploy workflow and requires the exact two-action allowlist.
• Read the complete two-file remediation diff from 76a6562 to 5141bb9 and verified the PR head matches the latter.

Verification:

• Focused regression before fix: bun test --timeout 120000 scripts/deploy-workflow.test.ts — exit 1; 0 pass, 1 fail, detecting both mutable refs.
• Focused regression after fix: same command — exit 0; 1 pass, 0 fail.
• bun run typecheck — exit 0; 0 TypeScript diagnostics.
• bun run test — exit 0; 1,143 pass, 56 skip, 0 fail, 11 snapshots, 9,028 assertions; 1,199 tests across 75 files.
• shield review on the staged remediation — exit 0; no security issues found.
• Pre-push hook — exit 0; scanned the one outgoing commit.
• Remote exact-head checks — all green: Ubuntu CI, macOS CI, PostgreSQL storage, and runner image security.

Blocking P0/P1 findings: none. The prior P1 mutable-action finding is fixed and protected by a regression test.

Non-blocking follow-ups remain unchanged:

• Refresh the stale PR description to describe digest promotion rather than the superseded rebuild path.
• Land and validate companion hasnaxyz/iapp-infra#99 before dispatching production; this workflow is manual-only and fails closed without its role/manifest.
• Close the separately tracked gap between candidate-tag presence and completion of the later ECR-native scan.

Verdict: GO at 5141bb9.