E2E tests broken: OIDC mint requires real GCP project with WIF and Secret Manager

[waynesun09]

Problem

The OIDC mint migration (#503) breaks E2E tests. The admin install E2E flow now provisions real GCP infrastructure (WIF pool/provider, Secret Manager secrets, Cloud Function), but the test suite uses a fake e2eDispatcher{} that doesn't create any of this.

The triage dispatch smoke test (line ~592 in e2e/admin/admin_test.go) triggers a real GitHub Actions workflow run. That workflow calls google-github-actions/auth@v3 to exchange a GitHub OIDC token for GCP credentials via the mint Cloud Function. Without a real WIF provider and deployed mint function, the token exchange fails and the workflow errors out.

What's faked vs what's real

Component	Current E2E	Needed
GitHub App creation	Real (Playwright manifest flow)	No change
App PEM storage	Skipped by fake dispatcher	Real Secret Manager secrets
WIF pool/provider	Skipped by fake dispatcher	Real WIF with GitHub OIDC
Cloud Function (mint)	Skipped by fake dispatcher	Real deployed function
FULLSEND_MINT_URL org var	Set to placeholder by fake	Must point to real function URL
Workflow dispatch	Real (gh workflow run)	No change
OIDC token exchange	Fails — no WIF/mint exists	Must succeed against real mint

Key code references

• e2e/admin/admin_test.go:835 — buildTestLayerStack passes &e2eDispatcher{} to NewOIDCDispatchLayer
• e2e/admin/admin_test.go:452 — verifyInstalled checks FULLSEND_MINT_URL org variable exists
• e2e/admin/admin_test.go:592 — triage dispatch smoke test triggers real workflow

Required GCP resources

A dedicated GCP project for E2E testing needs:

1. Workload Identity Federation

   • WIF pool (e.g., fullsend-e2e-pool)
   • WIF provider with GitHub OIDC issuer, attributeCondition scoped to E2E org
   • Allowed audiences: custom audience + IAM-format URI

2. Secret Manager

   • Org-scoped PEM secrets: fullsend-{org}--{role}-app-pem for each agent role
   • Service account must have secretmanager.versions.access

3. Cloud Functions (2nd gen)

   • Deployed mint function with env vars: ALLOWED_ORGS, ROLE_APP_IDS, WIF_POOL_NAME, WIF_PROVIDER_NAME, OIDC_AUDIENCE, GCP_PROJECT_NUMBER

4. Service Account

   • With roles: roles/cloudfunctions.admin, roles/iam.workloadIdentityPoolAdmin, roles/secretmanager.admin, roles/iam.serviceAccountUser
   • JSON key stored as GitHub Actions secret

New CI environment variables / secrets

Name	Type	Description
E2E_GCP_PROJECT_ID	Secret	GCP project ID for E2E mint infrastructure
E2E_GCP_PROJECT_NUMBER	Secret	GCP project number (for WIF audience)
E2E_GCP_SA_KEY	Secret	Service account JSON key for provisioning
E2E_WIF_POOL_NAME	Var	WIF pool name (e.g., fullsend-e2e-pool)
E2E_WIF_PROVIDER_NAME	Var	WIF provider name (e.g., github-oidc)

Options

Option A: Real GCP project in CI (recommended)

Stand up a dedicated GCP project for E2E. The provisioner already handles idempotent creation (409 = exists). Each E2E run would:

1. Use real GCFProvisioner instead of e2eDispatcher{}
2. Provision WIF + secrets + function (idempotent, fast on repeat runs)
3. Run the full admin install flow including triage dispatch
4. Tear down is optional — resources are scoped to E2E org

Pros: Tests the real path end-to-end. Catches issues like the WIF audience mismatch that unit tests miss.
Cons: Requires GCP project setup, costs (minimal — Cloud Functions free tier), CI secret management.

Option B: Enhanced fake dispatcher

Keep e2eDispatcher{} but make it smarter — deploy a mock HTTP endpoint that returns valid-looking tokens without real GCP. Skip the OIDC exchange in the workflow.

Pros: No GCP dependency.
Cons: Doesn't test the actual token exchange path. Misses real integration issues.

Related

• feat: OIDC token mint dispatch — deprecate PAT, migrate to OIDC #503 — OIDC mint with multi-org WIF support (introduces the GCP dependency)

[github-actions]

fullsend triage is working on this — view logs

[fullsend-ai-triage]

Triage Summary

Severity: High — E2E tests are broken for the triage dispatch path
Category: Bug (test infrastructure gap introduced by OIDC mint migration)

Problem

The E2E test suite uses a fake e2eDispatcher{} that does not provision the GCP infrastructure (WIF pool/provider, Secret Manager secrets, Cloud Function) now required by the OIDC mint flow. When the triage dispatch smoke test triggers a real workflow, the OIDC token exchange fails against non-existent infrastructure.

Root Cause

buildTestLayerStack in e2e/admin/admin_test.go passes a no-op e2eDispatcher{} to NewOIDCDispatchLayer. The dispatched workflow then calls google-github-actions/auth@v3 expecting a real WIF provider and mint function, which don't exist.

Recommended Fix

Option A from the issue (real GCP project) is the stronger approach — it validates the actual token exchange path. Key steps:

1. Provision a dedicated E2E GCP project with WIF pool/provider, Secret Manager, and Cloud Function
2. Add CI secrets/variables (E2E_GCP_PROJECT_ID, E2E_GCP_PROJECT_NUMBER, E2E_GCP_SA_KEY, E2E_WIF_POOL_NAME, E2E_WIF_PROVIDER_NAME)
3. Replace e2eDispatcher{} with real GCFProvisioner in buildTestLayerStack
4. Verify FULLSEND_MINT_URL points to the deployed function URL

Proposed Test Verification

After the fix, validate:
1. E2E triage dispatch smoke test passes end-to-end with real OIDC exchange
2. Run E2E twice without teardown to verify provisioner idempotency (409 skip)
3. Verify FULLSEND_MINT_URL org variable contains real Cloud Function URL
4. Confirm dispatched workflow completes successfully (not just dispatches)

[github-actions]

fullsend code is working on this — view logs

[ralphbean]

E2E pass rate: 23% (last 30 runs)

7 passed, 22 failed, 1 cancelled. Every runtime failure traces back to the missing OIDC mint infrastructure — the enrollment workflow exits 6, so secrets/variables/enrollment PRs never get created.

Both main failures are from this:

Run	Branch	Trigger	Failure
25716807495	main	Merge PR #808	enrollment exit 6, missing secrets/vars
25708950885	main	Merge PR #503	enrollment exit 6, missing secrets/vars
25753869016	fix/add-retro-agent-app-config	PR #828	same
25750901827	fix/retro-actions-permission	PR #835	same
25750738673	fix/add-retro-agent-app-config	PR #828	same

E2E has been broken on main since PR #503 merged. The remaining 11 failures are branch-specific compile errors (unrelated).

Also note #486 — the e2e bot account's session export breaks May 19 when 2FA becomes mandatory.