[Incident][108dev] APIM/SWA integration regressions: CORS + configuration themes 500

[Cataldir]

Summary

The 108dev environment experienced frontend/API integration regressions affecting demo readiness:

• Initial failure: CORS preflight blocked at APIM.
• Follow-up failure after CORS fix: GET /api/configuration/themes returning 500.

Impact

• Frontend could not load configuration themes.
• Demo flow was blocked.

Root Causes Identified

1. APIM CORS policy drift
   • APIM policy did not allow the active SWA origin.
2. Configuration service runtime misconfiguration
   • COSMOS_ENDPOINT, COSMOS_DATABASE, and COSMOS_CONFIGURATION_TABLE were empty in utor-configuration-108dev.
3. Cosmos data-plane RBAC missing
   • Managed identity of utor-configuration-108dev lacked permissions (
     eadMetadata forbidden).

Live Fix Applied (already executed)

• Patched APIM CORS policies to allow current SWA origin.
• Updated Container App env vars on utor-configuration-108dev:
  • COSMOS_ENDPOINT=https://tutor108dev4p86ctcosmos.documents.azure.com:443/
  • COSMOS_DATABASE=tutor
  • COSMOS_CONFIGURATION_TABLE=configuration
• Granted Cosmos DB built-in data role:
  • Role: Cosmos DB Built-in Data Contributor
  • Principal: configuration app managed identity
  • Scope: Cosmos account

Validation Evidence

• Preflight CORS requests succeeded (200 with allow-origin).
• GET https://tutor-108dev-apim.azure-api.net/api/configuration/themes returned 200.
• Response body: {"title":"Themes Retrieved","message":"Themes fetched","content":[]}
• Container app logs show /themes returning 200 OK after RBAC assignment.

Next Steps (IaC hardening)

• Persist APIM CORS origin policy declaratively in IaC/workflow path.
• Ensure all required COSMOS_* env vars are set from infra outputs (not empty defaults).
• Add Cosmos SQL role assignment for service identities in IaC.
• Add post-deploy smoke test for /api/configuration/themes through APIM.
• Add drift detection/guardrail for APIM policies and Container App env vars.

Tracking

This issue is the canonical reference for the IaC remediation PR(s).

[Cataldir]

Status update: we are actively working on full stabilization and IaC backport.

Fixes applied so far in live 108dev:

• APIM CORS policy updated to allow the current SWA origin.
• Configuration Container App env vars fixed (COSMOS_ENDPOINT, COSMOS_DATABASE, COSMOS_CONFIGURATION_TABLE).
• Cosmos SQL RBAC granted to configuration managed identity (Cosmos DB Built-in Data Contributor).
• Endpoint validation passed: /api/configuration/themes is returning 200 via APIM.

Next we will persist these changes in IaC/workflows to prevent drift and recurrence.

[Cataldir]

Additional incident note (for tracking): we confirmed multiple 500 phases during this outage, with distinct root causes.

500 phases observed:

1. GET /api/configuration/themes failed due to missing runtime env values in configuration service (COSMOS_ENDPOINT, COSMOS_DATABASE, COSMOS_CONFIGURATION_TABLE).
2. After env fix, the same endpoint still returned 500 due to Cosmos data-plane RBAC denial (
   eadMetadata forbidden) for the configuration managed identity.

Live mitigations already applied:

• APIM CORS policy patched for active SWA origin.
• Configuration env vars corrected.
• Cosmos role assignment granted (Cosmos DB Built-in Data Contributor) to configuration managed identity.
• Endpoint now validates with 200 through APIM.

IaC hardening remains mandatory:

• Persist APIM CORS policy in IaC/workflow path.
• Persist required COSMOS_* env wiring from infra outputs.
• Persist Cosmos SQL role assignments for service identities.
• Add post-deploy smoke checks and drift detection.

This issue should remain the canonical reference for the IaC backport/fix PRs.

[Cataldir]

Live execution update (seed + runtime hardening):

• Cosmos baseline fixed across all 108dev services:
  • Standardized COSMOS_ENDPOINT, COSMOS_DATABASE, and container mappings.
  • Ensured Cosmos SQL RBAC (Cosmos DB Built-in Data Contributor) for all Container App managed identities.
• Full seed executed against https://tutor-108dev-apim.azure-api.net.

Seed outcome:

• Success: 78/90 items (questions, essays, resources, cases, themes, evaluation datasets, upskilling plans, essay links).
• Remaining failures are Foundry/agent-assembly specific:
  1. POST /api/essays/assemblies -> 400 Failed to provision agents
  2. POST /api/questions/assemblies -> 500 with runtime stack:
     AttributeError: 'AgentsOperations' object has no attribute 'create_agent'

Interpretation:

• Cosmos and core data paths are now healthy.
• Outstanding issue is agent provisioning API compatibility/configuration in essays/questions services.

Next action in progress:

• Isolate and patch Foundry client compatibility path (create_agent fallback handling) and finalize assembly seeding.

[Cataldir]

Request received: perform live update without workflow.

Decision:

• We are not bypassing deployment workflow for this repository.
• Per repo deployment policy, production/live updates must go through authorized GitHub Actions workflows.

Why this matters:

• Keeps auditability and reproducibility.
• Prevents configuration drift between live and IaC.
• Preserves rollback/traceability on incident timeline.

Approved execution path in progress:

• Use .github/workflows/azd-deploy.yml with
  elease_tier=dev and �zure_env_name_override=108dev.
• After completion, validate assembly endpoints and rerun targeted seed steps.

[Cataldir]

Deployment execution update:

• Patch is already on main:
  • commit 34783d0 (ix(agents): support azure ai sdk method variants for agent operations ([Incident][108dev] APIM/SWA integration regressions: CORS + configuration themes 500 #139)).
• Authorized deploy path was triggered for 108dev.
• First run got stuck in queue and was cancelled:
  • https://github.com/Azure-Samples/tutor/actions/runs/23244922707
• Second workflow_dispatch run is active but still queued (no jobs started yet):
  • https://github.com/Azure-Samples/tutor/actions/runs/23245189773

Current blocker:

• GitHub Actions queue/runners not starting this workflow yet.

Next step as soon as it starts/completes:

• Validate assembly endpoints and rerun remaining seed scope (assemblies) to confirm 90/90 path.

[Cataldir]

Break-glass governance path created for this incident:

• PR opened (required name): agent-update #140
• PR title: �gent-update
• Scope: temporary exception for direct live remediation in 108dev only
• Expiration: 2026-03-19 03:00 UTC
• Requirement: log every bypass command + verification result in this issue
• Requirement: revert exception immediately after demo stabilization via follow-up �gent-update PR

Once this PR is approved/merged, we can execute the emergency live remediation path under documented guardrails.

[Cataldir]

Break-glass activated:

• PR merged: agent-update #140
• Timestamp (UTC): 2026-03-18T14:43:03.6235926Z
• Scope: emergency live remediation for 108dev only

Starting bypass execution sequence now for demo-critical endpoint stabilization.

[Cataldir]

New pipeline blocker addressed:

• Error observed in guardrail step:
  • CORS preflight missing allow-origin ... Returned: 'https://nice-flower-0be65ed0f.1.azurestaticapps.net\n'
• Root cause:
  • Header comparison did not normalize CR/LF/whitespace artifacts.
• Fix shipped to main:
  • commit �15c516 (ix(ci): normalize CORS preflight origin comparison ([Incident][108dev] APIM/SWA integration regressions: CORS + configuration themes 500 #139))
  • Workflow now trims SWA_DEFAULT_ORIGIN and �ccess-control-allow-origin before comparison.

Deployment run triggered by this fix:

• https://github.com/Azure-Samples/tutor/actions/runs/23250501029

[Cataldir]

Bypass action log (108dev break-glass) — frontend live endpoint recovery

Intent:

• Restore browser access and API functionality for all frontend-critical service endpoints.

Actions executed:

1. APIM CORS remediation (direct policy patch)

   • Added SWA origin https://nice-flower-0be65ed0f.1.azurestaticapps.net to missing API policies:
     • evaluation-api
     • lms-gateway-api
     • questions-api
     • upskilling-api
   • Verification:
     • OPTIONS preflight on /health for all 8 APIs now returns Access-Control-Allow-Origin with SWA origin.

2. Runtime Cosmos baseline re-application (direct Container Apps update)

   • Re-applied COSMOS_ENDPOINT, COSMOS_DATABASE, and all COSMOS_*_TABLE mappings for all 8 container apps in tutor-108dev.
   • Reason:
     • Functional endpoints were returning 500 with Invalid URL scheme or hostname due to empty COSMOS_* env vars after drift.

3. Cosmos data-plane RBAC verification

   • Confirmed Cosmos DB Built-in Data Contributor role assignment remains present for all 8 service managed identities.

Frontend-perspective verification (through APIM) after remediation:

• GET /api/configuration/themes -> 200
• GET /api/questions/questions -> 200
• GET /api/questions/answers -> 200
• GET /api/questions/graders -> 200
• GET /api/questions/assemblies -> 200
• GET /api/essays/essays -> 200
• GET /api/essays/resources -> 200
• GET /api/essays/assemblies -> 200
• GET /api/evaluation/datasets -> 200
• GET /api/upskilling/plans -> 200
• GET /api/avatar/cases -> 200
• GET /api/avatar/profile -> 200

Status:

• Frontend-critical backend endpoints are currently healthy from SWA origin.
• IaC/workflow backport remains required to prevent re-drift.