[FRD 0008 U3] Complete ACA sandbox GA qualification - #160
[FRD 0008 U3] Complete ACA sandbox GA qualification#160Laveesh Rohra (larohra) wants to merge 36 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Run Python 3.13 and 3.14 cold-start qualification before the deployed N=5 diagnostics, while retaining manual/scheduled nonblocking policy and human-only single-runtime N=100 safeguards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Reduce shared-group matrix contention, classify live SSE failures without reading bodies, preserve admission deadline evidence, and close review gaps in lost-backing snapshot cleanup, docs, and boundary coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Translate Sandbox Group 401/403 responses into a redacted runtime error, atomically settle reserved provision state, preserve deterministic idempotent replay, and document the controller Data Owner requirement. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Emit structured timer-pass summaries and per-session reclaim audit events after successful durable settlement, with focused coverage and operator KQL. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
Use a 90-second setup budget with renewable 120-second provision leases and retry guidance, add typed redacted timeout observability across HTTP and MCP paths, and align live qualification watchdogs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
There was a problem hiding this comment.
Pull request overview
Completes ACA Sandbox GA qualification by enabling the opt-in backend, hardening lifecycle behavior, and adding deployed qualification coverage.
Changes:
- Enables supported Linux x86_64 Python 3.13/3.14 execution.
- Adds timeout, authorization, reconciliation, and audit handling.
- Expands unit, Azurite, live Azure, documentation, and pipeline coverage.
Reviewed changes
Copilot reviewed 80 out of 80 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_transport_ports.py |
Updates SDK client contract coverage. |
tests/test_transport_models.py |
Tests sandbox-state validation. |
tests/test_transport_aca_sdk.py |
Covers reconciliation and authorization handling. |
tests/test_session_runtime_validation.py |
Tests supported-host startup. |
tests/test_registration_handlers.py |
Tests timeout telemetry emission. |
tests/test_registration_endpoints.py |
Covers MCP ACA timeout behavior. |
tests/test_observability.py |
Tests runtime event spans. |
tests/test_harness_bootstrap.py |
Updates delivered import-path coverage. |
tests/test_fake_session_runtime.py |
Tests operation lease selection. |
tests/test_execution_setup_budget.py |
Tests the 90-second setup budget. |
tests/test_convention_guards.py |
Enforces typed setup deadlines. |
tests/test_controller_readiness.py |
Covers provisioning reconciliation and authorization. |
tests/test_controller_http.py |
Tests timeout and authorization responses. |
tests/test_controller_budget.py |
Tests setup deadline origins. |
tests/test_app.py |
Covers timer deadlines and summaries. |
tests/test_aca_smoke_support.py |
Tests live-package and model configuration. |
tests/test_aca_deployed_runtime_matrix.py |
Validates deployed runtime matrices. |
tests/live/test_aca_real_agent_turn.py |
Adds lower-level live model qualification. |
tests/live/test_aca_deployed_loss.py |
Adds deployed backing-loss qualification. |
tests/live/test_aca_deployed_lifecycle.py |
Adds deployed lifecycle qualification. |
tests/live/test_aca_deployed_agent_turn.py |
Adds authenticated public-turn qualification. |
tests/live/aca_deployed_loss_support.py |
Defines backing-loss assertions. |
tests/live/aca_deployed_load_support.py |
Adds load metrics and safeguards. |
tests/live/aca_deployed_lifecycle_support.py |
Adds lifecycle observation helpers. |
tests/live/aca_deployed_cold_start_support.py |
Adds cold-start metrics and bounds. |
tests/fixtures/live_aca_model_turn/model_turn.agent.md |
Defines the live model agent. |
tests/fixtures/live_aca_model_turn/agents.config.yaml |
Configures the live model fixture. |
tests/fixtures/live_aca_deployed_agent_turn/tools/qualification_hold.py |
Adds a load-test hold tool. |
tests/fixtures/live_aca_deployed_agent_turn/host.json |
Configures deployed function timeout. |
tests/fixtures/live_aca_deployed_agent_turn/function_app.py |
Defines the deployed fixture app. |
tests/fixtures/live_aca_deployed_agent_turn/deployed_turn.agent.md |
Defines the public qualification agent. |
tests/fixtures/live_aca_deployed_agent_turn/deployed_load.agent.md |
Defines the load qualification agent. |
tests/fixtures/live_aca_deployed_agent_turn/agents.config.yaml |
Configures deployed ACA sessions. |
tests/fixtures/live_aca_deployed_agent_turn/.gitignore |
Excludes packaged dependencies. |
tests/fixtures/live_aca_deployed_agent_turn/.funcignore |
Excludes local deployment artifacts. |
tests/fixtures/config_scenarios/29_aca_sandbox_valid/main.agent.md |
Adds a valid ACA scenario. |
tests/fixtures/config_scenarios/29_aca_sandbox_valid/agents.config.yaml |
Marks the scenario as valid. |
tests/fixtures/config_scenarios/29_aca_sandbox_valid_but_unavailable/main.agent.md |
Removes obsolete gated fixture. |
tests/endtoend/test_session_state_store_azurite.py |
Verifies persisted lease durations. |
tests/doubles/fake_session_runtime.py |
Aligns fake lease semantics. |
tests/conftest.py |
Adds live qualification options. |
src/azure_functions_agents/transport/transport_models.py |
Adds provider state and reconciliation models. |
src/azure_functions_agents/transport/aca_sdk.py |
Hardens ACA SDK reconciliation and authorization. |
src/azure_functions_agents/session_state/store.py |
Applies kind-specific operation leases. |
src/azure_functions_agents/session_state/session_models.py |
Adds provisioning reconciliation state. |
src/azure_functions_agents/registration/endpoints.py |
Routes MCP calls through ACA execution. |
src/azure_functions_agents/registration/_handlers.py |
Emits setup-timeout observations. |
src/azure_functions_agents/execution/setup_budget.py |
Adds typed 90-second setup budgets. |
src/azure_functions_agents/execution/aca_sandbox.py |
Applies setup deadlines across execution. |
src/azure_functions_agents/execution/__init__.py |
Lazily imports ACA execution components. |
src/azure_functions_agents/controller/reconciler.py |
Handles backing loss, suspension, and audits. |
src/azure_functions_agents/controller/http.py |
Maps timeout and authorization responses. |
src/azure_functions_agents/controller/budget.py |
Preserves setup timing metadata. |
src/azure_functions_agents/config/validation.py |
Opens the supported ACA capability gate. |
src/azure_functions_agents/app.py |
Adds dependency validation and timer deadlines. |
src/azure_functions_agents/_observability.py |
Adds customer-queryable runtime events. |
README.md |
Documents ACA backend availability. |
eng/templates/official/jobs/e2e-tests.yml |
Adds deployed qualification jobs. |
eng/ci/variables/aca-deployed-runtime-targets.yml |
Defines runtime deployment targets. |
eng/ci/official-build.yml |
Forwards ACA qualification parameters. |
eng/ci/e2e-tests.yml |
Configures the ACA runtime matrix. |
docs/observability.md |
Documents timeout and reconciliation telemetry. |
docs/front-matter-spec.md |
Updates ACA availability guidance. |
docs/aca-sandbox-session-runtime.md |
Expands ACA deployment and operations guidance. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| except SessionActivationSetupTimeoutError as exc: | ||
| raise RunSubmissionIndeterminateError( | ||
| "Run launch may have started but journal acceptance was not confirmed." | ||
| ) from exc |
| "The resumed public session did not become ready within the bounded " | ||
| "setup-deadline retry window." | ||
| ) | ||
| await asyncio.sleep(_SETUP_RETRY_DELAY_SECONDS) |
| _log_session_reclaimed( | ||
| session_id=latest.record.session_id, | ||
| sandbox_id=latest.record.sandbox_id, | ||
| backing_deleted=backing_deleted, | ||
| deleted_snapshot_count=deleted_snapshot_count, | ||
| ) |
hallvictoria
left a comment
hallvictoria
left a comment
There was a problem hiding this comment.
Blocking findings
-
[P1] Accepted create can be terminalized after a recovery 403 -
src/azure_functions_agents/transport/aca_sdk.py:337. If_await_create_result()first raises a timeout/runtime error afterbegin_createwas accepted, the generic recovery calls_find_failed_create_sandboxes()directly. A 401/403 from that lookup replaces the original error withSandboxGroupAuthorizationError; readiness treats it as deterministic, fails the run, and marks the session deleting even though the labeled sandbox may exist. This violates FRD Decision 160. Route everycreate_acceptedrecovery through_recover_stable_accepted_create()and add timeout-then-403 coverage. -
[P1] Setup cancellation can retry an accepted create as a second create -
src/azure_functions_agents/controller/readiness.py:1618._within_setup_budget()can cancelprovider.create()after ACA accepts the request while the durable phase remainsprovision_create. After lease expiry, takeover retries withreconcile_only=False; if label-list visibility is delayed, the adapter issues another create with the same operation label. Persistprovision_reconcilefor this ambiguous timeout, or make stable-label retries reconcile-only until absence is authoritative, and test delayed list visibility. -
[P1] Status/result authorization failures escape as 500 -
src/azure_functions_agents/execution/aca_sandbox.py:468.get_run()invokes targeted reconciliation here and on nonterminal reads. The reconciler performs initiallist_sandboxes()/list_snapshots()calls outside its per-session handling, so provider 401/403 escapes asSandboxGroupAuthorizationError.read_status()andread_result()do not map it (orSessionActivationAuthorizationError) like submit/cancel do. Normalize this to the documented redactedsandbox_group_authorization_failed503 and add both management-read tests. -
[P1] The explicit <=2-second p95 visibility gate is not enforced -
tests/live/test_aca_deployed_load.py:341. The test computes submission-to-first-SSE-event percentiles and only renders them. It never asserts a threshold, and this end-to-end latency is not state-change-to-observation visibility at <=1 poll/s. Add timestamped state transitions plus bounded status/event polling and fail when p95 exceeds two seconds. -
[P1] The Python 3.13/3.14 matrix does not attest the remote runtime or artifact -
eng/templates/official/jobs/e2e-tests.yml:230. The matrix changes the test client interpreter and static app URL, but never deploys the build or verifies the remote Python minor/commit/package digest. A stale app, two apps on one Python minor, or an app without this PR can satisfy both labeled legs. Deploy the built artifact per leg or require remote runtime and package-digest attestation before qualification.
I did not repeat the three existing Copilot findings. I also did not flag the Manual/Scheduled nonblocking policy itself because FRD Decisions 156-157 explicitly retain it.
Use one flat sliding 120-second lease for every durable operation kind while preserving fencing, takeover, and journal recovery semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6242d7f-3c99-4dd6-b29d-3f43731e2245
| - '5' | ||
| - '100' | ||
| - name: acaProvisionConcurrency | ||
| displayName: 'Deployed ACA provisioning concurrency' |
There was a problem hiding this comment.
This one does not work with "both". Add that to the name or somewhere so its evident.
| - name: acaServiceConnection | ||
| displayName: 'ACA smoke: Azure service connection' | ||
| type: string | ||
| default: 'larohra-sandboxgroup-test' |
There was a problem hiding this comment.
revert default to the original value
| - name: acaServiceConnection | ||
| type: string | ||
| default: 'saf-foundry-connection' | ||
| default: 'larohra-sandboxgroup-test' |
There was a problem hiding this comment.
Revert default
| - job: "ACADeployedAgentTurn" | ||
| displayName: 'Run deployed ACA agent-turn, lifecycle, and backing-loss qualification' | ||
| dependsOn: ACADeployedColdStart | ||
| condition: >- |
There was a problem hiding this comment.
Remove this condition, it can run in PRs too now
| # The job default is the sole automated N=5 diagnostic. Both runtime | ||
| # legs share one Sandbox Group, so each may provision only one session. | ||
| # A human single-runtime N=100 request may use all four known safe slots. | ||
| case "${ACA_DEPLOYED_CONFIGURED_LOAD_CONCURRENCY:-}" in |
There was a problem hiding this comment.
Why only 5 or 100? Make this configurable to any value <=100
| fi | ||
| export AZURE_FUNCTIONS_AGENTS_ACA_LOAD_CONCURRENCY="${ACA_DEPLOYED_CONFIGURED_LOAD_CONCURRENCY}" | ||
| export AZURE_FUNCTIONS_AGENTS_ACA_PROVISION_CONCURRENCY="${ACA_DEPLOYED_CONFIGURED_PROVISION_CONCURRENCY}" | ||
| python - <<'PY' |
There was a problem hiding this comment.
Make this a script under eng/scripts
| image: 1es-ubuntu-22.04 | ||
| os: linux | ||
|
|
||
| strategy: |
There was a problem hiding this comment.
This block is being repeated, can we make it a common module somehow?
| scriptType: bash | ||
| scriptLocation: inlineScript | ||
| inlineScript: | | ||
| set -euo pipefail |
There was a problem hiding this comment.
These validations are repeated, make it a common script, avoid duplication
| esac | ||
| export AZURE_FUNCTIONS_AGENTS_ACA_COLD_START_SAMPLES="${ACA_DEPLOYED_COLD_START_SAMPLES}" | ||
| fi | ||
| python - <<'PY' |
There was a problem hiding this comment.
Move to a dedicated script
Summary
Completes U3, the final GA qualification unit for the ACA Sandbox Session Runtime from FRD 0008.
Retry-After: 120, and keeps the 180-second synchronous wall cap.Real Azure evidence
func-agent-func-twm2hp52kchdm: controllerContributorproduced repeatable data-plane 403/setup-504 loops; grantingContainer Apps SandboxGroup Data Ownerchanged list to 200 and the public model-backed turn completed withE2E_OK. The runtime now fails that misconfiguration fast, settles durable state, and maps submit/cancel/resume to the same redacted outcome.Observed fresh-session customer-path timing before the timeout expansion:
Safety boundaries
AgentExecutionBackendremains exactly four methods.transport/aca_sdk.pyremains the only production preview-SDK import boundary./var/lib/azurefunctions-agents-runtimeremains the sole sandbox root.Validation status
7aa4c424: Ruff clean, strict mypy clean for 97 files,2038 passed / 64 skipped / 82 deselected, coverage XML generated.Remaining before ready for merge
Target:
feature/aca-sandboxes. The eventual feature-to-mainmerge remains a separate human decision.