Skip to content

feat: activate Operations phase — trust-but-verify for extension compliance for v1 - #474

Open
steven-hooper wants to merge 2 commits into
awslabs:mainfrom
steven-hooper:operations-phase-and-extensions
Open

feat: activate Operations phase — trust-but-verify for extension compliance for v1#474
steven-hooper wants to merge 2 commits into
awslabs:mainfrom
steven-hooper:operations-phase-and-extensions

Conversation

@steven-hooper

@steven-hooper steven-hooper commented Jul 1, 2026

Copy link
Copy Markdown

Summary

Activates the Operations phase with a trust-but-verify approach for version 1 of AIDLC. Rather than trusting that Construction produces correct output, Operations independently verifies from three perspectives: static compliance checking (Rules Validation), deployment to a real environment (Deployment), and runtime testing that proves behaviour static analysis cannot (Post-Deployment Testing).

Also adds four extension domains (Observability, Recovery, Runbooks, Deployment), a rework loop that feeds gaps back to Construction for fixing, and Operations Retrofit for applying these rules to existing projects.

Resolves #473

Changes

New files (~40):

Directory Files Purpose
common/ design-rework.md, step-execution-accountability.md Rework loop mechanism, step tracking
operations/ rules-validation.md, deployment.md, post-deployment-testing.md Three Operations stages
extensions/observability/ 11 files (baseline, opt-in, metrics, logging, tracing, alarms, dashboards, differential, llm-rules, closed-loop, custom) Observability domain rules
extensions/recovery/ 5 files (baseline, opt-in, region, az, instance) Recovery domain rules
extensions/runbooks/ 6 files (baseline, opt-in, structure, automation, security, custom) Runbooks domain rules
extensions/deployment/ 6 files (baseline, opt-in, application, iac, pipeline, testing) Deployment domain rules
docs/ operations-phase.md, operations-changes-to-aidlc.md, operations-changes-to-extensions.md, operations-extensions.md, evaluator.md Documentation

Modified upstream files (12):

File Change
core-workflow.md Operations phase active (3 stages), Operations Retrofit stage, step-decision-log enforcement
common/depth-levels.md Blocking rules exclusion from adaptive depth
common/process-overview.md Operations phase description updated
common/terminology.md Extension and Operations terms added
construction/build-and-test.md Tool discovery and install enforcement
construction/code-generation.md Extension code plan steps
construction/infrastructure-design.md Extension rule mapping steps
construction/nfr-design.md Extension awareness
construction/nfr-requirements.md Rule × component matrix
inception/requirements-analysis.md Multi-round questions, extension opt-in process
inception/workflow-planning.md Operations in execution plan
operations/operations.md Placeholder removed (replaced by 3 stage files)

Modified evaluator files (6):

File Change
executor.py Operations-phase system prompt (common file reloading, artifact gates, handoff isolation)
simulator.py Hardened gap approval instructions for rework
config.py Extended timeout for Operations (8h vs 4h)
progress.py Handoff batch logging
runner.py Handoff stop patch (Strands Swarm workaround)
run_command.py AWS credential propagation into sandbox

Testing

Run Scenario Model Result
Arc Power Multi-region Lambda + DynamoDB + CodePipeline Sonnet 4.6 / Opus 4.6 25 handoffs, pipeline deployed and executed through pre-production, Human Approval gate reached
Retrofit Existing project without extensions Sonnet 4.6 / Opus 4.6 Detected, opted in, re-ran Construction, 128/128 rules PASS after 2 rework loops
Grid Spark Multi-region Lambda + DynamoDB + CodePipeline Sonnet 4.6 / Opus 4.6 25 handoffs, Rules Validation 125/125 PASS, pipeline deployed

Known Limitations

  • Context window pressure: Full rule set is large. Models with smaller context windows may degrade in quality during later stages.
  • Rework loop iteration limit: Complex gaps may not fully resolve within the configured limit (5 iterations). Remaining gaps are documented as known exceptions.
  • Model compliance: Significant effort has been made to improve deterministic execution (step accountability, validation checks, strong directives). However, models are probabilistic by nature and we cannot guarantee every instruction is followed in every run.
  • Evaluator recursion depth: Long Operations turns (>50 minutes) can overflow the Strands Swarm recursion limit. Does not affect real IDE usage.

Acknowledgement

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

…liance

Activate the Operations phase (previously a placeholder) with independent
verification of Construction output across three dimensions:

- Rules Validation: static compliance checking against extension rules
- Deployment: pipeline-first deployment to a real environment
- Post-Deployment Testing: runtime verification of behaviour

Add four extension domains: Observability (metrics, logging, tracing,
alarms, dashboards, synthetics), Recovery (region, AZ, instance),
Runbooks (structure, automation, security), Deployment (pipeline, IaC,
application, testing).

Rework loop feeds validation gaps back to Construction as requirements.
Operations Retrofit detects existing projects and offers to apply
extensions retrospectively without starting from scratch.

Tested with Claude Sonnet 4.6 / Opus 4.6 across 3 runs:
128/128 rules PASS, pipeline deployed to pre-production, 2 rework
iterations to full compliance.
@steven-hooper
steven-hooper requested review from a team as code owners July 1, 2026 08:43
@github-actions github-actions Bot added documentation Improvements or additions to documentation rules labels Jul 1, 2026
@steven-hooper steven-hooper changed the title feat: activate Operations phase — trust-but-verify for extension compliance feat: activate Operations phase — trust-but-verify for extension compliance for v1 Jul 1, 2026
Remove references to fork-specific run-evaluator.sh script.
Document credential injection via INJECT_AWS_CREDENTIALS env var
with upstream's uv run python run.py full command.
@steven-hooper
steven-hooper force-pushed the operations-phase-and-extensions branch from c021045 to 2aae771 Compare July 1, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC]: Activate Operations Phase — Trust-but-Verify for Extension Compliance

1 participant