diff --git a/AGENTS.md b/AGENTS.md index 439cf97..8b7011e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,6 +74,15 @@ v0.1 (core concepts) 5. Add validation test in `tests/` 6. State maturity clearly in the document +## Feature Completion Expectations + +- Every spec or schema behavior change must include tests or fixtures. +- Normative or user-guiding changes must include or update realistic examples. +- When a milestone item is completed or its status changes, update `README.md` + and `ROADMAP.md` in the same change. +- When a change affects an earlier layer, also update dependent milestone + documents, schemas, fixtures, and tests before considering the work done. + ## Pre-push Checklist Run `make check` before every push. It runs the schema validation tests. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d86b960..4f6cfc9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -15,6 +15,14 @@ AI Operations Specification contributions should improve interoperability for in A proposal SHOULD state the interoperability problem, affected concepts, compatibility impact, privacy implications, and at least one realistic example. Changes that affect an earlier layer MUST update dependent documents, schemas, fixtures, and tests in the same proposal. +Feature completion expectations: + +- Every spec or schema behavior change must include tests or fixtures. +- Normative or user-guiding changes should include or update realistic + examples in the same proposal. +- If a roadmap or milestone item is completed or its status changes, update + `README.md` and `ROADMAP.md` in the same pull request. + Do not mark an acceptance gate complete without recorded evidence. External review gates require evidence from an independent implementation or standard, not only internal agreement. ## Validation diff --git a/MILESTONE_CLOSEOUT_PLAN.md b/MILESTONE_CLOSEOUT_PLAN.md new file mode 100644 index 0000000..0297a09 --- /dev/null +++ b/MILESTONE_CLOSEOUT_PLAN.md @@ -0,0 +1,194 @@ +# AIOS Milestone Close-Out Plan + +This document turns the current `v0.1` through `v0.4` draft milestones into a concrete path to reviewed completion. + +It does not change milestone status by itself. A milestone is complete only when its acceptance criteria, dependency gates, repo updates, and validation evidence are all closed. + +## Current baseline + +- Documented milestone chain: + - `v0.1` core concepts + - `v0.2` relationships + - `v0.3` semantic conventions + - `v0.4` JSON Schemas +- Local validation baseline as of `2026-08-09`: + - `UV_CACHE_DIR=/tmp/uv-cache uv run pytest` + - `20 passed` +- Immediate blocker pattern: + - most remaining work is review evidence, cross-document consistency, and independent implementation validation + - later milestones cannot freeze before earlier review gates close + +## Done criteria by milestone + +### v0.1 + +Must close: + +- review definitions against at least two independent frameworks +- record decisions and update all `v0.1` documents consistently +- show that two independent implementers can classify the same runtime objects without package-specific knowledge + +Evidence package: + +- framework mapping notes for two independent frameworks +- classification worksheet using shared runtime examples +- maintainer decision log +- consistency pass across `README.md`, `core-concepts.md`, `terminology.md`, `examples.md`, and `acceptance-criteria.md` + +### v0.2 + +Must close: + +- validate the relationship vocabulary against at least two independent agent frameworks +- confirm ordering and retry semantics +- record decisions and update affected documents + +Evidence package: + +- graph reconstruction examples from the same scenario across two frameworks +- explicit decisions for parent-child, causal order, retry attempt, branch, join, and loop semantics +- updated examples and any test coverage needed for graph/document invariants + +### v0.3 + +Must close: + +- validate conventions against OpenTelemetry design principles +- validate event coverage against at least two reference implementations +- decide whether transport mappings belong in separate implementation profiles +- record decisions and update affected documents + +Evidence package: + +- event catalog review against OTel naming, cardinality, lifecycle, and extension principles +- coverage matrix showing each runtime area is representable by canonical or extension events +- written decision separating normative semantics from transport bindings + +### v0.4 + +Must close: + +- resolve all open `v0.1` through `v0.3` review gates +- validate artifacts produced by at least two independent implementations +- review identifier, reference, extension, and unknown-value behavior +- replace the draft version marker when the milestone freezes + +Evidence package: + +- artifact samples from at least two independent producers +- validation outputs against current schemas +- issue log for identifier/reference/extension edge cases +- final schema/version marker update and fixture refresh + +## Workstreams + +## 1. Repo-internal work + +These items can be completed directly in `ai-operations-spec`. + +- normalize milestone language across `README.md`, `ROADMAP.md`, `SPECIFICATION.md`, and milestone READMEs +- create or expand decision logs for unresolved reviewer questions +- add scenario-based review worksheets for object classification, graph reconstruction, and semantic event coverage +- add tests for any newly clarified document invariants or schema edge cases +- add a review ledger documenting which acceptance gates are closed, by whom, and with what evidence + +## 2. Cross-project coordination + +These items need evidence from sibling repos in this workspace. + +- `agenticlens` + - export draft-conforming artifacts + - prove consumer and producer use of relationships, events, and schemas + - supply examples for findings, traces, and conformance-oriented validation +- `agentic-chaos` + - exercise degraded, failed, retried, and recovered execution paths + - validate whether reliability and incident concepts are sufficient + - produce artifacts that stress event semantics and edge-case relationships +- `mcp-server` + - validate consumer-side reading, transformation, and interoperability expectations + - verify artifact references and unknown-value handling in a control-plane style consumer + +## 3. Independent review work + +These items cannot honestly be self-certified inside this repo alone. + +- choose two non-DeepAgentLabs frameworks or implementers for external review +- run the same scenario pack through both +- compare concept classification results for `v0.1` +- compare execution graph reconstruction results for `v0.2` +- compare event semantics interpretation results for `v0.3` +- compare schema artifact output and validation results for `v0.4` + +## Review sequence + +The fastest safe path is: + +1. Close `v0.1` first. +2. Use the finalized `v0.1` vocabulary to tighten `v0.2`. +3. Freeze `v0.3` only after `v0.2` ordering and retry semantics are explicit. +4. Freeze `v0.4` last after artifact validation from independent producers. + +Doing `v0.2` through `v0.4` first would create churn because each later layer inherits names and boundaries from `v0.1`. + +## Proposed close-out checklist + +### Phase A: close the `v0.1` foundation + +- create a single review packet with: + - concept glossary + - boundary decisions + - three to five shared runtime scenarios +- run the packet against two independent frameworks +- capture disagreements and resolve naming or boundary ambiguity +- update all `v0.1` docs together +- mark `v0.1` review gates complete only after evidence is recorded + +### Phase B: lock `v0.2` execution semantics + +- derive graph examples from the same shared scenarios +- write explicit rules for: + - ordering + - retries + - branches + - joins + - loops + - parent-child runs +- validate equivalent graph reconstruction across two implementations +- update relationship docs, examples, and any tests together + +### Phase C: lock `v0.3` semantics + +- produce a catalog review worksheet covering every runtime area +- align event naming and extension guidance with OTel design principles +- decide and document that transport mappings stay outside the core spec unless added as separate non-normative profiles +- validate event coverage using artifacts from at least two implementations + +### Phase D: freeze `v0.4` schemas + +- generate artifacts from at least two independent producers +- validate all artifacts against the schemas +- add missing edge-case fixtures for identifiers, references, extensions, and unknown values +- replace draft version markers only after `v0.1` through `v0.3` are closed + +## Recommended repo additions + +The repo would benefit from a small review infrastructure layer: + +- `reviews/decision-log.md` +- `reviews/evidence-matrix.md` +- `reviews/scenarios/` +- `reviews/framework-mappings/` + +These would let the project prove why a milestone is reviewed instead of only claiming it. + +## Exit standard + +Do not mark `v0.1` through `v0.4` as done when only the prose looks complete. + +Mark them done only when: + +- the acceptance criteria are checked off +- dependencies are closed in order +- tests and fixtures pass +- review evidence is committed in the repo +- at least the required independent implementation validation is recorded diff --git a/README.md b/README.md index 9304473..ab2a342 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,27 @@ DeepAgentLabs stewards the specification today and is its sole editor. AgenticLe **Pre-release draft. Do not claim stable conformance.** +## Release Status + +- **v0.1** 🏗️ In review — Core runtime concepts and boundaries +- **v0.2** 🚧 Exploratory draft — Relationships and execution graphs +- **v0.3** 🚧 Exploratory draft — Transport-neutral semantic events +- **v0.4** 🚧 Exploratory draft — JSON Schema artifacts +- **v0.5** 🚧 Planned — Versioning and compatibility rules +- **v0.6** 🚧 Planned — Canonical examples and extension model +- **v0.6.x** 🚧 Planned — Provenance, evidence, and operational artifacts +- **v1.0** 🚧 Planned — Stable specification + +## Next Steps + +- Close the remaining v0.1 external review gate so later layers can move + toward freeze. +- Validate the v0.4 draft artifacts against at least two independent + implementations. +- Formalize versioning/compatibility rules and the extension model. +- Standardize provenance/evidence concepts and publish producer-independent + conformance fixtures and rules. + The work is cumulative, not four competing formats: | Layer | Defines | Status | diff --git a/ROADMAP.md b/ROADMAP.md index b827413..df53c04 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,5 +1,67 @@ # AI Operations Specification Roadmap +## Release Status + +- **v0.1** 🏗️ In review — Core runtime concepts and boundaries +- **v0.2** 🚧 Exploratory draft — Relationships and execution graphs +- **v0.3** 🚧 Exploratory draft — Transport-neutral semantic events +- **v0.4** 🚧 Exploratory draft — JSON Schema artifacts +- **v0.5** 🚧 Planned — Versioning and compatibility rules +- **v0.6** 🚧 Planned — Canonical examples and extension model +- **v0.6.x** 🚧 Planned — Provenance, evidence, and operational artifacts +- **v1.0** 🚧 Planned — Stable specification + +## Next Steps + +- Resolve the remaining v0.1 review gate and keep later milestones clearly + labeled as draft until those dependencies close. +- Prove the v0.4 draft against at least two independent producers and + consumers. +- Define compatibility/versioning policy before claiming stable consumption + guarantees. +- Promote provenance, evidence, and conformance fixtures from implementation + practice into spec-standard artifacts once the underlying model settles. + +## Cross-Project Dependencies + +AIOS is the normative specification, not an implementation package, but its +roadmap still depends on evidence from sibling projects and independent +implementers. + +- `agenticlens` + Provides real producer/consumer evidence for traces, findings, conformance + tooling, and schema usage. +- `agentic-chaos` + Provides resilience and fault evidence that tests whether the model can + represent degraded, failed, and recovered behavior. +- `deep-agentic-core-mcp` + Provides a consumer/control-plane view that exercises artifact validation and + interoperability across tools. + +For roadmap planning, distinguish: + +- `Depends on`: an earlier AIOS layer that must freeze first. +- `Evidence from`: sibling repos or independent implementers that should + validate the design in practice. +- `Coordinate with`: implementation repos that may need doc, fixture, or + terminology updates when the spec changes. + +## Definition of Done + +A roadmap item is done only when all applicable work is complete: + +- normative text is written with scope, boundaries, and maturity clearly + stated +- examples, fixtures, and schema changes are added or updated together +- tests validate the intended structure or document invariants +- `README.md` and this roadmap are updated when milestone status or user + guidance changes +- dependent milestone documents are updated when an earlier layer changes +- implementation evidence or independent review is recorded for any acceptance + gate that requires it +- draft, reviewed, and stable claims remain consistent across all milestone + documents + ## Direction DeepAgentLabs builds the AI Operations Specification **specification-first**. The standard defines meaning before SDK ergonomics, exporter fields, dashboards, or integrations. @@ -123,6 +185,15 @@ Formalize concepts proven in implementation repositories. document from canonical `aiops.*` events and common attributes to OTel spans, span events, and resource attributes, so producers already emitting OTel do not need to invent their own bridge independently +- verification-signal schema — a standard shape for second-opinion/judge + results (verdict, confidence, evidence-grounding, safety concerns) as a + first-class signal object, once judge-output shapes stabilize across + implementation repos; candidate prior art: `devops-open-agent`'s + LLM-as-a-Judge verifier output +- audit-event schema — a standard shape for who-did-what operational events + (actor, action, target artifact reference, timestamp, no secret values), + feeding the incident-context/operator-facing summary semantics above; + candidate prior art: `devops-open-agent`'s structured audit log Success means implementations can attach provenance to findings using spec-standard objects, and producers can run conformance checks independently. diff --git a/reviews/EXTERNAL_REVIEW_INVITE.md b/reviews/EXTERNAL_REVIEW_INVITE.md new file mode 100644 index 0000000..a0375a4 --- /dev/null +++ b/reviews/EXTERNAL_REVIEW_INVITE.md @@ -0,0 +1,104 @@ +# External Review Invite + +This is a ready-to-send packet for closing `reviews/decision-log.md` decision +`D-001` — the one gate every `v0.1`–`v0.4` acceptance criteria still needs: +review from someone with no DeepAgentLabs authorship, funding, or editorial +role over AIOS. + +It exists so outreach doesn't stall on "who do we even ask" or "what do we +send them." Copy the message below, fill the brackets, and send it. + +## Who to send it to + +Anyone who fits the independence definition in `D-001` and has real exposure +to agent/LLM runtime instrumentation. Good candidate categories, roughly in +order of how directly comparable their domain is: + +- Maintainers or core contributors of **other agent/LLM observability or + tracing projects** (open-source tracing/eval tools, OpenTelemetry + GenAI semantic-conventions contributors, agent-framework tracing + maintainers). They will have the strongest opinions on `v0.2`/`v0.3`. +- **OpenTelemetry** contributors specifically, for the `v0.3` OTel-alignment + gate (`specification/v0.3/acceptance-criteria.md`) — someone active in + the OTel GenAI/AI semantic-conventions working group is close to ideal. +- Independent **AI reliability/safety practitioners** (incident response, + red-teaming, eval-tooling authors) for the `v0.1`/`v0.3` safety-and-incident + scenario. +- A **second, structurally different** agent framework or platform than the + first reviewer's — pick for maximum disagreement surface, not agreement. + +Two completed reviews close `D-001` (the closure rule in +`reviews/framework-mappings/README.md` requires at least two independent +mapping files against the same scenario set). They don't need to be from the +same category above — variety is more useful than volume. + +## Copy-paste message + +```text +Subject: 30–90 min external review request: AI Operations Specification (AIOS) + +Hi [name], + +I'm working on the AI Operations Specification (AIOS) — a vendor-neutral +draft contract for describing what happens when an AI/agentic system runs +(Runs, Steps, model interactions, tool calls, retrieval, memory, +evaluations, safety signals, incidents). Repo: +https://github.com/DeepAgentLabs/ai-operations-spec + +It's specification-first, not tied to one SDK, and it's explicitly not +"vendor-neutral" yet in fact — that label is only earned once independent +implementers outside DeepAgentLabs have reviewed it. Right now the only +reviewers have been DeepAgentLabs' own sibling projects, which we've been +upfront isn't enough (see reviews/decision-log.md, decision D-001). + +I'd like to ask for your independent read, with no obligation beyond what +you have time for: + +1. Pick 1–3 scenarios from reviews/scenarios/ (each is a short narrative, + ~5 min read): + - scenario-01-customer-support.md + - scenario-02-retry-and-recovery.md + - scenario-03-safety-and-incident.md +2. Work through how you'd classify the objects, relationships, and events + using specification/v0.1–v0.4, from your own framework's or your own + independent judgment — not ours. +3. Record it using reviews/framework-mappings/external-reviewer-template.md + (copy it to reviews/framework-mappings/.md). + +Rough time: 30–90 minutes depending on how many scenarios and how deep you +go. Partial answers or "this doesn't map, here's why" are exactly as useful +as full agreement — disagreement is the point of an independent review. + +To send it back: open a PR against the repo with your file under +reviews/framework-mappings/, or just reply with the filled template and +I'll commit it with attribution. Public credit in the decision log is +optional — happy to keep you anonymous if you'd rather. + +No pressure if this isn't a fit right now — thanks either way for +considering it. + +[your name] +``` + +## Short version (for a DM / issue comment / forum post) + +```text +Reviewing an early-draft spec for AI agent runtime telemetry (Runs, Steps, +model calls, tool calls, retrieval, safety signals, incidents — +https://github.com/DeepAgentLabs/ai-operations-spec) and looking for an +independent (non-DeepAgentLabs) read on whether the concepts hold up +against a real framework. ~30–90 min: pick a scenario from reviews/scenarios/, +fill reviews/framework-mappings/external-reviewer-template.md. Interested? +``` + +## After you get a response back + +1. Save their filled template as `reviews/framework-mappings/.md`. +2. Update `reviews/decision-log.md` `D-001`: move it toward `accepted` once + two such files exist, and summarize where the independent reviewers + agreed or disagreed with the DeepAgentLabs-internal mappings. +3. Update `reviews/evidence-matrix.md` rows that cite "at least two + independent frameworks" for `v0.1`, `v0.2`, and `v0.3` — flip to `closed` + only for rows their review actually covered, not the whole matrix at + once. +4. Only then reconsider the `README.md`/`ROADMAP.md` Release Status badges. diff --git a/reviews/README.md b/reviews/README.md new file mode 100644 index 0000000..7f225a1 --- /dev/null +++ b/reviews/README.md @@ -0,0 +1,38 @@ +# Review Pack + +This directory contains the evidence scaffolding needed to move `v0.1` through `v0.4` from draft status toward reviewed acceptance. + +These files do not by themselves close milestone gates. They provide a structured place to record: + +- shared review scenarios +- framework mapping results +- acceptance-gate evidence +- reviewer decisions + +## Contents + +- `decision-log.md` + Records review outcomes, unresolved questions, and accepted decisions. +- `evidence-matrix.md` + Maps each milestone acceptance gate to concrete evidence. +- `framework-mappings/` + Per-framework review notes showing how the same scenario is interpreted. +- `scenarios/` + Shared scenarios used across `v0.1` through `v0.4`. +- `EXTERNAL_REVIEW_INVITE.md` + Ready-to-send outreach packet for closing decision `D-001` — the + independent, non-DeepAgentLabs review gate every milestone still needs. + +## How to use this pack + +1. Pick one scenario from `scenarios/`. +2. Review it through at least two independent frameworks or implementations. +3. Record concept classification results under `framework-mappings/`. +4. Compare graph, event, and artifact outputs across implementations. +5. Update `evidence-matrix.md` with links to committed evidence. +6. Record accepted decisions and ambiguities in `decision-log.md`. + +## Review rule + +Do not mark a milestone as reviewed only because the prose looks complete. +Mark it reviewed only when the corresponding acceptance gates have linked evidence in this directory. diff --git a/reviews/artifacts/README.md b/reviews/artifacts/README.md new file mode 100644 index 0000000..b98c1fc --- /dev/null +++ b/reviews/artifacts/README.md @@ -0,0 +1,25 @@ +# Review Artifacts + +These artifacts are AIOS `v0.4-draft` review examples derived from real sibling-project outputs in this workspace. + +They are intended as producer-evidence fixtures for review and conformance work. + +## Included artifacts + +- `agenticlens-support-run.aios.json` + Derived from `agenticlens/examples/artifacts/support-run.json` +- `agentic-chaos-customer-support.aios.json` + Derived from `/tmp/agentic-chaos-review-report.json` +- `*.conformance.json` + Saved AIOS conformance results for the derived review artifacts + +## Important note + +These are reviewed mapping artifacts, not proof that the native producer formats are already identical to AIOS. + +They demonstrate that: + +- meaningful AIOS-aligned `run` artifacts can be derived from real producer output +- the current AIOS draft can express runtime structure, retries, degradation, tools, and reliability evidence from both sibling implementations +- both derived artifacts passed schema and semantic conformance checks on 2026-08-09 +- this evidence does not close the independent-producer review gate because both artifacts come from DeepAgentLabs-owned sibling projects diff --git a/reviews/artifacts/agentic-chaos-customer-support.aios.json b/reviews/artifacts/agentic-chaos-customer-support.aios.json new file mode 100644 index 0000000..22ab238 --- /dev/null +++ b/reviews/artifacts/agentic-chaos-customer-support.aios.json @@ -0,0 +1,433 @@ +{ + "spec_version": "0.4-draft", + "artifact_type": "run", + "id": "run-agentic-chaos-customer-support-20260809", + "workflow_id": "workflow-agentic-chaos-customer-support", + "started_at": "2026-08-09T02:32:58.452254Z", + "ended_at": "2026-08-09T02:33:03.454629Z", + "status": "completed", + "requests": [ + { + "id": "request-agentic-chaos-support-001", + "name": "customer support request under chaos", + "status": "completed" + } + ], + "steps": [ + { + "id": "step-agentic-chaos-planner", + "name": "Planner", + "category": "planning", + "status": "completed" + }, + { + "id": "step-agentic-chaos-retriever", + "name": "Retriever", + "category": "retrieval", + "status": "failed" + }, + { + "id": "step-agentic-chaos-final-response", + "name": "Final Response", + "category": "model", + "status": "completed" + } + ], + "agents": [], + "occurrences": [ + { + "id": "model-agentic-chaos-planner-001", + "type": "model_interaction", + "name": "planner call", + "status": "failed", + "attributes": { + "logical_activity_id": "planner-call", + "attempt": 1 + } + }, + { + "id": "model-agentic-chaos-planner-002", + "type": "model_interaction", + "name": "planner call", + "status": "failed", + "attributes": { + "logical_activity_id": "planner-call", + "attempt": 2 + } + }, + { + "id": "model-agentic-chaos-planner-003", + "type": "model_interaction", + "name": "planner call", + "status": "completed", + "attributes": { + "logical_activity_id": "planner-call", + "attempt": 3 + } + }, + { + "id": "retrieval-agentic-chaos-001", + "type": "rag_retrieval", + "name": "retriever call", + "status": "failed" + }, + { + "id": "model-agentic-chaos-final-response-001", + "type": "model_interaction", + "name": "final response call", + "status": "completed" + } + ], + "evidence": [ + { + "id": "reliability-event-planner-attempt-1", + "type": "reliability_event", + "name": "planner rate limit attempt 1", + "status": "detected", + "attributes": { + "fault_type": "rate_limit_storm", + "attempt": 1, + "retry_after": 1.0, + "source_event_id": "3ba644fd-b85a-4e05-8e1b-481ddfbeb6a4" + } + }, + { + "id": "reliability-event-planner-attempt-2", + "type": "reliability_event", + "name": "planner rate limit attempt 2", + "status": "detected", + "attributes": { + "fault_type": "rate_limit_storm", + "attempt": 2, + "retry_after": 1.0, + "source_event_id": "b2344d09-5f62-4efd-930e-fafee63e9293" + } + }, + { + "id": "reliability-event-planner-attempt-3", + "type": "reliability_event", + "name": "planner rate limit attempt 3", + "status": "recovered", + "attributes": { + "fault_type": "rate_limit_storm", + "attempt": 3, + "retry_after": 1.0, + "source_event_id": "57672be0-e1c4-47e4-adea-8617fb3397ce" + } + }, + { + "id": "reliability-event-retriever-timeout", + "type": "reliability_event", + "name": "retriever token timeout", + "status": "detected", + "attributes": { + "fault_type": "token_timeout", + "hang_seconds": 2.0, + "mode": "raise", + "source_event_id": "f5967188-716c-4fae-aad6-65b24714a102" + } + }, + { + "id": "reliability-event-final-degradation", + "type": "reliability_event", + "name": "final response silent degradation", + "status": "detected", + "attributes": { + "fault_type": "silent_degradation", + "source_event_id": "e31c1ad3-d051-4222-8f02-b63dd9874547" + } + } + ], + "incidents": [], + "relationships": [ + { + "id": "rel-agentic-chaos-run-of", + "type": "run-of", + "source": { + "id": "run-agentic-chaos-customer-support-20260809", + "type": "run" + }, + "target": { + "id": "workflow-agentic-chaos-customer-support", + "type": "workflow", + "external": true + } + }, + { + "id": "rel-agentic-chaos-contains-planner", + "type": "contains", + "source": { + "id": "run-agentic-chaos-customer-support-20260809", + "type": "run" + }, + "target": { + "id": "step-agentic-chaos-planner", + "type": "step" + } + }, + { + "id": "rel-agentic-chaos-contains-retriever", + "type": "contains", + "source": { + "id": "run-agentic-chaos-customer-support-20260809", + "type": "run" + }, + "target": { + "id": "step-agentic-chaos-retriever", + "type": "step" + } + }, + { + "id": "rel-agentic-chaos-contains-final", + "type": "contains", + "source": { + "id": "run-agentic-chaos-customer-support-20260809", + "type": "run" + }, + "target": { + "id": "step-agentic-chaos-final-response", + "type": "step" + } + }, + { + "id": "rel-agentic-chaos-planner-follows-retriever", + "type": "follows", + "source": { + "id": "step-agentic-chaos-planner", + "type": "step" + }, + "target": { + "id": "step-agentic-chaos-retriever", + "type": "step" + } + }, + { + "id": "rel-agentic-chaos-final-depends-on-retriever", + "type": "depends-on", + "source": { + "id": "step-agentic-chaos-final-response", + "type": "step" + }, + "target": { + "id": "step-agentic-chaos-retriever", + "type": "step" + } + }, + { + "id": "rel-agentic-chaos-planner-attempt-1-observed", + "type": "observed-in", + "source": { + "id": "model-agentic-chaos-planner-001", + "type": "model_interaction" + }, + "target": { + "id": "step-agentic-chaos-planner", + "type": "step" + } + }, + { + "id": "rel-agentic-chaos-planner-attempt-2-observed", + "type": "observed-in", + "source": { + "id": "model-agentic-chaos-planner-002", + "type": "model_interaction" + }, + "target": { + "id": "step-agentic-chaos-planner", + "type": "step" + } + }, + { + "id": "rel-agentic-chaos-planner-attempt-3-observed", + "type": "observed-in", + "source": { + "id": "model-agentic-chaos-planner-003", + "type": "model_interaction" + }, + "target": { + "id": "step-agentic-chaos-planner", + "type": "step" + } + }, + { + "id": "rel-agentic-chaos-planner-attempt-1-follows-attempt-2", + "type": "follows", + "source": { + "id": "model-agentic-chaos-planner-001", + "type": "model_interaction" + }, + "target": { + "id": "model-agentic-chaos-planner-002", + "type": "model_interaction" + } + }, + { + "id": "rel-agentic-chaos-planner-attempt-2-follows-attempt-3", + "type": "follows", + "source": { + "id": "model-agentic-chaos-planner-002", + "type": "model_interaction" + }, + "target": { + "id": "model-agentic-chaos-planner-003", + "type": "model_interaction" + } + }, + { + "id": "rel-agentic-chaos-retrieval-observed", + "type": "observed-in", + "source": { + "id": "retrieval-agentic-chaos-001", + "type": "rag_retrieval" + }, + "target": { + "id": "step-agentic-chaos-retriever", + "type": "step" + } + }, + { + "id": "rel-agentic-chaos-final-observed", + "type": "observed-in", + "source": { + "id": "model-agentic-chaos-final-response-001", + "type": "model_interaction" + }, + "target": { + "id": "step-agentic-chaos-final-response", + "type": "step" + } + }, + { + "id": "rel-agentic-chaos-signal-planner-1", + "type": "signals-on", + "source": { + "id": "reliability-event-planner-attempt-1", + "type": "reliability_event" + }, + "target": { + "id": "model-agentic-chaos-planner-001", + "type": "model_interaction" + } + }, + { + "id": "rel-agentic-chaos-signal-planner-2", + "type": "signals-on", + "source": { + "id": "reliability-event-planner-attempt-2", + "type": "reliability_event" + }, + "target": { + "id": "model-agentic-chaos-planner-002", + "type": "model_interaction" + } + }, + { + "id": "rel-agentic-chaos-signal-planner-3", + "type": "signals-on", + "source": { + "id": "reliability-event-planner-attempt-3", + "type": "reliability_event" + }, + "target": { + "id": "model-agentic-chaos-planner-003", + "type": "model_interaction" + } + }, + { + "id": "rel-agentic-chaos-signal-retriever", + "type": "signals-on", + "source": { + "id": "reliability-event-retriever-timeout", + "type": "reliability_event" + }, + "target": { + "id": "retrieval-agentic-chaos-001", + "type": "rag_retrieval" + } + }, + { + "id": "rel-agentic-chaos-signal-final", + "type": "signals-on", + "source": { + "id": "reliability-event-final-degradation", + "type": "reliability_event" + }, + "target": { + "id": "model-agentic-chaos-final-response-001", + "type": "model_interaction" + } + } + ], + "events": [ + { + "event_id": "event-agentic-chaos-run-started", + "event_name": "aiops.run.started", + "occurred_at": "2026-08-09T02:32:58.452254Z", + "object_id": "run-agentic-chaos-customer-support-20260809", + "object_type": "run" + }, + { + "event_id": "event-agentic-chaos-reliability-planner-1", + "event_name": "aiops.reliability.detected", + "occurred_at": "2026-08-09T02:32:58.453074Z", + "object_id": "reliability-event-planner-attempt-1", + "object_type": "reliability_event" + }, + { + "event_id": "event-agentic-chaos-reliability-planner-2", + "event_name": "aiops.reliability.detected", + "occurred_at": "2026-08-09T02:32:59.453427Z", + "object_id": "reliability-event-planner-attempt-2", + "object_type": "reliability_event" + }, + { + "event_id": "event-agentic-chaos-reliability-planner-3", + "event_name": "aiops.reliability.recovered", + "occurred_at": "2026-08-09T02:33:00.453688Z", + "object_id": "reliability-event-planner-attempt-3", + "object_type": "reliability_event" + }, + { + "event_id": "event-agentic-chaos-retrieval-failed", + "event_name": "aiops.retrieval.failed", + "occurred_at": "2026-08-09T02:33:03.454149Z", + "object_id": "retrieval-agentic-chaos-001", + "object_type": "rag_retrieval" + }, + { + "event_id": "event-agentic-chaos-reliability-timeout", + "event_name": "aiops.reliability.detected", + "occurred_at": "2026-08-09T02:33:03.454149Z", + "object_id": "reliability-event-retriever-timeout", + "object_type": "reliability_event" + }, + { + "event_id": "event-agentic-chaos-reliability-degradation", + "event_name": "aiops.reliability.detected", + "occurred_at": "2026-08-09T02:33:03.454607Z", + "object_id": "reliability-event-final-degradation", + "object_type": "reliability_event" + }, + { + "event_id": "event-agentic-chaos-run-completed", + "event_name": "aiops.run.completed", + "occurred_at": "2026-08-09T02:33:03.454629Z", + "object_id": "run-agentic-chaos-customer-support-20260809", + "object_type": "run" + } + ], + "measurements": [ + { + "name": "duration", + "value": 5002.375, + "unit": "ms", + "kind": "observed" + }, + { + "name": "chaos_event_count", + "value": 5, + "unit": "events", + "kind": "derived" + } + ] +} diff --git a/reviews/artifacts/agentic-chaos-customer-support.conformance.json b/reviews/artifacts/agentic-chaos-customer-support.conformance.json new file mode 100644 index 0000000..27d4c6c --- /dev/null +++ b/reviews/artifacts/agentic-chaos-customer-support.conformance.json @@ -0,0 +1,11 @@ +{ + "spec_version": "0.4", + "artifact_type": "run", + "artifact_path": "/home/pramodbn27/PyPi Projects/ai-operations-spec/reviews/artifacts/agentic-chaos-customer-support.aios.json", + "mode": "conformance", + "draft_alignment_claim": "Aligned with AI Operations Specification v0.4-draft when schema and semantic checks both pass.", + "schema_valid": true, + "semantic_valid": true, + "aligned": true, + "issues": [] +} diff --git a/reviews/artifacts/agenticlens-support-run.aios.json b/reviews/artifacts/agenticlens-support-run.aios.json new file mode 100644 index 0000000..267e3ed --- /dev/null +++ b/reviews/artifacts/agenticlens-support-run.aios.json @@ -0,0 +1,158 @@ +{ + "spec_version": "0.4-draft", + "artifact_type": "run", + "id": "run-agenticlens-support-20260809", + "workflow_id": "workflow-agenticlens-support", + "started_at": "2026-08-09T02:29:53.285034Z", + "ended_at": "2026-08-09T02:29:53.285121Z", + "status": "completed", + "requests": [ + { + "id": "request-agenticlens-support-001", + "name": "support request", + "status": "completed", + "attributes": { + "source_application": "support-agent" + } + } + ], + "steps": [ + { + "id": "step-agenticlens-planner", + "name": "planner", + "category": "planning", + "status": "completed", + "started_at": "2026-08-09T02:29:53.285069Z", + "ended_at": "2026-08-09T02:29:53.285089Z" + }, + { + "id": "step-agenticlens-lookup", + "name": "lookup", + "category": "tool", + "status": "completed", + "started_at": "2026-08-09T02:29:53.285107Z", + "ended_at": "2026-08-09T02:29:53.285118Z" + } + ], + "agents": [], + "occurrences": [ + { + "id": "tool-agenticlens-lookup-001", + "type": "tool_invocation", + "name": "knowledge-base-search", + "status": "completed", + "attributes": { + "tool_name": "knowledge-base-search", + "source_span_id": "a48d27d9-7951-46db-a2ec-faf8b5e2aeeb" + } + } + ], + "evidence": [], + "incidents": [], + "relationships": [ + { + "id": "rel-agenticlens-run-of", + "type": "run-of", + "source": { + "id": "run-agenticlens-support-20260809", + "type": "run" + }, + "target": { + "id": "workflow-agenticlens-support", + "type": "workflow", + "external": true + } + }, + { + "id": "rel-agenticlens-contains-planner", + "type": "contains", + "source": { + "id": "run-agenticlens-support-20260809", + "type": "run" + }, + "target": { + "id": "step-agenticlens-planner", + "type": "step" + } + }, + { + "id": "rel-agenticlens-contains-lookup", + "type": "contains", + "source": { + "id": "run-agenticlens-support-20260809", + "type": "run" + }, + "target": { + "id": "step-agenticlens-lookup", + "type": "step" + } + }, + { + "id": "rel-agenticlens-planner-follows-lookup", + "type": "follows", + "source": { + "id": "step-agenticlens-planner", + "type": "step" + }, + "target": { + "id": "step-agenticlens-lookup", + "type": "step" + } + }, + { + "id": "rel-agenticlens-tool-observed", + "type": "observed-in", + "source": { + "id": "tool-agenticlens-lookup-001", + "type": "tool_invocation" + }, + "target": { + "id": "step-agenticlens-lookup", + "type": "step" + } + } + ], + "events": [ + { + "event_id": "event-agenticlens-run-started", + "event_name": "aiops.run.started", + "occurred_at": "2026-08-09T02:29:53.285034Z", + "object_id": "run-agenticlens-support-20260809", + "object_type": "run" + }, + { + "event_id": "event-agenticlens-tool-completed", + "event_name": "aiops.tool.completed", + "occurred_at": "2026-08-09T02:29:53.285118Z", + "object_id": "tool-agenticlens-lookup-001", + "object_type": "tool_invocation" + }, + { + "event_id": "event-agenticlens-run-completed", + "event_name": "aiops.run.completed", + "occurred_at": "2026-08-09T02:29:53.285121Z", + "object_id": "run-agenticlens-support-20260809", + "object_type": "run" + } + ], + "measurements": [ + { + "name": "duration", + "value": 0.087, + "unit": "ms", + "kind": "observed" + }, + { + "name": "input_tokens", + "value": 132, + "unit": "tokens", + "kind": "derived" + }, + { + "name": "output_tokens", + "value": 36, + "unit": "tokens", + "kind": "derived" + } + ] +} diff --git a/reviews/artifacts/agenticlens-support-run.conformance.json b/reviews/artifacts/agenticlens-support-run.conformance.json new file mode 100644 index 0000000..9bd7ddb --- /dev/null +++ b/reviews/artifacts/agenticlens-support-run.conformance.json @@ -0,0 +1,11 @@ +{ + "spec_version": "0.4", + "artifact_type": "run", + "artifact_path": "/home/pramodbn27/PyPi Projects/ai-operations-spec/reviews/artifacts/agenticlens-support-run.aios.json", + "mode": "conformance", + "draft_alignment_claim": "Aligned with AI Operations Specification v0.4-draft when schema and semantic checks both pass.", + "schema_valid": true, + "semantic_valid": true, + "aligned": true, + "issues": [] +} diff --git a/reviews/decision-log.md b/reviews/decision-log.md new file mode 100644 index 0000000..d308ce9 --- /dev/null +++ b/reviews/decision-log.md @@ -0,0 +1,133 @@ +# Review Decision Log + +Use this file to record milestone-level review outcomes. + +## Open decisions + +### D-001: External framework set for acceptance review + +- Status: open +- Applies to: `v0.1`, `v0.2`, `v0.3`, `v0.4` +- Question: + Which two independent frameworks or implementations will be used for formal comparison? +- Suggested candidates: + - `agenticlens` + - `agentic-chaos` + - one non-DeepAgentLabs implementation or external reviewer +- Evidence needed: + committed framework mapping notes and scenario results +- Current state (2026-08-08): + `agenticlens` and `agentic-chaos` mappings are committed + (`reviews/framework-mappings/`), but both are DeepAgentLabs-owned repos. + Per `README.md`, vendor-neutrality is not established until review includes + a party outside DeepAgentLabs. This decision cannot close on repo-internal + work alone. +- Definition of "independent" for this gate: + a reviewer or implementation with no DeepAgentLabs authorship, funding, or + editorial control over AIOS, reviewing the scenario pack without being + briefed on the intended "correct" answer beforehand. +- Next action (external, not repo work): + identify and engage at least one such reviewer/implementer, run + `reviews/scenarios/` against their framework, and commit the result as + `reviews/framework-mappings/.md` using + `reviews/framework-mappings/external-reviewer-template.md`. A ready-to-send + outreach message is in `reviews/EXTERNAL_REVIEW_INVITE.md`. + +### D-002: v0.2 retry and ordering semantics + +- Status: accepted +- Applies to: `v0.2` +- Question: + Are retry attempts represented as repeated occurrences within one logical activity, and what minimal ordering guarantees are required for equivalent graph reconstruction? +- Decision: + Retry attempts should remain distinct runtime occurrences with their own identities, while still being attributable to one logical activity or step. Equivalent graph reconstruction requires at least explicit attempt ordering and a recoverability link between the failed occurrence and the subsequent retry path. +- Evidence: + `reviews/framework-mappings/agenticlens.md` + `reviews/framework-mappings/agentic-chaos.md` +- Follow-up: + Closed: `specification/v0.2/relationships.md` now requires an explicit + recoverability edge (`caused` or `follows`) between a failed occurrence and + its retry attempt. + +### D-003: v0.3 transport-boundary policy + +- Status: accepted +- Applies to: `v0.3` +- Question: + Should transport mappings remain outside the core spec and be published as separate non-normative profiles? +- Decision: + Yes. Core AIOS semantics should remain transport-neutral. Transport mappings such as OpenTelemetry bindings should live as separate non-normative profiles or companion documents. +- Evidence: + `reviews/framework-mappings/agenticlens.md` + `reviews/framework-mappings/agentic-chaos.md` +- Follow-up: + Closed: `specification/v0.3/semantic-conventions.md` now states the + transport-boundary rule under "Transport boundary". + +### D-004: v0.4 unknown-value behavior + +- Status: accepted (proposed by editor pass on 2026-08-08; maintainer should + confirm rather than treat as externally reviewed) +- Applies to: `v0.4` +- Question: + What unknown enum, extension, and forward-compatibility behavior should consumers preserve or reject? +- Decision: + Closed vocabularies (`object_type`, reference/relationship/occurrence/evidence + `type`, canonical `event_name`) are fixed by the schema; an out-of-enum value + is a schema violation and MUST be rejected, not tolerated as unknown. Forward + compatibility is scoped to the two existing open extension points instead: + freeform `attributes` objects and reverse-domain-namespaced `extensions` + objects. Consumers MUST preserve unrecognized keys in both rather than + discarding them. +- Evidence: + `specification/v0.4/conformance.md` ("Unknown-value behavior" section) + `specification/v0.4/examples/invalid/run-unknown-relationship-type.json` + `specification/v0.4/examples/invalid/run-unknown-object-type.json` + `specification/v0.4/examples/valid/run-namespaced-extensions.json` +- Follow-up: + This is a repo-internal design decision, not independent-implementer + evidence. It closes the "review identifier/reference/extension/unknown-value + behavior" row on its own merits but does not substitute for the still-open + independent-implementation validation gate. + +## Accepted decisions + +### D-100: `Run` remains broader than `Workflow` + +- Status: accepted +- Applies to: `v0.1` +- Notes: + Ad hoc execution may exist without a reusable workflow definition. + +### D-101: MCP is not a core runtime concept + +- Status: accepted +- Applies to: `v0.1` +- Notes: + MCP is treated as an integration protocol used by tool invocations. + +### D-102: `Model Interaction` is the general runtime concept + +- Status: accepted +- Applies to: `v0.1` +- Notes: + LLM calls are represented as a model family or operation, not as a separate core concept. + +## Review template + +Copy this block for each new decision: + +```text +### D-XXX: Short title + +- Status: open | accepted | rejected +- Applies to: v0.x +- Question: + ... +- Decision: + ... +- Evidence: + ... +- Follow-up: + ... +``` diff --git a/reviews/evidence-matrix.md b/reviews/evidence-matrix.md new file mode 100644 index 0000000..286b74c --- /dev/null +++ b/reviews/evidence-matrix.md @@ -0,0 +1,44 @@ +# Acceptance Evidence Matrix + +Use this matrix to show which acceptance gates are actually closed. + +Status values: + +- `open` +- `in progress` +- `ready for review` +- `closed` + +## v0.1 + +| Gate | Status | Evidence | Notes | +|---|---|---|---| +| Review definitions against at least two independent frameworks | ready for review | `reviews/framework-mappings/agenticlens.md`, `reviews/framework-mappings/agentic-chaos.md` | Two implementation mappings now exist for the shared scenario pack | +| Record decisions and update all `v0.1` documents consistently | in progress | `specification/v0.1/decisions.md`, `reviews/decision-log.md` | Final close requires cross-doc consistency pass | +| Two independent implementers classify the same runtime objects consistently | ready for review | `reviews/scenarios/`, `reviews/framework-mappings/agenticlens.md`, `reviews/framework-mappings/agentic-chaos.md` | Two mappings exist; remaining work is reviewer sign-off and any concept refinements | + +## v0.2 + +| Gate | Status | Evidence | Notes | +|---|---|---|---| +| Validate relationship vocabulary against at least two independent frameworks | ready for review | `reviews/framework-mappings/agenticlens.md`, `reviews/framework-mappings/agentic-chaos.md` | Two mappings now cover hierarchy, retries, and topology edges | +| Confirm ordering and retry semantics | closed | `reviews/framework-mappings/agenticlens.md`, `reviews/framework-mappings/agentic-chaos.md`, `reviews/decision-log.md` | Accepted in decision `D-002` | +| Record decisions and update affected documents | closed | `specification/v0.2/relationships.md`, `reviews/decision-log.md` | `D-002` wording folded into the Retry execution shape | + +## v0.3 + +| Gate | Status | Evidence | Notes | +|---|---|---|---| +| Validate conventions against OpenTelemetry design principles | ready for review | `reviews/framework-mappings/agenticlens.md`, `reviews/framework-mappings/agentic-chaos.md` | Evidence supports transport neutrality; a short formal memo is still advisable | +| Validate event coverage against at least two reference implementations | ready for review | `reviews/framework-mappings/agenticlens.md`, `reviews/framework-mappings/agentic-chaos.md` | Two mappings recorded, though native canonical `aiops.*` adoption remains partial | +| Decide whether transport mappings belong in separate implementation profiles | closed | `reviews/framework-mappings/agenticlens.md`, `reviews/framework-mappings/agentic-chaos.md`, `reviews/decision-log.md` | Accepted in decision `D-003` | +| Record decisions and update affected documents | closed | `specification/v0.3/semantic-conventions.md`, `reviews/decision-log.md` | `D-003` wording folded into a new "Transport boundary" section | + +## v0.4 + +| Gate | Status | Evidence | Notes | +|---|---|---|---| +| Resolve all open `v0.1` through `v0.3` gates | open | this matrix | Cannot close early | +| Validate artifacts produced by at least two independent implementations | open | `reviews/artifacts/agenticlens-support-run.aios.json`, `reviews/artifacts/agentic-chaos-customer-support.aios.json`, `reviews/artifacts/agenticlens-support-run.conformance.json`, `reviews/artifacts/agentic-chaos-customer-support.conformance.json` | Two implementation-derived AIOS draft artifacts now pass schema and semantic conformance, but both come from DeepAgentLabs-owned sibling repos. This evidence shows that two implementations produced valid artifacts; the gate stays open until a non-DeepAgentLabs producer artifact is recorded. | +| Review identifier, reference, extension, and unknown-value behavior | closed | `specification/v0.4/conformance.md`, `reviews/decision-log.md` (`D-004`), `specification/v0.4/examples/invalid/run-unknown-relationship-type.json`, `specification/v0.4/examples/invalid/run-unknown-object-type.json`, `specification/v0.4/examples/valid/run-namespaced-extensions.json` | Closed vocab vs. open-extension behavior is now normatively stated and fixture-tested | +| Replace the draft version marker when the milestone freezes | open | schema and fixture update | Only after earlier rows close | diff --git a/reviews/framework-mappings/README.md b/reviews/framework-mappings/README.md new file mode 100644 index 0000000..bf5aa57 --- /dev/null +++ b/reviews/framework-mappings/README.md @@ -0,0 +1,31 @@ +# Framework Mappings + +Each file in this directory should describe how one framework or implementation maps the shared review scenarios to: + +- `v0.1` concepts +- `v0.2` relationships +- `v0.3` semantic events +- `v0.4` artifacts or schema outputs + +The goal is not perfect sameness of field names. The goal is equivalent meaning. + +## Suggested file names + +- `agenticlens.md` +- `agentic-chaos.md` +- `external-framework-a.md` +- `external-framework-b.md` + +## Minimum contents per mapping + +For each scenario: + +- concept classification table +- relationship/graph notes +- semantic event notes +- artifact or schema notes +- ambiguities or disagreements + +## Closure rule + +Acceptance gates that require independent review are not closed until at least two mapping files are completed for the same shared scenario set. diff --git a/reviews/framework-mappings/agentic-chaos-template.md b/reviews/framework-mappings/agentic-chaos-template.md new file mode 100644 index 0000000..b4c3ef6 --- /dev/null +++ b/reviews/framework-mappings/agentic-chaos-template.md @@ -0,0 +1,54 @@ +# Agentic Chaos Review Mapping Template + +- Framework: `agentic-chaos` +- Reviewer: +- Date: +- Version or commit: + +## Scenario coverage + +- [ ] `scenario-01-customer-support.md` +- [ ] `scenario-02-retry-and-recovery.md` +- [ ] `scenario-03-safety-and-incident.md` + +## v0.1 concept classification + +Focus especially on: + +- Reliability Event +- Incident +- Evaluation +- Tool Invocation +- Model Interaction + +## v0.2 relationship notes + +Focus especially on: + +- retry chains +- causal order +- degraded versus recovered execution + +## v0.3 semantic event notes + +Focus especially on: + +- failure events +- retry events +- mitigation or recovery events +- extension event needs + +## v0.4 artifact notes + +Record: + +- schema validation result +- identifier stability +- reference integrity +- edge cases under failure and retry + +## Reviewer summary + +- Equivalent meaning achieved: +- Ambiguities: +- Changes requested: diff --git a/reviews/framework-mappings/agentic-chaos.md b/reviews/framework-mappings/agentic-chaos.md new file mode 100644 index 0000000..dd70f48 --- /dev/null +++ b/reviews/framework-mappings/agentic-chaos.md @@ -0,0 +1,169 @@ +# Agentic Chaos Review Mapping + +- Framework: `agentic-chaos` +- Reviewer: Codex +- Date: 2026-08-09 +- Version or commit: local workspace checkout + +## Scenario coverage + +- [x] `scenario-01-customer-support.md` +- [x] `scenario-02-retry-and-recovery.md` +- [~] `scenario-03-safety-and-incident.md` + +`agentic-chaos` strongly covers failure, retry, degradation, tool failure, memory corruption, and handoff corruption. It is weaker as direct evidence for first-class safety-signal and incident semantics because those remain closer to planned or adjacent resilience behavior than to a dedicated current object model. + +## Evidence reviewed + +- `README.md` +- `src/agentic_chaos/models/chaos_event.py` +- `src/agentic_chaos/models/report.py` +- `src/agentic_chaos/integrations/agenticlens.py` +- `examples/chaos_customer_support_demo.py` +- `examples/chaos_with_agenticlens_demo.py` +- `tests/test_report.py` +- `tests/test_agent_topology.py` +- `tests/test_integrations_agenticlens.py` + +## Executable evidence + +Generated a local standalone chaos report: + +- command: + `UV_CACHE_DIR=/tmp/uv-cache uv run agentic-chaos chaos run examples/chaos_customer_support_demo.py --inject rate_limit_storm,token_timeout,silent_degradation --save /tmp/agentic-chaos-review-report.json` +- output artifact: + `/tmp/agentic-chaos-review-report.json` + +Observed result: + +- planner recorded three `rate_limit_storm` events and recovered after retries +- retriever recorded one `token_timeout` event and degraded to zero chunks +- final response recorded one `silent_degradation` event and returned corrupted content + +This is strong direct evidence for retry, degraded completion, and reliability-event semantics. + +## v0.1 concept classification + +### Scenario 01: Customer support + +Strong alignment: + +- `Run` + Maps cleanly to `ChaosReport`, a standalone report for one chaos execution. +- `Step` + Represented indirectly through `step_id` and `step_name` correlation on each `ChaosEvent`. +- `Tool Invocation` + Strongly represented by tool-targeted faults such as `tool_failure`. +- `RAG Retrieval` + Strongly represented in the customer-support demo where the retriever is an explicit fault target. +- `Reliability Event` + Very strong alignment. `ChaosEvent` is effectively a resilience/fault occurrence model with timestamp, outcome, message, and detail payload. + +Partial alignment: + +- `Request` + Present narratively in examples but not a first-class native report object. +- `Model Interaction` + Strong for fault injection against model-like calls, but not a generic first-class object separate from step correlation. +- `Prompt` + Present as call input in examples, but not a first-class native report object. +- `Context` + Present as payload or result of retrieval, but not modeled separately. +- `Memory Operation` + Strongly implied by `memory_corruption` and memory-decay behavior, though the current report centers on the fault event more than on an independent memory-occurrence object. +- `Incident` + Not yet a first-class report object; events can show impact or degradation without opening a separate incident object. + +Weak alignment: + +- `Evaluation` + Present only indirectly through fidelity judges and optional scoring, not as a central standalone object in the base report. +- `Safety Signal` + Adjacent in spirit to resilience validation, but not strongly evidenced as a dedicated current object in the reviewed files. +- `Agent` + Topology tracking captures agents and edges well when enabled, but the base `ChaosReport` does not require agent objects. + +Conclusion for `v0.1`: + +- `agentic-chaos` strongly validates that AIOS needs distinct concepts for degraded completion, retries, tool failures, memory corruption, and reliability evidence. +- It is weaker as evidence for first-class request, evaluation, safety-signal, and incident objects. + +## v0.2 relationship notes + +Strong alignment: + +- retry chains + Explicitly demonstrated by repeated planner failures that later recover +- ordering + Event timestamps and attempt counters provide clear sequence semantics +- step correlation + `step_id` and `step_name` correlate each event to a runtime step +- topology edges + `TopologyTracker` records agent, tool, and handoff edges +- handoff relationships + edge-scoped corruption and `from_node` / `to_node` fields support non-node relationships + +Partial alignment: + +- generalized execution graph + `agentic-chaos` can describe significant relationships and topology, but its native report is event-centric and resilience-centric rather than a full neutral graph of all runtime objects + +Conclusion for `v0.2`: + +- `agentic-chaos` is strong evidence that AIOS must represent retries, causal order, topology edges, and handoff corruption distinctly. +- It meaningfully strengthens the case for non-tree relationships beyond what a pure span tree provides. + +## v0.3 semantic event notes + +Strong alignment: + +- event meaning is explicit + `fault_type`, `outcome`, `message`, `timestamp`, and `detail` convey stable semantics +- degraded-versus-errored distinction + outcomes such as `errored` and `degraded` stay distinct +- retry semantics + rate-limit events include `retry_after`, attempt number, and recovery context +- edge semantics + `edge_id`, `from_node`, and `to_node` support handoff-specific meaning + +Partial alignment: + +- canonical AIOS event namespace + `agentic-chaos` uses its own fault/event vocabulary rather than native `aiops.*` names +- transport neutrality + event meaning is portable, but not yet expressed as AIOS canonical event names + +Conclusion for `v0.3`: + +- `agentic-chaos` strongly supports the need for AI-native semantic events that are independent of transport. +- It also supports keeping transport mappings separate from the core semantic layer. +- It does not, by itself, prove adoption of the final AIOS canonical event catalog. + +## v0.4 artifact notes + +Strong alignment: + +- standalone JSON artifacts exist with stable identifiers and timestamps +- event references to step identity are explicit +- optional topology can capture agent and handoff structure +- optional AgenticLens integration can merge `chaos_events` into a shared workflow artifact + +Key findings: + +- the native `ChaosReport` is not itself an AIOS `run` artifact +- it intentionally shares JSON shape with AgenticLens workflow artifacts for interoperability +- optional integration shows a practical path from resilience evidence into a broader shared artifact + +Implication: + +- `agentic-chaos` is a strong evidence producer for AIOS design review +- it is not yet a direct AIOS artifact producer in its standalone report format + +## Reviewer summary + +- Equivalent meaning achieved: + strong for reliability, retries, degradation, topology edges, and resilience evidence +- Ambiguities: + request, safety-signal, evaluation, and incident concepts are not yet first-class in the native standalone report +- Changes requested: + add or document an explicit AIOS export path from `ChaosReport` to AIOS `run` artifacts, and add a safety/policy-oriented example if `agentic-chaos` will be used as formal evidence for those concepts diff --git a/reviews/framework-mappings/agenticlens-template.md b/reviews/framework-mappings/agenticlens-template.md new file mode 100644 index 0000000..62716dc --- /dev/null +++ b/reviews/framework-mappings/agenticlens-template.md @@ -0,0 +1,66 @@ +# AgenticLens Review Mapping Template + +- Framework: `agenticlens` +- Reviewer: +- Date: +- Version or commit: + +## Scenario coverage + +- [ ] `scenario-01-customer-support.md` +- [ ] `scenario-02-retry-and-recovery.md` +- [ ] `scenario-03-safety-and-incident.md` + +## v0.1 concept classification + +Record how AgenticLens identifies: + +- Run +- Request +- Step +- Agent +- Model Interaction +- Prompt +- Context +- Tool Invocation +- RAG Retrieval +- Memory Operation +- Evaluation +- Safety Signal +- Reliability Event +- Incident + +## v0.2 relationship notes + +Record: + +- containment +- dependency +- causal ordering +- retry linkage +- delegation or handoff +- parent-child runs if present + +## v0.3 semantic event notes + +Record: + +- canonical events that clearly fit +- extension events required +- naming ambiguities +- lifecycle coverage gaps + +## v0.4 artifact notes + +Record: + +- artifact types produced +- schema validation result +- reference resolution issues +- extension and unknown-value behavior + +## Reviewer summary + +- Equivalent meaning achieved: +- Ambiguities: +- Changes requested: diff --git a/reviews/framework-mappings/agenticlens.md b/reviews/framework-mappings/agenticlens.md new file mode 100644 index 0000000..10ab218 --- /dev/null +++ b/reviews/framework-mappings/agenticlens.md @@ -0,0 +1,166 @@ +# AgenticLens Review Mapping + +- Framework: `agenticlens` +- Reviewer: Codex +- Date: 2026-08-09 +- Version or commit: local workspace checkout + +## Scenario coverage + +- [x] `scenario-01-customer-support.md` +- [x] `scenario-02-retry-and-recovery.md` +- [~] `scenario-03-safety-and-incident.md` + +`agenticlens` has direct evidence for customer-support, retrieval, tools, hierarchical traces, retry analysis, and AIOS draft conformance tooling. Safety and incident semantics are named in high-level docs, but this review pass did not find a first-class native trace example that exercises them end to end. + +## Evidence reviewed + +- `README.md` +- `docs/aios-validation-and-otel.md` +- `schemas/trace.schema.json` +- `src/agenticlens/models/trace.py` +- `examples/rag_customer_support_demo.py` +- `examples/multiagent_support_demo.py` +- `examples/operational_intelligence_demo.py` +- `tests/test_trace.py` + +## Executable evidence + +Generated a local AgenticLens trace artifact: + +- command: `UV_CACHE_DIR=/tmp/uv-cache uv run python examples/operational_intelligence_demo.py` +- output artifact: `agenticlens/examples/artifacts/support-run.json` + +Attempted AIOS conformance: + +- command: `UV_CACHE_DIR=/tmp/uv-cache uv run agenticlens conformance examples/artifacts/support-run.json --version 0.4 --spec-root ../ai-operations-spec` +- result: + `artifact_type` is missing, so the native AgenticLens trace artifact is not directly an AIOS `run` or `workflow` artifact + +This is an important review finding: + +- `agenticlens` is a strong implementation evidence source for runtime meaning +- `agenticlens` already contains AIOS validation tooling +- but its native trace JSON is not yet equivalent to a frozen AIOS artifact format + +## v0.1 concept classification + +### Scenario 01: Customer support + +Strong alignment: + +- `Run` + Maps well to AgenticLens trace `Run` and to the profiler-level workflow execution record. +- `Request` + Present conceptually in the examples as the user question, but not yet a first-class native trace object in the `trace` schema. +- `Step` + Maps strongly to `step()` profiler steps and approximately to trace spans for execution occurrences. +- `Model Interaction` + Maps strongly to `SpanType.MODEL_CALL` and LLM profiler steps. +- `Prompt` + Present in profiler examples and can be captured or referenced, but is not required by the native trace schema. +- `Context` + Present conceptually through retrieved chunks and duplicated-context analysis, but not a first-class standalone trace object. +- `Tool Invocation` + Maps strongly to `SpanType.TOOL_CALL` and tool-oriented step metadata. +- `RAG Retrieval` + Maps strongly to `SpanType.RETRIEVAL` and retriever examples. +- `Evaluation` + Present strongly at the product level through evaluation suites and findings, though not a dedicated span type in the trace schema. + +Partial alignment: + +- `Agent` + Supported through `agent_name` metadata and multi-agent examples, but not as a top-level native object. +- `Memory Operation` + Maps strongly to `SpanType.MEMORY_READ` and `SpanType.MEMORY_WRITE`. +- `Safety Signal` + Mentioned in the README as a runtime area, but not evidenced here as a dedicated native trace object. +- `Reliability Event` + Represented operationally through retry analysis, failures, span status, and error metadata rather than a dedicated first-class object. +- `Incident` + Not evidenced as a first-class native trace object in the reviewed files. + +Conclusion for `v0.1`: + +- AgenticLens supports most runtime occurrences AIOS cares about. +- It is strongest on execution evidence, retrieval, memory, tools, retries, and evaluations. +- It is weaker on first-class native objects for `Request`, `Safety Signal`, `Reliability Event`, and `Incident`. + +## v0.2 relationship notes + +Strong alignment: + +- containment + Native trace spans support parent-child hierarchy through `parent_span_id` +- graph integrity + native validation rejects unknown parents and cycles +- delegation + supported by `SpanType.DELEGATION` and multi-agent examples +- retry linkage + supported by `SpanType.RETRY`, `retry_number`, and retry attribution analysis + +Partial alignment: + +- causal dependency between sibling steps + hierarchy is strong, but generalized causal relationships beyond parentage are not expressed as a separate native relation model +- joins, branches, and equivalent execution graphs + examples imply them, but the native trace structure is closer to a tree plus span metadata than to the full AIOS relationship vocabulary + +Conclusion for `v0.2`: + +- AgenticLens is good evidence that hierarchy, retries, and delegation matter. +- It is not yet full proof that two independent implementations will reconstruct the same execution graph semantics, especially for non-tree relationships. + +## v0.3 semantic event notes + +Strong alignment: + +- lifecycle meaning + runs and spans have explicit statuses and times +- transport boundary awareness + docs clearly separate AIOS semantics from OTLP export +- extension awareness + the project can export OTLP while also validating AIOS draft artifacts + +Partial alignment: + +- canonical AIOS event names + AgenticLens trace artifacts are span-oriented and status-oriented, not expressed natively as the AIOS `aiops.*` event catalog +- event coverage for safety and incident semantics + not directly demonstrated in reviewed examples + +Conclusion for `v0.3`: + +- AgenticLens provides strong evidence for the need for transport-neutral semantics. +- It also supports the AIOS position that transport bindings should remain separate from core meaning. +- It does not yet prove native adoption of the AIOS canonical event namespace. + +## v0.4 artifact notes + +Strong alignment: + +- structured, validated local JSON artifacts exist +- native trace schema enforces identity and parent-reference integrity +- AIOS `validate` and `conformance` tooling is implemented in AgenticLens + +Key finding: + +- the native `trace.schema.json` artifact is not itself an AIOS `run` artifact +- conformance failed on the generated demo trace because `artifact_type` was missing and the payload was not in AIOS artifact shape + +Implication: + +- AgenticLens is currently best treated as: + - a source of execution evidence for AIOS design review + - a validator/consumer of AIOS artifacts + - a partial producer candidate if an explicit AIOS export layer is added + +## Reviewer summary + +- Equivalent meaning achieved: + partial but meaningful +- Ambiguities: + `Request`, `Safety Signal`, `Reliability Event`, and `Incident` are not yet strongly represented as first-class native trace objects in the reviewed evidence +- Changes requested: + add or document an explicit AIOS export path from AgenticLens traces to AIOS `run` artifacts, and add at least one safety/incident example if AgenticLens is going to be used as a formal AIOS review implementation diff --git a/reviews/framework-mappings/external-reviewer-template.md b/reviews/framework-mappings/external-reviewer-template.md new file mode 100644 index 0000000..d78f855 --- /dev/null +++ b/reviews/framework-mappings/external-reviewer-template.md @@ -0,0 +1,91 @@ +# External Review Mapping Template + +Use this template if you are reviewing AIOS from outside DeepAgentLabs — +as an independent framework maintainer, a consultant, or an unaffiliated +practitioner. It mirrors the internal mapping template but does not assume +you are mapping onto a specific DeepAgentLabs product. + +Save your completed copy as `reviews/framework-mappings/.md`. + +- Framework or context: (name of your framework, product, or "N/A — reviewing as an unaffiliated practitioner") +- Reviewer: +- Affiliation: (confirm you have no DeepAgentLabs authorship, funding, or editorial role — see `reviews/decision-log.md` decision `D-001`) +- Date: +- Version or commit reviewed: + +## Scenario coverage + +- [ ] `scenario-01-customer-support.md` +- [ ] `scenario-02-retry-and-recovery.md` +- [ ] `scenario-03-safety-and-incident.md` + +For each scenario you cover, work from the narrative only — don't read the +other framework mappings in this directory first. The value of this exercise +is in seeing where your independent interpretation agrees or disagrees with +ours, not in converging on a pre-agreed answer. + +## v0.1 concept classification + +For each scenario, state how you would classify the runtime objects +described in `specification/v0.1/core-concepts.md`: + +- Run +- Request +- Step +- Agent +- Model Interaction +- Prompt +- Context +- Tool Invocation +- RAG Retrieval +- Memory Operation +- Evaluation +- Safety Signal +- Reliability Event +- Incident + +Flag anything that doesn't map cleanly, anything you'd split further, and +anything you'd merge. + +## v0.2 relationship notes + +Using `specification/v0.2/relationships.md`, record how you would represent: + +- containment +- dependency / causal order +- retry linkage (does the failed attempt and the retry need an explicit + edge, or is adjacency/timestamp enough in your model?) +- delegation or handoff +- parent-child runs, if applicable + +## v0.3 semantic event notes + +Using `specification/v0.3/semantic-conventions.md`, record: + +- which canonical `aiops.*` events clearly fit +- which scenario moments have no canonical event and would need an + extension (`.*`) +- any naming, cardinality, or lifecycle assumptions you disagree with +- whether the transport-neutrality stance (events are not OTel spans; OTel + bindings are a separate profile) matches how you'd actually instrument this + +## v0.4 artifact notes + +Using `specification/v0.4/schemas/` and `specification/v0.4/examples/`: + +- did you attempt to produce or hand-construct an artifact for one of the + scenarios? What broke or felt unnatural? +- schema validation result, if you ran one +- reference/identifier integrity issues +- reaction to the unknown-value policy in `specification/v0.4/conformance.md` + ("Unknown-value behavior") — closed enums reject unknown values, growth + happens via `attributes`/`extensions`. Does that match how you'd want a + consumer to behave? + +## Reviewer summary + +- Equivalent meaning achieved: (full / partial / minimal — and why) +- Ambiguities: +- Disagreements or changes requested: +- Would you be willing to be named as an external reviewer in + `reviews/decision-log.md`? (yes/no — either is fine, this is optional) diff --git a/reviews/scenarios/README.md b/reviews/scenarios/README.md new file mode 100644 index 0000000..768bc2b --- /dev/null +++ b/reviews/scenarios/README.md @@ -0,0 +1,25 @@ +# Shared Review Scenarios + +These scenarios are the common acceptance test pack for `v0.1` through `v0.4`. + +Use the same scenarios across all compared frameworks so differences in interpretation are visible. + +## Included scenarios + +- `scenario-01-customer-support.md` + Normal request, retrieval, answer, and evaluation. +- `scenario-02-retry-and-recovery.md` + Failed first attempt, retry, and successful completion. +- `scenario-03-safety-and-incident.md` + Safety detection, policy block, and possible incident escalation. + +## What reviewers should extract + +- `v0.1` + Object classification and boundaries. +- `v0.2` + Relationship graph and ordering semantics. +- `v0.3` + Canonical event interpretation and extension needs. +- `v0.4` + Structurally valid artifacts and edge-case behavior. diff --git a/reviews/scenarios/scenario-01-customer-support.md b/reviews/scenarios/scenario-01-customer-support.md new file mode 100644 index 0000000..02cb8e2 --- /dev/null +++ b/reviews/scenarios/scenario-01-customer-support.md @@ -0,0 +1,44 @@ +# Scenario 01: Customer Support Answer + +## Narrative + +A customer asks whether a refund is still possible for an order that arrived damaged. The system starts a run, classifies the request, retrieves policy passages, drafts an answer with a model interaction, and evaluates whether the answer is grounded in retrieved policy. + +## Minimum expected objects + +- one `Run` +- one initiating `Request` +- at least two `Step` occurrences: + - classification + - answer generation +- one or more `Model Interaction` occurrences +- one `RAG Retrieval` +- one `Prompt` +- one `Context` +- one `Evaluation` + +## v0.1 review questions + +- Is the `Request` distinct from the `Run`? +- Is retrieval modeled as `RAG Retrieval` instead of a generic tool by default? +- Is the evaluation distinct from runtime completion? + +## v0.2 review questions + +- Does the answer step depend on retrieval? +- Are retrieval and answer represented as separate observed occurrences? + +## v0.3 expected event areas + +- run lifecycle +- step lifecycle +- prompt rendering +- context assembly +- model request and response +- retrieval activity +- evaluation result + +## v0.4 artifact expectations + +- references resolve cleanly between run, request, steps, retrieval, and evaluation +- schema allows optional sensitive content omission diff --git a/reviews/scenarios/scenario-02-retry-and-recovery.md b/reviews/scenarios/scenario-02-retry-and-recovery.md new file mode 100644 index 0000000..6945e1f --- /dev/null +++ b/reviews/scenarios/scenario-02-retry-and-recovery.md @@ -0,0 +1,42 @@ +# Scenario 02: Retry and Recovery + +## Narrative + +A user asks for a product comparison. The system starts a run and begins a model-backed analysis step. The first model attempt times out. A reliability signal records the timeout. The system retries once, the second attempt succeeds, and the run completes with an answer. + +## Minimum expected objects + +- one `Run` +- one `Request` +- one logical analysis `Step` +- two distinct `Model Interaction` attempts +- at least one `Reliability Event` +- optional `Evaluation` + +## v0.1 review questions + +- Are the two attempts treated as separate model interaction occurrences? +- Is the timeout a `Reliability Event` rather than automatically an `Incident`? +- Can the run be completed even though it experienced degradation? + +## v0.2 review questions + +- How is the retry chain represented? +- What ordering relation proves the second attempt follows the first? +- Does the logical step stay one step while attempts remain separate occurrences? + +## v0.3 expected event areas + +- run lifecycle +- step lifecycle +- model requested +- model failed or timed out +- retry or recovery semantics +- model responded +- run completed + +## v0.4 artifact expectations + +- both attempts have stable identities +- the reliability event references the failed attempt +- graph references remain valid after retry diff --git a/reviews/scenarios/scenario-03-safety-and-incident.md b/reviews/scenarios/scenario-03-safety-and-incident.md new file mode 100644 index 0000000..bf1a74d --- /dev/null +++ b/reviews/scenarios/scenario-03-safety-and-incident.md @@ -0,0 +1,39 @@ +# Scenario 03: Safety Signal and Incident Escalation + +## Narrative + +A request appears to contain prompt injection aimed at exfiltrating sensitive data through a privileged tool. A detector emits a safety signal. Policy blocks the tool request. Later review determines no data left the system, so the event remains a blocked attempt and does not automatically become an incident. In a variant of the same scenario, if sensitive data is exposed or operational impact occurs, responders open an incident linked to the run and prior safety evidence. + +## Minimum expected objects + +- one `Run` +- one `Request` +- one or more `Step` occurrences +- one `Safety Signal` +- one blocked `Tool Invocation` +- optional `Incident` depending on outcome + +## v0.1 review questions + +- Is the safety signal distinct from the enforcement result? +- Is a blocked tool invocation distinct from an incident? +- Under what condition does an actual `Incident` begin? + +## v0.2 review questions + +- How do the safety signal and tool block relate to the request or step? +- If an incident is opened later, what relationship links it to prior evidence? + +## v0.3 expected event areas + +- safety detected +- tool requested +- authorization or denial +- mitigation or block +- incident opened if impact occurs + +## v0.4 artifact expectations + +- incident remains optional +- references can link signals, blocked tool attempts, and later incident evidence +- extension events can be namespaced if a producer needs provider-specific safety detail diff --git a/specification/v0.2/acceptance-criteria.md b/specification/v0.2/acceptance-criteria.md index 6262571..b49c13b 100644 --- a/specification/v0.2/acceptance-criteria.md +++ b/specification/v0.2/acceptance-criteria.md @@ -8,8 +8,9 @@ ## Review gates - [ ] Validate the vocabulary against at least two independent agent frameworks. -- [ ] Confirm ordering and retry semantics. -- [ ] Record decisions and update affected documents. +- [x] Confirm ordering and retry semantics (decision `D-002`). +- [x] Record decisions and update affected documents (`relationships.md` retry + section now states the required recoverability edge). ## Exit condition diff --git a/specification/v0.2/relationships.md b/specification/v0.2/relationships.md index 9a3d637..560803c 100644 --- a/specification/v0.2/relationships.md +++ b/specification/v0.2/relationships.md @@ -74,7 +74,7 @@ Connects operational evidence to an Incident. Incident association MUST NOT chan - **Parallel:** sibling Steps have no ordering edge and their lifecycles may overlap. - **Branch:** one occurrence causes or enables multiple alternatives; recorded branch evidence identifies the selected path. - **Join:** a Step depends on multiple predecessor Steps. -- **Retry:** a new attempt refers to the prior attempt and the shared logical activity; attempts retain distinct identities. +- **Retry:** a new attempt refers to the prior attempt and the shared logical activity; attempts retain distinct identities. The failed occurrence and the retry attempt MUST be linked by an explicit recoverability edge (for example `caused` or `follows`) so consumers can reconstruct the retry path without inferring it from timing or naming alone (decision `D-002`). - **Loop:** repeated occurrences have distinct identities and a shared loop identity plus iteration position. - **Delegation:** Agent responsibility is represented separately from Step structure. diff --git a/specification/v0.3/acceptance-criteria.md b/specification/v0.3/acceptance-criteria.md index ee19b22..a0a31e5 100644 --- a/specification/v0.3/acceptance-criteria.md +++ b/specification/v0.3/acceptance-criteria.md @@ -11,8 +11,10 @@ - [ ] Validate conventions against OpenTelemetry design principles. - [ ] Validate event coverage against at least two reference implementations. -- [ ] Decide whether transport mappings belong in separate implementation profiles. -- [ ] Record decisions and update affected documents. +- [x] Decide whether transport mappings belong in separate implementation profiles + (decision `D-003`). +- [x] Record decisions and update affected documents (`semantic-conventions.md` + now states the transport-boundary rule). ## Exit condition diff --git a/specification/v0.3/semantic-conventions.md b/specification/v0.3/semantic-conventions.md index 7e26f52..d3a2706 100644 --- a/specification/v0.3/semantic-conventions.md +++ b/specification/v0.3/semantic-conventions.md @@ -54,3 +54,7 @@ A failure, failed Evaluation, or Safety Signal MUST NOT automatically become an Durations use non-negative milliseconds. Token counts use non-negative integers. Cost MUST include a currency code. Scores MUST declare their scale or categorical domain. A measurement MUST state whether it is observed, estimated, or derived. Consumers MUST ignore unrecognized extension attributes unless configured to reject them. Producers MUST NOT reuse a canonical name with incompatible meaning. + +## Transport boundary + +AIOS event semantics are transport-neutral. Bindings to a specific wire format or telemetry pipeline (for example, an OpenTelemetry span/attribute mapping) MUST NOT be defined as part of this normative document. Such bindings belong in separate, non-normative companion profiles that map canonical `aiops.*` events and attributes onto a transport, and MAY evolve independently of this specification (decision `D-003`). diff --git a/specification/v0.4/acceptance-criteria.md b/specification/v0.4/acceptance-criteria.md index a8727da..0f0b87c 100644 --- a/specification/v0.4/acceptance-criteria.md +++ b/specification/v0.4/acceptance-criteria.md @@ -14,7 +14,10 @@ - [ ] Resolve all open v0.1 through v0.3 review gates. - [ ] Validate artifacts produced by at least two independent implementations. -- [ ] Review identifier, reference, extension, and unknown-value behavior. +- [x] Review identifier, reference, extension, and unknown-value behavior + (decision `D-004`; see `specification/v0.4/conformance.md` and the + `run-unknown-relationship-type.json`, `run-unknown-object-type.json`, and + `run-namespaced-extensions.json` fixtures). - [ ] Replace the draft version marker when the milestone freezes. ## Exit condition diff --git a/specification/v0.4/conformance.md b/specification/v0.4/conformance.md index 8f7eb5a..ed24efd 100644 --- a/specification/v0.4/conformance.md +++ b/specification/v0.4/conformance.md @@ -17,6 +17,9 @@ A producer claiming draft alignment MUST: 9. Support `workflow_id` with a matching `run-of` relationship. 10. Ensure event targets resolve with the declared object type. 11. Reject self-relationships and cycles in structural, causal, and ordering graphs. +12. When representing retries, preserve distinct occurrence identities for each + attempt and link each retry attempt to the prior attempt with an explicit + recoverability edge such as `follows` or `caused`. An external reference MUST set `external: true`. External resolution is the consumer's responsibility and MUST NOT be reported as verified unless it was resolved. @@ -24,6 +27,17 @@ An external reference MUST set `external: true`. External resolution is the cons A consumer claiming draft alignment MUST perform schema and semantic validation before treating an artifact as aligned. It MUST preserve unknown namespaced extensions and MUST NOT interpret structural validity as factual accuracy. +## Unknown-value behavior (decision `D-004`) + +Closed vocabularies — `object_type`, reference `type`, relationship `type`, `occurrence`/`evidence` `type`, and the canonical `event_name` catalog — are fixed by this draft's schemas. A value outside a closed vocabulary is a schema violation, not an unknown value to tolerate: producers MUST NOT invent new members of a closed enum, and consumers MUST reject an artifact that contains one rather than guessing its meaning. + +Forward-compatible growth belongs in the two open extension points instead: + +1. Freeform `attributes` objects, which MAY carry any producer-defined keys. +2. Namespaced `extensions` objects, which MUST use a reverse-domain name outside the `aiops` namespace. + +Consumers MUST preserve unrecognized `attributes` keys and namespaced `extensions` entries rather than discarding them, and MUST NOT treat their presence as a validation failure. + ## Claim format A claim SHOULD include the draft marker and observation date, for example: diff --git a/specification/v0.4/examples/invalid/run-unknown-object-type.json b/specification/v0.4/examples/invalid/run-unknown-object-type.json new file mode 100644 index 0000000..86a4f12 --- /dev/null +++ b/specification/v0.4/examples/invalid/run-unknown-object-type.json @@ -0,0 +1,19 @@ +{ + "spec_version": "0.4-draft", + "artifact_type": "run", + "id": "run-unknown-object-type", + "started_at": "2026-07-21T06:00:00Z", + "status": "completed", + "requests": [], + "steps": [], + "relationships": [], + "events": [ + { + "event_id": "event-1", + "event_name": "aiops.run.completed", + "occurred_at": "2026-07-21T06:00:00Z", + "object_id": "run-unknown-object-type", + "object_type": "widget" + } + ] +} diff --git a/specification/v0.4/examples/invalid/run-unknown-relationship-type.json b/specification/v0.4/examples/invalid/run-unknown-relationship-type.json new file mode 100644 index 0000000..9ff1a27 --- /dev/null +++ b/specification/v0.4/examples/invalid/run-unknown-relationship-type.json @@ -0,0 +1,17 @@ +{ + "spec_version": "0.4-draft", + "artifact_type": "run", + "id": "run-unknown-rel-type", + "started_at": "2026-07-21T06:00:00Z", + "status": "completed", + "requests": [], + "steps": [{"id": "step-a", "category": "model", "status": "completed"}], + "relationships": [ + { + "id": "rel-unknown", + "type": "supersedes", + "source": {"id": "run-unknown-rel-type", "type": "run"}, + "target": {"id": "step-a", "type": "step"} + } + ] +} diff --git a/specification/v0.4/examples/semantic-invalid/run-retry-merged-attempts.json b/specification/v0.4/examples/semantic-invalid/run-retry-merged-attempts.json new file mode 100644 index 0000000..9818d20 --- /dev/null +++ b/specification/v0.4/examples/semantic-invalid/run-retry-merged-attempts.json @@ -0,0 +1,113 @@ +{ + "spec_version": "0.4-draft", + "artifact_type": "run", + "id": "run-retry-merged-attempts", + "workflow_id": "workflow-retry-merged-attempts", + "started_at": "2026-08-09T00:00:00Z", + "ended_at": "2026-08-09T00:00:03Z", + "status": "completed", + "requests": [], + "steps": [ + { + "id": "step-retry-merged-attempts", + "name": "planner", + "category": "planning", + "status": "completed" + } + ], + "agents": [], + "occurrences": [ + { + "id": "model-retry-merged-attempts-001", + "type": "model_interaction", + "name": "planner call", + "status": "completed" + } + ], + "evidence": [ + { + "id": "reliability-retry-attempt-1", + "type": "reliability_event", + "name": "planner retry attempt 1", + "status": "detected", + "attributes": { + "attempt": 1 + } + }, + { + "id": "reliability-retry-attempt-2", + "type": "reliability_event", + "name": "planner retry attempt 2", + "status": "recovered", + "attributes": { + "attempt": 2 + } + } + ], + "incidents": [], + "relationships": [ + { + "id": "rel-run-of", + "type": "run-of", + "source": { + "id": "run-retry-merged-attempts", + "type": "run" + }, + "target": { + "id": "workflow-retry-merged-attempts", + "type": "workflow", + "external": true + } + }, + { + "id": "rel-contains-step", + "type": "contains", + "source": { + "id": "run-retry-merged-attempts", + "type": "run" + }, + "target": { + "id": "step-retry-merged-attempts", + "type": "step" + } + }, + { + "id": "rel-observed-in", + "type": "observed-in", + "source": { + "id": "model-retry-merged-attempts-001", + "type": "model_interaction" + }, + "target": { + "id": "step-retry-merged-attempts", + "type": "step" + } + }, + { + "id": "rel-signals-attempt-1", + "type": "signals-on", + "source": { + "id": "reliability-retry-attempt-1", + "type": "reliability_event" + }, + "target": { + "id": "model-retry-merged-attempts-001", + "type": "model_interaction" + } + }, + { + "id": "rel-signals-attempt-2", + "type": "signals-on", + "source": { + "id": "reliability-retry-attempt-2", + "type": "reliability_event" + }, + "target": { + "id": "model-retry-merged-attempts-001", + "type": "model_interaction" + } + } + ], + "events": [], + "measurements": [] +} diff --git a/specification/v0.4/examples/valid/run-namespaced-extensions.json b/specification/v0.4/examples/valid/run-namespaced-extensions.json new file mode 100644 index 0000000..8d8003d --- /dev/null +++ b/specification/v0.4/examples/valid/run-namespaced-extensions.json @@ -0,0 +1,21 @@ +{ + "spec_version": "0.4-draft", + "artifact_type": "run", + "id": "run-ext-root-001", + "started_at": "2026-07-21T06:00:00Z", + "status": "completed", + "requests": [], + "steps": [ + { + "id": "step-a", + "category": "model", + "status": "completed", + "attributes": {"unrecognized_future_field": "producers may add attributes freely"}, + "extensions": {"io.deepagentlabs.agenticlens.step_confidence": 0.87} + } + ], + "relationships": [ + {"id": "rel-contains-a", "type": "contains", "source": {"id": "run-ext-root-001", "type": "run"}, "target": {"id": "step-a", "type": "step"}} + ], + "extensions": {"io.deepagentlabs.agenticlens.trace_id": "trace-9f2c"} +} diff --git a/specification/v0.4/schemas/common.schema.json b/specification/v0.4/schemas/common.schema.json index 4192365..6f8f94a 100644 --- a/specification/v0.4/schemas/common.schema.json +++ b/specification/v0.4/schemas/common.schema.json @@ -41,7 +41,7 @@ "properties": { "event_id": {"$ref": "#/$defs/id"}, "event_name": { - "description": "Either a canonical aiops.* name from the reviewed catalog, or a reverse-domain namespaced extension event (e.g. io.deepagentlabs.agenticlens.custom_event). Extension names MUST NOT start with the aiops namespace.", + "description": "Either a canonical aiops.* name from the draft catalog, or a reverse-domain namespaced extension event (e.g. io.deepagentlabs.agenticlens.custom_event). Extension names MUST NOT start with the aiops namespace.", "anyOf": [{"$ref": "#/$defs/canonical_event_name"}, {"$ref": "#/$defs/extension_name"}] }, "occurred_at": {"$ref": "#/$defs/time"}, "object_id": {"$ref": "#/$defs/id"}, "object_type": {"enum": ["run", "request", "step", "agent", "model_interaction", "prompt", "context", "tool_invocation", "rag_retrieval", "memory_operation", "evaluation", "safety_signal", "reliability_event", "incident"]}, diff --git a/tests/test_review_artifacts.py b/tests/test_review_artifacts.py new file mode 100644 index 0000000..7a8e782 --- /dev/null +++ b/tests/test_review_artifacts.py @@ -0,0 +1,19 @@ +import json +from pathlib import Path + +from test_v04_schema import semantic_errors, validator + + +ROOT = Path(__file__).parents[1] +REVIEW_ARTIFACTS = ROOT / "reviews" / "artifacts" + + +def load(path: Path) -> dict: + return json.loads(path.read_text(encoding="utf-8")) + + +def test_review_aios_artifacts_validate() -> None: + for path in sorted(REVIEW_ARTIFACTS.glob("*.aios.json")): + artifact = load(path) + validator(artifact).validate(artifact) + assert not semantic_errors(artifact), f"{path.name} has semantic errors" diff --git a/tests/test_v04_schema.py b/tests/test_v04_schema.py index 0319a2e..bf142c4 100644 --- a/tests/test_v04_schema.py +++ b/tests/test_v04_schema.py @@ -47,6 +47,8 @@ def semantic_errors(artifact: dict) -> list[str]: if len(ids) != len(set(ids)): errors.append("object identities must be unique within a Run artifact") object_types = dict(objects) + occurrence_by_id = {item["id"]: item for item in artifact.get("occurrences", [])} + evidence_by_id = {item["id"]: item for item in artifact.get("evidence", [])} relationships = artifact.get("relationships", []) relationship_ids = [item["id"] for item in relationships] @@ -105,6 +107,59 @@ def count_edges(edge_type: str, source_id: str | None = None, target_id: str | N if object_types.get(event["object_id"]) != event["object_type"]: errors.append(f"Event {event['event_id']} must target an object of its declared type") + attempt_signal_targets: dict[str, list[tuple[int, str]]] = {} + for relationship in relationships: + if relationship["type"] != "signals-on": + continue + source = evidence_by_id.get(relationship["source"]["id"]) + if not source or source.get("type") != "reliability_event": + continue + attempt = source.get("attributes", {}).get("attempt") + if not isinstance(attempt, int): + continue + target_id = relationship["target"]["id"] + attempt_signal_targets.setdefault(target_id, []).append((attempt, source["id"])) + occurrence = occurrence_by_id.get(target_id) + if occurrence: + occurrence_attempt = occurrence.get("attributes", {}).get("attempt") + if isinstance(occurrence_attempt, int) and occurrence_attempt != attempt: + errors.append( + f"Retry evidence {source['id']} must target attempt {attempt}, not occurrence attempt {occurrence_attempt}" + ) + for target_id, attempts in attempt_signal_targets.items(): + if len(attempts) > 1: + ordered_attempts = ", ".join(str(attempt) for attempt, _ in sorted(attempts)) + errors.append( + f"Retry attempts {ordered_attempts} must not all signal the same occurrence {target_id}" + ) + + retry_groups: dict[str, dict[int, str]] = {} + for occurrence in artifact.get("occurrences", []): + attributes = occurrence.get("attributes", {}) + logical_activity_id = attributes.get("logical_activity_id") + attempt = attributes.get("attempt") + if not isinstance(logical_activity_id, str) or not isinstance(attempt, int): + continue + retry_groups.setdefault(logical_activity_id, {}) + if attempt in retry_groups[logical_activity_id]: + errors.append(f"Retry activity {logical_activity_id} duplicates attempt {attempt}") + continue + retry_groups[logical_activity_id][attempt] = occurrence["id"] + for logical_activity_id, attempts in retry_groups.items(): + ordered_attempts = sorted(attempts.items()) + if len(ordered_attempts) < 2: + continue + for index in range(len(ordered_attempts) - 1): + _, source_id = ordered_attempts[index] + next_attempt, target_id = ordered_attempts[index + 1] + if not ( + count_edges("follows", source_id, target_id) + or count_edges("caused", source_id, target_id) + ): + errors.append( + f"Retry activity {logical_activity_id} must link attempt {index + 1} to attempt {next_attempt}" + ) + for edge_types in ({"parent-of"}, {"caused", "follows", "depends-on"}): graph: dict[str, set[str]] = {} for item in relationships: