Phase 2 + Phase 3a/3b: session state, diagnostics, 5 new tools, docs (0.1.3 → 0.2.0) - #11
Merged
Merged
Conversation
- AdapterUnavailableError carries the adapter name + original exception so callers (tool wrappers, core.verify) can report which integration is down without parsing message text. - workspace_root() factors out the parents[4] computation ensure_repo_on_path already used, so other call sites (chaos.run_experiment's sandboxing, core.health's diagnostics) share one definition of "the workspace" instead of duplicating it.
SessionState holds the artifacts sequential tool calls need to share (workflow, baseline/candidate runs, last comparison, last chaos report) plus a capped call-history deque. Keyed by an optional session_id (default "default") - a stdio MCP server is one process per client, so this intentionally stops short of any cross-process or persistent design. Not wired into any tool yet; this is just the store.
…_runs/slo_summary/audit_report
- Import the sibling repo inside try/except now, capturing _IMPORT_ERROR
instead of letting a missing/broken agenticlens crash the whole server at
import time; probe() reports {available, version, error} for
core.verify/core.health.
- report_summary(): reuses agenticlens's own MarkdownExporter (via a temp
file, since it only writes to a path) rather than reimplementing report
formatting.
- compare_runs(): wraps comparison.runner.compare_runs() over Run traces.
- slo_summary(): wraps evaluation.gate.evaluate_gate() against a GateConfig
built from caller-supplied thresholds.
- audit_report(): returns full per-case evaluation detail, optionally with
evaluation.html_report's rendered HTML.
All four follow analyze_workflow's existing shape: validate with the
sibling's own pydantic models, delegate to its logic, return a plain dict -
this server stays a thin orchestration layer.
run_experiment() runs a target script inside a real chaos_session() (mirrors what the agentic-chaos CLI's `chaos run` does internally against the public chaos_session/runpy API - that command's own helper is private, so it isn't imported directly): - _resolve_sandboxed_script() confines the script path to the workspace root (resolved + relative_to check), rejecting anything outside it or missing. - resolve_faults() runs before anything executes, so an unknown fault name fails fast with a clear error. - The run happens on a worker thread so timeout_seconds can actually bound wall-clock time even when a fault sleeps past it; the executor is shut down with wait=False (not a `with` block) specifically so a timeout doesn't then block waiting for the thread anyway. Python can't force-kill a thread, so a timed-out script's thread may keep running in the background - documented, not swept under the rug. Same defensive-import + probe() treatment as the other adapters.
…resources - Loading the v0.4 schema documents now happens inside try/except; a missing sibling repo or empty schemas/ directory sets _IMPORT_ERROR instead of leaving SCHEMA_DOCUMENTS silently empty for callers to trip over later. - list_schema_resources() and the new schema_resource_content() both derive from a single _SCHEMA_RESOURCES mapping and skip any schema file that isn't actually loaded, instead of unconditionally advertising/indexing all three. This is what lets resources/list and resources/read degrade consistently with core.verify/core.health when the sibling is unavailable, rather than the caller (server.py) assuming all three schema documents always exist.
- core.health now returns adapter availability/version (via each adapter's
probe()), loaded tool/resource/prompt counts, the resolved workspace
root, and recent successful-call timestamps - not just
{"status": "ok"}. status flips to "degraded" if any adapter is down.
- core.verify checks all three adapters and reports {ok, adapters}, so a
new contributor can see what's connected in one call.
- core.session_state exposes a session's stored artifacts + call history,
the concrete way a client verifies shared context is working.
…report
Each wrapper validates its arguments, catches AdapterUnavailableError into a
structured {ok: false, error} payload, and records the call in the session
store. report_summary and compare_runs additionally read/write session
state (workflow, baseline_runs/candidate_runs, last_comparison) so an
'artifact' or 'baseline'/'candidate' argument can be omitted once a prior
call in the same session already supplied one.
… in list_faults
run_experiment validates 'script'/'faults', delegates to the adapter, stores
last_chaos_report in the session, and turns AdapterUnavailableError/
ValueError (unknown fault, path outside the workspace, missing script) into
a structured {ok: false, error} response instead of raising.
… mutates_session) Every tool entry now carries category, prerequisites (adapter names), expected_duration (instant/fast/slow), and mutates_session, plus schemas and dispatch entries for all the new Phase 2/3 tools. ToolDescriptor and PromptDescriptor (schemas/tooling.py) gain matching optional fields so the typed descriptors keep building. This metadata is what server.py maps onto Tool.annotations/Tool._meta, and what scripts/generate_tools_doc.py later renders into docs/tools.md.
…emplates list_prompts() entries now carry MCP PromptArgument-shaped 'arguments' instead of just name+description, and render_prompt() turns a name + argument values into the actual user-message text. Previously this module was data only and was never attached to the server; wiring happens in the next commit.
…or boundary
- _TOOL_DISPATCH gains core.verify, core.session_state, lens.report_summary/
compare_runs/slo_summary/audit_report, and chaos.run_experiment.
- _build_tools() maps registry metadata onto real MCP fields: mutates_session
-> annotations.read_only_hint, chaos.run_experiment specifically gets
destructive_hint/open_world_hint since it executes external code;
category/prerequisites/expected_duration ride in Tool._meta.
- New prompts/list and prompts/get handlers, following the exact
add_request_handler pattern already used for tools/resources.
- RESOURCE_CONTENT's schema entries now come from
ai_operations_spec.schema_resource_content() instead of indexing
SCHEMA_DOCUMENTS directly - the direct-index version raised KeyError at
import time whenever the sibling repo's schemas weren't loaded, which
defeated the adapters' whole degrade-instead-of-crash design.
- handle_call_tool now wraps handler(arguments) in try/except: any
exception a handler doesn't catch itself (e.g. a pydantic ValidationError
from malformed workflow/run input) becomes a structured
{ok: false, error} payload instead of propagating past the MCP dispatch
boundary.
Gives chaos.run_experiment a safe, in-repo target to run against out of the box, and lets tests exercise the real sandboxed-execution path without depending on a user's own scripts.
…the recommenders It previously failed Workflow validation outright (missing start_time) and was too thin to produce recommendations even if fixed. Now a valid 6-step workflow with real metrics that trips ExcessiveChunksRecommender, DuplicateToolCallsRecommender, and LongHistoryRecommender when run through lens.analyze_workflow/lens.report_summary.
Isolation between session ids, history recording/capping, last_successful_calls filtering out failures, reset, and the summary() view core.session_state returns.
… are missing Runs in a fresh subprocess with adapters.workspace_root() monkeypatched to an empty temp directory before deep_agentic_core_mcp.server is first imported - the only faithful way to test import-time behavior, since ai_operations_spec.py resolves its sibling repo and loads schema documents at import time. Directly guards against the KeyError-at-import-time regression fixed in the server.py commit above: reaching handle_call_tool at all is most of what this test proves.
Every registered tool has a valid category/expected_duration and list/bool typed prerequisites/mutates_session; chaos.run_experiment specifically is flagged slow + mutating + agentic_chaos-dependent; the new prompt arguments are present and correctly marked required.
…eview - core.verify, core.session_state (including cross-call context sharing) - lens.report_summary (session-reused artifact + missing-artifact error), compare_runs, slo_summary, audit_report against real agenticlens example fixtures (agenticlens/examples/pitch_demo/artifacts) - chaos.run_experiment success, path-outside-workspace rejection, unknown-fault rejection, and timeout enforcement (asserts wall-clock time stays bounded, not just the reported timed_out flag) - prompts/list and prompts/get, including the unknown-prompt case - malformed lens.analyze_workflow input returns a structured error instead of raising - an autouse fixture resets the default session between tests so session-sharing tests can't leak into each other
scripts/generate_tools_doc.py reads list_tools() (the same data MCP clients see via tools/list) plus server.py's _OPEN_WORLD_TOOLS set, so the doc can't drift out of sync with either - chaos.run_experiment's entry gets an 'executes external code, see SECURITY.md' callout derived straight from the server's own annotation logic. make docs regenerates it; make docs-check regenerates + git diff --exit-code to catch drift. Deliberately not folded into make check, so the default pre-push gate stays fast.
Covers what's mitigated (workspace sandboxing, timeout guard) and what isn't (the script's own filesystem/network/subprocess access, no MCP client auth) - and recommends keeping this server stdio/local-only until a real sandbox exists for a hypothetical remote deployment mode.
README's tool list, repository layout, and 'What's Next' section were still
describing the 0.1.x placeholder state (six-tool list, missing
ai_operations_spec.py/session.py/spec.py/test files, Phase 2 items listed
as upcoming). Now lists all 12 shipped tools, links docs/tools.md and
SECURITY.md, and trims two sections ('Initial Scope', 'Near-Term Build
Order') that had gone stale enough to contradict the current-state section
sitting right below them. AGENTS.md's tool-adding checklist gains the
registry-metadata and make docs steps.
Version bumped in pyproject.toml, __init__.py, server.json (and uv.lock's lockfile entry for the editable install). CHANGELOG.md gets a full 0.2.0 entry: Added (session state, core.verify, the 4 lens tools, chaos.run_experiment, prompts wiring, tool metadata, generated docs), Changed (defensive adapter imports), and Fixed (the timeout/degraded-boot/ malformed-input issues found in review, plus the sample_workflow.json validation bug).
…tations + backlog - Phase 2 (session mgmt & diagnostics) and Phase 3b (agentic-chaos) marked complete with what shipped in 0.2.0. - Phase 3a's 4 possible tools all checked off; remaining work narrowed to the one still-unverified success criterion (provenance on analyze_workflow's response shape). - Phase 3c's remaining work re-confirmed as blocked upstream (only v0.4 has populated schemas/; no defined conformance-rule format exists yet) rather than left ambiguous. - New 'Known Limitations' section: tool handlers block the event loop (fine for stdio, must be fixed before any remote/multi-session transport). - New 'Documentation Backlog' section: getting-started guide, session workflow walkthrough, and prompts overview - identified while writing docs/tools.md, deliberately deferred rather than missed.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5454b00ecb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
nitheshh405
approved these changes
Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements ROADMAP.md Phase 2 (Session Management & Diagnostics) and
Phase 3a/3b (AgenticLens / Agentic Chaos integration) in full, plus a
round of fixes for real bugs found during review of that work and a pass of
overdue documentation. Version bumped
0.1.3→0.2.0.Tool count: 5 → 12. Test count: 18 → 46.
What's in this PR
Phase 2 — Session Management & Diagnostics ✅
services/session.py) +core.session_statetool.lens.analyze_workflow→lens.compare_runs→chaos.run_experimentcan now share artifacts across calls via an optionalsession_id, without the client resending them.core.health: adapter availability/version, loaded tool/resource/prompt counts, workspace root, recent successful calls — not just{"status": "ok"}.core.verify: checks agenticlens/agentic-chaos/ai-operations-spec connectivity and reports readiness.category,prerequisites,expected_duration,mutates_sessionon every tool, surfaced to MCP hosts viaTool.annotations/Tool._meta.prompts/list/prompts/getare now actually wired into the server (previouslyprompts/registry.pywas data that nothing ever attached), with realPromptArguments and rendered templates.AdapterUnavailableError+ per-adapterprobe()). A missing/broken sibling repo used to crash server boot outright; now it surfaces as"available": falsethroughcore.verify/core.health.Phase 3a — AgenticLens (4 new tools, all shipped) ✅
lens.report_summary— Markdown workflow report via agenticlens's ownMarkdownExporterlens.compare_runs— baseline/candidate trace comparison + regression detectionlens.slo_summary— release-gate style SLO thresholds against an evaluation reportlens.audit_report— case-by-case evaluation detail, optionally with HTMLAll four reuse real agenticlens logic (no reimplementation) — see
docs/architecture.md's "thin orchestration layer" principle.Phase 3b — Agentic Chaos ✅
chaos.run_experiment— runs a workspace-sandboxed target script inside a realchaos_session(), mirroring what theagentic-chaosCLI'schaos rundoes internally. Confirmed live: fault injection visibly corrupts output when run againstexamples/chaos_target.py.SECURITY.md.Bug fixes (found in review of the above, before merge)
chaos.run_experiment'stimeout_secondsdidn't actually bound wall-clock time — the worker thread ran inside awith ThreadPoolExecutor(...)block whose__exit__blocks for the thread regardless of the timeout having fired. Fixed; verified a 0.3s timeout now returns in ~0.3s instead of the fault's full 2s hang.server.pyindexedSCHEMA_DOCUMENTS["workflow.schema.json"]directly at import time, which raisedKeyError(crashing server boot) wheneverai-operations-specwas unavailable — defeating the adapter resilience work above. Fixed viaschema_resource_content(), derived dynamically from what actually loaded.ValidationErrorfrom an invalid workflow artifact) could propagate pasthandle_call_toolinstead of becoming a structured MCP error. Fixed with a centralized try/except at the dispatch boundary.examples/sample_workflow.jsonfailedWorkflowvalidation outright (missingstart_time) and was too thin to produce recommendations even if fixed. Now valid and rich enough to trip 3 different recommenders.Regression tests added for all four (
test_handle_call_tool_run_experiment_honors_timeout,tests/test_degraded_boot.py,test_handle_call_tool_analyze_workflow_malformed_artifact, live-checkedsample_workflow.jsonoutput).Documentation
docs/tools.md— full tool reference (schemas, category, prerequisites, duration, mutation flag), generated fromtools/registry.pyviascripts/generate_tools_doc.pyso it can't drift.make docs/make docs-checkadded.README.md/AGENTS.mdbrought up to date — the tool list, repo layout, and "What's Next" section were still describing the 0.1.x placeholder state.SECURITY.md— new section onchaos.run_experiment's code-execution scope, mitigations, and what isn't mitigated.ROADMAP.md— Phase 2/3b marked complete, Phase 3a checkboxes flipped, Phase 3c's remaining work re-confirmed as blocked upstream (not left ambiguous), plus new Known Limitations (tool handlers block the event loop — fine for stdio, needs fixing before any remote/multi-session transport) and Documentation Backlog (getting-started guide, session walkthrough, prompts overview — deliberately deferred) sections.CHANGELOG.md— full0.2.0entry (Added/Changed/Fixed).Explicitly out of scope
ai-operations-speconly has populatedschemas/underv0.4.ROADMAP.md's new "Documentation Backlog" section.asyncio.to_thread()— tracked inROADMAP.md's new "Known Limitations", not a problem for today's single-client stdio transport.Testing
Also manually verified end-to-end: full tool chain (
analyze_workflow→report_summary→compare_runs→slo_summary→audit_report→run_experiment) sharing a session; server boots and degrades correctly with a sibling repo renamed away;chaos.run_experimentactually corrupts output undersilent_degradation.Commit structure
23 commits, organized by concern (adapters → services → tools → registry/prompts → server wiring → examples → tests → docs → version bump → roadmap) rather than by chronological session history — each is reviewable independently, though the full test suite only passes at
HEAD(later commits build on earlier ones in the same PR).🤖 Generated with Claude Code