feat(agent-advisor): standalone bundle for a headless, single-skill executor (stacked on #195) - #198
Closed
leon1418 wants to merge 4 commits into
Closed
feat(agent-advisor): standalone bundle for a headless, single-skill executor (stacked on #195)#198leon1418 wants to merge 4 commits into
leon1418 wants to merge 4 commits into
Conversation
added 2 commits
August 4, 2026 15:38
…k) + seeded runs Add a first-class Model Recommend phase to agent-advisor that selects a Bedrock model and API path per model-bearing workload, with dedicated provider modules. Runtime scoring is fully decoupled from model selection: the legacy model_recommendation field is removed from scoring.py and the scoring-result schema (runtime scoring owns compute only; Model Recommend owns the model). Anthropic: - dated path-aware catalog (anthropic-bedrock-2026-07-21) - joint (model, api_path) filtering/ranking, CRIS resolution without guessing - Claude version-hop analysis, structured compatibility/architecture findings - optional live verification (verify_model_path.py); no-AWS-account provisional flow OpenAI→Bedrock (new provider module): - source family detection (reasoning/legacy/unknown; opaque deployments stay unknown) - path rules: Responses+continuity->mantle_openai_responses; Chat->Responses reshape; governance->runtime_converse; continuity+runtime-only->decision_required - target-derived version/sampling/generation analysis; capability-aware fail-closed selection; separate-modality targets (embeddings/images/audio) as unresolved contracts; dated openai-bedrock-2026-07-21 catalog with honest "unknown" limits Seeded runs (reproducibility + provenance honesty): - scripts/schemas/seed.json: machine-readable answers for a non-interactive run (benchmark seed, CI regression, ATX transformation), so the deterministic engine receives byte-identical input every time. A malformed seed is a hard error, never a silent fall-through. - Clarify Step 2.5 resolves every dimension by an explicit precedence ladder and records provenance (seed|detected|asked|inherited|adapter|interview|assumed); a seeded value is copied byte-equal, never re-expressed; every `assumed` dimension must appear in UNANSWERED.md. Downstream Confirm/Design/Generate/POC carry the selected contract verbatim; POC never emits a runnable model id without passed live verification. Migration Plan validates the advisor (model, api_path) contract instead of reselecting. 301 tests passing; mise run build clean. Offline engine: no openai/boto3 imported at module load.
…xecutor
The plugin keeps the full skill, including the Migration Plan stage that reads
the sibling gcp-to-aws engine. An executor that installs one skill has no
siblings to read, so the variant published there has to be self-contained.
Rather than fork the prose -- which drifts, as this repo has learned more than
once -- the bundle is a strict subset of the canonical skill plus exactly one
generated file:
transformation_definition.md = preamble.md + SKILL.md (frontmatter stripped)
Every other file is byte-identical to its canonical source, so there is no
second copy of any instruction to keep in sync. preamble.md carries the only
environment-specific text: no interactive tools (values come from a run seed,
then repository prose, then UNANSWERED.md), no subagent dispatch (Estimate runs
inline), the ${CLAUDE_PLUGIN_ROOT} fallback, engine-produced scoring, optional
MCP, and the required validation-summary format.
`mise run atx:check` (now in the lint chain) makes "this bundle does not depend
on gcp-to-aws" a machine-verified property rather than a claim. It checks that
the manifest and the canonical tree agree, that no file resolves a path into
another skill, that the capability gate exists, that the phase graph closes
inside the shipped set, and that no runtime script needs a third-party import at
module load.
The Migration Plan stage resolves to not_applicable when the sibling engine is
absent, with a note to the user. The POC still runs -- design-backed rather than
plan-backed -- because losing the plan should not cost the user the deliverable
they most often want.
fixtures/agent-advisor-anthropic-demo is the target repository for a headless
run: an existing Anthropic Messages-API agent with a seed under seeds/, its own
check_recommendation.py, and a measurement of what the seed bought (six runs,
byte-identical scoring-result.json, and the three drift causes that surfaced on
the way there).
Verified: mise run lint clean (atx:check 68 files, 12 warnings), 301 pytest,
57 node tests, the fixture's own 3 tests.
added 2 commits
August 4, 2026 16:49
…venv from bandit Two things `mise run build` caught that `mise run lint` does not: - dprint pads markdown table columns; the seeds README's tables were authored by hand and were not padded. `fmt:check` is in `build` but not in `lint`. - The fixture has its own pyproject.toml and tests, so running them creates a .venv there, and bandit does not read .gitignore -- it walked the venv and failed on site-packages. The bandit invocation already names the other two .venv paths in the plugin for the same reason; this adds the third. CI never hit it (no venv on a fresh checkout), but anyone who runs the fixture's tests and then `mise run build` locally would. Also gave the Seed/Scenario table a heading -- it was left dangling under the CLAUDE.md section when the measurement section was added above it. Verified: mise run build exit 0 with the fixture .venv present, and no self-mutation afterwards (the check CI runs after build).
…-neutral The builder printed a publish command naming a specific internal registry CLI, and a comment explained the frontmatter strip in terms of that CLI's flags. The bundle itself is environment-neutral -- plain text and .py files with an entry point -- so its tooling should read that way too. The hint now says what the output IS (entry point plus the root its relative paths resolve against), which is the part a reader actually needs and is true of any executor.
Contributor
Author
|
Closing to restructure. Two problems with this PR as it stands:
Will reopen as a single commit carrying only the bundle: the preamble, the manifest, the builder, and the two mise tasks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #195. Until that merges, the diff here also shows #195's commit; the only commit this PR adds is
dcb0e80(23 files, +1529, no deletions). Review #195 first.Why
The plugin keeps the full
agent-advisorskill, including the Migration Plan stage that reads the siblinggcp-to-awsengine. An executor that installs one skill has no siblings to read, so the variant published there has to be self-contained.Rather than fork the prose — which drifts, as this repo has learned more than once — the bundle is a strict subset of the canonical skill plus exactly one generated file:
Every other file in the bundle is byte-identical to its canonical source, so there is no second copy of any instruction to keep in sync.
What's here
atx/agent-advisor/preamble.mdUNANSWERED.md), no subagent dispatch (Estimate runs inline), the${CLAUDE_PLUGIN_ROOT}fallback, engine-produced scoring, optional MCP, and the required validation-summary formatatx/agent-advisor/manifest.txttools/atx-bundle/build.tsmise run atx:build/atx:check)fixtures/agent-advisor-anthropic-demo/seeds/, and its owncheck_recommendation.pyatx:checkis in thelintchain, which makes "this bundle does not depend ongcp-to-aws" a machine-verified property rather than a claim. It checks that the manifest and the canonical tree agree, that no shipped file resolves a path into another skill, that the capability gate exists, that the phase graph closes inside the shipped set, and that no runtime script needs a third-party import at module load.Behaviour difference, stated plainly
The Migration Plan stage resolves to
not_applicablewhen the sibling engine is absent, with a note to the user. The POC still runs — design-backed rather than plan-backed, and labelled as such — because losing the plan should not cost the user the deliverable they most often want.The fixture, and what the seed bought
seeds/README.mdrecords the measurement rather than asserting the property. Six headless runs over the fixture with the same seed produced a byte-identicalscoring-result.json(sha2560fcf9967…). Getting there surfaced three drift causes — a seededregionobject reshaped into sibling keys, two source-scan false positives inflating the findings counts, andsource_pathslisting evidence files as call sites. All three are fixed in the skill prose (in #195), not pinned around in the seed, and the README shows the before/after per run.Verification
mise run lintclean (atx:check68 files / 12 warnings,shared:check,fixtures:check98 json, frontmatter, 57 node tests), 301 pytest, and the fixture's own 3 tests.