Skip to content

fix(llm-to-bedrock): sweep stale Sonnet 4 example model IDs to Sonnet 4.6 - #164

Open
herosjourney wants to merge 3 commits into
awslabs:mainfrom
herosjourney:fix/llm2bedrock-stale-model-ids
Open

fix(llm-to-bedrock): sweep stale Sonnet 4 example model IDs to Sonnet 4.6#164
herosjourney wants to merge 3 commits into
awslabs:mainfrom
herosjourney:fix/llm2bedrock-stale-model-ids

Conversation

@herosjourney

Copy link
Copy Markdown
Contributor

Summary

The llm2bedrock agent examples (code-rewriter, code-analyzer, report-generator) and the behavior-delta reference still used us.anthropic.claude-sonnet-4-20250514-v1:0 — Claude Sonnet 4 (May 2025) — as the example rewrite target. The plugin's own model catalog marks that model excluded, EOL Oct 14, 2026 (ai-model-lifecycle.md), and every recommendation elsewhere points at Sonnet 4.6. An agent pattern-matching the examples would produce rewrites targeting a model the plugin itself bans. This PR sweeps the examples to claude-sonnet-4-6 and adds a guard note so example IDs can't silently drive rewrites again.

Changes

  • Swept claude-sonnet-4-20250514-v1:0claude-sonnet-4-6 in:
    • agents/llm2bedrock-code-rewriter.md (7 sites: Vercel AI SDK / LangChain / boto3 examples, env-var block)
    • agents/llm2bedrock-code-analyzer.md (target_models example)
    • agents/llm2bedrock-report-generator.md (model-id list example, pricing comment — Sonnet 4.6 has identical $3/$15 rates so the comment's math is unchanged, dual-ARN prose example)
    • skills/llm-to-bedrock/references/helpers/behavior-delta-detection/references/openai-to-bedrock.md (converse example)
    • skills/llm-to-bedrock/scripts/test_iam_policy.py, test_validate_result.py (example IDs in tests; inference-profile logic is ID-agnostic)
  • Intentionally kept: the Sonnet 4 rows in pricing-cache.md and ai-model-lifecycle.md — those are catalog entries whose whole point is recording the model and its EOL/excluded status.
  • New guard note in llm2bedrock-code-rewriter.md above the rewrite examples: example IDs are illustrative; authoritative IDs come from the migration plan (aws_model_id); verify plan IDs resolve via aws bedrock list-foundation-models (or the preflight converse-ping); if an ID no longer resolves, STOP and surface it rather than substituting a model. This mirrors the agent-toolkit-for-aws guidance on verifying model IDs at runtime.

Relationship to other PRs

Independent — based on current main (post-#162). No overlap with open PRs (#141/#144/#149/#151/#155/#156/#161/#163).

Test plan

  • Repo-wide sweep verified: zero claude-sonnet-4-20250514 references outside the two catalog files
  • pytest on the two touched test scripts: 119 passed
  • Full mise run build green (160/160 checks)
  • Fresh-agent rewrite replay against the fixture repo (examples-only change; flagging honestly that no replay was run)

@herosjourney

Copy link
Copy Markdown
Contributor Author

Applied the Cursor review (commit 89eda24) — all five points, with two findings that came out of verifying point 2:

1. Regression lint — added. tools/model-id-lint.py (stdlib) + lint:model-ids mise task in the lint group, so it runs in every build. Two pattern classes: the EOL Sonnet 4 ID outside the two catalog files, and fabricated dated forms (sonnet-4-6-<date> / opus-4-8-<date>). Explicit per-pattern allowlists; extend BAD_PATTERNS on future EOLs.

2. The "dated Sonnet 4.6" IDs are fabricated, and one had a wrong price. claude-sonnet-4-6-20250514-v1:0 grafts Sonnet 4.6's name onto Sonnet 4's date stamp — that ID never existed. Swept in the three gcp design refs. Two things Cursor's list didn't surface:

  • bedrock_pricing.py's STATIC_FALLBACK carried the fabricated dated forms as executable lookup keys, plus a fabricated claude-opus-4-8-20250610 at $15/$75 per 1M — a 3x overstatement (Opus 4.8 is $5/$25 per the plugin's own pricing cache). Table now uses real undated IDs with the correct rate; _static_fallback strips date stamps so dated inputs still resolve; two new tests guard the rate and the date-strip (130 pass in the touched files; the 3 pre-existing test_preflight_bedrock.py failures reproduce on untouched main — env-only, missing boto3).
  • resolve-bedrock-model-id.md keeps its dated example on purpose (repairing broken IDs is that helper's job) — now labeled intentionally-invalid, and allowlisted in the lint.

Swept in this PR rather than a separate one: the lint has to be green to merge, so lint + sweep are one atomic change.

3. Analyzer guard — added as a _comment_target_models sibling key in the example JSON.

4. ID-form note — added to the rewriter guard: geo-prefixed inference-profile form preferred in rewritten code; bare foundation-model IDs belong in pricing tables (they often reject on-demand invocation — see the IAM known-fix).

5. Replay — left unchecked, per the review.

Full mise run build green (160/160) including the new lint.

@herosjourney
herosjourney marked this pull request as ready for review July 20, 2026 23:05
@herosjourney
herosjourney requested review from a team as code owners July 20, 2026 23:05
Comment on lines +355 to +356
"target_models": ["gpt-4o -> us.anthropic.claude-sonnet-4-6"],
"_comment_target_models": "example only — the real mapping comes from the migration plan, never from this illustration",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently Bedrock supports GPT-5.6, and Sonnet is positioned in the same tier as GPT-5.6 Terra. So, should we recommend migrating to a model from the same providor?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants