Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -685,7 +685,7 @@ The eval schema fields (validated, extras rejected):
- `no_golden_cases: true` — when `total_cases == 0` (T2-2 abort / paste / vision-no-images / embeddings paths). Tells T2-6 to render "no quality data" instead of "0% pass rate".
- Free-form addenda — manual-review notes, per-layer status hints. Keep total `notes` under ~500 chars to fit dashboards.
- **`live_source_baseline`** — MANDATORY. `true` ONLY when §9 produced live side-by-side responses for at least one prompt. `false` when §9 was skipped (no source key) or every live call failed. The report banner depends on this flag.
- **`judge_model`** — MANDATORY. Identifier of the LLM running THIS agent (e.g. `claude-opus-4-7`, `claude-haiku-4-5-20251001-v1:0`). The report discloses it so readers can assess same-family bias risk against the target Bedrock model. If you cannot identify the exact ID, pass `"unknown"` — never drop the field.
- **`judge_model`** — MANDATORY. Identifier of the LLM running THIS agent (e.g. `claude-opus-4-8`, `claude-haiku-4-5-20251001-v1:0`). The report discloses it so readers can assess same-family bias risk against the target Bedrock model. If you cannot identify the exact ID, pass `"unknown"` — never drop the field.
- **`source_baseline_quality`** — `'good'` (live baseline ran and looked fine), `'poor'` (live baseline ran but the source model's own output looked degraded — empty responses, error bodies, or obvious wrong-version behavior; per §9 step 3), or `'unknown'` (no live baseline ran). The orchestrator surfaces `'poor'` at the quality gate.

`live_source_baseline` and `judge_model` are MANDATORY — always include both, every time, even if uncertain. Never omit them.
Expand All @@ -704,7 +704,7 @@ When §7 routed past §10–§13 with `total_cases == 0` (no golden dataset to e
"failures": 1,
"notes": "live_source_baseline_used_model: \n1 prompt needs manual review — see notes for details.",
"live_source_baseline": true,
"judge_model": "claude-opus-4-7",
"judge_model": "claude-opus-4-8",
"source_baseline_quality": "good"
}
```
Expand All @@ -723,7 +723,7 @@ When `total_cases == 0` (T2-2 abort / paste / vision-no-images / embeddings path
"failures": 0,
"notes": "no_golden_cases: true\nreason: T2-2 reported total_golden_cases=0; layers 1/2 passed but no quality data to score.",
"live_source_baseline": false,
"judge_model": "claude-opus-4-7",
"judge_model": "claude-opus-4-8",
"source_baseline_quality": "unknown"
}
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,7 @@ <h3>High-level service mapping</h3>
<tr><td>Cloud SQL PostgreSQL</td><td>RDS PostgreSQL db.t4g.micro</td><td><span class="badge-conf badge-tail">Tailored to your setup</span> (single-AZ)</td></tr>
<tr><td>Cloud Storage</td><td>S3</td><td><span class="badge-conf badge-std">Standard pairing</span></td></tr>
<tr><td>BigQuery</td><td>Deferred — specialist engagement</td><td><span class="badge-conf badge-defer">Specialist required</span></td></tr>
<tr><td>Vertex AI / Gemini</td><td>Bedrock Claude Sonnet 4.6</td><td><span class="badge-conf badge-tail">Tailored to your setup</span> (quality priority)</td></tr>
<tr><td>Vertex AI / Gemini</td><td>Bedrock Claude Sonnet 5</td><td><span class="badge-conf badge-tail">Tailored to your setup</span> (quality priority)</td></tr>
<tr><td>Vertex AI Imagen</td><td>Bedrock Nova Canvas</td><td><span class="badge-conf badge-tail">Tailored to your setup</span></td></tr>
</tbody>
</table>
Expand Down Expand Up @@ -749,7 +749,7 @@ <h3>Model mappings</h3>
<caption>GCP AI models mapped to Amazon Bedrock targets.</caption>
<thead><tr><th scope="col">GCP model</th><th scope="col">Bedrock target</th><th scope="col">Workload</th><th scope="col">Complexity</th></tr></thead>
<tbody>
<tr><td>gemini-2.5-flash</td><td>anthropic.claude-sonnet-4-6</td><td>Text generation, structured output</td><td>Medium</td></tr>
<tr><td>gemini-2.5-flash</td><td>anthropic.claude-sonnet-5</td><td>Text generation, structured output</td><td>Medium</td></tr>
<tr><td>imagen-3.0-generate-001</td><td>amazon.nova-canvas-v1:0</td><td>Image generation (~500/mo)</td><td>Medium</td></tr>
</tbody>
</table>
Expand All @@ -759,13 +759,13 @@ <h3>Model cost comparison</h3>
<caption>Bedrock model options vs current GCP AI spend ($1,250/mo).</caption>
<thead><tr><th scope="col">Bedrock model</th><th scope="col">Monthly est.</th><th scope="col">vs GCP ($1,250/mo)</th><th scope="col">Quality</th><th scope="col">Notes</th></tr></thead>
<tbody>
<tr><td><strong>Claude Sonnet 4.6 (recommended)</strong></td><td>$780</td><td class="savings">-38%</td><td>High</td><td>Quality priority (Q16); selected target</td></tr>
<tr><td><strong>Claude Sonnet 5 (recommended)</strong></td><td>$780</td><td class="savings">-38%</td><td>High</td><td>Quality priority (Q16); selected target</td></tr>
<tr><td>Claude Haiku 4.5</td><td>$260</td><td class="savings">-79%</td><td>Medium-High</td><td>Balance of cost and quality</td></tr>
<tr><td>Nova Lite v1</td><td>$14</td><td class="savings">-99%</td><td>Medium</td><td>Backup if A/B test passes quality bar</td></tr>
</tbody>
</table>

<h3>Recommended model token breakdown (Claude Sonnet 4.6)</h3>
<h3>Recommended model token breakdown (Claude Sonnet 5)</h3>
<table>
<caption>Monthly token volume and cost for the recommended model.</caption>
<thead><tr><th scope="col">Component</th><th scope="col">Volume</th><th scope="col">Cost</th></tr></thead>
Expand All @@ -791,7 +791,7 @@ <h3>Code changes</h3>

<h3>Production readiness checklist</h3>
<ul class="compact">
<li>Bedrock model access enabled for Claude Sonnet 4.6 and Nova Canvas in us-west-2</li>
<li>Bedrock model access enabled for Claude Sonnet 5 and Nova Canvas in us-west-2</li>
<li>IAM role with <code>bedrock:InvokeModel</code> on ECS task role</li>
<li>Provider adapter deployed with <code>AI_PROVIDER</code> feature flag</li>
<li>A/B test with &ge;100 representative prompts; image quality human evaluation</li>
Expand Down Expand Up @@ -849,7 +849,7 @@ <h2>Your Configuration Choices</h2>
<td>What matters most for your AI workloads?</td>
<td>Quality</td>
<td>User answer</td>
<td>Claude Sonnet 4.6 recommended over lower-cost models</td>
<td>Claude Sonnet 5 recommended over lower-cost models</td>
</tr>
<tr>
<td>Detected: Cloud SQL ZONAL &rarr; single-AZ availability</td>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ the same topology discipline as Path 1:
- **Primary request flow (solid edges):** User → the app's entry surface (the runtime hosting
the migrated app + its `/invocations` entrypoint) → the app's orchestration layer (e.g.
LangChain) → the Bedrock target model. Show the provider swap explicitly — label the model
node with the migrated target (e.g. "Bedrock: Claude Sonnet 4.6") and note the source it
node with the migrated target (e.g. "Bedrock: Claude Sonnet 5") and note the source it
replaced where useful (e.g. "was: OpenAI gpt-3.5-turbo").
- **State / memory (solid edge to a store):** if the app has conversation memory, show it as
its own node mapped to its AWS target (e.g. in-process `ConversationBufferWindowMemory` →
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ summary first, technical detail after (single layered doc — do not fork by aud
`$RUN_DIR/recommendation.md`. Append the freshness footer.

For `migrate`: also fill Section 9 (Bedrock model) with the **coarse family mapping**
(e.g. "GPT-4o → Claude Sonnet 4.6 family") and a note that detailed pricing/TCO come from the
(e.g. "GPT-4o → Claude Sonnet 5 family") and a note that detailed pricing/TCO come from the
migration plugins — no dollar figures. Section 10 (cost magnitude) presents the per-unit
target-state bands from estimate.json and notes that the migration TCO comparison and
current-spend delta are produced by the migration plugins.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ contract but never replaces it. Resolve each unit's model and `api_path` from IT
never take one unit's decision from another or from a run-wide plan scalar.

For each unit, `design.json.units[<id>].model_recommendation.model` carries the selected
path-specific model ID (for example, `anthropic.claude-sonnet-4-6`).
path-specific model ID (for example, `anthropic.claude-sonnet-5`).
`model_recommendation.invocation_model_id` is the exact account-callable ID only after CRIS
resolution. These identities must remain separate.

Expand Down Expand Up @@ -155,11 +155,11 @@ The block's `target_bedrock_model` and path should equal `model_recommendation`
validated them. If they differ, stop and return to Model Recommend; do not choose one silently.

**Strip environment annotations from the model id.** The running assistant's model name may
carry a context-window annotation like `[1m]` (e.g. `us.anthropic.claude-sonnet-4-6[1m]`).
carry a context-window annotation like `[1m]` (e.g. `us.anthropic.claude-sonnet-5[1m]`).
That suffix is a harness label, NOT a valid Bedrock model/inference-profile id — a Bedrock
call with it 404s. Before writing the id into any generated file, strip any trailing
`[...]` bracket annotation. The id you emit must be exactly what Bedrock accepts (e.g.
`us.anthropic.claude-sonnet-4-6`). If unsure of the exact id, use the `TODO: verify model id`
`us.anthropic.claude-sonnet-5`). If unsure of the exact id, use the `TODO: verify model id`
placeholder rather than a bracket-tagged string.

**Single-unit path:** when `units[]` length is 1, the above logic applies to the one unit —
Expand Down
Loading