feat(heroku-to-aws): post-Estimate what-if workshop checkpoint by herosjourney · Pull Request #152 · awslabs/startups

herosjourney · 2026-07-20T00:17:19Z

Summary

Adds an optional what-if workshop checkpoint after Estimate: freeze discovery, edit region/HA/compute/Graviton knobs, refresh Design + Estimate as artifact-only inner runs, snapshot up to 5 scenarios, compare side-by-side.
Defers current_phase → generate until the workshop is declined or exited; mid-workshop state stays on estimate (warm starts must not recompute Estimate).
Adds a stakeholder migration-report.html for heroku (new generate-report.md fragment + validator script) with a conditional what-if-scenarios section — Heroku previously had no shareable HTML report at all.
Per-scenario shareable calculator.aws links via the aws-pricing-calculator MCP (registered in .mcp.json, verified live; best-effort, never blocks) — AWS computes regional prices server-side when the link is opened, which the us-east-1 cache cannot.

Motivation: in a live SA workshop, "what if us-west-2 / no Multi-AZ / Graviton?" previously meant re-running Estimate or re-Clarifying. Now inventory stays frozen, assumptions are a live sheet, and the customer walks out with priced, compared, shareable scenarios from one discovery run.

Relationship to other PRs

Stacked on feat(heroku-to-aws): live discovery via the authenticated Heroku CLI #148 (live Heroku discovery) — its lineage is included.
feat(vercel-to-aws): post-Estimate what-if workshop checkpoint #153 (Vercel) and feat(gcp-to-aws): post-Estimate what-if workshop checkpoint #154 (GCP) are the same feature for the other skills; refactor(migrate): canonical workshop invariants + estimate pricing-mode, vendored (merge after #152-#154) #157 (merge-last) extracts the shared invariants.

Test plan

Fixtures: fixtures/heroku-workshop/ (post-Estimate seed → arm64 reprice after-state); python3 check_expected_workshop.py after-arm64-reprice → PASS (inventory byte-frozen, fingerprint match, t3.small→t4g.small, $95→$88 balanced, scenario subset correct)
Fresh-agent fragment replay of generate-report.md against the workshop after-state: all sections rendered, scenario table with tiers + active marker, validate-heroku-migration-report.py → REPORT_OK (three spec gaps folded back)
aws-pricing-calculator MCP verified live: all 9 documented tools respond to a stdio probe
mise run build green (branch previously failed fmt:check/markdownlint/frontmatter — fixed here)
Full workshop-loop fresh-agent replay (enter → edit → apply → compare → exit) — SA demo script in the fixture README

Made with Cursor

…Discovery previously required Terraform with heroku_* resources, which most startups on Heroku don't have. Adds a consent-gated, read-only live discovery path: an interactive capture step (exact-command whitelist, config-var key names only, auth:token banned) writes raw CLI output to live-capture/, and a new 'live' fragment parses it into the inventory. When Terraform is also present, live wins for current state and disagreements surface as drift (unmanaged resources, plan changes, config conflicts) — never silently resolved. Includes replay fixtures with a drift-assertion set, validated by end-to-end fresh-agent runs of the Discover phase.

…heck_expected_drift.py (stdlib-only) machine-checks a scenario-B replay's heroku-resource-inventory.json against expected-drift.json: metadata, apps, merged resources, conflict fields, drift counts, forbidden clustering fields, and secret hygiene (no config-var value payloads, no raw config_vars, no secret-shaped strings). Validated PASS against a fresh fresh-agent replay of scenario B on this branch. Same pattern as the gcp-live-capture asserter (awslabs#149) so both live-discovery skills regress identically.

…ne Live discovery (awslabs#148) captures each add-on's actual billed plan price (monthly_price_usd), but Estimate could only build a current-spend baseline from a billing export -- and the migrate-or-stay comparison was gated on billing data, so live-discovered runs got projected AWS costs with no side-by-side at all. - New baseline rung between billing data and the rate-card cache: add-ons priced exactly from live-captured plan prices, dynos from the published-rate cache (the Platform API does not price formations). Distinct provenance (current_costs.source: live_prices_plus_cache) and a mandatory caveat: derived from plan prices, not an invoice; excludes usage-based charges. - The comparison (Part 3) and migration cost considerations (Part 4) now run for ANY determined baseline, labeled with baseline_source -- previously both were billing-only, which orphaned cache-derived baselines too. - Schema: add cached_stale to pricing_source.status (already mandated by both skills' estimate specs; the enum rejected it). - Fixtures: seed-estimate/ (the scenario-B inventory post-merge, plus clarify/design artifacts) + expected-estimate.json + check_expected_estimate.py asserting the exact $352 baseline math, provenance label, caveat, and unlocked comparison. Validated by a fresh-agent replay of the Estimate phase (asserter PASS).

… prices Review follow-up: 'live discovery ran' was ambiguous when the run captured zero priced add-ons (e.g. a dyno-only app) -- rung 2 would fire, price everything from the cache via fallthrough, and still label the baseline live_prices_plus_cache with 'exact for add-ons' accuracy. Right number, wrong provenance. The rung now requires at least one add-on resource carrying config.monthly_price_usd; otherwise fall to rung 3, whose pricing_cache label describes that baseline honestly.

…and the estimated rung Three review-observation cleanups from the estimate-phase replay: - EBS root volumes: new ebs block in the shared pricing file (gp3 $0.08/GB-mo, 8 GB AL2023 EB root default) + vendored sync. EB estimates now price root volumes instead of shipping a permanent 'known minor omission' warning. - Sign conventions: roi_analysis.monthly_difference_* (AWS minus Heroku, negative = AWS cheaper) and financial_summary.monthly_savings_* (Heroku minus AWS) are the same fact with opposite signs. Both specs now cross-reference the other, name their own convention, and require any presentation to label direction in words rather than print a bare signed value. - Pricing hierarchy: new row 4 'estimated' — the documented home of the services_by_source.estimated bucket the schema and assembler already carry. A rate stated verbatim by this file's own formulas may price a service (always with a warning naming the rate); only no-cache no-MCP no-formula is 'unavailable'.

Freeze discovery and reprice region/HA/compute/Graviton via Design+Estimate inner runs, with scenario snapshots and a deferred Generate advance. Stacked on the live-prices Estimate baseline so schema/pricing merges cleanly. Co-authored-by: Cursor <cursoragent@cursor.com>

Teach INTERPRETER deferred-advance checkpoint resume (Estimate completed + workshop pending/in_progress) and mirror the rule in heroku SKILL orchestration. Co-authored-by: Cursor <cursoragent@cursor.com>

herosjourney · 2026-07-20T00:40:58Z

Fresh-agent replay + resume fix

Resume rule (`17df8f5`)

skills/shared/dsl/INTERPRETER.md — deferred-advance checkpoint resume: Estimate completed + workshop pending/in_progress must not re-run Estimate
heroku-to-aws/SKILL.md — explicit Workshop resume (mandatory) + warm-start exception note
Vendored INTERPRETER synced (heroku + agent-advisor)

Fresh-agent replay (house bar)

Agent from seed/ only (no after-arm64-reprice peek) → enter workshop → arm64 Apply & reprice → snapshot scenario-002.

python3 check_expected_workshop.py /tmp/replay-heroku-fresh seed
→ PASS — expected-workshop.json assertions hold

Inventory bytes frozen
EB t3.small → t4g.small / cpu_architecture: arm64
current_phase: estimate, workshop: in_progress, Generate not completed
Transcript agent: Heroku replay

…rios Thin Generate report (decision, costs, optional what-if table) plus validator so workshop scenarios reach stakeholders instead of only scenarios/*.json. Co-authored-by: Cursor <cursoragent@cursor.com>

herosjourney · 2026-07-20T01:13:48Z

Stakeholder deliverable: `migration-report.html`

Heroku Generate now writes a thin stakeholder report (generate-report.md) with:

decision-summary / exec-costs / next-steps
conditional what-if-scenarios (workshop compare table) when scenarios/index.json has ≥2 entries
validator: scripts/validate-heroku-migration-report.py

Commit: 0c74594

…sh-agent replay of generate-report.md against the workshop after-arm64 fixture (validator REPORT_OK, all sections + scenario table rendered correctly) surfaced three spec gaps, all fixed: - the documented validator invocation omitted the required positional report path (exit 2 as written); full command + exit-code branching now inline instead of 'same way GCP does' - the Inputs table undersold what the scenario columns need: manifests carry cost tiers only; Region/HA/Compute/Arch come from each scenario's preferences/design copies -- now stated - exec-costs now states the fallback explicitly: Heroku-vs-AWS when a baseline exists, three-tier AWS table + note when current_costs.source is 'unavailable'

…ch failed CI three ways, all pre-existing: (1) ten files committed with dprint drift (fmt:check); (2) markdownlint MD025/MD029 in the fixtures README, generate-report scenario list, and workshop-assemble's fenced block splitting an ordered list; (3) the frontmatter validator rejects the workshop fragment files -- their _fragment ids (workshop-sheet/-refresh/-compare) did not match workshop.md's declared _ids (sheet/refresh/compare); renamed to match the house convention (fragment id == phase reference _id, as estimate/cost-engine does). mise run build now exits 0.

herosjourney · 2026-07-20T02:36:49Z

Verified + fixed on this branch (4c7667f, 63c3b4b): Report validated by fragment replay. Ran generate-report.md fresh-agent against the workshop after-arm64 fixture: all sections rendered, the what-if table showed both scenarios with all tier columns + active-row marker + knob-delta line, and validate-heroku-migration-report.py returned REPORT_OK | sections=3/3 | optional=what-if-scenarios. Three spec gaps folded back: the documented validator invocation omitted the required positional report path (exit 2 as written — full command + exit-code branching now inline); the Inputs table undersold the scenario columns' data source (Region/HA/Compute/Arch come from each scenario's preferences/design copies, not the manifests); and exec-costs now states the no-baseline fallback explicitly. Branch now passes mise run build — it previously failed three ways: ten files with dprint drift, markdownlint MD025/MD029, and the frontmatter validator rejecting the workshop fragments (_fragment ids didn't match workshop.md's declared _ids — renamed to the house convention). fixtures-check + the workshop asserter pass.

…rio Registers the aws-pricing-calculator MCP server (npx, no AWS credentials -- verified live: all 9 documented tools respond) in .mcp.json, and adds workshop-refresh 6b: after each scenario snapshot, best-effort build_estimate -> shareable calculator.aws URL stored as estimation_summary.calculator_url (null + one chat note on any failure; never blocks). The compare table and the stakeholder report render the link. This is the honest answer to the region knob's biggest limit: the cache is us-east-1-based, but the calculator computes REGIONAL prices server-side when the link is opened. Complements awslabs#49 (gcp Estimate-phase calculator integration -- same server, same degradation rules).

herosjourney · 2026-07-20T03:36:38Z

Pushed 7f87918 — shareable calculator.aws link per workshop scenario. Registers the aws-pricing-calculator MCP server in .mcp.json (npx, no AWS credentials; verified live — all 9 documented tools respond to a stdio probe) and adds refresh step 6b: after each scenario snapshot, a best-effort build_estimate produces a shareable calculator.aws URL stored as estimation_summary.calculator_url (null + one chat note on any failure — never blocks). The compare table and the stakeholder report render the links. This is the honest answer to the region knob's biggest limitation: our cache is us-east-1-based, but AWS computes regional prices server-side when the link is opened. Complements #49 (same server, same degradation rules; that PR documents the gcp Estimate-phase integration, this one registers the server plugin-wide).

Resolves conflicts after awslabs#148 + awslabs#150 merged to main: - SKILL.md description: merged both sides — kept main's live-CLI discovery phrasing (awslabs#148) AND awslabs#152's what-if-workshop trigger words + workshop sentence. Auto-merge already restored the live-first Philosophy bullet, file-tree discover-live entries, and Migration-mode line from main. - seed-estimate/.phase-status.json + estimate-assemble.md: kept awslabs#152's additive workshop content (workshop phase seed, optional workshop metadata block).

Brings the branch up to current main (through awslabs#153/awslabs#154/awslabs#159). Resolutions: - fixtures/README.md: union — keep heroku-workshop + vercel-workshop registry lines. - estimation-infra.schema.json (canonical + vendored): took main's generic workshop- metadata description (the 'heroku-to-aws pilot' note is stale now that vercel/gcp share the field); shared:sync keeps copies byte-identical. Full build green.

Union-resolve fixtures/README.md: all three workshop fixture registry lines (heroku-workshop, vercel-workshop, gcp-workshop) now coexist. Full build green.

…ode, vendored Merge-last PR: built from awslabs#152 + awslabs#153 + awslabs#154 (plus awslabs#148 via awslabs#152's lineage). After those merge, three near-identical copies of the workshop contract and the estimate Step 0 pricing-mode block would exist -- the exact drift pattern that produced the cached_stale enum bug and the vendored-edited-instead-of-canonical arm64 rates. - skills/shared/workshop/workshop-invariants.md (canonical): frozen discovery + fingerprint, checkpoint state semantics incl. the warm-start rule, the inner-runs allowed/forbidden contract, scenario store rules (max-5 warn-and-name eviction, working-tree==active, estimation_summary core fields), region honesty, and the calculator link procedure. Vendored into all three skills; gcp gains a references/vendored tree (now 4 skills under shared:check). - skills/shared/estimate/pricing-mode.md (canonical Step 0): cache staleness, MCP retry ladder, display modes, and the 5-rung pricing hierarchy incl. the estimated rung. heroku + vercel cost engines now execute the vendored copy as Step 0 instead of restating it; vercel keeps its skill-specific Step 0a-workshop knobs section. - Skill workshop files keep knobs/artifact names/engine refreshes and defer to the invariants file for every invariant (workshop.md hard rules note the canonical wins on disagreement; refresh inner-runs and 6b calculator sections are pointers with skill-specific lines only). - READMEs unioned across the three branches; canonical estimation-infra schema takes the workshop-metadata side + shared:sync. Validated: shared:check green across 4 vendored trees, fixtures-check green (123 json / 8 asserters / 10 seeds), all three workshop asserters PASS on the merged tree, frontmatter validation green, full mise build green.

…ode, vendored Merge-last PR: built from #152 + #153 + #154 (plus #148 via #152's lineage). After those merge, three near-identical copies of the workshop contract and the estimate Step 0 pricing-mode block would exist -- the exact drift pattern that produced the cached_stale enum bug and the vendored-edited-instead-of-canonical arm64 rates. - skills/shared/workshop/workshop-invariants.md (canonical): frozen discovery + fingerprint, checkpoint state semantics incl. the warm-start rule, the inner-runs allowed/forbidden contract, scenario store rules (max-5 warn-and-name eviction, working-tree==active, estimation_summary core fields), region honesty, and the calculator link procedure. Vendored into all three skills; gcp gains a references/vendored tree (now 4 skills under shared:check). - skills/shared/estimate/pricing-mode.md (canonical Step 0): cache staleness, MCP retry ladder, display modes, and the 5-rung pricing hierarchy incl. the estimated rung. heroku + vercel cost engines now execute the vendored copy as Step 0 instead of restating it; vercel keeps its skill-specific Step 0a-workshop knobs section. - Skill workshop files keep knobs/artifact names/engine refreshes and defer to the invariants file for every invariant (workshop.md hard rules note the canonical wins on disagreement; refresh inner-runs and 6b calculator sections are pointers with skill-specific lines only). - READMEs unioned across the three branches; canonical estimation-infra schema takes the workshop-metadata side + shared:sync. Validated: shared:check green across 4 vendored trees, fixtures-check green (123 json / 8 asserters / 10 seeds), all three workshop asserters PASS on the merged tree, frontmatter validation green, full mise build green. (#157) Co-authored-by: Logan Kleier <lkleier@amazon.com>

Per product decision, remove the vercel-to-aws migration skill entirely. This is a forward-removal (not a git-revert): vercel landed via shared/cross-skill commits (#153/#157/#162/#155/#152/#176) that also carried heroku/gcp/sidebar work, so reverting those merge commits was not viable. Removed: - skills/vercel-to-aws/ (full skill, 57 files incl. its references/vendored/ copies) - fixtures/vercel-capture/ + fixtures/vercel-workshop/ (replay fixture sets) - Orphaned vercel report validator + its test + golden fixtures (lived at plugin level, not under the skill dir): scripts/validate-assessment-report.py, tests/test_validate_assessment_report.py, fixtures/assessment-report-{reference,stub}.html, fixtures/tier1-signals-reference.json (vercel prescan fixture, no consumers) - Deregistered from all 3 plugin manifests (.claude/.codex/.cursor-plugin): dropped vercel/next-js/nextjs/opennext keywords, the 'Vercel' mentions in descriptions, and the codex 'Assess my Vercel migration' example prompt - Stripped vercel from both READMEs (intro bullets, triggers, Vercel→AWS mapping table, trigger-table row, requirement bullet, DSL-architecture section: 'five skills'→'four', removed the resumability-ledger note) + fixtures/README registry line - Dropped vercel asserter entries from tools/run-asserters.py (10→6 asserters) - Stripped stale vercel cross-refs: heroku SKILL.md workshop-arch note, gcp schema-workshop-scenarios arch-defaults line, canonical workshop-invariants {SKILL_LABEL}/{INVENTORY} placeholders (shared:sync propagated to vendored copies) PRESERVED (heroku + gcp depend on these — #157's shared architecture): canonical skills/shared/workshop/workshop-invariants.md + skills/shared/estimate/pricing-mode.md and their heroku/gcp vendored copies. shared:check drops 4→3 trees cleanly (the sync tool auto-discovers consumers). Left legit non-skill refs: gcp's vercel.json detection signal, 'Vercel AI SDK' framework name, historical PR-context notes. Verified: shared:check OK (3 trees byte-identical), fixtures:assert 6/6, all validators green (heroku/gcp/agent-advisor), dprint + markdownlint clean, full mise run build green.

Logan Kleier and others added 6 commits July 18, 2026 12:54

herosjourney requested a review from a team as a code owner July 20, 2026 00:17

herosjourney mentioned this pull request Jul 20, 2026

feat(gcp-to-aws): post-Estimate what-if workshop checkpoint #154

Merged

5 tasks

fix(heroku-to-aws): resume must not recompute Estimate mid-workshop

17df8f5

Teach INTERPRETER deferred-advance checkpoint resume (Estimate completed + workshop pending/in_progress) and mirror the rule in heroku SKILL orchestration. Co-authored-by: Cursor <cursoragent@cursor.com>

feat(heroku-to-aws): add stakeholder migration-report.html with scena…

0c74594

…rios Thin Generate report (decision, costs, optional what-if table) plus validator so workshop scenarios reach stakeholders instead of only scenarios/*.json. Co-authored-by: Cursor <cursoragent@cursor.com>

Logan Kleier added 2 commits July 19, 2026 18:59

icarthick added 2 commits July 20, 2026 14:21

icarthick merged commit d009d00 into awslabs:main Jul 20, 2026
8 checks passed

herosjourney mentioned this pull request Jul 20, 2026

feat(migrate): declared _gates key for sidebars + full lint:frontmatter coverage #166

Open

4 tasks

icarthick mentioned this pull request Jul 23, 2026

revert(migrate): remove the vercel-to-aws plugin #179

Merged

5 tasks

Navigation Menu

Uh oh!

feat(heroku-to-aws): post-Estimate what-if workshop checkpoint - #152

feat(heroku-to-aws): post-Estimate what-if workshop checkpoint#152
icarthick merged 13 commits into
awslabs:mainfrom
herosjourney:feat/heroku-what-if-workshop

herosjourney commented Jul 20, 2026 •

edited

Loading

Uh oh!

herosjourney commented Jul 20, 2026

Uh oh!

herosjourney commented Jul 20, 2026

Uh oh!

herosjourney commented Jul 20, 2026

Uh oh!

herosjourney commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

herosjourney commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Relationship to other PRs

Test plan

Uh oh!

herosjourney commented Jul 20, 2026

Fresh-agent replay + resume fix

Resume rule (17df8f5)

Fresh-agent replay (house bar)

Uh oh!

herosjourney commented Jul 20, 2026

Stakeholder deliverable: migration-report.html

Uh oh!

herosjourney commented Jul 20, 2026

Uh oh!

herosjourney commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

herosjourney commented Jul 20, 2026 •

edited

Loading

Resume rule (`17df8f5`)

Stakeholder deliverable: `migration-report.html`