ci(migrate): fixtures integrity check + pricing-cache staleness report - #151
Merged
icarthick merged 2 commits intoJul 22, 2026
Merged
Conversation
…t Two zero-dep TS tools (Node 24 type-stripping, same pattern as the existing validators), both wired into the lint aggregate: fixtures:check -- the replay fixture sets are the plugin's regression harness, and they rot in ways the build never saw: capture manifests referencing files that were never committed (a repo-root .gitignore build/ rule once silently swallowed three canned .next manifests), seed .phase-status.json files lagging behind the phases a skill declares, and asserters carrying syntax newer than the ambient python. Now CI checks: every fixture JSON parses, every asserter ast-parses (no bytecode litter), every manifest captures[]/api[] entry with status ok resolves to a file on disk (failed/skipped may be absent) and build.files exist, nothing under fixtures/ is gitignored, and every seed phase list exactly matches the owning skill's declared _phase set. Validated against all four in-flight fixture sets (heroku/gcp/vercel x2) -- and it immediately caught a real one: the vercel seed omitted the declared scaffold phase (fixed on that branch as d11db7b). pricing:staleness -- every pricing cache declares its own freshness contract (_meta.last_updated + staleness_days, or a Last updated line with a documented 30-day window) but nothing enforced it; caches quietly crossed their own thresholds and every estimate degraded to fallback accuracy. Warn-only in build (a stale cache must not fail unrelated PRs); --strict exits 1 for a scheduled freshness workflow. Currently reports 3/4 caches stale, which is the point.
This was referenced Jul 19, 2026
Merged
Merged
Clean merge (no conflicts) bringing the branch to current main. The two new zero-dep CI tools validated against main's current fixture sets: - fixtures:check OK (139 json, 9 asserters, 3 manifests / 51 refs, 10 seeds) - pricing:staleness reports 1/5 stale (heroku cache) — WARN-ONLY in build by design; awslabs#155 (stacked on this) does the actual refresh. Full mise build green.
icarthick
added a commit
to herosjourney/startups
that referenced
this pull request
Jul 22, 2026
Brings the branch to current main (after awslabs#151's staleness tool merged). awslabs#155's cache refresh now takes effect: pricing:staleness reports 5/5 fresh (was 1/5 stale), --strict exits 0, and the new weekly .github/workflows/pricing-staleness.yml is present. Also shared:sync'd vercel's vendored aws-infra-pricing.json to canonical (it lacked the new _meta.last_verification provenance field awslabs#155 adds). shared:check OK (4 trees), full build green.
icarthick
added a commit
that referenced
this pull request
Jul 22, 2026
…flow (stacked on #151) (#155) * ci(migrate): fixtures integrity check + pricing-cache staleness report Two zero-dep TS tools (Node 24 type-stripping, same pattern as the existing validators), both wired into the lint aggregate: fixtures:check -- the replay fixture sets are the plugin's regression harness, and they rot in ways the build never saw: capture manifests referencing files that were never committed (a repo-root .gitignore build/ rule once silently swallowed three canned .next manifests), seed .phase-status.json files lagging behind the phases a skill declares, and asserters carrying syntax newer than the ambient python. Now CI checks: every fixture JSON parses, every asserter ast-parses (no bytecode litter), every manifest captures[]/api[] entry with status ok resolves to a file on disk (failed/skipped may be absent) and build.files exist, nothing under fixtures/ is gitignored, and every seed phase list exactly matches the owning skill's declared _phase set. Validated against all four in-flight fixture sets (heroku/gcp/vercel x2) -- and it immediately caught a real one: the vercel seed omitted the declared scaffold phase (fixed on that branch as d11db7b). pricing:staleness -- every pricing cache declares its own freshness contract (_meta.last_updated + staleness_days, or a Last updated line with a documented 30-day window) but nothing enforced it; caches quietly crossed their own thresholds and every estimate degraded to fallback accuracy. Warn-only in build (a stale cache must not fail unrelated PRs); --strict exits 1 for a scheduled freshness workflow. Currently reports 3/4 caches stale, which is the point. * chore(migrate): refresh pricing caches; weekly strict staleness workflow All caches had crossed their own declared freshness windows (the new pricing:staleness tool reports 3/4 stale), so every estimate was degrading to cached_fallback accuracy. Refresh: spot-verified the highest-traffic rates against published pricing pages -- Fargate Linux/x86 us-east-1 per-second rates convert exactly to the cached $0.04048/vCPU-hr and $0.004445/GB-hr, and Heroku dyno flat rates (Eco $5 flat, Basic $7, Standard-1X $25, Standard-2X $50, Performance-M $250, Performance-L $500) match devcenter -- no rate changes needed, so this is a verified date bump with provenance recorded in _meta.last_verification / the cache header. Vendored copies synced. New .github/workflows/pricing-staleness.yml: weekly scheduled run of the zero-dep staleness tool with --strict (build keeps the warn-only mode so a stale cache never fails unrelated PRs; the schedule makes staleness a visible failure someone owns, prompting a #131-style data refresh). --------- Co-authored-by: Logan Kleier <lkleier@amazon.com> Co-authored-by: Karthik Balasubramanian <carthick@amazon.com>
icarthick
added a commit
to herosjourney/startups
that referenced
this pull request
Jul 22, 2026
Resolves conflicts after the workshop stack + awslabs#149/awslabs#151/awslabs#155/awslabs#162 landed: - heroku-to-aws SKILL.md + heroku-live-capture/README.md: took main (stale awslabs#148/awslabs#150 lineage; main has the merged workshop/what-if + Scenario C content). - gcp-live-capture/README.md: kept THIS branch's added 'Scenario C — derived GCP baseline' block (its genuine new content; main's side empty there). - migrate/README.md: took main (adds vercel + agent-advisor reqs; awslabs#149's gcloud GCP requirement lines already present above the conflict). New gcp-infra-pricing-cache.md auto-enrolls in pricing:staleness (6/6 fresh) and the seed-baseline fixtures pass fixtures:check (141 json / 10 asserters). Full build green.
icarthick
added a commit
to herosjourney/startups
that referenced
this pull request
Jul 22, 2026
Union-resolved mise.toml lint group: fixtures:assert (this PR) + fixtures:check + pricing:staleness (from awslabs#151/awslabs#155, now on main) all run in build. Registered the two gcp-live-capture asserters that landed via awslabs#149/awslabs#156 after this branch was cut (the PR's own registry-completeness guard correctly flagged them): - gcp-live-capture/check_expected_baseline.py (smoke — already exits 1 cleanly) - gcp-live-capture/check_expected_drift.py (smoke — added the same missing-artifact guard this PR applies to the other 4, replacing a FileNotFoundError traceback with a clean one-line FAIL + exit 1) fixtures:assert now 10/10 (3 golden, 7 smoke). Full build green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lintaggregate somise run buildruns them:mise run fixtures:check— structural integrity of the committed replay fixtures.mise run pricing:staleness— reports pricing caches past their own declared freshness windows (warn-only in build;--strictfor a scheduled gate).Motivation: the fixture sets are the plugin's regression harness, and they rot in ways the build never saw — all three of these happened in practice: a capture manifest referencing files that were never committed (the repo-root
.gitignore'sbuild/rule silently swallowed three canned.nextmanifests), a seed.phase-status.jsonlagging behind the phases the skill declares, and an asserter using Python-3.10-only syntax that crashes on 3.9. Separately, every pricing cache declares a freshness contract that nothing enforced — caches quietly went stale and every estimate degraded to fallback accuracy.Changes
tools/fixtures-check.tsChecks: every fixture JSON parses; every asserter
ast.parses (no bytecode litter); every manifestcaptures[]/api[]entry with statusokresolves to a file on disk (failed/skippedmay legitimately have none) andbuild.files[]exist; nothing underfixtures/is gitignored; every seed phase list exactly matches the owning skill's declared_phaseset. Takes an optional root argument to point at another worktree.tools/pricing-staleness.tsReads each cache's own declared date and window (
_meta.last_updated+staleness_days, or the markdown**Last updated:**header). Warn-only inbuild(a stale cache must not fail unrelated PRs);--strictexits 1 for a scheduled workflow. Currently reports 3 of 4 caches stale on main — the condition that degraded this week's estimate replays.Supporting
node-shims.d.tsextended (basename/relative/spawnSync) to keep the zero-dependency typing approach; tsconfig includes the new tools solint:typescovers them.Relationship to other PRs
Test plan
mise run buildgreen (both tasks run insidelint;lint:typestype-checks the tools)fixtures:checkPASS against all four in-flight fixture sets — after catching a real bug on feat(migrate): add vercel-to-aws migration assessment skill #141 (seed omitted the declaredscaffoldphase; fixed there asd11db7b)build.filesentry, gitignored fixture, and an invalid seed — all 9 problems reported, exit 1pricing:stalenesswarn-mode exit 0;--strictexit 1 against main's stale caches