Skip to content

ci(migrate): fixtures integrity check + pricing-cache staleness report - #151

Merged
icarthick merged 2 commits into
awslabs:mainfrom
herosjourney:feat/fixtures-integrity-check
Jul 22, 2026
Merged

ci(migrate): fixtures integrity check + pricing-cache staleness report#151
icarthick merged 2 commits into
awslabs:mainfrom
herosjourney:feat/fixtures-integrity-check

Conversation

@herosjourney

@herosjourney herosjourney commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds two zero-dependency TypeScript CI tools (Node 24 native type-stripping, same pattern as the existing validators), both wired into the lint aggregate so mise run build runs them:
    • mise run fixtures:check — structural integrity of the committed replay fixtures.
    • mise run pricing:staleness — reports pricing caches past their own declared freshness windows (warn-only in build; --strict for a scheduled gate).

Motivation: the fixture sets are the plugin's regression harness, and they rot in ways the build never saw — all three of these happened in practice: a capture manifest referencing files that were never committed (the repo-root .gitignore's build/ rule silently swallowed three canned .next manifests), a seed .phase-status.json lagging behind the phases the skill declares, and an asserter using Python-3.10-only syntax that crashes on 3.9. Separately, every pricing cache declares a freshness contract that nothing enforced — caches quietly went stale and every estimate degraded to fallback accuracy.

Changes

tools/fixtures-check.ts

Checks: every fixture JSON parses; every asserter ast.parses (no bytecode litter); every manifest captures[]/api[] entry with status ok resolves to a file on disk (failed/skipped may legitimately have none) and build.files[] exist; nothing under fixtures/ is gitignored; every seed phase list exactly matches the owning skill's declared _phase set. Takes an optional root argument to point at another worktree.

tools/pricing-staleness.ts

Reads each cache's own declared date and window (_meta.last_updated + staleness_days, or the markdown **Last updated:** header). Warn-only in build (a stale cache must not fail unrelated PRs); --strict exits 1 for a scheduled workflow. Currently reports 3 of 4 caches stale on main — the condition that degraded this week's estimate replays.

Supporting

  • node-shims.d.ts extended (basename/relative/spawnSync) to keep the zero-dependency typing approach; tsconfig includes the new tools so lint:types covers them.

Relationship to other PRs

Test plan

  • mise run build green (both tasks run inside lint; lint:types type-checks the tools)
  • fixtures:check PASS against all four in-flight fixture sets — after catching a real bug on feat(migrate): add vercel-to-aws migration assessment skill #141 (seed omitted the declared scaffold phase; fixed there as d11db7b)
  • Negative tests: synthetic tree with broken JSON, broken asserter, missing ok-status capture, missing build.files entry, gitignored fixture, and an invalid seed — all 9 problems reported, exit 1
  • pricing:staleness warn-mode exit 0; --strict exit 1 against main's stale caches

…t Two zero-dep TS tools (Node 24 type-stripping, same pattern as the existing validators), both wired into the lint aggregate: fixtures:check -- the replay fixture sets are the plugin's regression harness, and they rot in ways the build never saw: capture manifests referencing files that were never committed (a repo-root .gitignore build/ rule once silently swallowed three canned .next manifests), seed .phase-status.json files lagging behind the phases a skill declares, and asserters carrying syntax newer than the ambient python. Now CI checks: every fixture JSON parses, every asserter ast-parses (no bytecode litter), every manifest captures[]/api[] entry with status ok resolves to a file on disk (failed/skipped may be absent) and build.files exist, nothing under fixtures/ is gitignored, and every seed phase list exactly matches the owning skill's declared _phase set. Validated against all four in-flight fixture sets (heroku/gcp/vercel x2) -- and it immediately caught a real one: the vercel seed omitted the declared scaffold phase (fixed on that branch as d11db7b). pricing:staleness -- every pricing cache declares its own freshness contract (_meta.last_updated + staleness_days, or a Last updated line with a documented 30-day window) but nothing enforced it; caches quietly crossed their own thresholds and every estimate degraded to fallback accuracy. Warn-only in build (a stale cache must not fail unrelated PRs); --strict exits 1 for a scheduled freshness workflow. Currently reports 3/4 caches stale, which is the point.
Clean merge (no conflicts) bringing the branch to current main. The two new
zero-dep CI tools validated against main's current fixture sets:
- fixtures:check OK (139 json, 9 asserters, 3 manifests / 51 refs, 10 seeds)
- pricing:staleness reports 1/5 stale (heroku cache) — WARN-ONLY in build by design;
  awslabs#155 (stacked on this) does the actual refresh.
Full mise build green.
@icarthick
icarthick merged commit 88e6404 into awslabs:main Jul 22, 2026
8 checks passed
icarthick added a commit to herosjourney/startups that referenced this pull request Jul 22, 2026
Brings the branch to current main (after awslabs#151's staleness tool merged). awslabs#155's
cache refresh now takes effect: pricing:staleness reports 5/5 fresh (was 1/5 stale),
--strict exits 0, and the new weekly .github/workflows/pricing-staleness.yml is present.

Also shared:sync'd vercel's vendored aws-infra-pricing.json to canonical (it lacked
the new _meta.last_verification provenance field awslabs#155 adds). shared:check OK (4 trees),
full build green.
icarthick added a commit that referenced this pull request Jul 22, 2026
…flow (stacked on #151) (#155)

* ci(migrate): fixtures integrity check + pricing-cache staleness report Two zero-dep TS tools (Node 24 type-stripping, same pattern as the existing validators), both wired into the lint aggregate: fixtures:check -- the replay fixture sets are the plugin's regression harness, and they rot in ways the build never saw: capture manifests referencing files that were never committed (a repo-root .gitignore build/ rule once silently swallowed three canned .next manifests), seed .phase-status.json files lagging behind the phases a skill declares, and asserters carrying syntax newer than the ambient python. Now CI checks: every fixture JSON parses, every asserter ast-parses (no bytecode litter), every manifest captures[]/api[] entry with status ok resolves to a file on disk (failed/skipped may be absent) and build.files exist, nothing under fixtures/ is gitignored, and every seed phase list exactly matches the owning skill's declared _phase set. Validated against all four in-flight fixture sets (heroku/gcp/vercel x2) -- and it immediately caught a real one: the vercel seed omitted the declared scaffold phase (fixed on that branch as d11db7b). pricing:staleness -- every pricing cache declares its own freshness contract (_meta.last_updated + staleness_days, or a Last updated line with a documented 30-day window) but nothing enforced it; caches quietly crossed their own thresholds and every estimate degraded to fallback accuracy. Warn-only in build (a stale cache must not fail unrelated PRs); --strict exits 1 for a scheduled freshness workflow. Currently reports 3/4 caches stale, which is the point.

* chore(migrate): refresh pricing caches; weekly strict staleness workflow All caches had crossed their own declared freshness windows (the new pricing:staleness tool reports 3/4 stale), so every estimate was degrading to cached_fallback accuracy. Refresh: spot-verified the highest-traffic rates against published pricing pages -- Fargate Linux/x86 us-east-1 per-second rates convert exactly to the cached $0.04048/vCPU-hr and $0.004445/GB-hr, and Heroku dyno flat rates (Eco $5 flat, Basic $7, Standard-1X $25, Standard-2X $50, Performance-M $250, Performance-L $500) match devcenter -- no rate changes needed, so this is a verified date bump with provenance recorded in _meta.last_verification / the cache header. Vendored copies synced. New .github/workflows/pricing-staleness.yml: weekly scheduled run of the zero-dep staleness tool with --strict (build keeps the warn-only mode so a stale cache never fails unrelated PRs; the schedule makes staleness a visible failure someone owns, prompting a #131-style data refresh).

---------

Co-authored-by: Logan Kleier <lkleier@amazon.com>
Co-authored-by: Karthik Balasubramanian <carthick@amazon.com>
icarthick added a commit to herosjourney/startups that referenced this pull request Jul 22, 2026
Resolves conflicts after the workshop stack + awslabs#149/awslabs#151/awslabs#155/awslabs#162 landed:
- heroku-to-aws SKILL.md + heroku-live-capture/README.md: took main (stale awslabs#148/awslabs#150
  lineage; main has the merged workshop/what-if + Scenario C content).
- gcp-live-capture/README.md: kept THIS branch's added 'Scenario C — derived GCP
  baseline' block (its genuine new content; main's side empty there).
- migrate/README.md: took main (adds vercel + agent-advisor reqs; awslabs#149's gcloud GCP
  requirement lines already present above the conflict).

New gcp-infra-pricing-cache.md auto-enrolls in pricing:staleness (6/6 fresh) and the
seed-baseline fixtures pass fixtures:check (141 json / 10 asserters). Full build green.
icarthick added a commit to herosjourney/startups that referenced this pull request Jul 22, 2026
Union-resolved mise.toml lint group: fixtures:assert (this PR) + fixtures:check +
pricing:staleness (from awslabs#151/awslabs#155, now on main) all run in build.

Registered the two gcp-live-capture asserters that landed via awslabs#149/awslabs#156 after this
branch was cut (the PR's own registry-completeness guard correctly flagged them):
- gcp-live-capture/check_expected_baseline.py (smoke — already exits 1 cleanly)
- gcp-live-capture/check_expected_drift.py (smoke — added the same missing-artifact
  guard this PR applies to the other 4, replacing a FileNotFoundError traceback with
  a clean one-line FAIL + exit 1)

fixtures:assert now 10/10 (3 golden, 7 smoke). Full build green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants