Skip to content

Require golden fixture cutover gates #105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mrjf merged 4 commits into from
Jun 4, 2026
Merged

Require golden fixture cutover gates #105

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e14a028
Require golden fixture cutover gates
mrjf Jun 3, 2026
0fbef5d
Fix test file formatting for CI
Copilot Jun 3, 2026
190b4f2
Fix parity coverage manifest for new Python tests
Copilot Jun 3, 2026
f4b4f12
Merge remote-tracking branch 'origin/main' into codex/golden-fixture-…
mrjf Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 60 additions & 2 deletions .crane/scripts/score.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ type CutoverGates struct {
FunctionalContracts float64 `json:"functional_contracts"`
StateDiffContracts float64 `json:"state_diff_contracts"`
PythonBehaviorContracts float64 `json:"python_behavior_contracts"`
GoldenFixtureCorpus string `json:"golden_fixture_corpus"`
AllGoGoldenTests string `json:"all_go_golden_tests"`
NoPythonRuntime string `json:"no_python_runtime_dependency"`
KnownExceptions int `json:"known_exceptions"`
GoTests string `json:"go_tests"`
PythonTests string `json:"python_tests"`
Expand Down Expand Up @@ -99,6 +102,9 @@ type Score struct {
PythonTestsPassing bool `json:"python_tests_passing"`
GoTestsPassing bool `json:"go_tests_passing"`
BenchmarksPassing bool `json:"benchmarks_passing"`
GoldenFixtureCorpus bool `json:"golden_fixture_corpus"`
AllGoGoldenTests bool `json:"all_go_golden_tests"`
NoPythonRuntime bool `json:"no_python_runtime_dependency"`
ParityPassing int `json:"parity_passing"`
ParityTotal int `json:"parity_total"`
SourceTestsPassing int `json:"source_tests_passing"`
Expand Down Expand Up @@ -143,6 +149,9 @@ func computeScore(input scanInput, getenv getenvFunc) (Score, error) {
functional := RatioGate{}
stateDiff := RatioGate{}
behaviorContracts := RatioGate{}
goldenFixtureCorpus := BoolGate{}
allGoGoldenTests := BoolGate{}
noPythonRuntime := BoolGate{}

for scanner.Scan() {
line := scanner.Text()
Expand All @@ -151,7 +160,21 @@ func computeScore(input scanInput, getenv getenvFunc) (Score, error) {
}
if gate, ok := parseGateEvent(line); ok {
eventsSeen++
applyGateEvent(gate, &pythonReference, &surface, &help, &functional, &stateDiff, &behaviorContracts, &knownExceptions, &pythonTests, &benchmarks)
applyGateEvent(
gate,
&pythonReference,
&surface,
&help,
&functional,
&stateDiff,
&behaviorContracts,
&goldenFixtureCorpus,
&allGoGoldenTests,
&noPythonRuntime,
&knownExceptions,
&pythonTests,
&benchmarks,
)
continue
}

Expand All @@ -163,7 +186,21 @@ func computeScore(input scanInput, getenv getenvFunc) (Score, error) {

if ev.Output != "" {
if gate, ok := parseGateEvent(ev.Output); ok {
applyGateEvent(gate, &pythonReference, &surface, &help, &functional, &stateDiff, &behaviorContracts, &knownExceptions, &pythonTests, &benchmarks)
applyGateEvent(
gate,
&pythonReference,
&surface,
&help,
&functional,
&stateDiff,
&behaviorContracts,
&goldenFixtureCorpus,
&allGoGoldenTests,
&noPythonRuntime,
&knownExceptions,
&pythonTests,
&benchmarks,
)
}
if n, ok := approvedExceptionCount(ev.Output); ok && n > knownExceptions {
knownExceptions = n
Expand Down Expand Up @@ -253,6 +290,9 @@ func computeScore(input scanInput, getenv getenvFunc) (Score, error) {
FunctionalContracts: functional.Percent(),
StateDiffContracts: stateDiff.Percent(),
PythonBehaviorContracts: behaviorContracts.Percent(),
GoldenFixtureCorpus: passFail(goldenFixtureCorpus.OK()),
AllGoGoldenTests: passFail(allGoGoldenTests.OK()),
NoPythonRuntime: passFail(noPythonRuntime.OK()),
KnownExceptions: knownExceptions,
GoTests: passFail(goTestsPass),
PythonTests: passFail(pythonTests.OK()),
Expand All @@ -275,6 +315,9 @@ func computeScore(input scanInput, getenv getenvFunc) (Score, error) {
gates.FunctionalContracts == 1.0 &&
gates.StateDiffContracts == 1.0 &&
gates.PythonBehaviorContracts == 1.0 &&
gates.GoldenFixtureCorpus == "pass" &&
gates.AllGoGoldenTests == "pass" &&
gates.NoPythonRuntime == "pass" &&
gates.KnownExceptions == 0 &&
gates.GoTests == "pass" &&
gates.PythonTests == "pass" &&
Expand Down Expand Up @@ -315,6 +358,9 @@ func computeScore(input scanInput, getenv getenvFunc) (Score, error) {
PythonTestsPassing: gates.PythonTests == "pass",
GoTestsPassing: gates.GoTests == "pass",
BenchmarksPassing: gates.Benchmarks == "pass",
GoldenFixtureCorpus: gates.GoldenFixtureCorpus == "pass",
AllGoGoldenTests: gates.AllGoGoldenTests == "pass",
NoPythonRuntime: gates.NoPythonRuntime == "pass",
ParityPassing: metrics.ParityPassing,
ParityTotal: metrics.ParityTotal,
SourceTestsPassing: metrics.SourceTestsPassing,
Expand Down Expand Up @@ -344,6 +390,9 @@ func applyGateEvent(
functional *RatioGate,
stateDiff *RatioGate,
behaviorContracts *RatioGate,
goldenFixtureCorpus *BoolGate,
allGoGoldenTests *BoolGate,
noPythonRuntime *BoolGate,
knownExceptions *int,
pythonTests *BoolGate,
benchmarks *RatioGate,
Expand All @@ -361,6 +410,12 @@ func applyGateEvent(
*stateDiff = RatioGate{Seen: true, Passing: gate.Passing, Total: gate.Total}
case "python_behavior_contracts":
*behaviorContracts = RatioGate{Seen: true, Passing: gate.Passing, Total: gate.Total}
case "golden_fixture_corpus":
*goldenFixtureCorpus = BoolGate{Seen: true, Passed: gate.Passed}
case "all_go_golden_tests":
*allGoGoldenTests = BoolGate{Seen: true, Passed: gate.Passed}
case "no_python_runtime_dependency":
*noPythonRuntime = BoolGate{Seen: true, Passed: gate.Passed}
case "known_exceptions":
*knownExceptions = gate.Count
case "python_tests":
Expand Down Expand Up @@ -412,6 +467,9 @@ func gateResults(gates CutoverGates) []GateResult {
{Name: "functional_contracts", Passing: gates.FunctionalContracts == 1.0},
{Name: "state_diff_contracts", Passing: gates.StateDiffContracts == 1.0},
{Name: "python_behavior_contracts", Passing: gates.PythonBehaviorContracts == 1.0},
{Name: "golden_fixture_corpus", Passing: gates.GoldenFixtureCorpus == "pass"},
{Name: "all_go_golden_tests", Passing: gates.AllGoGoldenTests == "pass"},
{Name: "no_python_runtime_dependency", Passing: gates.NoPythonRuntime == "pass"},
{Name: "python_tests_pass", Passing: gates.PythonTests == "pass"},
{Name: "benchmarks_pass", Passing: gates.Benchmarks == "pass"},
{Name: "no_known_exceptions", Passing: gates.KnownExceptions == 0},
Expand Down
34 changes: 25 additions & 9 deletions cmd/apm/CUTOVER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,12 @@ via PyInstaller packaging and `pip install apm-cli`.
The Go CLI currently implements:
- `apm --help` / `apm --version` (full parity with Python)
- `apm init [--yes] [PROJECT_NAME]` (functional, creates apm.yml)
- Per-command `--help` for all 26 commands (golden-file verified)
- Per-command `--help` for all 26 commands (initial golden-file coverage)

The checked-in `cmd/apm/testdata/golden/` files are the start of the
cutover corpus, not final completion proof. Final completion requires the
full command matrix below to be represented as committed fixtures and replayed
by Go without invoking the Python runtime.

Most remaining commands are wired at the CLI surface. That is not enough for
cutover. A command that prints success without writing the expected files,
Expand Down Expand Up @@ -84,8 +89,18 @@ are true:
4. Python-vs-Go parity tests pass for all commands in the matrix
5. Migration benchmarks pass real fixture-backed command workloads and emit a
passing counted `benchmarks` gate
6. `go build ./cmd/apm` produces a single static binary
7. CI passes on the crane PR branch (`crane/crane-migration-python-to-go-full-apm-cli-rewrite`)
6. The final Python-reference parity run has been frozen into a committed,
versioned golden fixture corpus. The corpus must include CLI inventory,
help and usage output, error output, exit codes, generated files, lockfiles,
config files, managed-file manifests, deterministic cache/config layout, and
audit artifacts for the full command matrix.
7. An all-Go golden replay passes against that corpus with no live Python
oracle. The replay must build `cmd/apm` and compare only the Go binary
against checked-in fixtures.
8. A no-Python-runtime check passes: `APM_PYTHON_BIN` is unset, the Python CLI
is hidden or unavailable to the replay, and the golden replay still passes.
9. `go build ./cmd/apm` produces a single static binary
10. CI passes on the crane PR branch (`crane/crane-migration-python-to-go-full-apm-cli-rewrite`)

## Cutover Steps

Expand All @@ -102,13 +117,14 @@ When conditions are met:

## Python Compatibility Shim

Until all commands are implemented in Go, the Python CLI remains the
authoritative `apm` command. The Go binary is available as `apm-go`
for testing.
Until all commands are implemented in Go and the golden replay gate passes, the
Python CLI remains the authoritative `apm` command. The Go binary is available
as `apm-go` for testing.

The shim removal plan: once the command matrix passes functional tests,
the Python entrypoint is replaced by the Go binary in the same PR that
passes the final parity tests.
The shim removal plan: once the command matrix passes functional tests, the
final Python-reference behavior is frozen into golden fixtures. Only after the
all-Go replay passes without a Python runtime can the Python entrypoint be
replaced by the Go binary.

## Timeline

Expand Down
Loading
Loading