Skip to content

Commit 44c3df8

Browse files
runkidsrunkids
committed
feat: add mdproof.json support for step-setup/teardown, E2E tests, and update-docs skill
- Add step_setup/step_teardown fields to Config struct and Merge function - CLI flags override config values; RunOptions now reads from merged config - Fix CHANGELOG CLI examples (remove nonexistent `run` subcommand, correct flag ordering) - Add E2E runbook (11-step-setup-subcmd-proof.md) with 12 steps validating both v0.0.4 features - Add test fixtures: with-subcmds-proof.md, subcmd-fail-proof.md, setup-target-proof.md - Update skills/ docs with sub-command separator, per-step hooks, and config support - Add update-docs skill for cross-validating docs against Go source
1 parent fb05891 commit 44c3df8

13 files changed

Lines changed: 549 additions & 16 deletions

File tree

‎.skillshare/skills/update-docs/SKILL.md‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
---
2+
name: mdproof-update-docs
3+
description: >-
4+
Update skills/ documentation and CHANGELOG to match recent code changes,
5+
cross-validating every flag and config field against Go source. Use this skill
6+
whenever the user asks to: update docs, sync docs with code, document a new
7+
flag or config option, fix stale docs, or refresh the skills reference after
8+
implementing a feature. This skill covers skills/SKILL.md (main doc),
9+
skills/references/ (assertions, advanced features), and CHANGELOG.md. If you
10+
just implemented a feature and need to update documentation, this is the skill
11+
to use. Never manually edit skills/ docs without cross-validating against Go
12+
source first.
13+
argument-hint: "[feature-name | commit-range]"
14+
targets: [claude, codex]
15+
---
16+
17+
Sync skills/ documentation with recent code changes. $ARGUMENTS specifies scope: a feature name (e.g., `step-setup`), commit range, or omit to auto-detect from `git diff HEAD~1`.
18+
19+
**Scope**: This skill updates `skills/`, `CHANGELOG.md`. It does NOT write Go code (use `implement-feature`).
20+
21+
## Workflow
22+
23+
### Step 1: Detect Changes
24+
25+
```bash
26+
# Auto-detect recently changed code
27+
git diff HEAD~1 --stat -- cmd/mdproof/ internal/
28+
29+
# Check config struct changes
30+
git diff HEAD~1 -- internal/config/config.go
31+
32+
# Check new/changed types
33+
git diff HEAD~1 -- internal/core/types.go
34+
```
35+
36+
Map changed files to affected documentation:
37+
38+
**Main skill doc** (`skills/SKILL.md`):
39+
- `cmd/mdproof/main.go` → Quick Reference, CLI Flags table
40+
- `internal/core/types.go` → Writing Runbooks section (new step fields, report structure)
41+
- `internal/config/config.go` → mention config support in CLI Flags table
42+
- `internal/executor/` → Persistent Session, Code Blocks, Sub-Commands sections
43+
44+
**Assertions reference** (`skills/references/assertions-guide.md`):
45+
- `internal/assertion/` → assertion type changes, new matchers
46+
- `internal/executor/` → changes to how assertions interact with execution
47+
48+
**Advanced features reference** (`skills/references/advanced-features.md`):
49+
- `internal/executor/session.go` → directives, hooks, retry, sub-commands
50+
- `internal/config/config.go` → Configuration File section
51+
- `cmd/mdproof/main.go` → new flags → new sections
52+
- `internal/report/` → report format changes (plain, JSON, JUnit)
53+
54+
**CHANGELOG** (`CHANGELOG.md`):
55+
- Any user-visible change → use the `changelog` skill instead
56+
57+
### Step 2: Cross-Validate Flags
58+
59+
For each affected area, verify docs match source:
60+
61+
1. **CLI flags** — extract all flags from `cmd/mdproof/main.go`:
62+
```bash
63+
grep -n 'flag\.' cmd/mdproof/main.go
64+
```
65+
66+
2. **Config fields** — extract from `internal/config/config.go`:
67+
```bash
68+
grep -n 'json:"' internal/config/config.go
69+
```
70+
71+
3. **Report types** — extract from `internal/core/types.go`:
72+
```bash
73+
grep -n 'json:"' internal/core/types.go
74+
```
75+
76+
4. Compare each against what's documented in `skills/SKILL.md` and `skills/references/`:
77+
- **New flag in code** → add to CLI Flags table + Quick Reference
78+
- **New config field** → add to Configuration File section in advanced-features.md
79+
- **Removed flag/field** → remove from docs
80+
- **Changed behavior** → update description
81+
- **Every `--flag` / `-flag` in docs** must have a matching hit in source
82+
83+
### Step 3: Update Documentation
84+
85+
Apply changes following the established structure:
86+
87+
#### skills/SKILL.md structure:
88+
89+
| Section | What to update |
90+
|---------|---------------|
91+
| Quick Reference | One-liner examples for new flags |
92+
| Container Safety | Sandbox config changes |
93+
| Writing Runbooks | New runbook syntax (separators, directives) |
94+
| Assertions | New assertion types (update count if changed) |
95+
| CLI Flags | Flag table — must match `cmd/mdproof/main.go` exactly |
96+
| Advanced Features | Pointer to references/ (add new topics if needed) |
97+
| Workflow | Rarely changes |
98+
| Self-Learning | Rarely changes |
99+
| Rules | Add rules for new features if needed |
100+
101+
#### skills/references/assertions-guide.md structure:
102+
103+
| Section | What to update |
104+
|---------|---------------|
105+
| Type sections | New assertion types get their own section |
106+
| Choosing table | Add rows for new assertion use cases |
107+
| Tips | New gotchas or best practices |
108+
109+
#### skills/references/advanced-features.md structure:
110+
111+
| Section | What to update |
112+
|---------|---------------|
113+
| Directives | New HTML comment directives |
114+
| Lifecycle Hooks | Per-runbook hooks (`--build`, `--setup`, `--teardown`) |
115+
| Per-Step Setup/Teardown | Per-step hooks (`-step-setup`, `-step-teardown`) |
116+
| Sub-Command Separator | `---` execution model, report format |
117+
| Configuration File | `mdproof.json` fields — must match Config struct |
118+
| Inline Testing | `--inline` mode |
119+
| Coverage Analysis | `--coverage` mode |
120+
| Watch Mode | `--watch` mode |
121+
| Step Filtering | `--steps`, `--from` |
122+
| Full Examples | Update or add examples for new features |
123+
124+
### Step 4: Consistency Checks
125+
126+
After updating, verify cross-references are consistent:
127+
128+
1. **Assertion count** — `skills/SKILL.md` says "Six types" → count actual types in assertions-guide.md
129+
2. **Config example** — JSON example in advanced-features.md must include all Config struct fields
130+
3. **Flag table** — CLI Flags in SKILL.md must be a complete subset of flags in main.go
131+
4. **Quick Reference** — examples must use correct flag ordering (flags before file path — Go's `flag` package requirement)
132+
5. **Report fields** — any jq assertion examples must reference fields that actually exist in the JSON report
133+
134+
### Step 5: Verify
135+
136+
```bash
137+
# Ensure code still builds (catches any accidental code edits)
138+
go build ./...
139+
140+
# Ensure tests pass
141+
go test ./...
142+
```
143+
144+
### Step 6: Report
145+
146+
List all changes made with rationale:
147+
148+
```
149+
== Documentation Updates ==
150+
151+
Modified:
152+
skills/SKILL.md
153+
- Added -step-setup/-step-teardown to CLI Flags table
154+
- Added sub-command separator section to Writing Runbooks
155+
156+
skills/references/advanced-features.md
157+
- Added Per-Step Setup/Teardown section
158+
- Updated Configuration File example with step_setup/step_teardown
159+
160+
No code changes.
161+
```
162+
163+
## Source-to-Doc Mapping Quick Reference
164+
165+
| Source file | Doc file | What to check |
166+
|------------|----------|---------------|
167+
| `cmd/mdproof/main.go` | `skills/SKILL.md` | CLI Flags table, Quick Reference |
168+
| `internal/config/config.go` | `skills/references/advanced-features.md` | Configuration File section |
169+
| `internal/core/types.go` | `skills/references/assertions-guide.md` | Report field references in jq examples |
170+
| `internal/assertion/` | `skills/references/assertions-guide.md` | Assertion types and behavior |
171+
| `internal/executor/session.go` | `skills/references/advanced-features.md` | Execution model, hooks, sub-commands |
172+
| `internal/report/plain.go` | `skills/references/advanced-features.md` | Plain text report behavior |
173+
| `internal/report/junit.go` | `skills/references/advanced-features.md` | JUnit report behavior |
174+
| `CHANGELOG.md` | (use `changelog` skill) | User-facing release notes |
175+
176+
## Rules
177+
178+
- **Source of truth is Go code** — docs must match what the code actually does
179+
- **Every flag/config claim must be verified** — grep source before writing docs
180+
- **No speculative docs** — never document planned but unimplemented features
181+
- **No code changes** — this skill only touches `skills/` and `CHANGELOG.md`
182+
- **Preserve style** — match existing doc structure and tone
183+
- **Flag ordering** — all CLI examples must put flags before file paths
184+
- **Config field names** — use the exact `json:"..."` tag from Config struct (snake_case)

‎CHANGELOG.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,13 @@
44

55
### New Features
66

7-
- **Per-step setup/teardown** — `-step-setup` and `-step-teardown` CLI flags run a command before/after each step. Setup failure marks the step as failed and skips the body; teardown failure is informational only.
7+
- **Per-step setup/teardown** — `-step-setup` and `-step-teardown` CLI flags run a command before/after each step. Setup failure marks the step as failed and skips the body; teardown failure is informational only. Also configurable in `mdproof.json` via `step_setup` / `step_teardown`.
88
```bash
9-
mdproof run test.md -step-setup "rm -rf /tmp/test-*"
10-
mdproof run test.md -step-teardown "echo cleanup"
9+
mdproof -step-setup 'rm -rf /tmp/test-*' test.md
10+
mdproof -step-teardown 'echo cleanup' test.md
11+
```
12+
```json
13+
{ "step_setup": "reset-db", "step_teardown": "dump-logs" }
1114
```
1215

1316
- **Sub-command granular report** — steps with `---` separators now execute each block independently in its own subshell. The JSON report includes a `sub_commands` array with per-sub-command `exit_code`, `stdout`, `stderr`, and `command`. Plain text and JUnit reporters surface sub-command failure details.

‎cmd/mdproof/main.go‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -197,7 +197,7 @@ func main() {
197197
strictExplicit = true
198198
}
199199
})
200-
cfg := mdproof.MergeConfig(fileCfg, cliBuild, cliSetup, cliTeardown, timeout, strict, strictExplicit)
200+
cfg := mdproof.MergeConfig(fileCfg, cliBuild, cliSetup, cliTeardown, cliStepSetup, cliStepTeardown, timeout, strict, strictExplicit)
201201

202202
// Strict mode off or watch mode → allow local execution.
203203
if !cfg.IsStrict() || watchMode {
@@ -245,8 +245,8 @@ func main() {
245245
From: fromFlag,
246246
FailFast: failFast,
247247
SnapshotUpdate: updateSnapshots,
248-
StepSetup: cliStepSetup,
249-
StepTeardown: cliStepTeardown,
248+
StepSetup: cfg.StepSetup,
249+
StepTeardown: cfg.StepTeardown,
250250
}, reportFmt, int(verbose), inlineMode)
251251
if errs > 0 {
252252
exitCode = 1

‎internal/config/config.go‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,10 @@ type SandboxConfig struct {
2121
type Config struct {
2222
Build string `json:"build,omitempty"` // command to run once before all runbooks
2323
Setup string `json:"setup,omitempty"` // command to run before each runbook
24-
Teardown string `json:"teardown,omitempty"` // command to run after each runbook
25-
Timeout string `json:"timeout,omitempty"` // per-step timeout (e.g., "5m")
24+
Teardown string `json:"teardown,omitempty"` // command to run after each runbook
25+
StepSetup string `json:"step_setup,omitempty"` // command to run before each step
26+
StepTeardown string `json:"step_teardown,omitempty"` // command to run after each step
27+
Timeout string `json:"timeout,omitempty"` // per-step timeout (e.g., "5m")
2628
Env map[string]string `json:"env,omitempty"` // environment variables seeded into all steps
2729
Strict *bool `json:"strict,omitempty"` // container-only execution (default: true)
2830
Sandbox *SandboxConfig `json:"sandbox,omitempty"` // sandbox subcommand settings
@@ -63,7 +65,7 @@ func Load(dir string) (Config, error) {
6365
// Merge applies CLI flag overrides on top of file-based config.
6466
// CLI flags take precedence when non-empty. strictExplicit indicates
6567
// whether --strict was explicitly passed on the command line.
66-
func Merge(file Config, cliBuild, cliSetup, cliTeardown string, cliTimeout time.Duration, cliStrict bool, strictExplicit bool) Config {
68+
func Merge(file Config, cliBuild, cliSetup, cliTeardown, cliStepSetup, cliStepTeardown string, cliTimeout time.Duration, cliStrict bool, strictExplicit bool) Config {
6769
merged := file
6870
if cliBuild != "" {
6971
merged.Build = cliBuild
@@ -74,6 +76,12 @@ func Merge(file Config, cliBuild, cliSetup, cliTeardown string, cliTimeout time.
7476
if cliTeardown != "" {
7577
merged.Teardown = cliTeardown
7678
}
79+
if cliStepSetup != "" {
80+
merged.StepSetup = cliStepSetup
81+
}
82+
if cliStepTeardown != "" {
83+
merged.StepTeardown = cliStepTeardown
84+
}
7785
if cliTimeout != 0 {
7886
merged.Timeout = cliTimeout.String()
7987
}

‎internal/config/config_test.go‎

Lines changed: 42 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ func TestLoad_WithBuild(t *testing.T) {
2424

2525
func TestMerge_CLIBuildOverrides(t *testing.T) {
2626
file := Config{Build: "file-build"}
27-
merged := Merge(file, "cli-build", "", "", 0, true, false)
27+
merged := Merge(file, "cli-build", "", "", "", "", 0, true, false)
2828
if merged.Build != "cli-build" {
2929
t.Errorf("build = %q, want %q", merged.Build, "cli-build")
3030
}
@@ -81,7 +81,7 @@ func TestMerge_CLIOverrides(t *testing.T) {
8181
Timeout: "1m",
8282
}
8383

84-
merged := Merge(file, "", "cli-setup", "", 0, true, false)
84+
merged := Merge(file, "", "cli-setup", "", "", "", 0, true, false)
8585
if merged.Setup != "cli-setup" {
8686
t.Errorf("setup = %q, want %q", merged.Setup, "cli-setup")
8787
}
@@ -92,7 +92,7 @@ func TestMerge_CLIOverrides(t *testing.T) {
9292

9393
func TestMerge_CLITimeoutOverrides(t *testing.T) {
9494
file := Config{Timeout: "1m"}
95-
merged := Merge(file, "", "", "", 5*time.Minute, true, false)
95+
merged := Merge(file, "", "", "", "", "", 5*time.Minute, true, false)
9696
if merged.Timeout != "5m0s" {
9797
t.Errorf("timeout = %q, want %q", merged.Timeout, "5m0s")
9898
}
@@ -116,7 +116,7 @@ func TestIsStrict_ConfigFalse(t *testing.T) {
116116
func TestMerge_CLIStrictOverridesConfig(t *testing.T) {
117117
f := false
118118
file := Config{Strict: &f}
119-
merged := Merge(file, "", "", "", 0, true, true) // CLI explicit --strict=true
119+
merged := Merge(file, "", "", "", "", "", 0, true, true) // CLI explicit --strict=true
120120
if !merged.IsStrict() {
121121
t.Error("CLI --strict=true should override config strict=false")
122122
}
@@ -125,7 +125,7 @@ func TestMerge_CLIStrictOverridesConfig(t *testing.T) {
125125
func TestMerge_ConfigStrictNotOverriddenByDefault(t *testing.T) {
126126
f := false
127127
file := Config{Strict: &f}
128-
merged := Merge(file, "", "", "", 0, true, false) // CLI not explicit
128+
merged := Merge(file, "", "", "", "", "", 0, true, false) // CLI not explicit
129129
if merged.IsStrict() {
130130
t.Error("config strict=false should be preserved when CLI --strict is not explicit")
131131
}
@@ -187,6 +187,43 @@ func TestLoadSandboxConfigDefaults(t *testing.T) {
187187
}
188188
}
189189

190+
func TestLoad_StepSetup(t *testing.T) {
191+
dir := t.TempDir()
192+
data := `{"step_setup": "reset-db", "step_teardown": "dump-logs"}`
193+
if err := os.WriteFile(filepath.Join(dir, "mdproof.json"), []byte(data), 0644); err != nil {
194+
t.Fatal(err)
195+
}
196+
cfg, err := Load(dir)
197+
if err != nil {
198+
t.Fatal(err)
199+
}
200+
if cfg.StepSetup != "reset-db" {
201+
t.Errorf("step_setup = %q, want %q", cfg.StepSetup, "reset-db")
202+
}
203+
if cfg.StepTeardown != "dump-logs" {
204+
t.Errorf("step_teardown = %q, want %q", cfg.StepTeardown, "dump-logs")
205+
}
206+
}
207+
208+
func TestMerge_CLIStepSetupOverrides(t *testing.T) {
209+
file := Config{StepSetup: "file-setup", StepTeardown: "file-teardown"}
210+
merged := Merge(file, "", "", "", "cli-setup", "", 0, true, false)
211+
if merged.StepSetup != "cli-setup" {
212+
t.Errorf("step_setup = %q, want %q", merged.StepSetup, "cli-setup")
213+
}
214+
if merged.StepTeardown != "file-teardown" {
215+
t.Errorf("step_teardown should keep file value, got %q", merged.StepTeardown)
216+
}
217+
}
218+
219+
func TestMerge_ConfigStepSetupPreserved(t *testing.T) {
220+
file := Config{StepSetup: "file-setup"}
221+
merged := Merge(file, "", "", "", "", "", 0, true, false)
222+
if merged.StepSetup != "file-setup" {
223+
t.Errorf("step_setup = %q, want %q", merged.StepSetup, "file-setup")
224+
}
225+
}
226+
190227
func TestTimeoutDuration(t *testing.T) {
191228
tests := []struct {
192229
input string

‎mdproof.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -124,8 +124,8 @@ func LoadConfig(dir string) (Config, error) {
124124

125125
// MergeConfig applies CLI flag overrides on top of file-based config.
126126
// strictExplicit indicates whether --strict was explicitly passed on the CLI.
127-
func MergeConfig(file Config, cliBuild, cliSetup, cliTeardown string, cliTimeout time.Duration, cliStrict bool, strictExplicit bool) Config {
128-
return config.Merge(file, cliBuild, cliSetup, cliTeardown, cliTimeout, cliStrict, strictExplicit)
127+
func MergeConfig(file Config, cliBuild, cliSetup, cliTeardown, cliStepSetup, cliStepTeardown string, cliTimeout time.Duration, cliStrict bool, strictExplicit bool) Config {
128+
return config.Merge(file, cliBuild, cliSetup, cliTeardown, cliStepSetup, cliStepTeardown, cliTimeout, cliStrict, strictExplicit)
129129
}
130130

131131
// --- Report ---

0 commit comments

Comments
 (0)