Add govkit-metrics-emit skill (Tier 1 metric event producer)

[marty916]

Summary

Adds a second skill to the govkit plugin: govkit-metrics-emit, an org-agnostic producer that reads GovKit exhaust (feature packages, .govkit/marker.json, git history, and optional CI-run / PR JSON exports) and emits NDJSON metric events per the AIPOS tier1_metrics_catalog v1 vocabulary. Aggregation stays out of scope by design — producers stay org-agnostic.

Contents:

• SKILL.md — usage, the org-agnostic law, event vocabulary, interpretation guidance
• scripts/emit_metrics.py — the bundled producer (5 event types, --validate mode)
• references/event_schema.md — event field definitions + spec-completeness rubric
• evals/evals.json — gate-readiness, aggregator-export, and org-agnostic checks

Also:

• Bumps govkit plugin version 0.1.0 → 0.2.0 so installed users receive the new skill (per CONTRIBUTING).
• Adds Python build artifacts (__pycache__/, *.py[cod], .pytest_cache/) to .gitignore, since the plugin now ships Python.

Validation

• claude plugin validate . passes.
• The bundled producer was run end-to-end against a constructed GovKit fixture and reproduces the evals' documented outputs: checkout_assistant completeness = 100, incomplete_feature = 15 (with named missing components), PR AI-assisted detection true/false via Co-Authored-By trailers, a 10-event stream (2 snapshot / 2 rework / 4 gate / 2 PR), non-gate CI runs filtered out, and --validate returns valid: true.

Notes

• The /tmp/testrepo path in evals/evals.json is intentional — an external eval harness populates that fixture at run time; it is not a bundled fixture.

🤖 Generated with Claude Code

[qodo-code-review]

PR Summary by Qodo

Add govkit-metrics-emit skill to emit Tier 1 metric NDJSON events

✨ Enhancement 📝 Documentation 🧪 Tests ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Add govkit-metrics-emit skill emitting org-agnostic Tier 1 metric events as NDJSON.
• Include Python producer to derive snapshot, rework, gate-run, and PR-merge events.
• Document event vocabulary and add eval harness prompts for gate-readiness and exports.

Diagram

graph TD
  repo[("GovKit repo")]
  ci[("CI runs JSON")]
  prs[("PRs JSON")]
  script["emit_metrics.py"]
  out[("NDJSON events")]
  validate{{"--validate?"}}
  summary[("Validation summary")]

  repo --> script --> out --> validate
  validate -- "yes" --> summary
  ci --> script
  prs --> script

  subgraph Legend
    direction LR
    _data[("Data input/output")] ~~~ _proc["Process"] ~~~ _opt{{"Option"}}
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Drive validation from JSON Schema + jsonschema

• ➕ Clear, machine-readable contract for events and nested fields
• ➕ Better error messages and future-proofing as the vocabulary evolves
• ➖ Adds a dependency and schema files to ship/maintain
• ➖ More overhead than a simple allowed-field check for a Tier 1 producer

2. Use GitPython (or libgit2 bindings) for history scanning

• ➕ Avoids brittle subprocess parsing and edge cases in log/numstat formats
• ➕ Easier to unit-test with mocked repositories
• ➖ Heavier dependency footprint for a plugin-bundled script
• ➖ May complicate installation in minimal environments compared to plain git CLI

Recommendation: The current approach (lightweight git CLI + minimal vocabulary validation) is a good fit for an org-agnostic Tier 1 producer with low dependency goals. If validation needs to expand beyond unknown-field checks (types, required nested fields), consider migrating validation to JSON Schema; if rework attribution becomes a major focus, consider a library-based git parser to reduce log-format edge cases.

Files changed (5) +622 / -1

Enhancement (1) +418 / -0

emit_metrics.pyAdd Python NDJSON producer for Tier 1 metric events +418/-0

Add Python NDJSON producer for Tier 1 metric events

• Implements a CLI that reads GovKit feature packages, marker level, git history, and optional CI/PR exports to emit NDJSON events. Produces feature.package.snapshot, rework.observed, gate.run.completed, and pr.merged events, and supports a --validate mode that checks events against an allowed-field vocabulary.

plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py

Tests (1) +26 / -0

evals.jsonAdd eval prompts for gate readiness, export, and org-agnostic checks +26/-0

Add eval prompts for gate readiness, export, and org-agnostic checks

• Introduces three eval scenarios covering gate-readiness audit, aggregator export with validation, and org-agnostic verification including AI-assisted PR detection. References an external fixture path populated by the eval harness.

plugins/govkit/skills/govkit-metrics-emit/evals/evals.json

Documentation (2) +177 / -0

SKILL.mdAdd SKILL documentation for govkit-metrics-emit +92/-0

Add SKILL documentation for govkit-metrics-emit

• Defines the new skill, its org-agnostic constraints, and when to trigger it. Documents how to run the bundled producer script, the five-event vocabulary, and interpretation guidance for users.

plugins/govkit/skills/govkit-metrics-emit/SKILL.md

event_schema.mdDocument tier1_metrics_catalog v1 event schema and rubric +85/-0

Document tier1_metrics_catalog v1 event schema and rubric

• Adds a reference describing event envelopes, per-event fields, and the spec-completeness rubric used in snapshot events. Includes the metric-pair mapping and notes about approximations (e.g., file-overlap rework).

plugins/govkit/skills/govkit-metrics-emit/references/event_schema.md

Other (1) +1 / -1

plugin.jsonBump govkit plugin version to 0.2.0 +1/-1

Bump govkit plugin version to 0.2.0

• Updates the plugin version from 0.1.0 to 0.2.0 so the new skill is delivered to installed users.

plugins/govkit/.claude-plugin/plugin.json

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Optional JSON crashes ✓ Resolved 🐞 Bug ☼ Reliability

Description

gate_events() and pr_events() parse user-provided JSON exports without handling file read or JSON
decode errors, so a missing/malformed export crashes the entire run. This is especially risky
because these inputs are documented as optional.

Code

plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[R237-261]

+def gate_events(ci_runs_path: Path):
+    runs = json.loads(ci_runs_path.read_text(encoding="utf-8"))
+    for r in runs:
+        raw = (r.get("name") or r.get("workflowName") or "").strip()
+        base = raw.lower().removesuffix(".yml").removesuffix(".yaml")
+        if base not in KNOWN_GATES:
+            continue
+        blob = " ".join(str(r.get(k, "")) for k in
+                        ("displayTitle", "headBranch", "title"))
+        m = re.search(r"features?/([A-Za-z0-9_\-]+)", blob)
+        yield envelope("gate.run.completed", {
+            "feature_id": m.group(1) if m else None,
+            "gate_name": base,
+            "run_attempt": r.get("attempt") or r.get("runAttempt") or 1,
+            "conclusion": r.get("conclusion") or r.get("result"),
+            "ts_start": r.get("createdAt") or r.get("startTime"),
+            "ts_end": r.get("updatedAt") or r.get("finishTime"),
+            "commit_sha": r.get("headSha") or r.get("sourceVersion"),
+        })
+
+
+# ------------------------------------------------------------------ PRs
+def pr_events(prs_path: Path):
+    prs = json.loads(prs_path.read_text(encoding="utf-8"))
+    for pr in prs:

Evidence

Both functions directly json.loads() file content with no exception handling; any read/decode
error will raise and terminate the process.

plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[237-261]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The script unconditionally does `json.loads(path.read_text(...))` for `--ci-runs` and `--prs`. If the file is missing, unreadable, or invalid JSON, the script raises and aborts.

## Issue Context
These inputs are optional and user-supplied; failures should be turned into a clear error message (and a deterministic exit code) rather than a stack trace.

## Fix Focus Areas
- plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[237-261]
- plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[382-415]

## Suggested fix
- Wrap both reads/loads in try/except catching `OSError` and `json.JSONDecodeError`.
- On failure, either:
 - print a concise message to stderr and `sys.exit(2)`, OR
 - skip that event stream with a warning (but be explicit in stderr output that optional inputs were skipped).
- Consider validating that the parsed JSON is a list before iterating.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Null commit/ts output ✓ Resolved 🐞 Bug ≡ Correctness

Description

snapshot_events() emits commit_sha and ts as null when git log returns empty, but the documented
vocabulary defines both as strings. This can break downstream ingestion/lead-time computations while
still passing the script’s current validation.

Code

plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[R195-198]

+        rel = f"features/{fdir.name}"
+        sha = git(repo, "log", "-1", "--format=%H", "--", rel) or None
+        ts = git(repo, "log", "-1", "--format=%cI", "--", rel) or None
+        artifacts = ["acceptance.feature", "nfrs.md", "plan.md",

Evidence

The producer explicitly assigns None when git returns empty output, while the vocabulary document
requires commit_sha and ts to be strings.

plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[189-198]
plugins/govkit/skills/govkit-metrics-emit/references/event_schema.md[12-14]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`feature.package.snapshot` events may contain `commit_sha: null` and `ts: null`, violating the documented event vocabulary (strings) and creating ambiguous timestamps for downstream metric calculations.

## Issue Context
This happens whenever `git log -1 --format=... -- features/<id>` returns no output.

## Fix Focus Areas
- plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[189-217]
- plugins/govkit/skills/govkit-metrics-emit/references/event_schema.md[10-14]

## Suggested fix
Choose one consistent behavior and enforce it:
- If git metadata is missing, either:
 - (Preferred) fall back to `git rev-parse HEAD` for `commit_sha` and `datetime.now(timezone.utc).isoformat()` for `ts`, OR
 - omit the event / record a validation error and exit non-zero.
- Extend `validate()` to flag null/empty `commit_sha` and `ts` for snapshot events so `--validate` fails when these are missing.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Prediction type crash ✓ Resolved 🐞 Bug ☼ Reliability

Description

extract_evaluation_prediction() can return a non-dict YAML value for evaluation_prediction, but
completeness() and snapshot_events() call pred.get(...), which will raise AttributeError and abort
emission. This makes a malformed plan.md able to crash the producer instead of producing a
null/invalid prediction object.

Code

plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[R156-168]

+    add("evaluation_prediction_block", 15, pred is not None,
+        "evaluation_prediction parsed from plan.md" if pred is not None
+        else "plan.md missing or evaluation_prediction block absent/invalid")
+
+    thr = bool(pred and pred.get("thresholds_met") is True)
+    add("thresholds_met", 15, thr,
+        f"thresholds_met={pred.get('thresholds_met') if pred else None}")
+
+    fa = pred.get("first", {}).get("average") if pred else None
+    va = pred.get("virtues", {}).get("average") if pred else None
+    ok_scores = isinstance(fa, (int, float)) and isinstance(va, (int, float)) \
+        and fa >= 4.0 and va >= 4.0
+    add("score_minima", 15, ok_scores, f"first.average={fa}, virtues.average={va}")

Evidence

The extractor returns data["evaluation_prediction"] without ensuring it’s a dict, while later code
unconditionally calls pred.get(...) in multiple places.

plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[113-128]
plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[141-168]
plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[219-223]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`extract_evaluation_prediction()` may return a non-dict value (string/list/etc.) from YAML, but later code assumes `pred` is a mapping and calls `.get()`, causing a runtime crash.

## Issue Context
The crash can occur in both `completeness()` and snapshot payload construction whenever the YAML block contains `evaluation_prediction: <non-object>`.

## Fix Focus Areas
- plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[113-168]
- plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[189-233]

## Suggested fix
- In `extract_evaluation_prediction()`, only return the value if it is a `dict` (otherwise return `None`).
- Additionally (defensive), normalize `pred = pred if isinstance(pred, dict) else None` at the start of `completeness()` and before building `evaluation_prediction` in `snapshot_events()`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Same-author rework misflag 🐞 Bug ≡ Correctness

Description

rework_events() sets rework_author_same=true whenever any later overlapping commit is by the same
author, even if that later commit contributes zero deletions (i.e., no rework under this
approximation). This inflates the self-rework signal.

Code

plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[R337-342]

+            overlap = set(c["files"]) & set(later["files"])
+            for fname in overlap:
+                reworked += min(c["files"][fname]["added"],
+                                later["files"][fname]["deleted"])
+                if later["author"] == c["author"]:
+                    same_author = True

Evidence

The flag is set on author match regardless of the deletion overlap amount, so commits with 0
deletions can still set rework_author_same=true.

plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[337-343]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`rework_author_same` is marked true even when there is no deletion overlap (no rework counted), because the flag is set based solely on author match, not on whether rework was attributed.

## Issue Context
Under the file-overlap approximation, “rework” is represented by later deletions in files the origin commit added to.

## Fix Focus Areas
- plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[328-356]

## Suggested fix
- Only set `same_author = True` when the later commit both:
 - matches the author, and
 - contributes a positive deletion overlap (e.g., `later_deleted > 0` and/or `min(...) > 0`).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Validation scope mismatch 🐞 Bug ⚙ Maintainability

Description

SKILL.md claims --validate reports pairing coverage, but validate() only checks unknown top-level
fields and missing envelope keys. This can provide false confidence because required payload fields,
types, and null-vs-string mismatches are not checked.

Code

plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[R361-379]

+def validate(events: list) -> dict:
+    summary = {"total": len(events), "by_event": {}, "errors": []}
+    for i, e in enumerate(events):
+        name = e.get("event")
+        summary["by_event"][name] = summary["by_event"].get(name, 0) + 1
+        if name not in EVENT_FIELDS:
+            summary["errors"].append(f"event {i}: unknown event '{name}'")
+            continue
+        allowed = EVENT_FIELDS[name] | ENVELOPE_FIELDS
+        unknown = set(e) - allowed
+        if unknown:
+            summary["errors"].append(
+                f"event {i} ({name}): unknown fields {sorted(unknown)}")
+        missing_env = ENVELOPE_FIELDS - set(e)
+        if missing_env:
+            summary["errors"].append(
+                f"event {i} ({name}): missing envelope {sorted(missing_env)}")
+    summary["valid"] = not summary["errors"]
+    return summary

Evidence

The skill documentation explicitly mentions pairing coverage, but the validator only computes counts
and unknown/missing envelope fields.

plugins/govkit/skills/govkit-metrics-emit/SKILL.md[44-49]
plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[361-379]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The documented `--validate` behavior promises more than the current implementation provides. As implemented, validation does not check required per-event fields, field types, nullability, or any "pairing coverage" concept.

## Issue Context
Users may rely on `--validate` as a quality gate before exporting events to an aggregator.

## Fix Focus Areas
- plugins/govkit/skills/govkit-metrics-emit/SKILL.md[44-49]
- plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[361-415]

## Suggested fix
Pick one:
1) Implement what the docs promise:
  - Add required-field checks per event (presence + non-null where required).
  - Add basic type checks (e.g., strings for timestamps/SHAs).
  - Implement and report "pairing coverage" (or remove that term if not applicable).
2) Narrow the documentation to exactly what is validated today (unknown fields + envelope only).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

4. Same-author rework misflag 🐞 Bug ≡ Correctness

rework_events() sets rework_author_same=true whenever any later overlapping commit is by the same
author, even if that later commit contributes zero deletions (i.e., no rework under this
approximation). This inflates the self-rework signal.

Agent Prompt

## Issue description
`rework_author_same` is marked true even when there is no deletion overlap (no rework counted), because the flag is set based solely on author match, not on whether rework was attributed.

## Issue Context
Under the file-overlap approximation, “rework” is represented by later deletions in files the origin commit added to.

## Fix Focus Areas
- plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[328-356]

## Suggested fix
- Only set `same_author = True` when the later commit both:
  - matches the author, and
  - contributes a positive deletion overlap (e.g., `later_deleted > 0` and/or `min(...) > 0`).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-code-review]

5. Validation scope mismatch 🐞 Bug ⚙ Maintainability

SKILL.md claims --validate reports pairing coverage, but validate() only checks unknown top-level
fields and missing envelope keys. This can provide false confidence because required payload fields,
types, and null-vs-string mismatches are not checked.

Agent Prompt

## Issue description
The documented `--validate` behavior promises more than the current implementation provides. As implemented, validation does not check required per-event fields, field types, nullability, or any "pairing coverage" concept.

## Issue Context
Users may rely on `--validate` as a quality gate before exporting events to an aggregator.

## Fix Focus Areas
- plugins/govkit/skills/govkit-metrics-emit/SKILL.md[44-49]
- plugins/govkit/skills/govkit-metrics-emit/scripts/emit_metrics.py[361-415]

## Suggested fix
Pick one:
1) Implement what the docs promise:
   - Add required-field checks per event (presence + non-null where required).
   - Add basic type checks (e.g., strings for timestamps/SHAs).
   - Implement and report "pairing coverage" (or remove that term if not applicable).
2) Narrow the documentation to exactly what is validated today (unknown fields + envelope only).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools