Skip to content

fix(claude-code-hermit): narrow the unattended metrics writer to typed ledger verbs - #666

Merged
gtapps merged 3 commits into
mainfrom
fix/narrow-metrics-writer-grant
Jul 25, 2026
Merged

fix(claude-code-hermit): narrow the unattended metrics writer to typed ledger verbs#666
gtapps merged 3 commits into
mainfrom
fix/narrow-metrics-writer-grant

Conversation

@gtapps

@gtapps gtapps commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Summary

append-metrics.ts took an arbitrary path and an arbitrary JSON payload, validating only that JSON.parse succeeded. It sat in the sealed unattended permission allow-list, so the grant it held amounted to write any JSON to any path. This replaces it with two writers scoped to the ledger each one owns, and moves three observation sources into the scripts that actually compute them.

The security narrowing was the motivating change, but the larger find came out of verification: the prose-mediated observation path had never worked. Across the live fleet, observations.jsonl contained only script-written startup-drift rows — not a single cost-spike or behavior-digest row had ever been recorded. Both asked the model to re-read a log, redo arithmetic a script had already done, and hand-format the result. Those are now written by the code that derives them.

Changes

New writers

  • observations.ts observe <dir> <source> — stdin-only, typed. Sources are an enum, ts is stamped by the script, session_id resolves from runtime.json. Only the three model-authored sources are invocable; cost-spike, behavior-digest and startup-drift are rejected because the scripts that compute them own them.
  • proposal.ts event <dir> responded|resolved --id= — the same treatment for the two proposal-metrics rows skill prose used to assemble as JSON. Named event, not metric, so it can't be confused with the existing metrics reader one character away in a dispatcher whose unknown-verb branch exits 0.

Deterministic sources move to their owners

  • reflect-precheck.ts writes cost-spike and startup-drift; checkCostSpike now returns its figures instead of discarding them.
  • transcript-digest.ts --record-observation writes behavior-digest, with the wakes >= 20 && productive/wakes < 0.25 threshold moved out of prose. The flag is opt-in so the digest stays a pure read for ad-hoc runs.

Consequences

  • Every remaining prose caller now passes free text, so the writer is stdin-only — the argv quoting hazard and the lint that policed it are deleted, not retargeted.
  • The cost-spike label is date-scoped (cost-spike:<YYYY-MM-DD>) with today_total/median_7d as fields. Carrying the running total in the label, as the prose row did, would have defeated the ledger's exact-pattern dedup and written a fresh row on every precheck run of a spike day.
  • metrics_event is gone from the reflect eval-runner schema — a model-authored JSON string appended after only a JSON.parse check, and always derivable from proposal_id. A stale runner sending one is now rejected rather than silently trusted.
  • HERMIT_ALLOW swaps to Bash(bun */scripts/observations.ts observe *); the old entry is retired via HERMIT_OBSOLETE so permissions-sync scrubs it from installed hermits.

Test plan

  • bun test from plugins/claude-code-hermit3230 pass, 0 fail (exit 0).
  • bunx tsc --noEmit from the repo root — exit 0.
  • bash tests/run-all.sh in plugins/claude-code-dev-hermit — all suites 0 failed.
  • New coverage: stdin apostrophe/$/quote survival, deterministic sources refused from the CLI, per-source origin rules, empty/multiline/oversized labels writing nothing, ERROR|<token> at exit 0 vs exit 1 on mis-invocation, all four proposal.ts event error paths, and a dedup regression test asserting two precheck runs on the same spike day with different running totals produce exactly one row.
  • Live smoke in a scratch hermit dir: both writers produce well-formed rows; a label containing bob's spend spiked: $5.00 round-trips verbatim.
  • Grep sweep: zero live append-metrics invocations remain; the only hits are the HERMIT_OBSOLETE entry, its assertion, and comments explaining the replacement.

Notes for review

  • Readers were left alone deliberately — historical responded/resolved rows still parse, and reader outcomes are asserted by field rather than by byte comparison.
  • One reviewer finding was refuted with evidence and not applied: tightening the id regex to /^PROP-\d+$/ would reject 81 of the 106 proposal_id values in the live ledger, including every created row proposal.ts writes itself.
  • Out of scope, flagged for separate work: defer→accept (and dismiss→accept) never emit an accept event because responded is first-response-only, so those proposals are permanently counted as unaccepted — an existing bug that feeds a 30% kill gate.

gtapps added 3 commits July 25, 2026 23:17
…d ledger verbs

append-metrics.ts took an arbitrary path and arbitrary JSON, so the unattended
grant it held let any caller write anything anywhere. Replaced by two writers
scoped to the ledger each owns: observations.ts observe and proposal.ts event.

Moving cost-spike, behavior-digest and startup-drift into the scripts that
compute them also fixes a feature that had never worked. Across the live fleet,
observations.jsonl held only script-written startup-drift rows — not one
prose-authored cost-spike or behavior-digest row had ever been recorded, because
each asked the model to recompute and reformat a number a script had already
derived and discarded. With those sources script-owned, every remaining caller
passes free text, so the writer is stdin-only and the argv quoting hazard (and
the lint that policed it) disappears rather than moving.

The cost-spike label is date-scoped with the figures as fields. Carrying the
running total in the label, as the prose row did, would have defeated the
ledger's exact-pattern dedup and written a fresh row on every precheck run of a
spike day.
… identity-safe

Three gaps the review pass surfaced in the typed-writer change:

The digest's --record-observation append ran unguarded and ahead of the
stdout write, so an ENOENT from hermitDir()'s fail-open state dir would kill
the process before any digest JSON emerged — reflect would lose the whole
anomaly checklist over one telemetry row. Wrapped fail-open, matching every
other writer this change introduced.

observationRow assigned `extra` last, so an extra key named `source` or
`session_id` silently rewrote the very fields the module exists to own.
Colliding keys are now rejected outright.

The freshness-gate comment still claimed cost-spike rows do not self-trigger;
they are written by the block directly above it now. Both SKILL.md passages
distinguished a rejected row (exit 0) from a mis-invocation (exit 1 by
design), and permissions-sync's scrub list gained the entry this change
retires.
Both sides added a verb to proposal.ts's dispatcher in the same three spots:
main added `quality-gate`, this branch added `event`. They are disjoint, so
the resolution keeps both — `event` stays next to `metrics` (writer beside
reader, as its comment explains) and `quality-gate` follows it.

proposal-act/SKILL.md auto-merged correctly: main rewrote the quality-gate
sections, this branch rewrote the proposal-metrics call sites, and the two
regions do not touch.

bun test 3332 pass / 0 fail, bunx tsc --noEmit clean.
@gtapps
gtapps merged commit 833835f into main Jul 25, 2026
6 checks passed
@gtapps
gtapps deleted the fix/narrow-metrics-writer-grant branch July 25, 2026 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant