fix(claude-code-hermit): narrow the unattended metrics writer to typed ledger verbs - #666
Merged
Conversation
…d ledger verbs append-metrics.ts took an arbitrary path and arbitrary JSON, so the unattended grant it held let any caller write anything anywhere. Replaced by two writers scoped to the ledger each owns: observations.ts observe and proposal.ts event. Moving cost-spike, behavior-digest and startup-drift into the scripts that compute them also fixes a feature that had never worked. Across the live fleet, observations.jsonl held only script-written startup-drift rows — not one prose-authored cost-spike or behavior-digest row had ever been recorded, because each asked the model to recompute and reformat a number a script had already derived and discarded. With those sources script-owned, every remaining caller passes free text, so the writer is stdin-only and the argv quoting hazard (and the lint that policed it) disappears rather than moving. The cost-spike label is date-scoped with the figures as fields. Carrying the running total in the label, as the prose row did, would have defeated the ledger's exact-pattern dedup and written a fresh row on every precheck run of a spike day.
… identity-safe Three gaps the review pass surfaced in the typed-writer change: The digest's --record-observation append ran unguarded and ahead of the stdout write, so an ENOENT from hermitDir()'s fail-open state dir would kill the process before any digest JSON emerged — reflect would lose the whole anomaly checklist over one telemetry row. Wrapped fail-open, matching every other writer this change introduced. observationRow assigned `extra` last, so an extra key named `source` or `session_id` silently rewrote the very fields the module exists to own. Colliding keys are now rejected outright. The freshness-gate comment still claimed cost-spike rows do not self-trigger; they are written by the block directly above it now. Both SKILL.md passages distinguished a rejected row (exit 0) from a mis-invocation (exit 1 by design), and permissions-sync's scrub list gained the entry this change retires.
Both sides added a verb to proposal.ts's dispatcher in the same three spots: main added `quality-gate`, this branch added `event`. They are disjoint, so the resolution keeps both — `event` stays next to `metrics` (writer beside reader, as its comment explains) and `quality-gate` follows it. proposal-act/SKILL.md auto-merged correctly: main rewrote the quality-gate sections, this branch rewrote the proposal-metrics call sites, and the two regions do not touch. bun test 3332 pass / 0 fail, bunx tsc --noEmit clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
append-metrics.tstook an arbitrary path and an arbitrary JSON payload, validating only thatJSON.parsesucceeded. It sat in the sealed unattended permission allow-list, so the grant it held amounted to write any JSON to any path. This replaces it with two writers scoped to the ledger each one owns, and moves three observation sources into the scripts that actually compute them.The security narrowing was the motivating change, but the larger find came out of verification: the prose-mediated observation path had never worked. Across the live fleet,
observations.jsonlcontained only script-writtenstartup-driftrows — not a singlecost-spikeorbehavior-digestrow had ever been recorded. Both asked the model to re-read a log, redo arithmetic a script had already done, and hand-format the result. Those are now written by the code that derives them.Changes
New writers
observations.ts observe <dir> <source>— stdin-only, typed. Sources are an enum,tsis stamped by the script,session_idresolves fromruntime.json. Only the three model-authored sources are invocable;cost-spike,behavior-digestandstartup-driftare rejected because the scripts that compute them own them.proposal.ts event <dir> responded|resolved --id=— the same treatment for the two proposal-metrics rows skill prose used to assemble as JSON. Namedevent, notmetric, so it can't be confused with the existingmetricsreader one character away in a dispatcher whose unknown-verb branch exits 0.Deterministic sources move to their owners
reflect-precheck.tswritescost-spikeandstartup-drift;checkCostSpikenow returns its figures instead of discarding them.transcript-digest.ts --record-observationwritesbehavior-digest, with thewakes >= 20 && productive/wakes < 0.25threshold moved out of prose. The flag is opt-in so the digest stays a pure read for ad-hoc runs.Consequences
cost-spikelabel is date-scoped (cost-spike:<YYYY-MM-DD>) withtoday_total/median_7das fields. Carrying the running total in the label, as the prose row did, would have defeated the ledger's exact-pattern dedup and written a fresh row on every precheck run of a spike day.metrics_eventis gone from the reflect eval-runner schema — a model-authored JSON string appended after only aJSON.parsecheck, and always derivable fromproposal_id. A stale runner sending one is now rejected rather than silently trusted.HERMIT_ALLOWswaps toBash(bun */scripts/observations.ts observe *); the old entry is retired viaHERMIT_OBSOLETEsopermissions-syncscrubs it from installed hermits.Test plan
bun testfromplugins/claude-code-hermit— 3230 pass, 0 fail (exit 0).bunx tsc --noEmitfrom the repo root — exit 0.bash tests/run-all.shinplugins/claude-code-dev-hermit— all suites 0 failed.$/quote survival, deterministic sources refused from the CLI, per-sourceoriginrules, empty/multiline/oversized labels writing nothing,ERROR|<token>at exit 0 vs exit 1 on mis-invocation, all fourproposal.ts eventerror paths, and a dedup regression test asserting two precheck runs on the same spike day with different running totals produce exactly one row.bob's spend spiked: $5.00round-trips verbatim.append-metricsinvocations remain; the only hits are theHERMIT_OBSOLETEentry, its assertion, and comments explaining the replacement.Notes for review
responded/resolvedrows still parse, and reader outcomes are asserted by field rather than by byte comparison./^PROP-\d+$/would reject 81 of the 106proposal_idvalues in the live ledger, including everycreatedrowproposal.tswrites itself.defer→accept(anddismiss→accept) never emit an accept event becauserespondedis first-response-only, so those proposals are permanently counted as unaccepted — an existing bug that feeds a 30% kill gate.