emulo verify: trace every profile quote back to a real session - #35
Merged
Conversation
resolve_emulo_home chose ~/.emulo whenever that directory merely existed. Emulo Pro and the release tooling both create ~/.emulo for unrelated reasons (pro/, autopilot/, release-keys/, release-builds/), so on any machine that upgraded from Ditto the miner stopped seeing the profile store and segment cache still living in ~/.ditto. Measured on a real upgraded machine before the fix: cached segments 0 (272 reports orphaned in ~/.ditto) worker calls 112 profiles 0 (59 files orphaned) After: cached segments 104 worker calls 9 A 12x cost reduction on every re-mine, and re-mining was previously running against an empty profile store. The intent in the original comment was right; the test was wrong. Decide on the presence of mined data (profiles/, cache/reports, cache/segments), not on whether someone happened to mkdir the folder. tests/test_home_resolution.py covers all five paths: fresh machine, upgraded user, the regression itself, a genuinely migrated home, and both env-var overrides. NOTE: test_release_runtime_hashes_match_canonical_repository_bytes now fails, because .agents/skills/emulo/runtime.json pins emulo.py for the v0.6.1 release and this changes those bytes. That pin controls what installs on other people's machines, so it is deliberately NOT updated here. Shipping this needs a version bump and a real release, not an edited pin. The new digest, if that release is cut, is: emulo.py e696bec245be88317a8136abb9bc03a7b36611bdbaaeffe0e226f3bec2c3dddb Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A mined profile earns trust from its receipts, and the worst failure it can have is not a missing rule but a confident rule quoting something the person never said. Nothing checked that. `emulo verify you.md` extracts every quoted span from a profile and searches the mined corpus for it. Unfound quotes are reported and the command exits non-zero. Quotes resting on a single session are flagged separately, since one session is context rather than a rule. `--json` emits the supporting session ids per quote. Straight quotes are paired in document order rather than by regex alternation. The first implementation used alternation and invented "quotes" out of the prose sitting between one span's closing mark and the next one's opening mark; against a real profile it reported nonsense like `". **No trust-badge filler** - he HATES"` as an unfound receipt. There is a regression test for exactly that. Verified against a real 2,177-session corpus and a real profile, where it immediately earned its keep: it caught quotes that had been silently tidied up. The source said "its feel static beucase you just dont move the character" and the profile rendered it in corrected English, which makes the receipt untraceable. That is the class of error this exists to find, and no reviewer reading fifteen chunks by hand reliably catches it. Scope is deliberately narrow. This verifies what is mechanically checkable. Whether a rule is vague, generic, or true of every developer alive stays a judgment call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_release_runtime_hashes_match_canonical_repository_bytes hashed the working tree and compared it to the digest in runtime.json. That equality only holds at a release commit, so every branch that edited emulo.py went red in CI by construction. CI that is always failing stops carrying information, and the next real regression gets waved through. What the bootstrap actually promises is narrower: runtime.json describes the bytes a user downloads at the pinned tag. That is now checked directly, by reading emulo.py and MINING_PROMPT.md out of `git show <ref>:<file>` rather than off disk. It holds on any branch and it verifies the released artifact instead of assuming the tree equals it. The useful half of the old assertion is kept where it is actually true, in a second test that enforces the working tree against the pin only when HEAD is the release commit. That is what catches editing emulo.py and forgetting to re-pin before tagging, and it self-activates at exactly the moment it matters. CI now checks out with fetch-depth: 0. Without tags the pin test cannot resolve the release bytes and skips, which would have retired the check silently, and a guard that disappears without saying so is worse than one that fails. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`emulo verify --json` is not a local diagnostic. In a paid Profile Build the client runs it on their own machine and sends the report back, because that is the only way to check receipts without the reviewer ever seeing raw session logs. The report was embedding the absolute profile path and the output directory, which meant emailing it disclosed a home directory, a username, and frequently a client's project name. It now reports the profile's file name and drops the output path entirely. Everything that remains is quotes already present in the draft plus opaque session ids, so the file proves which rules are supported while disclosing nothing about what the person works on. A regression test asserts no temporary or output path survives into the payload. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A mined profile earns trust from its receipts. The worst failure it can have is not a missing rule, it is a confident rule quoting something the person never said. Nothing checked that until now.
What this adds
emulo verify you.mdextracts every quoted span from a profile and searches the mined corpus for it. Unfound quotes are reported and the command exits non-zero. Quotes resting on a single session are flagged separately, since one session is context rather than a rule.--jsonemits the supporting session ids per quote.Why it matters beyond the CLI
The JSON report is safe to share: it carries quotes already present in the draft plus opaque session ids, and no session text. That makes it possible to check someone else's profile receipts without ever seeing their logs.
Proof it works
Run against a real 2,177-session corpus and a real profile, it immediately caught quotes that had been silently tidied up. The source said
its feel static beucase you just dont move the characterand the profile rendered it in corrected English, which makes the receipt untraceable. 24 of 31 quotes failed on the first pass of a profile written carefully the same day.Also in this branch
emulo.pywent red in CI by construction. It now checks the bytes at the pinned tag viagit show, and the working-tree assertion is kept in a second test that self-activates only when HEAD is the release commit. CI checks out withfetch-depth: 0so the tag resolves.--jsonreport no longer embeds local paths, which previously disclosed a home directory and username in a file meant to be sent to someone else.Scope
Verifies what is mechanically checkable. Whether a rule is vague, generic, or true of every developer alive stays a judgment call.
415 tests pass.