把 Pass-7 和 Pass-8 的修复真正落到数据上 - #21
Merged
Merged
Conversation
…graph First run against a real provider (DeepSeek). All three model-backed evals had a scoring bug of the same shape as the Pass-7 tautology this series set out to fix — the eval graded its own output, or graded a window that excluded the evidence it was asking about. 1. entailment reported 35.8%. It sent the judge `chunk_text[:1500]`, but Alice's chunks run to ~4,800 characters, so the evidence span usually fell past the cut and the judge correctly answered "that quote is not in the passage you gave me". The same claim was judged both ways in a single run depending on where its span landed. Centre the window on the span: 0/120 samples now fall outside it, and the real rate is 85.0%. It also vindicates the stratification: both multi_step claims fail (0/2), one_step strata sit at 0.5–0.75, explicit at 0.8+. That is exactly why Pass-7 verifies every multi_step claim instead of sampling it. 2. contamination reported the graph at 12/12. `ground_truth` was a restatement of `graph_answer`, so the judge was asked whether an answer matched itself. Grade both arms against the source passage instead: closed-book 0/12, graph 5/12. 3. perturbation scored plainly correct answers as reversions — "Pellwyn is a curious young girl who follows a White Rabbit down a rabbit-hole" was marked as having reverted to the published text. Cause: an instruction fed into the answer-scoring judge's ground-truth slot. Entailment and reversion now have their own prompts rather than borrowing one built for a different question. Rescored: memory follows the altered source 2/6, reading it follows 4/6. The raw answers say more than the ratio. Asked whether the text states X, the memory arm answers yes and quotes the published wording — wording absent from the excerpt it was given. And both failures of the reading arm quote the altered clause and then contradict it: it cites "the Queen tried the little golden key" and answers "Alice has the key"; it cites "she had *not* peeped into the book" and answers "yes, she did". Putting the source in the context window does not make the answer faithful to it. `LOREGRAPH_COVE_SUPPORTED_FLOOR` 0.85 -> 0.80. The 0.85 was a guess and the measurement landed exactly on it, so it would have aborted every run. 0.80 is alice's measured rate less a 5-point margin; the comment records the date, the judge model, and that one book is not a distribution. `model_arm.SPEND` accumulates across every arm and judge, so a run's cost comes out of the harness. The whole session, including the three wasted runs: 0.65 CNY.
…y hit The Pass-8 fix was merged as code and never reached the data — Alice's note still carried "Her ultimate fate and how she returns home are not given here", which is false about a book whose twelfth chapter is her waking up. There is a local PostgreSQL 18 with pgvector on this machine. I had run `docker info`, seen it fail, and concluded the database was unavailable — without checking whether Postgres was installed directly. It was, with both books fully extracted. Docker is needed for the integration tests (testcontainers); it is not needed to run the pipeline. Rather than re-run Pass-8 over all 477 entities (~4.1 CNY), computed which ones the chapter-spread fix actually changes: 8 of 477, and only Alice's chapter coverage genuinely moves — from chapters 1-9 to all 12. Regenerated those eight for 0.09 CNY. Alice's [GAPS] now reads "The evidence does not say how Alice returns to the real world or what happens after she wakes up." It knows she wakes up. That is only in chapter 12, which the old prefix-cap never showed it. The phrasing also moved from a verdict on the work to a statement about the evidence, which is what the prompt change asked for. Also re-ran Pass-7 (60 claims per family, 0.40 CNY) so the export carries a measured quality figure instead of an absent one: supported_rate 0.7333 ← the number that means something literal_match_rate 1.0 ← the invariant, unfailable as predicted weakest strata INFLUENCES/one_step 0/2, both multi_step strata 0/1 Two judges, twelve points apart. `loregraph eval entailment` scored the same book at 85.0%; Pass-7's own prompt scores it at 73.3%. Same claims, same spans, different judge. So the 0.80 floor calibrated from the eval would have aborted this run had it been enforced — it was not, because .env sets the floor to 0 while the number is being established. The floor is not trustworthy until the two judges are reconciled or one is chosen.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loregraph | 4b14e65 | Commit Preview URL Branch Preview URL |
Aug 05 2026, 03:17 AM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
接上 #20。前几轮我一直说 Pass-7 和 Pass-8 的修复「只在代码里，落不到数据上，因为要数据库，数据库要 Docker，本机没装」。
这个结论是错的。本机装了 PostgreSQL 18 + pgvector 并在 5432 上运行，
loregraph库里两本书八遍全跑完，数据一直都在。我只试了docker info，失败就断言数据库不可用，没查本机是不是直接装了 Postgres。Docker 只有tests/integration/需要（testcontainers），流水线不需要。Pass-8:假话没了
没有全量重跑 477 个实体（约 4.1 元）。先算出章节铺开这个修复到底会改变哪些实体的输入：477 个里只有 8 个，而且只有爱丽丝的章节覆盖真的变了，从第 1–9 章变成全部 12 章。重跑这 8 条，0.09 元。
爱丽丝的
[GAPS]前后对比：Her ultimate fate and how she returns home are not given here.The evidence does not say how Alice returns to the real world or what happens after she wakes up.它现在知道爱丽丝会醒来。这件事只写在第 12 章，正是旧的前缀截断从没喂给它的那一章。措辞也从对作品下判断变成了对手上证据的陈述，正是提示词改动要求的。
Pass-7:导出里第一次有了真实质量数
重跑 Pass-7（每类抽 60 条，0.40 元）：
metadata.quality现在带着这些数出厂，前端因此把爱丽丝显示为verified加真实速率，西游记仍是extracted加空值——因为它没在新闸口下审过。一个必须说明的矛盾
同一批数据，两个判分器差 12 个点。
loregraph eval entailment判 85.0%，Pass-7 自己的提示词判 73.3%。同样的断言、同样的引文，只是判分提示词不同。所以我根据评测校准出来的 0.80 阈值，如果启用，会把这次运行拦下。它没被拦下，是因为
.env里阈值设成 0（记录不拦截）。在两个判分器对齐、或者明确选定一个之前，这个阈值不可信。 这是下一步要解决的问题，我没有就地拍一个数糊过去。花费
这一轮 0.49 元（笔记 0.09 + Pass-7 0.40）。本次会话累计约 1.15 元。
🤖 Generated with Claude Code