跑起来之后：我自己的三项评测都在量自己 - #20
Merged
Merged
Conversation
…graph First run against a real provider (DeepSeek). All three model-backed evals had a scoring bug of the same shape as the Pass-7 tautology this series set out to fix — the eval graded its own output, or graded a window that excluded the evidence it was asking about. 1. entailment reported 35.8%. It sent the judge `chunk_text[:1500]`, but Alice's chunks run to ~4,800 characters, so the evidence span usually fell past the cut and the judge correctly answered "that quote is not in the passage you gave me". The same claim was judged both ways in a single run depending on where its span landed. Centre the window on the span: 0/120 samples now fall outside it, and the real rate is 85.0%. It also vindicates the stratification: both multi_step claims fail (0/2), one_step strata sit at 0.5–0.75, explicit at 0.8+. That is exactly why Pass-7 verifies every multi_step claim instead of sampling it. 2. contamination reported the graph at 12/12. `ground_truth` was a restatement of `graph_answer`, so the judge was asked whether an answer matched itself. Grade both arms against the source passage instead: closed-book 0/12, graph 5/12. 3. perturbation scored plainly correct answers as reversions — "Pellwyn is a curious young girl who follows a White Rabbit down a rabbit-hole" was marked as having reverted to the published text. Cause: an instruction fed into the answer-scoring judge's ground-truth slot. Entailment and reversion now have their own prompts rather than borrowing one built for a different question. Rescored: memory follows the altered source 2/6, reading it follows 4/6. The raw answers say more than the ratio. Asked whether the text states X, the memory arm answers yes and quotes the published wording — wording absent from the excerpt it was given. And both failures of the reading arm quote the altered clause and then contradict it: it cites "the Queen tried the little golden key" and answers "Alice has the key"; it cites "she had *not* peeped into the book" and answers "yes, she did". Putting the source in the context window does not make the answer faithful to it. `LOREGRAPH_COVE_SUPPORTED_FLOOR` 0.85 -> 0.80. The 0.85 was a guess and the measurement landed exactly on it, so it would have aborted every run. 0.80 is alice's measured rate less a 5-point margin; the comment records the date, the judge model, and that one book is not a distribution. `model_arm.SPEND` accumulates across every arm and judge, so a run's cost comes out of the harness. The whole session, including the three wasted runs: 0.65 CNY.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loregraph | d1817a5 | Commit Preview URL Branch Preview URL |
Aug 05 2026, 02:35 AM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
接上 #18。第一次真的接上模型（DeepSeek）跑,三项需要模型的评测全都有 bug,而且犯的是这个系列一开始批评的同一类错——评测在给自己打分,或者在量一个把证据切掉了的窗口。
蕴含率:35.8% 是假的,真实值 85.0%
判分时发的是
chunk_text[:1500],而爱丽丝的段落有四千八百多字符,引文经常落在截断之外。判分模型如实回答「这句话不在你给我的段落里」。同一条断言在同一次运行里判了两次——引文落在窗口内就通过,落在窗口外就不通过。改成以引文为中心开窗:120 条抽样里 0 条落在窗口外。真实值 102/120 = 85.0%。
它同时验证了分层抽样的意义:两条
multi_step全部不成立(0/2),one_step各层在 0.5 到 0.75,explicit在 0.8 以上。这正是让 Pass-7 对 multi_step 全查不抽样的理由。污染探针:图谱 12/12 是构造出来的
ground_truth是graph_answer的改写,判分模型被问「这个答案跟它自己一致吗」。改成两边都对着原文段落判:闭卷 0/12,图谱 5/12。扰动测试:判分把明显正确的答案算成了错
「Pellwyn 是一个跟着白兔跳进兔子洞的好奇女孩」被判成「退回了出版版本」。原因是把一句指令塞进了「给答案打分」那个判分器的标准答案栏。蕴含判断和退回判断现在各有自己的提示词。
重判后:凭记忆跟上改动 2/6,拿到改后文本跟上 4/6。
原始回答比这两个数说明得多。问「原文是不是说了 X」,凭记忆那一路答「是」并引用原文——引的是出版版本的句子,根本不在给它的材料里。而读到文本仍然失败的那两次,模型引用了改动后的句子,紧接着说了相反的话:引「the Queen tried the little golden key」然后答「Alice 有钥匙」;引「她没有偷看那本书」然后答「是的,她看了」。
把原文放进上下文,不等于答案会忠于原文。这对一个强制片段级绑定的系统是好消息——它有真实的靶子。
阈值现在是测出来的
LOREGRAPH_COVE_SUPPORTED_FLOOR从 0.85 改成 0.80。0.85 是猜的,实测正好落在这个数上,会让每次运行都中止。0.80 是爱丽丝实测值减 5 个点,注释记下了日期、判分模型,以及一本书不构成分布。记账
model_arm.SPEND跨所有调用累计,花费从评测自身报出来。整轮包括三次白跑,共 0.65 元。没做的
🤖 Generated with Claude Code