跑起来之后：我自己的三项评测都在量自己 - #19
Closed
YunyueLi wants to merge 8 commits into
Closed
Conversation
Pass-8 caps each entity's evidence at 30 mentions / 25 edges / 15 facts. The cap was `items[:N]` over rows ordered by id — i.e. book order — so it took the earliest chapters and nothing else. On Alice that meant her outgoing edges and implicit facts came from chapter 1 alone, out of 12. The cap bites hardest on whoever has the most evidence, which is the protagonist. Pass-8 is then asked to list what "the source does not tell us", and duly reported the unread chapters: Alice's note claims her fate and how she returns home are never given. The full text is in the database, all 42 chunks through THE END. A confident false claim about the work, in the voice of evidence, on the character the reader most wants to read about — the worst failure mode available to a project whose whole argument is that its claims are checkable. Round-robin over chapters instead of taking a prefix. Same budget, same cost; Alice now sees all 12 chapters in every evidence stream. Tell the model when its evidence is a sample and which chapters are missing, and require [GAPS] to be phrased about the evidence shown rather than about the work.
Pass-7 is described as the hallucination gate and enforces one number: `literal_match_rate < 0.95` aborts the run. That number cannot fail. Pass-2/5/6 already drop any claim whose evidence span is not a literal substring, using this same `is_literal_match`, so on a healthy pipeline it is 1.0 by construction — as measured: 19,868 of 19,868 spans across alice and xyj match exactly. The gate proves the quote is real. It says nothing about whether the quote supports the claim. `supported_rate` is the number that does — and Pass-7 already computed it, recorded it, and then ignored it. It never left `pass_runs.stats`. - Enforce `supported_rate` with a configurable floor (LOREGRAPH_COVE_SUPPORTED_FLOOR, 0 disables). Provisional at 0.85: no calibrated distribution exists yet, which is what `eval entailment` is for. Keep the literal check as an invariant tripwire and say in the error that it points at a span-handling bug, not at quality. - Stratify the sample by (relation|dimension) x inference_depth instead of drawing uniformly. A uniform draw of Alice's edges is 63% INTERACTS and 80% explicit; PREDICTS is 1.3% and a 150-row uniform sample can miss it entirely. Verify every multi_step claim rather than sampling it — deepest inference, likeliest to be wrong, rare enough to afford. - Raise the default sample from 50 to 150 per claim family. 50 was 0.5% of xyj's 18,868 claims. - Report per-stratum rates and the weakest strata, so an operator learns which kind of claim is failing rather than a pooled average that hides it. - Ship `metadata.quality` in the book export. The graph's quality number belongs with the graph; until now the only number a reader could see was a count of edges.
Pass-5 asks for "a specific verb-phrase … stay close to the verb in the
evidence_span" and gets one: alice and xyj between them carry **2627
distinct predicates over 9843 edges, 1554 of them singletons**. A third
of every edge in the graph is labelled with a relation name that appears
nowhere else in it. You cannot ask "who betrayed whom" when betrayal is
spread across BETRAYS, DECEIVES and TRICKS at one occurrence each. The
5-way `relation` enum above it cannot compensate — 65% of edges are
INTERACTS.
Taking the specific verb away would be the wrong fix; it is the most
informative thing on the edge and what a reader wants to see. Add a
second, closed axis instead:
relation INTERACTS 5 values — too coarse
predicate_class CONFLICT 25 values — queryable
predicate THREATENS_TO_BURN open — readable
`classify` resolves the open verb by longest leading stem, so the
model's habit of gluing the object on survives: CAUSES_SNEEZING ->
CAUSATION, THREATENS_TO_BURN -> CONFLICT, PREDICTS_RESURRECTION ->
FORESIGHT. Modal and tense prefixes and both regular and irregular
inflections are peeled (TAUGHT -> TEACH was a real miss).
The vocabulary is derived from the observed distribution, not invented:
every family has a real head in the data, and it was grown by measuring
the misses and adding what they showed. Anything still unplaceable goes
to `<RELATION>_OTHER` rather than a plausible wrong bucket — the share
landing there is the coverage metric, and it now reads 90.1% on alice
and 94.0% on xyj into 25 real classes.
Both shipped exports are backfilled, and `metadata.predicate_coverage`
carries the score so it is visible rather than asserted.
…k good Every figure in the catalogue was invented. All 85 rows in data.js carried hard-coded entity/edge/glucose counts, a matchRate, a lastRun timestamp, a provider and a cost — $116.12 of claimed spend across 84 timestamps, none of which came from a run. Only alice and xyj have real exports, and even xyj's row disagreed with its own: 502/1158/2876 claimed against 600/8833/ 10035 actual. Twelve rows were marked `verified` while displaying a matchRate of 0.93 or 0.94, below the 0.95 gate that would have aborted their runs. `build_frontend_data.py` was the other half of it, stamping `status: "verified"` and `matchRate: 0.96` onto any book the moment it was exported — before Pass-7 had said anything at all. For a project whose entire argument is that its claims are checkable, the shop window was the one place nothing was. - Derive run facts from the export's own Pass-7 record. A book that is extracted but not audited reads `extracted` with no rate, which is what alice and xyj actually are. - Null the run fields on the other 83. `queued` is the honest state, and the card already had a "not extracted yet" treatment for null stats — 166 cells now use it. - Library header gains an "extracted" count so 1,077 entities reads as the total for 2 books rather than for 85, and drops the cost stat when no cost was recorded. - Sidebar budget shows the real default ceiling ($20, the LOREGRAPH_COST_CEILING_USD default) with spend absent, not $14.27/$50. - Label the pipeline view's hand-written run as an illustration. It is the one screen still showing sample data; now it says so instead of sitting next to real figures looking equally sourced. Bibliographic fields are untouched — title, author, year, tokens, chapters are properties of the work and were never the problem. Also fixes a pre-existing overflow this made visible: the three card stats need ~222px of labels and the grid dropped to 160px columns under 720px, so labels ran into the neighbouring card. Verified 0 overflowing cells and 0 horizontal page scroll at 320/375/430/592/900/1280.
The project could report 8,833 edges and a 100% literal-match rate and
still not answer either question that matters: is the extraction correct,
and — on a corpus where every work is canonical — does it beat simply
asking a model that has read a thousand essays on each one.
Five evaluations, `loregraph eval [name] --book [id]`:
perturbation Alter the source (rename a character to a name never in
print, move an object, negate a clause) and ask a
question that turns on the change. A system reading the
text follows the edit; one reciting its memory reverts.
The edit IS the ground truth, so no annotation is needed.
This is the only experiment that can show the pipeline
earning its cost on a famous book — or that it cannot.
contamination The same questions closed-book. Scored per book, never
averaged: the useful output is a list of which works
justify the pipeline, not a mean that hides them.
entailment The check Pass-7 was not making. Stratified, multi_step
exhaustive, and it emits a suggested floor so
LOREGRAPH_COVE_SUPPORTED_FLOOR can be set on a measured
distribution rather than a guess.
gaps Every [GAPS] bullet against the source.
graph Twenty analyst questions, answered by traversal alone.
`graph` and `gaps` need no model or credentials and produce numbers now.
The other three fall back to a dry preview that prints exactly what would
be sent, so the battery can be inspected before anything is spent.
Two of these evals caught their own first version being useless, which is
recorded in the code because it is the point:
- `graph` scored 20/20 on both books while returning "her brother's Latin
Grammar -> the Mouse" for "which characters come into conflict".
"Returns something" is not a threshold. It now also scores whether the
endpoints are the kind of thing the question asked for: 78% on alice,
91% on xyj, and the misses are real extraction defects — an Object as
the aggressor in a fight, 倒身下拜 ("prostrating oneself") as a
character present from chapter 1 to 100.
- `perturbation`'s possession swap renamed the holder globally: 464
occurrences of "Alice" became "the Queen", leaving a text about nobody.
It now rewrites the one passage, and reports any probe where the
original wording survives, since a leaky probe can be answered from the
leftovers instead of by reading.
`gaps` independently found the Pass-8 defect: Alice's note claims whether
her adventures were a dream "is not resolved in the provided text", and
"dream" and "reality" both occur in chapter 12 — which the note never
read. On xyj, 267 of 599 profiled entities were profiled without citing
every chapter they appear in.
Every result carries a `skipped` list and the renderer always prints it.
An eval that silently scores the subset it could reach is worse than none,
because the number looks like coverage.
…hange The spec described Pass-7's literal-match rate as the hard gate, which is how it came to be trusted as a quality figure. It is an invariant Pass-2/5/6 already guarantee. Name both numbers, say which one measures anything, and document the eval harness as the way to check an extraction change rather than counting rows.
The literal-gate test asserted the message contained "gate fails". The message now explains that the literal check is an invariant tripwire and not a quality figure — which is the point of the change, since taking that number for a quality score is how it came to be trusted as one — so the phrase went and CI went red on the wording, not the behaviour. Assert on what the message has to convey instead, and cover what was never covered: the entailment gate had no test at all. Added, in both tiers. - Integration: the entailment gate aborts on literal-perfect spans the judge rejects (exactly the run the old single-gate Pass-7 waved through); `supported_floor=0` records without enforcing; per-stratum counts reach the stats. - Unit: the same three paths plus the floor boundary and settings defaults, with no database. A gate this load-bearing should not be reachable only through a Postgres testcontainer.
…graph Ran the harness against DeepSeek for the first time. All three model-backed evals had a scoring bug of the same shape as the Pass-7 tautology this PR set out to fix — the eval graded its own output, or graded a window that excluded the evidence. 1. entailment reported 35.8%. It sends the judge `chunk_text[:1500]`, but Alice's chunks run to ~4,800 characters, so the evidence span usually fell past the cut and the judge correctly answered "that quote is not in the passage you gave me". The same claim was judged both ways in one run depending on where its span landed. Centre the window on the span instead: 0/120 samples now fall outside it, and the real rate is 85.0%. 2. contamination reported the graph at 12/12. `ground_truth` was a restatement of `graph_answer`, so the judge was asked whether an answer matched itself. Grade both arms against the source passage: closed-book 0/12, graph 5/12. 3. perturbation scored two plainly correct answers as reversions — "Pellwyn is a curious young girl who follows a White Rabbit down a rabbit-hole" was marked as having reverted to the published text. Cause: an instruction was being fed into the answer-scoring judge's ground-truth slot. Gave reversion and entailment their own judges and prompts rather than borrowing one built for a different question. Also: `LOREGRAPH_COVE_SUPPORTED_FLOOR` drops 0.85 -> 0.80. The 0.85 was a guess, and the measurement landed exactly on it — it would have aborted every run. 0.80 is alice's measured rate less a 5-point margin, and the comment says so, along with the fact that one book is not a distribution. Spend for the whole session, including the three wasted runs: 0.65 CNY. `model_arm.SPEND` now accumulates across every arm and judge so the figure comes out of the harness rather than the provider dashboard.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loregraph | 721168d | Commit Preview URL Branch Preview URL |
Aug 05 2026, 02:30 AM |
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
接上 #18。第一次真的接上模型（DeepSeek）跑,三项需要模型的评测全都有 bug,而且犯的是这个 PR 系列一开始批评的同一类错——评测在给自己打分,或者在量一个把证据切掉了的窗口。
蕴含率:35.8% 是假的,真实值 85.0%
判分时我发的是
chunk_text[:1500],而爱丽丝的段落有四千八百多字符,引文经常落在截断之外。判分模型如实回答「这句话不在你给我的段落里」。同一条断言在同一次运行里判了两次——引文落在窗口内就通过,落在窗口外就不通过。改成以引文为中心开窗:120 条抽样里 0 条落在窗口外。真实蕴含率 102/120 = 85.0%。
而且它验证了分层抽样的意义:两条
multi_step全部不成立(0/2),one_step各层在 0.5 到 0.75,explicit在 0.8 以上。这正是让 Pass-7 对 multi_step 全查不抽样的理由。污染探针:图谱 12/12 是构造出来的
ground_truth是graph_answer的改写,判分模型被问「这个答案跟它自己一致吗」。改成两边都对着原文段落判:闭卷 0/12,图谱 5/12。扰动测试:判分把明显正确的答案算成了错
「Pellwyn 是一个跟着白兔跳进兔子洞的好奇女孩」被判定为「退回了出版版本」。原因是我把一句指令塞进了「给答案打分」那个判分器的标准答案栏。给蕴含判断和退回判断各写了专用提示词,不再挪用为别的问题设计的那一个。
重判后:凭记忆跟上改动 2/6,拿到改后文本跟上 4/6。
原始回答比这两个数说明得多。问「原文是不是说了 X」,凭记忆那一路答「是」并引用了原文——引的是出版版本的句子,根本不在给它的材料里。而失败的两次里,模型引用了改动后的句子,紧接着说了相反的话:引了「the Queen tried the little golden key」然后回答「Alice 有钥匙」,引了「她没有偷看那本书」然后回答「是的,她看了」。
所以把原文放进上下文不等于答案会忠于原文。证据落地是个没解决的真问题,这对一个强制片段级绑定的系统是好消息——它有真实的靶子。
阈值现在是测出来的
LOREGRAPH_COVE_SUPPORTED_FLOOR从 0.85 改成 0.80。0.85 是我猜的,实测正好落在这个数上,会让每次运行都中止。0.80 是爱丽丝实测值减 5 个点,注释里写明来源、日期、判分模型,以及一本书不构成分布。记账
model_arm.SPEND现在跨所有调用累计,花费从评测自身报出来而不是去服务商后台翻。整轮包括三次白跑的运行,共 0.65 元。没做的
🤖 Generated with Claude Code