让数字对得上：修抽取端的三处失真，加评测台 - #18
Merged
Merged
Conversation
Pass-8 caps each entity's evidence at 30 mentions / 25 edges / 15 facts. The cap was `items[:N]` over rows ordered by id — i.e. book order — so it took the earliest chapters and nothing else. On Alice that meant her outgoing edges and implicit facts came from chapter 1 alone, out of 12. The cap bites hardest on whoever has the most evidence, which is the protagonist. Pass-8 is then asked to list what "the source does not tell us", and duly reported the unread chapters: Alice's note claims her fate and how she returns home are never given. The full text is in the database, all 42 chunks through THE END. A confident false claim about the work, in the voice of evidence, on the character the reader most wants to read about — the worst failure mode available to a project whose whole argument is that its claims are checkable. Round-robin over chapters instead of taking a prefix. Same budget, same cost; Alice now sees all 12 chapters in every evidence stream. Tell the model when its evidence is a sample and which chapters are missing, and require [GAPS] to be phrased about the evidence shown rather than about the work.
Pass-7 is described as the hallucination gate and enforces one number: `literal_match_rate < 0.95` aborts the run. That number cannot fail. Pass-2/5/6 already drop any claim whose evidence span is not a literal substring, using this same `is_literal_match`, so on a healthy pipeline it is 1.0 by construction — as measured: 19,868 of 19,868 spans across alice and xyj match exactly. The gate proves the quote is real. It says nothing about whether the quote supports the claim. `supported_rate` is the number that does — and Pass-7 already computed it, recorded it, and then ignored it. It never left `pass_runs.stats`. - Enforce `supported_rate` with a configurable floor (LOREGRAPH_COVE_SUPPORTED_FLOOR, 0 disables). Provisional at 0.85: no calibrated distribution exists yet, which is what `eval entailment` is for. Keep the literal check as an invariant tripwire and say in the error that it points at a span-handling bug, not at quality. - Stratify the sample by (relation|dimension) x inference_depth instead of drawing uniformly. A uniform draw of Alice's edges is 63% INTERACTS and 80% explicit; PREDICTS is 1.3% and a 150-row uniform sample can miss it entirely. Verify every multi_step claim rather than sampling it — deepest inference, likeliest to be wrong, rare enough to afford. - Raise the default sample from 50 to 150 per claim family. 50 was 0.5% of xyj's 18,868 claims. - Report per-stratum rates and the weakest strata, so an operator learns which kind of claim is failing rather than a pooled average that hides it. - Ship `metadata.quality` in the book export. The graph's quality number belongs with the graph; until now the only number a reader could see was a count of edges.
Pass-5 asks for "a specific verb-phrase … stay close to the verb in the
evidence_span" and gets one: alice and xyj between them carry **2627
distinct predicates over 9843 edges, 1554 of them singletons**. A third
of every edge in the graph is labelled with a relation name that appears
nowhere else in it. You cannot ask "who betrayed whom" when betrayal is
spread across BETRAYS, DECEIVES and TRICKS at one occurrence each. The
5-way `relation` enum above it cannot compensate — 65% of edges are
INTERACTS.
Taking the specific verb away would be the wrong fix; it is the most
informative thing on the edge and what a reader wants to see. Add a
second, closed axis instead:
relation INTERACTS 5 values — too coarse
predicate_class CONFLICT 25 values — queryable
predicate THREATENS_TO_BURN open — readable
`classify` resolves the open verb by longest leading stem, so the
model's habit of gluing the object on survives: CAUSES_SNEEZING ->
CAUSATION, THREATENS_TO_BURN -> CONFLICT, PREDICTS_RESURRECTION ->
FORESIGHT. Modal and tense prefixes and both regular and irregular
inflections are peeled (TAUGHT -> TEACH was a real miss).
The vocabulary is derived from the observed distribution, not invented:
every family has a real head in the data, and it was grown by measuring
the misses and adding what they showed. Anything still unplaceable goes
to `<RELATION>_OTHER` rather than a plausible wrong bucket — the share
landing there is the coverage metric, and it now reads 90.1% on alice
and 94.0% on xyj into 25 real classes.
Both shipped exports are backfilled, and `metadata.predicate_coverage`
carries the score so it is visible rather than asserted.
…k good Every figure in the catalogue was invented. All 85 rows in data.js carried hard-coded entity/edge/glucose counts, a matchRate, a lastRun timestamp, a provider and a cost — $116.12 of claimed spend across 84 timestamps, none of which came from a run. Only alice and xyj have real exports, and even xyj's row disagreed with its own: 502/1158/2876 claimed against 600/8833/ 10035 actual. Twelve rows were marked `verified` while displaying a matchRate of 0.93 or 0.94, below the 0.95 gate that would have aborted their runs. `build_frontend_data.py` was the other half of it, stamping `status: "verified"` and `matchRate: 0.96` onto any book the moment it was exported — before Pass-7 had said anything at all. For a project whose entire argument is that its claims are checkable, the shop window was the one place nothing was. - Derive run facts from the export's own Pass-7 record. A book that is extracted but not audited reads `extracted` with no rate, which is what alice and xyj actually are. - Null the run fields on the other 83. `queued` is the honest state, and the card already had a "not extracted yet" treatment for null stats — 166 cells now use it. - Library header gains an "extracted" count so 1,077 entities reads as the total for 2 books rather than for 85, and drops the cost stat when no cost was recorded. - Sidebar budget shows the real default ceiling ($20, the LOREGRAPH_COST_CEILING_USD default) with spend absent, not $14.27/$50. - Label the pipeline view's hand-written run as an illustration. It is the one screen still showing sample data; now it says so instead of sitting next to real figures looking equally sourced. Bibliographic fields are untouched — title, author, year, tokens, chapters are properties of the work and were never the problem. Also fixes a pre-existing overflow this made visible: the three card stats need ~222px of labels and the grid dropped to 160px columns under 720px, so labels ran into the neighbouring card. Verified 0 overflowing cells and 0 horizontal page scroll at 320/375/430/592/900/1280.
The project could report 8,833 edges and a 100% literal-match rate and
still not answer either question that matters: is the extraction correct,
and — on a corpus where every work is canonical — does it beat simply
asking a model that has read a thousand essays on each one.
Five evaluations, `loregraph eval [name] --book [id]`:
perturbation Alter the source (rename a character to a name never in
print, move an object, negate a clause) and ask a
question that turns on the change. A system reading the
text follows the edit; one reciting its memory reverts.
The edit IS the ground truth, so no annotation is needed.
This is the only experiment that can show the pipeline
earning its cost on a famous book — or that it cannot.
contamination The same questions closed-book. Scored per book, never
averaged: the useful output is a list of which works
justify the pipeline, not a mean that hides them.
entailment The check Pass-7 was not making. Stratified, multi_step
exhaustive, and it emits a suggested floor so
LOREGRAPH_COVE_SUPPORTED_FLOOR can be set on a measured
distribution rather than a guess.
gaps Every [GAPS] bullet against the source.
graph Twenty analyst questions, answered by traversal alone.
`graph` and `gaps` need no model or credentials and produce numbers now.
The other three fall back to a dry preview that prints exactly what would
be sent, so the battery can be inspected before anything is spent.
Two of these evals caught their own first version being useless, which is
recorded in the code because it is the point:
- `graph` scored 20/20 on both books while returning "her brother's Latin
Grammar -> the Mouse" for "which characters come into conflict".
"Returns something" is not a threshold. It now also scores whether the
endpoints are the kind of thing the question asked for: 78% on alice,
91% on xyj, and the misses are real extraction defects — an Object as
the aggressor in a fight, 倒身下拜 ("prostrating oneself") as a
character present from chapter 1 to 100.
- `perturbation`'s possession swap renamed the holder globally: 464
occurrences of "Alice" became "the Queen", leaving a text about nobody.
It now rewrites the one passage, and reports any probe where the
original wording survives, since a leaky probe can be answered from the
leftovers instead of by reading.
`gaps` independently found the Pass-8 defect: Alice's note claims whether
her adventures were a dream "is not resolved in the provided text", and
"dream" and "reality" both occur in chapter 12 — which the note never
read. On xyj, 267 of 599 profiled entities were profiled without citing
every chapter they appear in.
Every result carries a `skipped` list and the renderer always prints it.
An eval that silently scores the subset it could reach is worse than none,
because the number looks like coverage.
…hange The spec described Pass-7's literal-match rate as the hard gate, which is how it came to be trusted as a quality figure. It is an invariant Pass-2/5/6 already guarantee. Name both numbers, say which one measures anything, and document the eval harness as the way to check an extraction change rather than counting rows.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loregraph | d9dc332 | Commit Preview URL Branch Preview URL |
Aug 03 2026, 10:18 AM |
The literal-gate test asserted the message contained "gate fails". The message now explains that the literal check is an invariant tripwire and not a quality figure — which is the point of the change, since taking that number for a quality score is how it came to be trusted as one — so the phrase went and CI went red on the wording, not the behaviour. Assert on what the message has to convey instead, and cover what was never covered: the entailment gate had no test at all. Added, in both tiers. - Integration: the entailment gate aborts on literal-perfect spans the judge rejects (exactly the run the old single-gate Pass-7 waved through); `supported_floor=0` records without enforcing; per-stratum counts reach the stats. - Unit: the same three paths plus the floor boundary and settings defaults, with no database. A gate this load-bearing should not be reachable only through a Postgres testcontainer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
起因是一个问题：这套方案现在真的 work 吗？对知名作品，直接问顶级模型会不会更好？
读代码读出四处问题，都修了，外加一套能回答那个问题的评测。
目录里的数字是编的
data.js85 行书目全部硬编码了实体数、边数、matchRate、lastRun、provider 和 cost——84 个时间戳、$116.12 的花销，没有一个来自真实运行。只有 alice 和 xyj 有真实导出，连 xyj 那行都跟自己的导出对不上（502/1158/2876 对 600/8833/10035）。12 行标着verified却显示 0.93/0.94 的 matchRate，低于会让它们中止的 0.95 闸口。build_frontend_data.py是另一半：任何书一导出就被盖上status: "verified"和matchRate: 0.96，在 Pass-7 说话之前。对一个全部立论都在「断言可核查」的项目，橱窗是唯一什么都没核查的地方。
现在运行事实从导出自带的 Pass-7 记录派生。抽取了但没审计的书显示
extracted且不报速率——alice 和 xyj 的真实状态。其余 83 本置空，卡片已有的「尚未抽取」样式接管了 166 个格子。书库头部加了「已抽取」计数，1077 个实体读起来是 2 本书的总数而不是 85 本的。侧栏预算显示真实的默认上限（$20），花销留空。流水线视图那份手写运行数据打上了「示意」标签。Pass-7 守的是同义反复
literal_match_rate < 0.95中止运行——但这个数不可能失败。Pass-2/5/6 用同一个is_literal_match在落库前就丢掉了非字面 span，所以健康流水线上它恒等于 1.0。实测：两本书 19,868 条 span 全中。这道闸证明引文是真的，不证明引文支持断言。supported_rate才是那个数，而 Pass-7 一直在算它、记它、然后忽略它——它从没离开过pass_runs.stats。现在它参与闸口（
LOREGRAPH_COVE_SUPPORTED_FLOOR，默认 0.85，0 关闭），样本按 (relation|dimension) × inference_depth 分层，multi_step全查不抽样，默认样本从 50 提到 150（原来是 xyj 18,868 条断言的 0.5%），并报出最弱分层。质量数随导出发布。Pass-8 在用证据的语气说假话
每个实体的证据上限是
items[:N]，行按书序排列，所以取的是开头几章。爱丽丝的出边和推断只喂了第 1 章，共 12 章。截断对证据最多的实体最狠，也就是主角。然后 Pass-8 被要求列出「原文没有告诉我们什么」，它照做了：爱丽丝的笔记称她的命运和如何回家从未交代。全文 42 个 chunk 完整到 THE END，就在库里。
改成按章节轮转。同样的预算，同样的成本，爱丽丝现在每条证据流都覆盖 12 章。提示词会被告知证据是抽样的、缺哪些章，
[GAPS]必须针对看到的证据措辞。图不可查询
两本书 9,843 条边用了 2,627 种谓词，1,554 种只出现一次。三分之一的边挂着全图独一份的关系名——「谁背叛过谁」查不出来，因为背叛散在各出现一次的 BETRAYS/DECEIVES/TRICKS 里。上层五分类太粗，65% 是 INTERACTS。
拿走具体动词是错的修法，它是边上最有信息量的东西。改成加第二条闭合轴：
词表从观测分布里长出来，不是凭空造的，靠量缺失再补而成。落地率 alice 90.1%、xyj 94.0%，其余老实标
_OTHER。评测台
loregraph eval [name] --book [id]，五项：LOREGRAPH_COVE_SUPPORTED_FLOOR建立在测量而不是猜测上。[GAPS]对着原文核。graph和gaps不需要模型和密钥，现在就出数。另外三项无 provider 时打印干跑预览，把要发的内容原样列出来，先看再花钱。有两项评测抓到了自己第一版没用，这写进代码里了，因为这正是要点：
graph第一版在两本书上都是 20/20，同时把「her brother's Latin Grammar → the Mouse」当成人物冲突返回。「返回非空」不是门槛。现在它同时评估端点是不是该类东西：alice 78%，xyj 91%，漏掉的都是真实抽取缺陷——物件当打架的主体，「倒身下拜」当从第 1 章贯穿到第 100 章的角色。perturbation的换物那条做成了全局改名：全书 464 个「Alice」变成「the Queen」，剩下一篇关于没有人的文本。现在只改那一处，并报告任何原文措辞残留的探针，因为漏了的探针可以靠残留回答而不必阅读。gaps独立发现了 Pass-8 那个缺陷：爱丽丝的笔记说是梦是真「在所提供的文本中未有定论」，而 dream 和 reality 都出现在第 12 章——笔记从没读过那章。xyj 上 599 个有笔记的实体里有 267 个没有引全自己出现过的章节。每个结果都带
skipped列表，渲染器一定打印。评测悄悄只给它够得着的那部分打分，比没有评测更糟，因为那个数看起来像覆盖率。验证
ruff format --check/ruff check全过；mypy 11 条错，全部既有（基线 13，改动顺手修掉两条）没做的
available()一路老实上报。扰动测试该第一个跑——它一天能出结果，而且它回答的正是这个 PR 起因的那个问题。suggested_floor还没有测量支撑。 0.85 是暂定值，等eval entailment在几本书上跑过再定。' was '→' was not '），在中文源上不产出，评测会明说这一点，两者的结果不可比。🤖 Generated with Claude Code