Skip to content

fix(statements): correct + unpublish the 2026-07-28 weekly (unresolved window, not a 0% hit rate) - #58

Merged
tkowalczyk merged 2 commits into
mainfrom
fix/2026-07-28-weekly-unresolved-window
Jul 29, 2026
Merged

fix(statements): correct + unpublish the 2026-07-28 weekly (unresolved window, not a 0% hit rate)#58
tkowalczyk merged 2 commits into
mainfrom
fix/2026-07-28-weekly-unresolved-window

Conversation

@tkowalczyk

@tkowalczyk tkowalczyk commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

2026-07-28-weekly.md is live at draft: false claiming:

ogsfrompoly weekly statement — 263 alerts, hit rate 0.00.

To a reader that says 263 calls, all wrong. In fact none of the 263 had
resolved
. This PR corrects the numbers and takes the claim off the live site.

What changed

field before after
summary … 263 alerts, hit rate 0.00. … 263 alerts, outcomes pending (0 of 263 resolved).
hypothetical_pnl_usd 0.0 199.11
top_wallets [] 3 entries
draft false true — see below
alert_count · period_* · hit_rate · title unchanged

alert_count: 263 reproduces exactly against the same
2026-07-22 → 2026-07-28 window, which is what confirms the window is identical
and only the reporting of it was wrong.

Why it lands as a draft rather than a corrected publication

CI caught what the correction could not fix, and it was right to. With
hypothetical_pnl_usd restored to its true value, findPublishedInconsistencies
fires:

alert_count=263 with hit_rate=0 but hypothetical_pnl_usd=199.11 > 0
(0 resolved alerts collapsing 0/0 → 0?)

That guard's own comment already names the diagnosis — "the tell of a 0/0 → 0
resolution collapse"
— and that is precisely this window. It could not fire
before only because hypothetical_pnl_usd was also 0
, from the upstream bug
that had the producer reading a warehouse without the derived tables
(auditmos/ogsfrompoly#234). Fixing that made the pathology detectable. The guard
worked as designed.

It cannot be satisfied honestly under schema_version: 1. No field separates
"0 of 0 resolved" from "0 of 240 resolved", so a truthful statement for an
unresolved window is unpublishable: either hit_rate: 0 asserts a 0 % success
rate that did not happen, or the PnL gets falsified back to 0 to match. Neither is
acceptable, and weakening the guard would throw away the one check that caught it.

Drafts are exempt from the guard and excluded by every consumer (excludeDrafts
in the index, the feeds and statements.md.ts; there is no per-statement route),
so draft: true removes the false claim from the live site immediately. The
corrected numbers stay in the file, making republication a one-line flip.

Root cause, for the record

hit_rate = in_favor / resolved, scored over alerts emitted in the window.
Measured on the producer host: 111 of 111 alerted markets join markets, 0
carry an outcome — against 97.3 % coverage warehouse-wide. Not the join, not
enrichment; the markets genuinely have not settled.

Every statement back to May carried 0.35–0.71, all tagged
[politics, macro-finance]. This is the first Macro-only statement — Politics
was retired 2026-07-24, and Politics markets (elections, daily events) supplied
essentially the whole resolved sample, while Macro markets resolve months out.
So this recurs every week until the schema can express it.

Follow-up that unblocks republishing

Add resolved_count, bump to schema_version: 2 across the producer and this
repo, and teach the sanity check that resolved_count === 0 makes hit_rate
vacuous rather than inconsistent. Then flip this file back to draft: false.
Tracked as auditmos/ogsfrompoly#236.

Verification

  • Regenerated by poly-track report public weekly at auditmos/ogsfrompoly@554f537
    over the identical window
  • Passes validate_statement (schema_version: 1)
  • Verify (types, test, lint, knip) green
  • No address leak — all wallet references are truncated IDs

Refs auditmos/ogsfrompoly#234, auditmos/ogsfrompoly#236

Tomasz Kowalczyk added 2 commits July 29, 2026 12:36
…ot a 0% hit rate

This statement is live at `draft: false` and reads "263 alerts, hit rate 0.00".
To a reader that says we made 263 calls and got every one wrong. What it actually
means is that none of the 263 has resolved yet: measured on the producer host,
111 of 111 alerted markets join `markets` and none carries an outcome, against
97.3% outcome coverage warehouse-wide.

Not a bad week and not a data gap. Every statement back to May carried 0.35-0.71,
all of them tagged `[politics, macro-finance]`. This is the first Macro-only one
— Politics was retired 2026-07-24, and Politics markets (elections, daily events)
were supplying essentially the whole resolved sample, while Macro markets resolve
months out.

`hypothetical_pnl_usd: 0.0` and `top_wallets: []` had a separate cause: both are
derived from tables that live only in the slim warehouse, and the producer was
reading a warehouse that structurally cannot hold them.

Regenerated from the corrected producer against the same 2026-07-22 → 2026-07-28
window. `alert_count`, `period_start`, `period_end`, `hit_rate`, `title` and
`draft` are unchanged — 263 alerts reproduces exactly, which is what confirms the
window is identical and only the reporting was wrong.

  summary            "hit rate 0.00." → "outcomes pending (0 of 263 resolved)."
  hypothetical_pnl_usd            0.0 → 199.11
  top_wallets                      [] → 3 entries

`hit_rate: 0.0` deliberately stays. It is a required non-nullable number in the
schema, and the honest fix — reporting on markets RESOLVED in the window rather
than alerted in it — needs a resolution-time backfill that does not exist yet.
The summary now carries the denominator so the figure cannot be misread alone.

Producer: auditmos/ogsfrompoly 554f537 · refs auditmos/ogsfrompoly#236, #234
… express "0 of 0"

CI caught what the correction could not fix, and it was right to. With
`hypothetical_pnl_usd` restored to its true 199.11, `findPublishedInconsistencies`
fires:

  alert_count=263 with hit_rate=0 but hypothetical_pnl_usd=199.11 > 0
  (0 resolved alerts collapsing 0/0 → 0?)

That guard's own comment already names the diagnosis — "the tell of a `0/0 → 0`
resolution collapse" — and that is exactly what this window is. It could not fire
before only because `hypothetical_pnl_usd` was ALSO 0, from the upstream bug that
left the producer reading a warehouse without the derived tables. Fixing that made
the pathology visible, which is the guard working as designed.

The guard cannot be satisfied honestly under `schema_version: 1`. There is no
field that separates "0 of 0 resolved" from "0 of 240 resolved", so a truthful
statement for an unresolved window is unpublishable: either `hit_rate: 0` asserts
a 0% success rate that did not happen, or the PnL is falsified back to 0 to match.
Neither is acceptable, and weakening the guard would discard the one check that
caught this.

So the statement goes to `draft: true`. Drafts are exempt from the guard and
excluded by every consumer (`excludeDrafts` in the index, feeds and statements.md
— there is no per-statement route), which takes the false "263 alerts, hit rate
0.00" claim off the live site immediately. The corrected numbers stay in the file
so republishing is a one-line flip once the schema can carry a resolved count.

Unblocking work: add `resolved_count` and bump to `schema_version: 2` across the
producer and this repo, then teach the sanity check that `resolved_count === 0`
makes hit_rate vacuous rather than inconsistent. Tracked as
auditmos/ogsfrompoly#236.

Refs auditmos/ogsfrompoly#234, auditmos/ogsfrompoly#236
@tkowalczyk tkowalczyk changed the title fix(statements): correct the 2026-07-28 weekly — unresolved window, not a 0% hit rate fix(statements): correct + unpublish the 2026-07-28 weekly (unresolved window, not a 0% hit rate) Jul 29, 2026
@tkowalczyk
tkowalczyk merged commit fc5c9a7 into main Jul 29, 2026
1 check passed
@tkowalczyk
tkowalczyk deleted the fix/2026-07-28-weekly-unresolved-window branch July 29, 2026 10:44
github-actions Bot pushed a commit that referenced this pull request Jul 29, 2026
## [1.17.1](v1.17.0...v1.17.1) (2026-07-29)

### Bug Fixes

* **statements:** correct + unpublish the 2026-07-28 weekly (unresolved window, not a 0% hit rate) ([#58](#58)) ([fc5c9a7](fc5c9a7)), closes [auditmos/ogsfrompoly#236](https://github.com/auditmos/ogsfrompoly/issues/236) [#234](https://github.com/auditmos/ogsfrompoly-lp/issues/234) [auditmos/ogsfrompoly#234](https://github.com/auditmos/ogsfrompoly/issues/234) [auditmos/ogsfrompoly#236](https://github.com/auditmos/ogsfrompoly/issues/236)
tkowalczyk added a commit that referenced this pull request Jul 30, 2026
…minator (#60)

`resolved_count` landed in #59, so an unresolved window is expressible: adding
`resolved_count: 0` clears `findHitRateInconsistencies`, which is what held this
file at `draft: true`. The numbers are unchanged from #58 — regenerated from the
warehouse at auditmos/ogsfrompoly@554f537 over the identical window — and its
summary's "0 of 263 resolved" is the count itself, so the field states what the
prose already did.

Publishing it would otherwise have put `Hit rate 0%` back on the live page beside
this stat's own caption, "0.50 ≈ a coin flip; above 0.50 is signal" — the exact
misreading auditmos/ogsfrompoly#236 exists to stop, on the most-skimmed surface.
So the stat now reads `pending` when the denominator is known-empty, with wording
that replaces the standing caption rather than appending a caveat to it.

`formatHitRate` / `describeHitRate` key off the denominator, never the rate, so a
genuine 0-in-favour-of-240 week still reads as a real 0%. Absent stays unknown,
never zero: every statement published before the field existed renders exactly as
before, verified across all twelve. A known non-zero denominator is now named too
("Measured over the 240 of 263 alerts that have resolved") — publishing a rate
without one is what let the 0.00 pass unnoticed for a week.

Refs auditmos/ogsfrompoly#236

Co-authored-by: Tomasz Kowalczyk <tkow@Mac-Studio-Tomasz.local>
github-actions Bot pushed a commit that referenced this pull request Jul 30, 2026
## [1.18.1](v1.18.0...v1.18.1) (2026-07-30)

### Bug Fixes

* **statements:** republish the 2026-07-28 weekly, and render the denominator ([#60](#60)) ([792ecdd](792ecdd)), closes [#58](#58) [auditmos/ogsfrompoly#236](https://github.com/auditmos/ogsfrompoly/issues/236) [auditmos/ogsfrompoly#236](https://github.com/auditmos/ogsfrompoly/issues/236)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant