Skip to content

fix(services): share one positive/negative/pending split across the recommendation aggregators - #9890

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
bitfathers94:fix/issue-9701
Jul 29, 2026
Merged

fix(services): share one positive/negative/pending split across the recommendation aggregators#9890
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
bitfathers94:fix/issue-9701

Conversation

@bitfathers94

Copy link
Copy Markdown
Contributor

fix(services): share one positive/negative/pending split across the recommendation aggregators

buildRecommendationOutcomeCalibration and qualityTotals both read
agent_recommendation_outcomes and expose positiveRate, but classified the
same states oppositely: the quality report counted stale/ignored as negative
and returned 0 on an empty ledger, so 5 accepted + 5 stale read as a 100%
success rate on the calibration signal and a 50% failure rate on the report.

Group the states once in outcome-calibration.ts as
RECOMMENDATION_{POSITIVE,NEGATIVE,PENDING}_STATES and import them into the
quality report. stale/ignored are now pending (dropped from the positiveRate
denominator and from failureCategories, added as a RecommendationQualityTotals.pending
count), and positiveRate is number | null, returning null on a zero denominator
to match the calibration module. Guard the operator-dashboard tile on the null.

Closes #9701

…ecommendation aggregators

buildRecommendationOutcomeCalibration and qualityTotals both read
agent_recommendation_outcomes and expose positiveRate, but classified the
same states oppositely: the quality report counted stale/ignored as negative
and returned 0 on an empty ledger, so 5 accepted + 5 stale read as a 100%
success rate on the calibration signal and a 50% failure rate on the report.

Group the states once in outcome-calibration.ts as
RECOMMENDATION_{POSITIVE,NEGATIVE,PENDING}_STATES and import them into the
quality report. stale/ignored are now pending (dropped from the positiveRate
denominator and from failureCategories, added as a RecommendationQualityTotals.pending
count), and positiveRate is number | null, returning null on a zero denominator
to match the calibration module. Guard the operator-dashboard tile on the null.
@bitfathers94
bitfathers94 requested a review from JSONbored as a code owner July 29, 2026 19:10
@loopover-orb

loopover-orb Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Tip

✅ LoopOver review result - approve/merge recommended

Review updated: 2026-07-29 19:21:43 UTC

5 files · 1 AI reviewer · no blockers · readiness 93/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This PR centralizes the recommendation-outcome state classification (positive/negative/pending) into shared exported constants in outcome-calibration.ts and has recommendation-quality-report.ts import and use them, fixing the #9701 bug where stale/ignored outcomes were treated as failures in the quality report but excluded (pending) in the calibration module, producing contradictory positiveRate values off the same ledger. The change is well-scoped: positiveRate is now `number | null` in both modules (null on empty denominator), the operator-dashboard tile is correctly updated to guard on null instead of the old `.empty` flag, and a new dedicated test directly compares `buildRecommendationOutcomeCalibration` and `buildRecommendationQualityReportFromOutcomes` output on the same stale-heavy and empty ledgers to prevent regression. One subtlety worth a maintainer note: `RecommendationQualityTotals.total` (positive+negative, excluding pending) and `RecommendationOutcomeCalibration.total` (all scoped outcomes, including pending) still mean different things despite sharing the same state classification — this is pre-existing behavior, not introduced by this diff, but the PR's framing ('share one split') could mislead a reader into assuming full parity between the two `total` fields.

Nits — 5 non-blocking
  • outcome-calibration.ts and recommendation-quality-report.ts now share the positive/negative/pending classification but still diverge on what `total` means (calibration's `total` = all scoped outcomes incl. pending; quality report's `total` = positive + negative only) — worth a one-line doc comment on `RecommendationQualityTotals.total` to head off future confusion given the PR's 'share one split' framing.
  • recommendation-quality-report.ts:191-196 (failureCategoryRows) drops the `stale`/`ignored` category rows entirely rather than replacing them with a `pending` category row — the count survives on `totals.pending` but per-repo/failure-category granularity for pending outcomes is lost from `failureCategories`; confirm that's intended and not something callers relied on.
  • The 'rejected only' and 'mixed' test cases in recommendation-quality-report.test.ts (lines ~130-158) weren't updated to assert the new `pending` field even though they exercise `qualityTotals`, though `toMatchObject` makes this a coverage gap rather than a failure.
  • Add a short comment above `RecommendationQualityTotals.total` clarifying it excludes pending, mirroring the comment already added for the shared state constants in outcome-calibration.ts:15-18.
  • Consider asserting `report.totals.pending` explicitly in the pre-existing 'rejected only'/'mixed' tests (recommendation-quality-report.test.ts) now that the field exists, for symmetry with the new orb(calibration): two aggregators over one ledger publish contradictory positiveRate values #9701 regression tests.

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #9701
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ⚠️ 20/25 Preflight is ready, but the PR body does not name the validation run.
Contributor workload ✅ 10/10 Author activity: 94 registered-repo PR(s), 65 merged, 1 issue(s).
Contributor context ✅ Confirmed Gittensor contributor bitfathers94; Gittensor profile; 94 PR(s), 1 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: moderate
Linked issue satisfaction

Addressed
The diff defines RECOMMENDATION_POSITIVE/NEGATIVE/PENDING_STATES once in outcome-calibration.ts and imports them into recommendation-quality-report.ts, removes stale/ignored from failureCategories, adds a pending count, and changes positiveRate to number|null with null on zero denominator, matching the required tests (5 accepted+5 stale agreement, empty-ledger null, no stale/ignored failure rows)

Review context
  • Author: bitfathers94
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 94 PR(s), 1 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Add validation command/output.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 29, 2026
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.36%. Comparing base (e3247ed) to head (5275dec).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9890      +/-   ##
==========================================
+ Coverage   79.28%   79.36%   +0.07%     
==========================================
  Files         281      284       +3     
  Lines       58516    58746     +230     
  Branches     6777     6856      +79     
==========================================
+ Hits        46393    46621     +228     
  Misses      11840    11840              
- Partials      283      285       +2     
Flag Coverage Δ
backend 99.13% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/services/operator-dashboard.ts 100.00% <100.00%> (ø)
src/services/outcome-calibration.ts 98.52% <100.00%> (ø)
src/services/recommendation-quality-report.ts 99.14% <100.00%> (ø)

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoopOver approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit 38e37ec into JSONbored:main Jul 29, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

orb(calibration): two aggregators over one ledger publish contradictory positiveRate values

1 participant