Skip to content

chore(observability): admin metrics & error logging (#5 review)#17

Merged
DevHDI merged 5 commits into
mainfrom
chore/observability-improvements
May 13, 2026
Merged

chore(observability): admin metrics & error logging (#5 review)#17
DevHDI merged 5 commits into
mainfrom
chore/observability-improvements

Conversation

@DevHDI

@DevHDI DevHDI commented May 13, 2026

Copy link
Copy Markdown
Owner

Summary

Addresses 3 observability findings from the Raven's Verdict review on bmad-method-ui#5. All three were UX/visibility issues that presented fabricated or invisible information; the fixes restore honesty without changing behavior.

Findings addressed

# Severity Issue Fix
#17 🟢 Minor Admin dashboard showed 0% for "Parsing Errors" — a hardcoded zero presented as a real KPI Return null from getUsageMetrics, render "N/A — Not yet tracked" in the stats card
#16 🟢 Minor When a detectBmadRepos GraphQL batch failed, the catch handler set every repo in that chunk to false (indistinguishable from confirmed-absent) Change return type to boolean | null, mark failed batches as null so callers can distinguish "unknown" from "confirmed no"
#15 🟡 Moderate fetchFileContent wrapped provider.extendBmadDirs(topSegment) in a silent catch {} — the downstream assertSafePath is still the authoritative invariant but logs had zero signal Replace the silent catch with a console.warn capturing owner/name/segment/error; no behavior change

Commits

  • 85c527c fix(admin): show parsingErrorRate as N/A until tracking exists
  • 8090e03 fix(actions): represent detectBmadRepos batch failures as null
  • 14113bb chore(actions): log extendBmadDirs failures instead of silently catching

Verification

  • pnpm tsc --noEmit — clean
  • pnpm lint — 0 errors (2 pre-existing warnings unrelated)
  • pnpm test — 151/151 passing

Note on #16

The single in-repo consumer (add-repo-dialog.tsx) does bmadResult.data[r.fullName] ?? false at the boundary, so the null values are coerced to false for the existing badge rendering — current behavior is preserved. Surfacing the unknown count to the user (e.g. "BMAD detection failed for 12 repos") was kept out of this PR because the dialog is also modified by PR #14 for the LIMIT_EXCEEDED warning, and the merge story is cleaner if the UI surface comes as a follow-up after both PRs land.

Context

PR 4 of 4 addressing bmad-method-ui#5 review. This completes the upstream fix series.

Findings not addressed (intentional):

Follow-up issues tracked: #12 CSP and #13 role enum.

@DevHDI

DevHDI commented May 13, 2026

Copy link
Copy Markdown
Owner Author

Follow-up: actually surface the unknown state in the UI

Reviewer's finding was correct. Returning null from the server but letting the dialog do ?? false at the boundary meant the data contract was honest but the user still saw a confirmed-absent label on repos we never confirmed. That's only half of finding #16.

Fix in 56567db

In add-repo-dialog.tsx:

  • Count null values in bmadResult.data after a successful detection call
  • Track via new unknownDetectCount: number state (distinct from the planned detectWarning state in #14 so both surfaces coexist after merge)
  • When > 0, render an amber warning next to the "Detecting BMAD files..." status: "BMAD detection failed for N repositor(y/ies) — these may be mis-labelled as 'no BMAD'."
  • The repo list itself stays fully usable; only the misleading badge gets a disclaimer.

The repo's hasBmad still coerces null → false at the boundary, so existing sorting/display logic is unchanged (no need to widen GitHubRepo.hasBmad to boolean | null in this PR).

Verification

  • pnpm tsc --noEmit — clean
  • pnpm lint — 0 errors (2 pre-existing warnings)
  • pnpm test — 151/151

@DevHDI

DevHDI commented May 13, 2026

Copy link
Copy Markdown
Owner Author

JSX whitespace fix

Reviewer was right — splitting the word as `repositor` + `{count === 1 ? "y" : "ies"}` rendered with a space in the middle because JSX preserves whitespace between text and expressions.

Fix in 33c5e95 (pushed)

Move the whole word inside the conditional and use explicit `{" "}` to control the space between the count and the noun:

```tsx
BMAD detection failed for {unknownDetectCount}{" "}
{unknownDetectCount === 1 ? "repository" : "repositories"} —
these may be mis-labelled as “no BMAD”.
```

Renders correctly as BMAD detection failed for 3 repositories — ….

Verification

  • pnpm tsc --noEmit — clean
  • pnpm lint — 0 errors (2 pre-existing warnings)
  • pnpm test — 151/151

DevHDI added 5 commits May 14, 2026 00:00
The admin dashboard displayed `0%` for "Parsing Errors" with the
description "Target KPI: < 1%", suggesting the metric was working when
in fact no parsing-error log table exists yet — the value was a
hardcoded zero. Admins reading "0%" would conclude the system has no
parsing errors when the reality is the data is not collected.

Change `parsingErrorRate` from `number` to `number | null` in the
UsageMetrics interface, return `null` from getUsageMetrics(), and have
the stats card render "N/A" with the description "Not yet tracked"
when the value is null. The existing display path (number + "%") is
preserved for when a real metric is wired up.

Addresses finding #17 from bmad-method-ui#5 review.
When a GraphQL batch fails (transient network error, GitHub hiccup),
the catch handler set every repo in that chunk to `false`, which is
the same value used for "confirmed no BMAD". A caller asking the UI
"does this repo have BMAD?" got "no" — when the truthful answer was
"we don't know, the API call failed".

Change the result map type from `Record<string, boolean>` to
`Record<string, boolean | null>`:
  - true  → BMAD detected
  - false → confirmed absent
  - null  → detection failed for this repo (unknown)

The single in-repo consumer (add-repo-dialog) already coerces
`bmadResult.data[r.fullName] ?? false` at the boundary so existing
behavior is preserved; surfacing the unknown count in the UI is a
follow-up that can build on this contract.

Addresses finding #16 from bmad-method-ui#5 review.
fetchFileContent wraps provider.extendBmadDirs in `try { ... } catch {}`
with a comment relying on the downstream getFileContent →
assertSafePath check to enforce the whitelist invariant. That
reasoning is still correct (assertSafePath refuses any path whose
top-level segment is not in the whitelist, so an extension that
failed to apply cannot be exploited), but discarding the error gives
zero visibility when the validation path is exercised in production.

Replace the silent catch with a console.warn that captures the
owner / name / segment / error message. No behavior change — the
authoritative refusal still happens in assertSafePath if the segment
was not added.

Addresses finding #15 from bmad-method-ui#5 review.
Commit 8090e03 had the server return `null` for repos in a failed
batch, but add-repo-dialog still applied `?? false` at the boundary
and rendered the repo identically to a confirmed "no BMAD". The data
contract was honest but the user saw the same lie as before.

Count the null values returned by detectBmadRepos and surface that
count via a new `unknownDetectCount` state. When > 0 the dialog
shows an amber warning ("BMAD detection failed for N
repositor(y/ies)…") in the same area as the existing detection
status. The repo list itself stays usable; only the misleading badge
gets the disclaimer.

State name is intentionally distinct from existing/planned warning
state in PR #14 so the two surfaces can coexist when
both PRs merge.

Addresses the follow-up review on 8090e03.
The previous JSX split the word as `repositor` + `{count === 1 ? "y"
: "ies"}`. React preserves whitespace between text and expressions,
so the rendered output had a space in the middle of the word
("repositor y" / "repositor ies").

Move the whole word into the conditional ("repository" /
"repositories") and use an explicit `{" "}` to control the space
between the count and the word. The dash and trailing text are also
on their own lines, which JSX collapses to single spaces correctly.

Addresses the follow-up review on 56567db.
@DevHDI DevHDI force-pushed the chore/observability-improvements branch from 0742e9d to df08f8d Compare May 13, 2026 22:02
@DevHDI DevHDI merged commit 575f6f5 into main May 13, 2026
1 check passed
@DevHDI DevHDI deleted the chore/observability-improvements branch May 13, 2026 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant