Skip to content

feat: add flaky board, test detail and rca panel to the dashboard#114

Merged
AKogut merged 1 commit into
mainfrom
feat/dashboard-analytics
Jul 21, 2026
Merged

feat: add flaky board, test detail and rca panel to the dashboard#114
AKogut merged 1 commit into
mainfrom
feat/dashboard-analytics

Conversation

@AKogut

@AKogut AKogut commented Jul 21, 2026

Copy link
Copy Markdown
Owner

Closes #25
Closes #24
Closes #26

Summary

Adds the three analytics screens on top of the dashboard foundation. Together they answer the SDET's actual questions: which tests are eroding trust, what is this test's story, and why did this failure happen.

Flaky board (#25)

Tests ranked by score, each row carrying the signals behind it: flip rate, pass-on-rerun, last flaked, and a trend arrow. Quick filters for quarantine candidates and getting worse, and every row links to the test detail page.

Test detail (#24)

  • Header with identity, current score (colour-coded by severity) and quarantine flag
  • Sparkline of the last 60 executions — pass renders short, fail full-height, flaky amber — so a flaky test is recognisable at a glance
  • Reason codes panel explaining the score in words, straight from the scoring engine
  • Execution table with attempts, retries, commit, branch, duration and timing
  • Failed executions link into the RCA panel

The testHistoryPoint contract gained executionId, branch, attempt, errorMessage and hasRca — without them the execution list and the RCA link could not be built. The API's test.get gains the same richness.

RCA panel (#26)

Rendered for the selected failed execution: the error message, then the analysis — summary, likely cause, suggested action, confidence, model and token cost, plus prior occurrences. It is explicitly labelled as model-generated ("treat it as a lead, not a verdict").

When no analysis exists — which is the case today, since AI RCA is deliberately last in the plan — the panel explains that it runs asynchronously and is budget-gated, rather than showing a broken or empty box.

Deliberately not included: the thumbs up/down feedback affordance from the issue scope. It needs storage and can only be exercised once real reports exist, so building it now would ship UI that no code path can reach. It belongs in the AI RCA PR, where it can be built and tested for real.

Verification

Seeded eight runs with deliberately different behaviours and drove them through real OTLP → API → queue → worker, then checked the rendered pages:

Test Score Reason codes
auth/logs in (fails and recovers on the same commit) 0.80 SAME_SHA_VARIANCE, PASS_ON_RERUN, HIGH_FLIP_RATE, FAIL_ISOLATION
checkout/completes payment (regressed recently) 0.23 FAIL_ISOLATION
two stable tests 0.02 STABLE
  • The board rendered all four in the right order, and the getting worse filter selected exactly the test that was deliberately regressed
  • Test detail rendered 12 sparkline bars, all four reason codes and the execution table
  • RCA panel verified in both states: the empty state, and a populated report rendering summary, cause, action, 78% confidence, model and prior occurrences
  • Access control re-checked on the new routes: foreign project 307 → /, no session 307 → /sign-in

Testing

  • 42/42 turbo tasks, 128 tests
  • Also stops tracking tsconfig.tsbuildinfo, a build artifact that had been committed

@AKogut AKogut self-assigned this Jul 21, 2026
@AKogut
AKogut merged commit 65d7f6f into main Jul 21, 2026
2 checks passed
@AKogut
AKogut deleted the feat/dashboard-analytics branch July 21, 2026 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

apps/web — RCA panel on failed execution apps/web — Flaky board apps/web — Test detail page with history sparkline + reason codes

1 participant