feat: add flaky board, test detail and rca panel to the dashboard#114
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #25
Closes #24
Closes #26
Summary
Adds the three analytics screens on top of the dashboard foundation. Together they answer the SDET's actual questions: which tests are eroding trust, what is this test's story, and why did this failure happen.
Flaky board (#25)
Tests ranked by score, each row carrying the signals behind it: flip rate, pass-on-rerun, last flaked, and a trend arrow. Quick filters for quarantine candidates and getting worse, and every row links to the test detail page.
Test detail (#24)
The
testHistoryPointcontract gainedexecutionId,branch,attempt,errorMessageandhasRca— without them the execution list and the RCA link could not be built. The API'stest.getgains the same richness.RCA panel (#26)
Rendered for the selected failed execution: the error message, then the analysis — summary, likely cause, suggested action, confidence, model and token cost, plus prior occurrences. It is explicitly labelled as model-generated ("treat it as a lead, not a verdict").
When no analysis exists — which is the case today, since AI RCA is deliberately last in the plan — the panel explains that it runs asynchronously and is budget-gated, rather than showing a broken or empty box.
Deliberately not included: the thumbs up/down feedback affordance from the issue scope. It needs storage and can only be exercised once real reports exist, so building it now would ship UI that no code path can reach. It belongs in the AI RCA PR, where it can be built and tested for real.
Verification
Seeded eight runs with deliberately different behaviours and drove them through real OTLP → API → queue → worker, then checked the rendered pages:
auth/logs in(fails and recovers on the same commit)checkout/completes payment(regressed recently)307 → /, no session307 → /sign-inTesting
tsconfig.tsbuildinfo, a build artifact that had been committed