Skip to content

Python: Add flaky test trend reporting to CI workflows - #5339

Closed
Giles Odigwe (giles17) wants to merge 9 commits into
microsoft:mainfrom
giles17:flaky-test-report
Closed

Python: Add flaky test trend reporting to CI workflows#5339
Giles Odigwe (giles17) wants to merge 9 commits into
microsoft:mainfrom
giles17:flaky-test-report

Conversation

@giles17

@giles17 Giles Odigwe (giles17) commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a flaky test trend reporting system to CI, following the same pattern as sample validation reports.

What it does

  • After all integration test jobs complete, a new Flaky Test Report job parses the JUnit XML results from each provider
  • Aggregates results with cached history from previous runs (up to 5 runs)
  • Posts a markdown trend table to the GitHub Job Summary showing per-test status over time

Changes

  • python/scripts/flaky_report/ - New package: JUnit XML parser + trend report generator
  • .github/workflows/python-merge-tests.yml - Upload artifacts from 6 integration test jobs + new aggregation job
  • .github/workflows/python-integration-tests.yml - Same changes + add --junitxml=pytest.xml (was missing)
  • Fix: Cosmos job --junitxml now uses absolute path (pre-existing bug where uv run --directory wrote XML to wrong location)

Report covers

Only the 6 integration test jobs (OpenAI, Azure OpenAI, Misc, Functions, Foundry, Cosmos). Unit tests are excluded.

How to test

Trigger Integration Tests (Manual) workflow against this branch. The Flaky Test Report job summary will show the trend table.

@giles17
Add flaky test trend reporting to CI workflows
f60a77c 
Parse JUnit XML (pytest.xml) from each integration test job and
aggregate results into a markdown trend report showing per-test
pass/fail/skip status across the last 5 runs.

Changes:
- Add python/scripts/flaky_report/ package (JUnit XML parser + trend
  report generator following the sample_validation pattern)
- Add upload-artifact steps to all 6 integration test jobs in both
  python-merge-tests.yml and python-integration-tests.yml
- Add python-flaky-test-report aggregation job with history caching
- Add --junitxml=pytest.xml to integration-tests.yml jobs (already
  present in merge-tests.yml)
- Fix Cosmos job --junitxml path (use absolute path since uv run
  --directory changes cwd)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@moonbox3 Evan Mattson (moonbox3) added the python Usage: [Issues, PRs], Target: Python label Apr 17, 2026
@github-actions github-actions Bot changed the title Add flaky test trend reporting to CI workflows Python: Add flaky test trend reporting to CI workflows Apr 17, 2026
@giles17
Fix flaky report: handle missing test results gracefully
1cd4a98 
- Guard against missing reports directory in load_current_run()
- Only run report job when at least one integration test job completed
  (skip when all jobs are skipped, e.g. on pull_request events)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@giles17
Giles Odigwe (giles17) marked this pull request as ready for review April 17, 2026 21:45
Copilot AI review requested due to automatic review settings April 17, 2026 21:45
Copilot AI reviewed Apr 17, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a Python CI “flaky test trend” report, similar in spirit to the existing sample validation trend report, by collecting per-job JUnit XML artifacts, persisting short history via Actions cache, and publishing a markdown trend table to the job summary.

Changes:

  • Introduces python/scripts/flaky_report/ to parse JUnit XML and generate a multi-run trend markdown report.
  • Updates Python CI workflows to emit pytest.xml artifacts from integration jobs and adds a final aggregation/reporting job.
  • Fixes Cosmos integration job JUnit output location by switching to an absolute --junitxml path.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
python/scripts/flaky_report/aggregate.py New aggregator/parser and markdown trend report generator for JUnit XML.
python/scripts/flaky_report/__main__.py Module entrypoint wiring python -m scripts.flaky_report to the aggregator.
python/scripts/flaky_report/__init__.py Package docstring/usage for the flaky report tool.
.github/workflows/python-merge-tests.yml Uploads per-job JUnit artifacts and adds a final “Flaky Test Report” aggregation job; fixes Cosmos junitxml path.
.github/workflows/python-integration-tests.yml Adds missing --junitxml, uploads artifacts, and adds the final aggregation/reporting job.

Comment thread python/scripts/flaky_report/aggregate.py
Comment thread python/scripts/flaky_report/aggregate.py
Comment thread python/scripts/flaky_report/aggregate.py
Comment thread .github/workflows/python-merge-tests.yml Outdated
Comment thread .github/workflows/python-integration-tests.yml Outdated
@giles17
Giles Odigwe (giles17) marked this pull request as draft April 17, 2026 21:54
@giles17
Address PR review: fix provider names and if-expression precedence
f5d7a88 
- Use explicit provider name mapping in _derive_provider() so OpenAI
  renders correctly instead of 'Openai'
- Fix operator precedence in workflow if-expressions by wrapping
  success/failure checks in parentheses

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@giles17
Giles Odigwe (giles17) marked this pull request as ready for review April 17, 2026 21:57
@giles17
Giles Odigwe (giles17) marked this pull request as draft April 17, 2026 22:12
Giles Odigwe (giles17) and others added 4 commits April 17, 2026 17:03
@giles17
Add File column and xfail detection to flaky test report
f9c8ebc 
- Add File column showing module name (e.g., test_openai_chat_client)
  to disambiguate tests with the same function name across files
- Detect pytest xfail tests in JUnit XML (type=pytest.xfail) and
  show them with a distinct warning emoji instead of skip emoji
- Update legend to include xfail explanation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@giles17
Merge branch 'main' into flaky-test-report
5083902
@giles17
Add Foundry embedding env vars to merge-tests workflow
086dbf7 
Sync the Foundry integration job in python-merge-tests.yml with
python-integration-tests.yml by adding FOUNDRY_MODELS_ENDPOINT,
FOUNDRY_MODELS_API_KEY, FOUNDRY_EMBEDDING_MODEL, and
FOUNDRY_IMAGE_EMBEDDING_MODEL. Once the repo variables/secrets
are configured, the embedding integration test will run in CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@giles17
Merge branch 'main' of https://github.com/giles17/agent-framework int…
f8b0caa 
…o flaky-test-report
@giles17
Fix File column showing class name instead of module name
e5c15b6 
When a test is inside a class, pytest writes the classname as e.g.
'pkg.test_file.TestClass'. The previous rsplit logic extracted
'TestClass' instead of 'test_file'. Now detect uppercase-starting
segments as class names and use the preceding segment instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@giles17
Address PR review: UTC timestamps, XML error handling, summary fix, d…
001d2ec 
…ocstring

- Use datetime.now(timezone.utc) for accurate UTC timestamps
- Catch ET.ParseError per-file so corrupt XML doesn't crash the report
- Remove separate 'error' key from summary (errors folded into 'failed')
- Fix _short_name docstring to show actual dotted classname::name format

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@giles17 @moonbox3