Skip to content

Python: Improve python sample validation workflow - #7350

Merged
chetantoshniwal merged 38 commits into
mainfrom
workflow/improve-python-sample-validation-workflow
Aug 3, 2026
Merged

Python: Improve python sample validation workflow#7350
chetantoshniwal merged 38 commits into
mainfrom
workflow/improve-python-sample-validation-workflow

Conversation

@TaoChenOSU

@TaoChenOSU Tao Chen (TaoChenOSU) commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Motivation & Context

Addresses #7348

Description & Review Guide

  • What are the major changes?
    • Temporarily disables automatic validation on a subset of samples due to GHCP instability.
    • Breaks jobs into multiple ones so that they are easier to be managed and become more stable. For example, this PR breaks the hosting sample job into Foundry hosting agent samples and other hosting samples.
    • Adds playbook capability where previously validated samples that remain unchanged (gated by the sample hashes) such that the sample can be replayed deterministically. GHCP will only get invoked if a sample is changed or replay fails.
    • Fixes a few samples that are broken.
    • Remove the Foundry skill sample and only maintain the toolbox skill sample
  • What is the impact of these changes?
    • A more stable and efficient sample validation workflow.
  • What do you want reviewers to focus on?
    • The fixes on the broken samples and format of the playbook.

Run result:

image

Related Issue

Fixes #7348

We plan to reenable the remaining jobs in the future. Item tracking this effort: #7349

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

@TaoChenOSU Tao Chen (TaoChenOSU) self-assigned this Jul 27, 2026
@chetantoshniwal
chetantoshniwal added this pull request to the merge queue Aug 3, 2026
Merged via the queue into main with commit 8d37916 Aug 3, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs github_actions Pull requests that update GitHub Actions code python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable Automated Sample Validation and file issues on failure for P0 samples including FHA & Toolbox

6 participants