Skip to content

ci: reflect trigger reason in pipeline run name and warn on ADO YAML changes - #62

Merged
saurabhrb merged 10 commits into
mainfrom
users/saurabhrb/eval-pipeline-runname-and-warn
Jul 21, 2026
Merged

ci: reflect trigger reason in pipeline run name and warn on ADO YAML changes#62
saurabhrb merged 10 commits into
mainfrom
users/saurabhrb/eval-pipeline-runname-and-warn

Conversation

@saurabhrb

Copy link
Copy Markdown
Contributor

Summary

Two CI-side improvements to the eval pipeline plumbing — no skill or plugin changes.

Pipeline run name (.azdo/DataversePluginEvals_PR.yml)

PR run buildId=20097004 still rendered as ... • plugin-main • ... because the YAML used a label parameter (default: 'plugin-main') in the run name format. Parameter defaults are baked at queue time and auto-triggered (pr / ci) runs can't override them — only "Run pipeline" manual queues can — so every auto run leaks the default label.

Fix: drop the label parameter and use Azure Pipelines runtime variables instead:

name: $(Date:yyyyMMdd)$(Rev:.r) • ${{ parameters.mode }} • $(Build.Reason)$(System.PullRequest.PullRequestNumber) • ${{ parameters.testFile }}

Resulting run names:

  • PR build → 20260516.1 • dry-run • PullRequest61 • dv_data.biceval.json
  • CI on main20260516.1 • dry-run • IndividualCI • dv_data.biceval.json
  • Manual queue → 20260516.1 • dry-run • Manual • dv_data.biceval.json

The mode and testFile parameters stay — those are user-facing knobs for manual runs.

ADO YAML change warning workflow (.github/workflows/ado-pipeline-yaml-pr-warning.yml)

Modeled on the equivalent workflow in microsoft/PowerPlatform-DataverseClient-Python. Triggers on PRs that touch .azdo/DataversePluginEvals_PR.yml and:

  1. Emits a ::warning:: line in the workflow log
  2. Adds a Step Summary
  3. Posts a resolvable PR review comment pointing at the file, reminding the reviewer to re-enable / approve the updated YAML in ADO (definition 32010) post-merge

Closes the manual recall step that bit us when the pipeline default branch was migrated to main.

Not in scope

  • No skill files touched → no version bump required
  • Static eval (python .github/evals/static_checks.py) passes

…changes

- Drop the `label` parameter (default 'plugin-main' was leaking into auto-triggered run names) and use Build.Reason + System.PullRequest.PullRequestNumber so PR runs show e.g. `PullRequest61` and CI runs show `IndividualCI`. Manual queues will show `Manual` (run owner can still describe via the existing mode/testFile params).

- Add a PR warning workflow modeled on PowerPlatform-DataverseClient-Python that fires when `.azdo/DataversePluginEvals_PR.yml` changes. Surfaces a workflow summary and a resolvable PR review comment so the reviewer remembers to re-enable / approve the pipeline YAML in ADO (definition 32010) after merge.
@saurabhrb
saurabhrb requested a review from a team May 16, 2026 02:23
Reverts the no-op newline appended to dv-overview/SKILL.md in commit c0b490c (which was only there to force the skill-evals CI to fire on the eval pipeline PR).
Comment thread .azdo/DataversePluginEvals_PR.yml
@microsoft microsoft deleted a comment from github-actions Bot May 16, 2026
Workflow now checks PR review threads for an unresolved comment carrying the hidden marker <!-- ado-pipeline-yaml-pr-warning:DataversePluginEvals_PR --> and skips posting if one already exists. If the prior comment was resolved (or none exists), a new one is posted. Prevents the warning from re-appearing on every commit push during an in-progress PR.
arorashivam96
arorashivam96 previously approved these changes May 22, 2026
@saurabhrb
saurabhrb enabled auto-merge (squash) June 8, 2026 00:11
@saurabhrb
saurabhrb requested a review from arorashivam96 July 20, 2026 21:01
suyask-msft
suyask-msft previously approved these changes Jul 20, 2026
Updated the SKILL.md file to enhance the description and clarify the functionalities of the dv-overview skill, including detailed tool capabilities, hard rules, and the change lifecycle for Dataverse/Power Platform work.
@saurabhrb
saurabhrb dismissed stale reviews from suyask-msft and arorashivam96 via ec11f9f July 21, 2026 20:40
suyask-msft
suyask-msft previously approved these changes Jul 21, 2026
Comment thread .github/workflows/skill-evals.yml
@saurabhrb
saurabhrb removed the request for review from arorashivam96 July 21, 2026 22:40
@saurabhrb
saurabhrb merged commit f4d02be into main Jul 21, 2026
6 of 7 checks passed
@saurabhrb
saurabhrb deleted the users/saurabhrb/eval-pipeline-runname-and-warn branch July 21, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants