Fix agentic workflow markers removed from safe outputs - #132011
Open
vitek-karas wants to merge 2 commits into
Open
Fix agentic workflow markers removed from safe outputs#132011vitek-karas wants to merge 2 commits into
vitek-karas wants to merge 2 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2038caaa-4cae-49b0-84e0-a077f5c91aa0
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2038caaa-4cae-49b0-84e0-a077f5c91aa0
|
Azure Pipelines: Successfully started running 1 pipeline(s). 15 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 15 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates several agentic workflow prompts and regenerated lockfiles to stop using agent-authored HTML comments (<!-- ... -->) as persisted markers (since safe-output sanitization strips them), replacing them with either safe-outputs.data structured identifiers or stable visible Markdown fields.
Changes:
- Introduces
safe-outputs.dataschemas and corresponding prompt guidance for workflows that need machine-readable identity/deduplication. - Updates the shared KBE templates and eval specs to use collapsed
<details>blocks for authoring guidance and workflow-owned match-count metadata (while preserving the “exactly one fenced JSON block” KBE constraint). - Regenerates the affected
*.lock.ymlworkflows to the newer gh-aw compiler/runtime versions.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/shared/create-kbe.instructions.md | Moves KBE guidance + verification metadata into collapsed <details> blocks while keeping exactly one JSON signature block. |
| .github/workflows/README.md | Documents that safe outputs strip HTML/XML comments and recommends safe-outputs.data or visible fields. |
| .github/workflows/evals/README.md | Updates eval expectations to require collapsed guidance + metadata blocks. |
| .github/workflows/evals/ci-failure-scan.eval.yaml | Updates graders to validate the new <details>-based KBE template requirements. |
| .github/workflows/closed-issue-reference-check.md | Adds safe-outputs.data schema + updates dedup logic to detect prior advisory comments via structured or legacy visible markers. |
| .github/workflows/ci-failure-scan.md | Updates scanner match-count gating language to require the new collapsed verification block. |
| .github/workflows/ci-failure-scan.lock.yml | Regenerated workflow lockfile (compiler/runtime/tooling version and infra script updates). |
| .github/workflows/ci-failure-scan-feedback.md | Updates feedback workflow prompt to identify artifacts via structured data or legacy visible blocks; stores tracker metadata in visible collapsed block. |
| .github/workflows/ci-failure-scan-feedback.lock.yml | Regenerated workflow lockfile (compiler/runtime/tooling version and infra script updates). |
| .github/workflows/ci-failure-fix.md | Adds safe-outputs.data schema + updates prompt guidance for dedup/identity via structured data with legacy fallback. |
| .github/aw/actions-lock.json | Updates pinned gh-aw setup action entry to v0.83.5 and removes older entries. |
| .github/agents/agentic-workflows.agent.md | Adds repo guidance explaining sanitization behavior and when to use safe-outputs.data vs visible fields. |
Comment on lines
+157
to
+161
| data_workflow='"workflow_artifact": "closed-issue-reference-check"' | ||
| data_kind='"artifact_kind": "advice"' | ||
| legacy_call_id='gh-aw-workflow-call-id: dotnet/runtime/closed-issue-reference-check' | ||
| legacy_workflow_id='gh-aw-workflow-id: closed-issue-reference-check' | ||
| legacy_agentic_id='workflow_id: closed-issue-reference-check' |
| 5. **At most one open `[ci-fix]` PR and one `ci-fix` loop-in comment per KBE, ever.** Before opening a PR, run the Step 3 PR dedup. Before commenting, search for a prior `ci-fix` comment on that KBE (the marker `<!-- ci-fix:handoff -->`). If a comment already exists, skip with `-> skipped: loop-in comment already posted`. Build Analysis tracks occurrence counts in the KBE body; do not add occurrence chatter. | ||
| 6. **Every PR title starts with `[ci-fix] `.** Every PR body and every loop-in comment carries the artifact marker block (see Output markers). | ||
| 5. **At most one open `[ci-fix]` PR and one `ci-fix` loop-in comment per KBE, ever.** Before opening a PR, run the Step 3 PR dedup. Before commenting, search for a prior `ci-fix` handoff using the structured-data or legacy-visible signatures in Step 3.6. If a comment already exists, skip with `-> skipped: loop-in comment already posted`. Build Analysis tracks occurrence counts in the KBE body; do not add occurrence chatter. | ||
| 6. **Every PR title starts with `[ci-fix] `.** Every PR body and every loop-in comment carries the visible artifact block and the matching `safe-outputs.data` object (see Output markers). |
PureWeen
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Several agentic workflows used HTML comments as machine-readable markers. That does not work: gh-aw removes agent-provided HTML comments when it sanitizes safe-output text. The comments were therefore not present in the posted issue or PR content, so later workflow runs could not reliably recognize earlier work. On #128405, this allowed
ci-failure-fixto post two different handoff comments.This change replaces those comments with markers that survive publishing:
safe-outputs.datawhere the output type supports structured data.Existing visible markers are still recognized so older workflow output continues to work.
Changes by workflow
ci-failure-fix: Adds structured identifiers to fix PRs, help-wanted PRs, and handoff comments. Deduplication now checks every comment instead of assuming comments are at a fixed position, and it still recognizes older visible markers.closed-issue-reference-check: Adds structured identifiers to advisory comments. The pre-check skips issues that contain either the new structured marker or the older workflow-specific marker and advisory heading.ci-failure-scan: Moves KBE authoring guidance and the verified-match count into collapsed, clearly labeled sections. KBE bodies still contain exactly one JSON block, as required by Build Analysis.ci-failure-scan-feedback: Reads the newci-failure-fixidentifiers while keeping compatibility with older markers. Its tracker identity and window are stored in a collapsed visible section, and tracker updates explicitly replace the body instead of appending to it.The shared workflow guidance now documents that HTML comments are removed and explains when to use structured data or visible fields. The affected generated workflows were refreshed with gh-aw v0.83.5.
Related change
#131996 is the repository-wide mechanical refresh to gh-aw v0.83.5. This PR contains the workflow behavior and authoring changes, and regenerates the affected workflows with the same compiler version.
Validation
Note
This PR description was generated with GitHub Copilot.