feat: add INFRASTRUCTURE classification for environmental failures#4732
Open
rnetser wants to merge 4 commits intoRedHatQE:mainfrom
Open
feat: add INFRASTRUCTURE classification for environmental failures#4732rnetser wants to merge 4 commits intoRedHatQE:mainfrom
rnetser wants to merge 4 commits intoRedHatQE:mainfrom
Conversation
The AI analysis prompt only allowed CODE ISSUE and PRODUCT BUG classifications, forcing environmental/infrastructure failures to be mislabeled as CODE ISSUE with disclaimer text. The rootcoz backend already supports INFRASTRUCTURE in OverrideClassificationLiteral. Changes: - Add INFRASTRUCTURE as a third allowed classification value - Update all environmental blocker guidance to use INFRASTRUCTURE - Add container runtime failures and missing operators to the environmental blockers list - Remove workaround guidance about forced binary-label fallbacks Signed-off-by: rnetser <rnetser@redhat.com>
Contributor
📝 WalkthroughWalkthroughThis PR expands the failure classification prompt system from a two-label binary (CODE ISSUE or PRODUCT BUG) to a three-label system that introduces INFRASTRUCTURE as a distinct classification. It adds a mandatory pre-classification check for test teardown failures and updates pattern guidance throughout to reflect this distinction and refine when each label applies. ChangesFailure Classification System Expansion
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
Report bugs in Issues Welcome! 🎉This pull request will be automatically processed with the following features: 🔄 Automatic Actions
📋 Available CommandsPR Status Management
Review & Approval
Testing & Validation
Container Operations
Cherry-pick Operations
Label Management
✅ Merge RequirementsThis PR will be automatically approved when the following conditions are met:
📊 Review ProcessApprovers and ReviewersApprovers:
Reviewers:
Available Labels
AI Features
💡 Tips
For more information, please refer to the project documentation or contact the maintainers. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Collaborator
Author
|
/approve |
Contributor
Test execution plan request detailsCRITICAL: You MUST post an inline review comment on the first changed line of the first file. As an expert software testing engineer, analyze all modified files in this PR and create a targeted test execution plan. Analysis Requirements:
Your deliverable: Test Execution Plan
Guidelines:
Hardware-Related Checks (SR-IOV, GPU, DPDK): When PR modifies fixtures for hardware-specific resources:
CRITICAL WORKFLOW COMPLETION RULES: When responding to this test execution plan request, you MUST follow these rules EXACTLY:
ABSOLUTE PROHIBITIONS (violating these creates empty/meaningless reviews):
Remember: The pull request review is visible to users. Additional comments are redundant noise. CRITICAL: Before setting "Run smoke tests: True", you MUST use shell scripts to: |
Contributor
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 22 🏁 Script executed: Length of output: 5455 🏁 Script executed: Length of output: 41 🏁 Script executed: Length of output: 676 🏁 Script executed: Length of output: 3297 |
1 similar comment
Contributor
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 22 🏁 Script executed: Length of output: 5455 🏁 Script executed: Length of output: 41 🏁 Script executed: Length of output: 676 🏁 Script executed: Length of output: 3297 |
openshift-virtualization-qe-bot
added
the
tox:verify-tc-requirement-polarion:running
openshift-virtualization-qe-bot
added
tox:verify-tc-requirement-polarion:passed
dshchedr
previously approved these changes
May 4, 2026
Collaborator
Author
|
/verified cancel |
3 tasks
…zation-tests into feat/add-infrastructure-classification
…zation-tests into feat/add-infrastructure-classification
openshift-virtualization-qe-bot-4
added
size/M
and removed
lgtm-coderabbitai[bot]
size/S
needs-rebase
labels
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@JOB_INSIGHT_FAILURE_HISTORY_ANALYSIS_PROMPT.md`:
- Around line 1-7: The markdown headings are indented and the file lacks a
top-level heading at the start; remove leading spaces and left-align all
headings so they begin at column 0, convert the intended main title to a
top-level heading "# Pre-Classification Check: Did a Previous Test Break the
Cluster?" (remove the surrounding HTML comment block so the title is real), and
ensure the subsequent heading "## MANDATORY: Before classifying any failure,
answer this question first:" and the other flagged headings (the ones matching
the same heading texts elsewhere) are also left-aligned with no leading
indentation to satisfy MD041/MD023 lint rules.
- Around line 35-40: Update the teardown-root-cause guidance so the root
teardown classification can be CODE ISSUE, PRODUCT BUG, or INFRASTRUCTURE:
modify the sentence under "The test whose teardown failed" to include
INFRASTRUCTURE as a valid classification (e.g., "Classify based on why its
teardown failed — CODE ISSUE, PRODUCT BUG, or INFRASTRUCTURE"), and ensure the
later rule for "All other tests that failed after it" still instructs using the
same classification and reason format ("Caused by [test_name] teardown failure —
[resource] was not reverted.") so INFRASTRUCTURE is propagated the same way as
the other labels.
In `@JOB_INSIGHT_PROMPT.md`:
- Around line 44-46: The PR description is missing required PR template headers
causing validation failures; update the PR body to restore the mandatory
sections exactly as specified: add the headers "##### What this PR does / why we
need it:", "##### Which issue(s) this PR fixes:", "##### Special notes for
reviewer:", and "##### jira-ticket:" (each present even if empty), and populate
"What this PR does / why we need it:" with meaningful content summarizing the
change; ensure these exact header strings appear in the PR description so the
repo's PR Template Validation rules recognize them.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: d8c15a18-a5b3-44af-a58b-b1d543a692e3
📒 Files selected for processing (2)
JOB_INSIGHT_FAILURE_HISTORY_ANALYSIS_PROMPT.mdJOB_INSIGHT_PROMPT.md
Comment on lines
+1
to
+7
| <!-- To be used with https://github.com/myk-org/rootcoz | ||
| Complements the main analysis prompt — history-aware classification. | ||
| --> | ||
|
|
||
| # Pre-Classification Check: Did a Previous Test Break the Cluster? | ||
|
|
||
| ## MANDATORY: Before classifying any failure, answer this question first: |
Contributor
There was a problem hiding this comment.
LOW: Fix markdown heading structure to satisfy lint and rendering consistency
Static analysis flags heading formatting (MD041, MD023): headings are indented and the first line is not a top-level heading. Left-aligned headings improve predictable rendering and keep CI lint clean.
Proposed fix
- # Pre-Classification Check: Did a Previous Test Break the Cluster?
+# Pre-Classification Check: Did a Previous Test Break the Cluster?
...
- ## MANDATORY: Before classifying any failure, answer this question first:
+## MANDATORY: Before classifying any failure, answer this question first:
...
- ## How to Check
+## How to Check
...
- ## Classification
+## Classification
...
- ## When This Check Does NOT Apply
+## When This Check Does NOT ApplyAlso applies to: 14-14, 33-33, 42-42
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 1-1: First line in a file should be a top-level heading
(MD041, first-line-heading, first-line-h1)
[warning] 5-5: Headings must start at the beginning of the line
(MD023, heading-start-left)
[warning] 7-7: Headings must start at the beginning of the line
(MD023, heading-start-left)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@JOB_INSIGHT_FAILURE_HISTORY_ANALYSIS_PROMPT.md` around lines 1 - 7, The
markdown headings are indented and the file lacks a top-level heading at the
start; remove leading spaces and left-align all headings so they begin at column
0, convert the intended main title to a top-level heading "# Pre-Classification
Check: Did a Previous Test Break the Cluster?" (remove the surrounding HTML
comment block so the title is real), and ensure the subsequent heading "##
MANDATORY: Before classifying any failure, answer this question first:" and the
other flagged headings (the ones matching the same heading texts elsewhere) are
also left-aligned with no leading indentation to satisfy MD041/MD023 lint rules.
Comment on lines
+35
to
+40
| - **The test whose teardown failed**: Classify based on why its teardown failed — | ||
| CODE ISSUE if the cleanup logic is wrong, PRODUCT BUG if the product blocked | ||
| the revert. | ||
| - **All other tests that failed after it**: Use the **same classification** as the | ||
| root-cause test. In the reason, state: "Caused by [test_name] teardown failure — | ||
| [resource] was not reverted." |
Contributor
There was a problem hiding this comment.
HIGH: Root teardown classification still excludes INFRASTRUCTURE
Line 35–37 limits the failed-teardown root cause to CODE ISSUE or PRODUCT BUG, which conflicts with the new 3-label model. Teardown can fail due to cluster/lab outages too, and forcing binary classification will regress the exact behavior this PR is adding.
Proposed fix
-- **The test whose teardown failed**: Classify based on why its teardown failed —
- CODE ISSUE if the cleanup logic is wrong, PRODUCT BUG if the product blocked
- the revert.
+- **The test whose teardown failed**: Classify based on why its teardown failed —
+ `CODE ISSUE` if cleanup logic is wrong, `PRODUCT BUG` if product behavior blocks
+ revert under valid conditions, or `INFRASTRUCTURE` if cluster/lab conditions
+ (API outage, node failure, missing operators, storage/network outage) prevent cleanup.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@JOB_INSIGHT_FAILURE_HISTORY_ANALYSIS_PROMPT.md` around lines 35 - 40, Update
the teardown-root-cause guidance so the root teardown classification can be CODE
ISSUE, PRODUCT BUG, or INFRASTRUCTURE: modify the sentence under "The test whose
teardown failed" to include INFRASTRUCTURE as a valid classification (e.g.,
"Classify based on why its teardown failed — CODE ISSUE, PRODUCT BUG, or
INFRASTRUCTURE"), and ensure the later rule for "All other tests that failed
after it" still instructs using the same classification and reason format
("Caused by [test_name] teardown failure — [resource] was not reverted.") so
INFRASTRUCTURE is propagated the same way as the other labels.
Comment on lines
+44
to
+46
| Your goal is to classify each failure as `CODE ISSUE`, `PRODUCT BUG`, or | ||
| `INFRASTRUCTURE` based on the available evidence. Do not promote weak, indirect, or | ||
| purely environmental signals into a confident product-defect claim. |
Contributor
There was a problem hiding this comment.
HIGH: Restore required PR template sections in the PR description
The PR body does not include the required template headers, especially ##### What this PR does / why we need it: (with meaningful content), plus ##### Which issue(s) this PR fixes:, ##### Special notes for reviewer:, and ##### jira-ticket:. Please restore them to satisfy repo policy and reviewer workflow expectations.
As per coding guidelines ## PR Template Validation ... Required sections (must be present, even if empty).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@JOB_INSIGHT_PROMPT.md` around lines 44 - 46, The PR description is missing
required PR template headers causing validation failures; update the PR body to
restore the mandatory sections exactly as specified: add the headers "##### What
this PR does / why we need it:", "##### Which issue(s) this PR fixes:", "#####
Special notes for reviewer:", and "##### jira-ticket:" (each present even if
empty), and populate "What this PR does / why we need it:" with meaningful
content summarizing the change; ensure these exact header strings appear in the
PR description so the repo's PR Template Validation rules recognize them.
servolkov
reviewed
May 8, 2026
|
|
||
| ## Classification | ||
|
|
||
| - **The test whose teardown failed**: Classify based on why its teardown failed — |
Contributor
There was a problem hiding this comment.
Don't we need to add INFRASTRUCTURE here? A teardown can fail due to infrastructure as well.
| 4. **Prefer direct evidence over wrapper location.** | ||
| File paths like `tests/`, `utilities/`, `libs/`, and `conftest.py` are useful clues, | ||
| but they are not verdicts. Those modules often wrap product, cluster, or node state. | ||
| 5. **Separate test-owned, product-owned, and environment-owned problems.** |
Contributor
There was a problem hiding this comment.
INFRASTRUCTURE has no entry in the decision order. Per this PR logic it should.
| - NodeNetworkConfigurationPolicy status: | ||
| `oc get nncp -o yaml` | ||
|
|
||
| ### For environmental blockers, suggest collecting |
Contributor
There was a problem hiding this comment.
Suggested change
| ### For `INFRASTRUCTURE`, suggest collecting` |
?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The AI analysis prompt only allows
CODE ISSUEandPRODUCT BUGclassifications. When a failure is caused by an environmental/infrastructure issue (broken node, OCI hook failure, storage outage, etc.), the AI is forced to mislabel it asCODE ISSUEwith disclaimer text like:The rootcoz backend already supports
INFRASTRUCTUREinOverrideClassificationLiteral, but the prompt never tells the AI it can use it.Solution
Add
INFRASTRUCTUREas a third allowed classification value in the prompt:CODE ISSUE— Test framework, test code, or test-owned configuration problemPRODUCT BUG— Actual product defectINFRASTRUCTURE— Environmental blocker, lab/cluster infrastructure failure, or external dependency outageChanges
INFRASTRUCTUREto the allowed classification values in Section 2INFRASTRUCTUREinstead of vague "environmental" wordingINFRASTRUCTURElabelAssisted-by: Claude noreply@anthropic.com
Summary by CodeRabbit
Documentation