OCPEDGE-2727: Add eval-skills presubmit and enhance agent-eval ref#81166
OCPEDGE-2727: Add eval-skills presubmit and enhance agent-eval ref#81166dhensel-rh wants to merge 2 commits into
Conversation
- Add EVAL_REPO_DIR env var to ref (backward compatible, defaults to /opt/ai-helpers) - Reorder commands.sh: plugins install → setup script → config check - Support setup script overriding EVAL_CONFIG or setting EVAL_SNAPSHOT_DIR - Add eval-skills presubmit for edge-tooling (optional, triggers on plugins skill/eval changes) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
openshift-ci-robot
added
the
jira/valid-reference
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
@dhensel-rh: This pull request references OCPEDGE-2727 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Skipping CI for Draft Pull Request. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughA new optional ChangesEval-skills job and step flow
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 14 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: dhensel-rh The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh`:
- Around line 58-66: The setup-script output handling in
openshift-claude-agent-eval-commands.sh is too permissive and can misinterpret
bad stdout as a snapshot directory. Update the EVAL_SETUP_SCRIPT result handling
in the EVAL_SETUP_OUTPUT block to accept only valid existing directories (using
the existing EVAL_SNAPSHOT_DIR path), and explicitly fail fast for any other
non-empty output; keep the EVAL_CONFIG override path in the same flow and use
the existing EVAL_SETUP_OUTPUT, EVAL_SNAPSHOT_DIR, and EVAL_CONFIG symbols to
locate the change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 3f31e6a5-d159-4800-8b20-5fe860f14de7
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift-eng/edge-tooling/openshift-eng-edge-tooling-main-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (3)
ci-operator/config/openshift-eng/edge-tooling/openshift-eng-edge-tooling-main.yamlci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.shci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-ref.yaml
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
2 participants
Summary
EVAL_REPO_DIRenv var toopenshift-claude-agent-evalref (backward compatible, defaults to/opt/ai-helpers)commands.sh: plugins install → setup script → config check (setup script can now override config)EVAL_CONFIG, directory setsEVAL_SNAPSHOT_DIReval-skillsoptional presubmit foropenshift-eng/edge-tooling— triggers on^plugins/.*/(skills|evals)changesTest plan
eval-skillsjob on this PR/opt/ai-helpers)Supersedes #80177.
🤖 Generated with Claude Code
Summary by CodeRabbit
This updates the OpenShift CI eval setup for
openshift-eng/edge-toolingso theopenshift-claude-agent-evaljob can run from a configurable repository directory instead of assuming/opt/ai-helpers, while keeping that path as the default. It also changes the eval startup flow so the plugin install runs before the setup script, and the setup script can now override the eval config or provide a snapshot directory through its output.In addition, it adds an optional
eval-skillspresubmit that triggers on changes under pluginskillsandevalspaths, enabling targeted eval coverage for those areas.