test(aeh): add pairwise mode hello-world submission#108
Draft
GuyZivRH wants to merge 2 commits into
Draft
Conversation
Add aeh-hello-world-pairwise submission to test Agent-Eval-Harness integration in pairwise/A/B mode with control and treatment variants. Structure: - metadata.yaml: Pipeline configuration (n_trials=3) - eval-control.yaml: Control variant config (baseline) - eval-treatment.yaml: Treatment variant config (with plugin_dirs) - cases/case-001/input.yaml: Shared task prompt - cases/case-001/annotations.yaml: Shared expected results Variants: - Control: Basic configuration (no plugin_dirs) - Treatment: Enhanced with plugin_dirs for skills Judges (both variants): - exit_success: Verifies Claude Code completed successfully - file_created: Validates output file content Test task: Create greeting.txt with "Hello, World!" content Pairwise comparison via score.py will determine if treatment wins over control based on judge results across trials.
Use claude-sonnet, add treatment skill package, clear control slash-skill, drop file_created gate, and add pairwise LLM judge for score.py.
GuyZivRH
force-pushed
the
eval/aeh-hello-world-pairwise
branch
from
July 19, 2026 12:04
012b9dc to
6ad7fff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Test Agent-Eval-Harness (AEH) integration in pairwise mode with A/B comparison between control (no skill) and treatment (with skill).
Changes
Added
aeh-hello-world-pairwisesubmission with:metadata.yaml—eval_engine: aeh, n_trials=3, security_scan: warneval-control.yaml— baseline with emptyskill(raw prompt, no slash-command)eval-treatment.yaml—skill: aeh-hello-world-pairwise+plugin_dirs: [skills]skills/aeh-hello-world-pairwise/SKILL.md— treatment skill packagecases/case-001/— shared prompt + annotationsModels
Both configs use LiteLLM proxy IDs:
claude-sonnet(notclaude-sonnet-4-5).Variants
skill: ""), no plugin_dirs — agent gets the raw prompt/{skill} {prompt}with skills mounted via plugin_dirsJudges
Testing
Pipeline params:
Related