Skip to content

Add AgentCheck deterministic evaluation MVP - #1

Merged
WaseemGhanem98 merged 1 commit into
mainfrom
agent/agentcheck-phase1
Aug 16, 2026
Merged

Add AgentCheck deterministic evaluation MVP#1
WaseemGhanem98 merged 1 commit into
mainfrom
agent/agentcheck-phase1

Conversation

@WaseemGhanem98

Copy link
Copy Markdown
Owner

Summary

  • add the framework-neutral AgentCheck Phase 1 domain, runner, evaluator, failure analysis, artifacts, and offline HTML reporting
  • add a fail-closed OpenAI Agents SDK adapter that reconstructs function tools around the mock gateway and never invokes original handlers
  • add 12 deterministic account-support scenarios and a deliberately flawed offline example that produces 7 passes and 5 high-confidence failures
  • integrate the agentcheck inspect and agentcheck test CLI, packaging, documentation, and CI coverage

Why

AgentLens already observes production agent behavior. This adds the complementary deterministic pre-deployment evaluation slice without coupling evaluation semantics to AgentLens collector internals.

Safety and developer impact

  • trusted local code only; explicitly not an untrusted-code sandbox
  • every supported tool call is intercepted, fixture-controlled, schema-validated, and isolated in a child process
  • unknown or unsupported tool surfaces fail closed
  • raw instructions are hidden in reports by default; structured artifacts and diagnostics use AgentLens redaction
  • no paid provider call or API key is required for the bundled suite

Validation

  • 266 passed across the complete repository test suite
  • Ruff passed
  • mypy passed across 56 source files
  • existing AgentLens end-to-end smoke passed
  • secret scan and pip check passed
  • isolated sdist/wheel build and installed CLI smoke passed
  • real offline CLI regression verified 14 distinct worker processes, 12 scenarios, exact 7 PASS / 5 FAIL / 0 inconclusive / 0 infra, complete artifacts/report, and zero original tool handler invocations

Review fixes included

The pre-merge review found and resolved full auth-header/cookie redaction, canonical world-path evaluation, fixture-effect dry-run linting, target entrypoint containment, base-wheel CLI dependency correctness, AgentCheck dependency auditing, and end-to-end CLI coverage.

@WaseemGhanem98
WaseemGhanem98 force-pushed the agent/agentcheck-phase1 branch from ae1a363 to afdd586 Compare August 16, 2026 04:19
@WaseemGhanem98
WaseemGhanem98 marked this pull request as ready for review August 16, 2026 04:21
@WaseemGhanem98
WaseemGhanem98 merged commit 5f0218f into main Aug 16, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant