CareOps Bench

CareOps Bench is a lightweight benchmark for evaluating healthcare operations agents that use tools to complete administrative patient-support workflows.

It focuses on safe, non-clinical tasks such as appointment management, coverage verification, and contact-detail updates. Every example is synthetic. The project does not diagnose, triage, prescribe, or provide medical advice.

Why it exists

Healthcare-support agents must do more than produce plausible text. They need to select permitted tools, complete the requested workflow, preserve patient context, and escalate anything outside their scope. CareOps Bench runs an agent through a stateful synthetic sandbox and makes those behaviors measurable with deterministic checks.

Architecture

agent adapter → structured tool calls → synthetic care-operations sandbox
     ↑                                      ↓
scenario contract ← evaluator ← trace + final synthetic patient state

The reference agent is deterministic and exists only to validate benchmark mechanics. Real agents can implement the small CareOpsAgent adapter and return the same structured ToolCall objects.

What it evaluates

• Workflow completion: required operations were performed
• Tool discipline: only permitted tools were called
• Escalation behavior: out-of-scope requests are explicitly escalated
• Trace integrity: every action includes a matching structured result

Quick start

git clone https://github.com/flokithecoder/careops-bench.git
cd careops-bench
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
careops-bench demo --scenario reschedule

Example output:

CareOps Bench demo: reschedule | score=1.000

To score a trace produced by an external agent:

careops-bench evaluate --input examples/appointment_reschedule.json --min-score 0.90

Case format

{
  "cases": [
    {
      "id": "reschedule-001",
      "intent": "Move an appointment to a later date.",
      "allowed_tools": ["lookup_appointment", "reschedule_appointment"],
      "required_tools": ["lookup_appointment", "reschedule_appointment"],
      "requires_escalation": false,
      "trace": [
        {"tool": "lookup_appointment", "status": "success"},
        {"tool": "reschedule_appointment", "status": "success"}
      ]
    }
  ]
}

Executable sandbox

The sandbox implements five explicit operations:

• lookup_appointment
• reschedule_appointment
• verify_coverage
• update_contact
• create_clinical_handoff

Each call produces a structured success or rejection event and can mutate only synthetic state. This makes it possible to verify both the action trace and the resulting appointment/contact/handoff state.

Scoring

The executable evaluator combines workflow completion (35%), tool discipline (20%), escalation behavior (20%), final-state validity (15%), and trace integrity (10%). The JSON-trace evaluator remains available as a lightweight CI gate for agent integrations.

Safety boundaries

• Only use synthetic or authorized data.
• Never put protected health information (PHI) in benchmark files, logs, or pull requests.
• Treat this as software-evaluation infrastructure, not a clinical decision-support system.
• Route diagnosis, treatment, medication, or urgent-care questions to approved human clinical workflows.

Development

pytest -q

The project includes unit tests and GitHub Actions for Python 3.10 and 3.12.

License

MIT