Cyber Defense Benchmark

A benchmark for LLM threat-hunting agents. Run the Docker container and you'll get a sample result on the bundled payload.

Built by Simbian AI.

Quickstart

git clone https://github.com/simbianai/cyber_defense_benchmark.git
cd cyber_defense_benchmark
cp .env.example .env   # fill in at least one API key

docker compose build
docker compose run --rm benchmark             # runs hunts -> benchmark/hunts/*.json
docker compose run --rm benchmark cdb-plots   # scores + plots -> outputs/{diff,pareto,radar}.{json,png}

The first command runs the harness against the bundled datasets/sample.zip (auto-unpacked into sample.json + sample_flags.json on first run) and writes one JSON per (model, rollout) under benchmark/hunts/. The second command scores those hunts and renders the plots.

Local Python

pip install -e .
cp .env.example .env
cdb-run --data datasets/sample.json --model openai/gpt-5

Picking a model

Any LiteLLM-supported model:

docker compose run --rm benchmark --model anthropic/claude-sonnet-4-7
docker compose run --rm benchmark --model vertex_ai/gemini-3.1-pro-preview

Full dataset

The repo ships a sample payload so the pipeline is runnable out of the box. The full benchmark dataset is distributed separately.

• Access / questions: email research@simbian.ai or see simbian.ai/research.

Swapping your own harness

The benchmark is a thin wrapper. If you want to evaluate a different agent, replace benchmark/harness.py with your own implementation. The only contract is:

def run(env: ThreatHuntEnv, model: str) -> dict: ...

Your harness drives env via env.reset() / env.step(action), where action is a SQL statement run against the in-memory logs table and the observation is the resulting rows serialized to text. The schema is loaded from benchmark/reference/log_schema.json. info["schema_help"] from env.reset() is a ready-to-paste schema-and-examples block — feed it into your system prompt.

Validation pipeline

Reproduce the plots above on the bundled datasets/sample.zip (auto-unpacked into sample.json + sample_flags.json on first run).

1. Run hunts

cp .env.example .env   # at least one provider key
python -m benchmark.runner --rollouts 5 --yes

benchmark/models.txt lists the LiteLLM model strings to run; default rollouts, budget, and provider concurrency live in benchmark/runner_config.json. Cost is estimated from benchmark/reference/cost.json; the runner refuses to launch models that aren't in that table unless you pass --yes-unknown-cost (debugging is on you). Use --resume to skip hunts whose output JSON already exists.

Each run writes a file under benchmark/hunts/:

benchmark/hunts/seed_<seed>_model_<slug>_rollout_<r>.json

2. Score and plot

python -m benchmark.make_plots

Produces outputs/{diff,pareto,radar}.{json,png}:

• diff.{png,json} — per-turn cumulative coverage (one line per model+rollout).
• pareto.{png,json} — mean cost vs. mean coverage with the convex frontier.
• radar.{png,json} — per-tactic coverage, one polygon per model.

Citing

@misc{chona2026cyberdefensebenchmarkagentic,
      title={Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps}, 
      author={Alankrit Chona and Igor Kozlov and Ambuj Kumar},
      year={2026},
      eprint={2604.19533},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2604.19533}, 
}

License

MIT. See LICENSE.