title	Agentrology Environment Server
emoji	🛡️
colorFrom	gray
colorTo	red
sdk	docker
pinned	false
app_port	8000
base_path	/web
tags	openenv security linux

Agentrology: A live Linux training ground for AI security agents

Can a 7B model learn to be a SOC analyst during a live Linux attack?

Agentrology

We drop an AI agent into a real Linux machine that is actively infected with malware, and we see if the agent can fix it.

Most AI tests are basically school quizzes. They ask a model to read a text file and answer a multiple choice question. But real security work does not happen on a multiple choice test. In the real world, things break, viruses hide, and if you type the wrong command, you destroy the system.

So we built a real fighting ring for AI. The agent gets a Linux terminal, a time limit, and one job: hunt down the malware and kill it.

How the agent learns

Episode 1. The AI has no idea what to do. It runs random commands. It tries to delete the whole hard drive. Our safety sandbox stops it. It gets a terrible score.

Episode 15. The AI figures out how to look at running programs. It finds a fake crypto miner. It kills the process. The threat is gone. The score goes up.

Episode 40. The AI deals with a virus that keeps reviving itself. It realizes that just killing the program is not enough. It hunts down the hidden startup files, deletes them, and then kills the virus for good. It actually learned how to think.

The evaluation engine checks the live process table and file system. The model is allowed to make mistakes, but it cannot permanently destroy the host machine. A security layer blocks fatal system commands.

Why we built this

Human security workers are burned out. There are too many alerts and not enough people to check them. If we want AI to actually help, we need to know if it can do the job without breaking the server.

Agentrology proves whether an AI is ready to touch a real production machine.

Architecture: The Arena

┌─────────────────────────────────────────────────────────────────┐
│                    AGENTROLOGY ENVIRONMENT                      │
│                                                                 │
│  ┌─────────────┐    ┌─────────────────┐    ┌─────────────────┐  │
│  │  Frontier   │    │ OpenEnv Server  │    │  Live Linux     │  │
│  │   Model     │───►│ (FastAPI :8000) │───►│  Container      │  │
│  │ (Qwen/Llama)│◄───│                 │◄───│ (Threat Arena)  │  │
│  └─────────────┘    └────────┬────────┘    └────────┬────────┘  │
│         ▲                    │                      │           │
│         │             ┌──────▼──────┐        ┌──────▼──────┐    │
│         │             │   Command   │        │ Stateful    │    │
│         └─────────────┤  Validator  │        │ Grader      │    │
│            Reward     │ (Sandbox)   │        │ (No LLM)    │    │
│                       └─────────────┘        └─────────────┘    │
└─────────────────────────────────────────────────────────────────┘

The Threat Catalogue

We built 12 different scenarios. They start easy. Find a bad program and stop it. They get incredibly hard. Find a rootkit that hides its tracks, rebuilds itself every three seconds, and spreads across different system folders.

Task ID	Difficulty	Severity	Threat	Agent Objective
T01	Easy	LOW	Unauthorized HTTP listener	Identify process bound to port 8080 and terminate it.
T02	Easy	LOW	Rogue crypto-miner	Kill the miner process and delete its payload script.
T03	Easy	LOW	Rogue SSH reverse-tunnel	Locate unauthorized SSH process on port 2222, terminate it, and delete its payload script.
T04	Medium	MEDIUM	Persistent cron backdoor	Kill the process, remove the crontab entry, and delete the payload script.
T05	Medium	MEDIUM	In-memory keylogger	Terminate the process, delete hidden log output, and remove the payload script.
T06	Hard	CRITICAL	Multi-stage APT infiltration	Neutralise 3 concurrent threats: sever cron dropper before killing C2 beacon, purge rogue SSH key, and eradicate secrets dump.
T07	Medium	MEDIUM	Data exfiltration agent	Terminate the process, delete staged data, and remove the payload script.
T08	Medium	MEDIUM	Fake update service	Terminate the rogue service and clear its binaries and persistence files.
T09	Hard	HIGH	Disguised syslog daemon	Kill the process first, then remove the persistence hook and payload script.
T10	Hard	HIGH	Self-healing rootkit	Eradicate the process, clean poisoned environment state, and remove persistence hooks.
T11	Hard	MEDIUM	Credential harvester	Multi-directory cleanup across captured data, staging files, and payload scripts.
T12	Hard	CRITICAL	Self-healing privilege kit	Handle order-dependent remediation under time pressure before artifacts regenerate.

Results

Displays the percentage of completely neutralized threats across the different LLMs tested.

A. The overall success rate of each model. A high score indicates the agent consistently neutralized the active malware and removed any hidden persistence mechanisms.

B. Agent efficiency by tracking the average number of terminal commands required to fully resolve each specific threat scenario.

Reward Trajectory

C. Step-by-step performance of an agent during a single task. The rising curve tracks the agent's progress, while red dashed lines indicate security violations where the sandbox intervened to block an unauthorized or destructive command.

(Note: Individual trajectory graphs are saved to assets/trajectories)

What We Observed

These notes are based on the benchmark JSONs currently saved in benchmarks/ and log files captured on April 11-12, 2026. They are useful directional observations, not a perfectly controlled leaderboard: task coverage is uneven across models, and some runs used different step budgets.

Strongest runs in the current saved set

Model	Saved run config	Coverage in saved set	What stood out
OpenAI/GPT-5.3	reasoning on, temp 0.08, max steps 45	T01 T02 T03 T04 T05 T09	Best complete multi-task run in the repo right now: 6/6 successes, average score 0.9489, average 7.67 steps.
minimax/minimax-m2.5	reasoning off, temp 0.9, max steps 45	T02 only	Very strong single-task pilot: solved T02 in 7 steps with score 0.9533. Promising, but not enough coverage to generalize.
openai/gpt-oss-20b:free	reasoning on, temp 0.08, max steps 25	T01 T02 T03 T04 T05 T06	Could solve simple one-process tasks, but degraded sharply on persistence and multi-stage cleanup.

Behaviour patterns from the logs

• The best runs followed a simple loop: ps first, kill one specific PID, then inspect /tmp, cron, or persistence paths, then remove the exact artifact that kept the threat alive.
• OpenAI/GPT-5.3 was the cleanest operator in the saved logs. It avoided sandbox hits entirely in its successful run set and usually completed medium tasks in 6-11 steps.
• minimax/minimax-m2.5 showed a similar pattern on T02: narrow enumeration, one targeted kill, then focused filesystem inspection until it found the leftover script.
• openai/gpt-oss-20b:free handled obvious “find PID, kill PID” cases, but often stalled after the first neutralization and did not reliably finish the cleanup phase.
• qwen/qwen3-32b and arcee-ai/trinity-large-preview:free often got the first step right on easy tasks, then drifted into long grep/find loops or started probing protected server processes and port 8000.
• llama-3.1-8b-instant mostly failed on control and format discipline. In several logs it tried to kill the environment server PID (17) or produced invalid command formatting after the first step.
• Prompt format mattered more than expected. Earlier ReAct-style outputs using [THOUGHT] and [COMMAND] were workable on short histories, but several models became less reliable as context grew and started dropping the command block or drifting out of format.
• Switching to the JSON response format defined in prompts.py improved stability because many models were more consistent when asked for a structured object like {"thought": "...", "command": "..."} or {"command": "..."}. The current inference path is built around these JSON prompts and the tolerant JSON extractor.

Rough config that worked best

If you want a practical starting point for new runs, the current logs suggest:

• Use a low temperature around 0.08 for larger frontier models.
• Give the agent enough room to recover from a wrong hypothesis: 45 max steps was much more forgiving than 25.
• Enable reasoning for stronger proprietary/frontier models when the provider supports it.
• Prefer the JSON prompt format over the older [THOUGHT] / [COMMAND] ReAct-style format, especially for longer trajectories.
• Keep the action format strict: one shell command per turn, no interactive tools, no broad kill patterns.
• Expect hard tasks to require both process neutralization and persistence cleanup. A model that only kills PIDs will look good briefly and still fail the grader.

Scoring

Grading is fully deterministic. The evaluator inspects the live Linux process table and filesystem — no string matching, no LLM-as-judge.

Per-Task Grading

Each task's grade() returns a float in [0.0001, 0.9999] based on independently weighted conditions:

Condition	Score contribution
Process killed	0.3 – 0.5 (task-dependent)
Payload script deleted	0.15 – 0.5
Persistence artefacts removed (log/config/DB/hook)	Weighted per artefact
All conditions met	~0.9999

Step RewardComponents

Component	Value	Trigger
Score delta	Varies	Sum of per-threat grade changes this step (primary signal)
Diagnostic bonus	+0.05 to +0.01 (decaying)	ps, ls, grep, netstat, etc. — decays after 3 unique uses
Non-diagnostic bonus	+0.01 to +0.002 (decaying)	Other successful commands with no score change
Execution error penalty	-0.04	Non-zero exit on non-diagnostic, non-kill commands
Security violation penalty	-0.1 to -0.5	Command blocked by CommandValidator
Intra-command repetition	-0.1	Command string contains repeated sub-commands

Rewards are clamped to [-1.0, 10.0]. Episode final score is clamped to [0.001, 0.9999].

Quick Start

Prerequisites

• Docker (required — see warning above)
• Python >= 3.10 + uv
• A model provider API token (API_KEY, HF_TOKEN, or OPENAI_API_KEY) or a local Ollama instance

1. Configure

cp .env.example .env
# Set API_KEY (or HF_TOKEN / OPENAI_API_KEY), and optionally MODEL_NAME

2. Build and Run the Container

Warning: Do not run the environment server directly on your host machine. Tasks intentionally spawn background processes that mimic malware behavior. Always use Docker.

chmod +x scripts/docker_build_and_run.sh

# Headless (for inference)
./scripts/docker_build_and_run.sh

# With interactive web UI
./scripts/docker_build_and_run.sh --web

Flag	Description
--web	Enable ENABLE_WEB_INTERFACE (mounts dashboard and terminal UI)
--skip-build	Skip the Docker build phase
--build-only	Build image only, do not start container
--bash	Attach a shell to a running container

3. Run Inference

inference.py is the main agent loop. Make it executable or run via uv:

# Make executable (Linux)
chmod +x inference.py
./inference.py --hf --model moonshotai/Kimi-K2-Instruct

# Or using uv
uv run inference.py --hf --model moonshotai/Kimi-K2-Instruct

Key flags:

Flag	Default	Description
--hf / --ollama	HF	Toggle between HuggingFace Router and local Ollama
--model <name>	moonshotai/Kimi-K2-Instruct	LLM identifier
--max-steps <n>	45	Max steps per episode
--task-ids <ids...>	T01 T02 T03 T04 T05 T06	Space-separated task IDs to run
--reasoning	off	Enable reasoning mode that uses a prompt that requires thought
--reasoning-effort <none|low|medium|high>	low	Configure provider-specific reasoning effort
--temperature <f>	0.08	Sampling temperature
--max-tokens <n>	500	Max tokens per model response
--dev	off	Verbose colour-coded console output
--port <n>	auto	Expose environment UI on a fixed port
--benchmark <name>	agentrology-benchmark	Label for the run (affects log/result filenames)
--benchmark-dir <path>	benchmarks/	Directory for JSON benchmark result files
--interactive	off	Bridge-UI human-in-the-loop mode
--api-url <url>	provider default	Override the LLM API base URL
--image <name>	env-dependent	Override the Docker image used for the environment

Logs are timestamped and saved to logs/ automatically.

Web Interface

Path	Description
/web	Interactive terminal UI for manual step-by-step control
/dashboard	Live threat dashboard: real-time neutralization status and scores
/benchmarks	Benchmark viewer: browse and inspect saved run results
/docs	OpenAPI / Swagger UI

Security Sandbox

The environment blocks commands before subprocess execution. Blocked categories:

• Interactive / TTY commands (vim, top, htop, nano, less, ...)
• System destruction (rm -rf /, reboot, shutdown, ...)
• Mass process termination via pipes or xargs
• Commands targeting uvicorn, port 8000, or /app/env

Following are test cases that can be run to validate the effectiveness:

uv run python -m tests.test_command_validator
uv run python -m tests.self_kill_protection

Development

uv sync # Install dependencies
./scripts/run_dev.sh # Start server locally (no Docker)

License

BSD 3-Clause - see LICENSE. Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.