From 55d42f99ec4825158001a2cc078d86c185e63140 Mon Sep 17 00:00:00 2001 From: aktasbatuhan Date: Mon, 8 Jun 2026 16:42:15 +0100 Subject: [PATCH 1/3] docs(examples): add vulnerable-vault example + product-first README - examples/vulnerable-vault: a tiny, self-contained, intentionally-vulnerable Solidity target (reentrancy + unchecked ERC-20 return) with a ready-made threat_context.yaml and a README documenting the planted bugs, the run command, and the expected findings. Lets anyone try the full audit -> view/report loop without a private repo or large spend. - examples/README.md indexes the targets with a responsible-use note. - README: add a top-of-file Quickstart (install -> audit the example -> view/report), an Examples section, and a Benchmarking pointer that makes the evaluation/ harness discoverable while keeping it secondary to the product. --- README.md | 38 ++++++++++++++++ examples/README.md | 21 +++++++++ examples/vulnerable-vault/README.md | 44 +++++++++++++++++++ examples/vulnerable-vault/foundry.toml | 5 +++ examples/vulnerable-vault/src/Vault.sol | 35 +++++++++++++++ examples/vulnerable-vault/threat_context.yaml | 22 ++++++++++ 6 files changed, 165 insertions(+) create mode 100644 examples/README.md create mode 100644 examples/vulnerable-vault/README.md create mode 100644 examples/vulnerable-vault/foundry.toml create mode 100644 examples/vulnerable-vault/src/Vault.sol create mode 100644 examples/vulnerable-vault/threat_context.yaml diff --git a/README.md b/README.md index 07cd441..6156836 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,28 @@ Kai runs a multi-stage pipeline: a **setup agent** prepares and builds the targe Built on [ra](src/ra/), a recursive language model framework where LLMs write code that launches other LLMs. +## Quickstart + +```bash +git clone https://github.com/firstbatchxyz/kai-security.git +cd kai-security +uv sync +cp .env.example .env # add OPENROUTER_API_KEY (or OPENAI_API_KEY) + +# Audit the bundled, intentionally-vulnerable example target +uv run kai audit --repo-path examples/vulnerable-vault --verbose + +# Explore the findings + the agent's reasoning in your browser... +uv run kai view output/state/ --open +# ...or print a Markdown report (or a styled HTML one) +uv run kai report output/state/ +``` + +`` is printed during the run (the directory created under +`output/state/`). Point `--repo-path` at any local checkout you're authorized +to test. See [`examples/`](examples/) for more, and the [full CLI](#command-line-interface) +and [Usage](#usage) below for every option. + ## Installation Requires Python 3.12+ and [uv](https://docs.astral.sh/uv/). @@ -55,6 +77,14 @@ uv run kai report output/state/ --format html -o report.html (equivalently `uv run python -m kai.main ...`). Run `kai -h` for per-command options. +## Examples + +The [`examples/`](examples/) directory has small, self-contained, +intentionally-vulnerable targets you can audit end to end without a private +repo or large spend — start with +[`vulnerable-vault`](examples/vulnerable-vault/) (a Solidity vault with a +reentrancy and an unchecked-transfer bug, plus a ready-made threat context). + ### API keys | Key | Required | Used by | @@ -401,6 +431,14 @@ make lint make typecheck ``` +## Benchmarking + +Kai ships an optional harness for scoring it against external security +benchmarks (CyberGym, BountyBench, EVMBench) and for running fleets of audits +in parallel. It drives `kai` as a subprocess and lives entirely in +[`evaluation/`](evaluation/) — see [`evaluation/README.md`](evaluation/README.md). +Most users don't need it; it's for measuring and improving Kai itself. + ## Related Work Kai uses ideas from the Recursive Language Models paper. To cite that diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..a5d5dbb --- /dev/null +++ b/examples/README.md @@ -0,0 +1,21 @@ +# Examples + +Runnable targets for trying `kai` end to end — no private repos, minimal API +spend. + +| Example | What it is | Highlights | +|---------|------------|------------| +| [`vulnerable-vault/`](vulnerable-vault/) | A tiny Solidity vault with two planted bugs | reentrancy + unchecked ERC-20 return; ships a `threat_context.yaml` | + +Each example is **intentionally vulnerable** and is for authorized +demonstration only — do not deploy them. + +Quick run (see each example's README for details): + +```bash +uv run kai audit --repo-path examples/vulnerable-vault --verbose +uv run kai view output/state/ --open +``` + +Running an audit makes real LLM calls, so it needs an API key configured (see +the project [README](../README.md#api-keys)) and incurs some cost. diff --git a/examples/vulnerable-vault/README.md b/examples/vulnerable-vault/README.md new file mode 100644 index 0000000..8aecd21 --- /dev/null +++ b/examples/vulnerable-vault/README.md @@ -0,0 +1,44 @@ +# vulnerable-vault + +A tiny, self-contained, **intentionally vulnerable** Solidity project — a +target you can point `kai audit` at to see the whole pipeline run end to end +without a private repo or a large API spend. + +> ⚠️ Intentionally insecure. Do not deploy. For authorized demonstration only. + +## Planted bugs + +| # | Bug | Location | Expected category | +|---|-----|----------|-------------------| +| 1 | **Reentrancy** — external call before the balance is zeroed, no guard | `src/Vault.sol` · `withdraw()` | `active_exploit` | +| 2 | **Unchecked ERC-20 return** — `transfer()` boolean ignored | `src/Vault.sol` · `sweepToken()` | `active_exploit` | + +## Run it + +```bash +# From the kai-security repo root +uv run kai audit --repo-path examples/vulnerable-vault \ + --threat-context examples/vulnerable-vault/threat_context.yaml --verbose +``` + +Then look at the results: + +```bash +# Interactive HTML (findings + the agent's reasoning trace) +uv run kai view output/state/ --open + +# Or a Markdown report (stdout), or a styled HTML document +uv run kai report output/state/ +uv run kai report output/state/ --format html -o report.html +``` + +`` is printed during the run and is the directory name under +`output/state/`. + +## What to expect + +Kai should surface the reentrancy in `withdraw()` as a confirmed +`active_exploit` (typically the highest-CVSS finding, with a working PoC and a +suggested patch that moves the balance update before the external call), and +flag the unchecked `transfer()` return in `sweepToken()`. Exact wording, CVSS +scores, and ordering depend on the models configured for the run. diff --git a/examples/vulnerable-vault/foundry.toml b/examples/vulnerable-vault/foundry.toml new file mode 100644 index 0000000..cdbe10f --- /dev/null +++ b/examples/vulnerable-vault/foundry.toml @@ -0,0 +1,5 @@ +[profile.default] +src = "src" +out = "out" +libs = ["lib"] +solc = "0.8.20" diff --git a/examples/vulnerable-vault/src/Vault.sol b/examples/vulnerable-vault/src/Vault.sol new file mode 100644 index 0000000..0d0ce5b --- /dev/null +++ b/examples/vulnerable-vault/src/Vault.sol @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: MIT +pragma solidity ^0.8.20; + +interface IERC20 { + function transfer(address to, uint256 amount) external returns (bool); +} + +/// @title Vault +/// @notice INTENTIONALLY VULNERABLE example target for kai-security demos. +/// Do NOT deploy. The bugs below are planted so `kai audit` has +/// something real to find on a tiny, self-contained codebase. +contract Vault { + mapping(address => uint256) public balances; + + function deposit() external payable { + balances[msg.sender] += msg.value; + } + + /// BUG 1 — reentrancy: the ETH is sent *before* the balance is zeroed, + /// and there is no reentrancy guard, so a malicious receiver can re-enter + /// withdraw() and drain the contract. + function withdraw(uint256 amount) external { + require(balances[msg.sender] >= amount, "insufficient"); + (bool ok, ) = msg.sender.call{value: amount}(""); + require(ok, "transfer failed"); + balances[msg.sender] -= amount; + } + + /// BUG 2 — unchecked return value: ERC-20 transfer() can return false + /// instead of reverting; ignoring it lets a failed transfer look like a + /// success. + function sweepToken(IERC20 token, address to, uint256 amount) external { + token.transfer(to, amount); + } +} diff --git a/examples/vulnerable-vault/threat_context.yaml b/examples/vulnerable-vault/threat_context.yaml new file mode 100644 index 0000000..61fe952 --- /dev/null +++ b/examples/vulnerable-vault/threat_context.yaml @@ -0,0 +1,22 @@ +# Threat context for the vulnerable-vault example. +# Tells kai who can interact with the contract and what the trust boundaries +# are, so it ranks an unprivileged-attacker drain above admin-only concerns. + +deployment_type: smart-contract +environment: on-chain + +access_roles: + - name: anyone + trust: none + description: "Permissionless caller — any EOA or contract" + - name: depositor + trust: none + description: "An account that has deposited ETH into the vault" + +boundaries: + - "User input → contract storage (deposit/withdraw accounting)" + - "External call to msg.sender during withdraw (reentrancy boundary)" + +known_constraints: + - "No admin or owner role exists; every function is permissionless" + - "ERC-20 tokens passed to sweepToken may be non-reverting (return false)" From f0e8529381c5702a0d3414ec201c418c2f158884 Mon Sep 17 00:00:00 2001 From: aktasbatuhan Date: Mon, 8 Jun 2026 18:46:16 +0100 Subject: [PATCH 2/3] fix(examples): make vault reentrancy genuinely exploitable + honest README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A real kai audit showed the original withdraw(amount) reentrancy was NOT exploitable under Solidity 0.8.x: the checked '-= amount' underflows and reverts during the reentrant unwind, and kai correctly disproved it. Switch to the canonical drainable form (zero the balance with '= 0' AFTER the external call) so the bug is real. Rewrite the example README to match verified output instead of a fabricated findings table: show the actual confirmed Critical (CVSS 9.8) reentrancy and its Check-Effects-Interaction patch, and state plainly that kai is non-deterministic — which bugs are confirmed and their scores vary by run and model. --- examples/vulnerable-vault/README.md | 46 +++++++++++++++++++------ examples/vulnerable-vault/src/Vault.sol | 16 +++++---- 2 files changed, 46 insertions(+), 16 deletions(-) diff --git a/examples/vulnerable-vault/README.md b/examples/vulnerable-vault/README.md index 8aecd21..c88ea88 100644 --- a/examples/vulnerable-vault/README.md +++ b/examples/vulnerable-vault/README.md @@ -8,10 +8,10 @@ without a private repo or a large API spend. ## Planted bugs -| # | Bug | Location | Expected category | -|---|-----|----------|-------------------| -| 1 | **Reentrancy** — external call before the balance is zeroed, no guard | `src/Vault.sol` · `withdraw()` | `active_exploit` | -| 2 | **Unchecked ERC-20 return** — `transfer()` boolean ignored | `src/Vault.sol` · `sweepToken()` | `active_exploit` | +| # | Bug | Location | +|---|-----|----------| +| 1 | **Reentrancy** — the caller's balance is zeroed *after* the external call, no guard (a re-entrant caller drains the contract) | `src/Vault.sol` · `withdraw()` | +| 2 | **Unchecked ERC-20 return** — `transfer()`'s boolean result is ignored | `src/Vault.sol` · `sweepToken()` | ## Run it @@ -35,10 +35,36 @@ uv run kai report output/state/ --format html -o report.html `` is printed during the run and is the directory name under `output/state/`. -## What to expect +## What a real run produced -Kai should surface the reentrancy in `withdraw()` as a confirmed -`active_exploit` (typically the highest-CVSS finding, with a working PoC and a -suggested patch that moves the balance update before the external call), and -flag the unchecked `transfer()` return in `sweepToken()`. Exact wording, CVSS -scores, and ordering depend on the models configured for the run. +This isn't hypothetical — here's an actual result. With the reentrancy bug, +Kai built a Foundry proof-of-concept, confirmed the drain, and proposed a fix: + +``` +| CVSS | Severity | Finding | Location | Status | +| 9.8 | critical | Reentrancy in withdraw() (CEI violation) | Vault.sol:withdraw | verified_and_fixed ✓ | +``` + +with the correct Check-Effects-Interaction patch (move the balance update +*before* the external call): + +```diff + function withdraw() external { + uint256 amount = balances[msg.sender]; + require(amount > 0, "nothing to withdraw"); ++ balances[msg.sender] = 0; + (bool ok, ) = msg.sender.call{value: amount}(""); + require(ok, "transfer failed"); +- balances[msg.sender] = 0; + } +``` + +> **Kai is an agentic system, so runs are not deterministic.** Which bugs get +> confirmed, their CVSS scores, and the exact wording vary by run and by the +> models you configure. In one run Kai confirmed the reentrancy as Critical +> (above); in another it confirmed the unchecked-return in `sweepToken()` as +> Medium instead. It also reasons about *exploitability*, not just patterns — +> given a `withdraw()` that used a checked `-= amount`, it correctly **disproved** +> a textbook-looking reentrancy because the subtraction underflows and reverts +> under Solidity 0.8.x. Treat the output as a strong signal to investigate, not +> a fixed checklist. diff --git a/examples/vulnerable-vault/src/Vault.sol b/examples/vulnerable-vault/src/Vault.sol index 0d0ce5b..5bf3b0f 100644 --- a/examples/vulnerable-vault/src/Vault.sol +++ b/examples/vulnerable-vault/src/Vault.sol @@ -16,14 +16,18 @@ contract Vault { balances[msg.sender] += msg.value; } - /// BUG 1 — reentrancy: the ETH is sent *before* the balance is zeroed, - /// and there is no reentrancy guard, so a malicious receiver can re-enter - /// withdraw() and drain the contract. - function withdraw(uint256 amount) external { - require(balances[msg.sender] >= amount, "insufficient"); + /// BUG 1 — reentrancy: the caller's balance is zeroed *after* the external + /// call, with no reentrancy guard. A malicious receiver can re-enter + /// withdraw() from its fallback and drain the contract, because the balance + /// is still non-zero on each re-entry. (Zeroing with `= 0` rather than a + /// checked `-=` is what makes this genuinely exploitable under Solidity + /// 0.8.x — a checked subtraction would underflow and revert the drain.) + function withdraw() external { + uint256 amount = balances[msg.sender]; + require(amount > 0, "nothing to withdraw"); (bool ok, ) = msg.sender.call{value: amount}(""); require(ok, "transfer failed"); - balances[msg.sender] -= amount; + balances[msg.sender] = 0; } /// BUG 2 — unchecked return value: ERC-20 transfer() can return false From d83772cbbb7d1bd8bb60ac403235efc8c5a49c47 Mon Sep 17 00:00:00 2001 From: aktasbatuhan Date: Mon, 15 Jun 2026 14:53:08 +0100 Subject: [PATCH 3/3] docs(examples): use `kai-security` command in example walkthroughs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follows the console-script rename (`kai` → `kai-security`, which avoids shadowing kai-agent's `kai`). Updates the copy-pasteable example commands in the walkthrough docs; the import package / module names are unchanged. --- README.md | 22 +++++++++++----------- examples/README.md | 4 ++-- examples/vulnerable-vault/README.md | 10 +++++----- 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 6156836..83fcc7d 100644 --- a/README.md +++ b/README.md @@ -17,12 +17,12 @@ uv sync cp .env.example .env # add OPENROUTER_API_KEY (or OPENAI_API_KEY) # Audit the bundled, intentionally-vulnerable example target -uv run kai audit --repo-path examples/vulnerable-vault --verbose +uv run kai-security audit --repo-path examples/vulnerable-vault --verbose # Explore the findings + the agent's reasoning in your browser... -uv run kai view output/state/ --open +uv run kai-security view output/state/ --open # ...or print a Markdown report (or a styled HTML one) -uv run kai report output/state/ +uv run kai-security report output/state/ ``` `` is printed during the run (the directory created under @@ -46,7 +46,7 @@ uv sync cp .env.example .env ``` -`uv sync` installs the `kai` command (the distribution is published as +`uv sync` installs the `kai-security` command (the distribution is published as `kai-security`; the command and import package are `kai`). Common developer commands are available through `make`: @@ -62,19 +62,19 @@ make run REPO_PATH=/path/to/target ```bash # Audit a repository you're authorized to test (setup → exploit pipeline) -uv run kai audit --repo-path /path/to/target --verbose +uv run kai-security audit --repo-path /path/to/target --verbose # Open a finished run as an interactive HTML report (findings + agent trace) -uv run kai view output/state/ --open +uv run kai-security view output/state/ --open # Render a run's findings — Markdown to stdout, or a styled HTML document -uv run kai report output/state/ -uv run kai report output/state/ --format html -o report.html +uv run kai-security report output/state/ +uv run kai-security report output/state/ --format html -o report.html ``` -`kai audit` is the friendly alias for the full pipeline; `kai pipeline` and -`kai agent` expose the complete interface documented under [Usage](#usage) -(equivalently `uv run python -m kai.main ...`). Run `kai -h` for +`kai-security audit` is the friendly alias for the full pipeline; `kai-security pipeline` and +`kai-security agent` expose the complete interface documented under [Usage](#usage) +(equivalently `uv run python -m kai.main ...`). Run `kai-security -h` for per-command options. ## Examples diff --git a/examples/README.md b/examples/README.md index a5d5dbb..d09de49 100644 --- a/examples/README.md +++ b/examples/README.md @@ -13,8 +13,8 @@ demonstration only — do not deploy them. Quick run (see each example's README for details): ```bash -uv run kai audit --repo-path examples/vulnerable-vault --verbose -uv run kai view output/state/ --open +uv run kai-security audit --repo-path examples/vulnerable-vault --verbose +uv run kai-security view output/state/ --open ``` Running an audit makes real LLM calls, so it needs an API key configured (see diff --git a/examples/vulnerable-vault/README.md b/examples/vulnerable-vault/README.md index c88ea88..4f0b496 100644 --- a/examples/vulnerable-vault/README.md +++ b/examples/vulnerable-vault/README.md @@ -1,7 +1,7 @@ # vulnerable-vault A tiny, self-contained, **intentionally vulnerable** Solidity project — a -target you can point `kai audit` at to see the whole pipeline run end to end +target you can point `kai-security audit` at to see the whole pipeline run end to end without a private repo or a large API spend. > ⚠️ Intentionally insecure. Do not deploy. For authorized demonstration only. @@ -17,7 +17,7 @@ without a private repo or a large API spend. ```bash # From the kai-security repo root -uv run kai audit --repo-path examples/vulnerable-vault \ +uv run kai-security audit --repo-path examples/vulnerable-vault \ --threat-context examples/vulnerable-vault/threat_context.yaml --verbose ``` @@ -25,11 +25,11 @@ Then look at the results: ```bash # Interactive HTML (findings + the agent's reasoning trace) -uv run kai view output/state/ --open +uv run kai-security view output/state/ --open # Or a Markdown report (stdout), or a styled HTML document -uv run kai report output/state/ -uv run kai report output/state/ --format html -o report.html +uv run kai-security report output/state/ +uv run kai-security report output/state/ --format html -o report.html ``` `` is printed during the run and is the directory name under