Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ You can pass a TOML table instead of such a prefixed string:
model = { provider = "openai", model_id = "gpt-5.6" }
```

Note that models require specific configuration based on the provider. For details, see the `quantiles.toml` file under the provider of your choice in [`cli/examples/configs`](./cli/examples/configs).
Note that models require specific configuration based on the provider. For details, see the `quantiles.toml` file under the provider of your choice in the [provider configuration examples](./cli/examples/configs).

### `custom_code`

Expand Down Expand Up @@ -177,4 +177,4 @@ input = { greeting = "world" }

### Custom evaluation with failure simulation

See [`cli/examples/configs/custom_code/quantiles.toml`](./cli/examples/configs/custom_code/quantiles.toml) for a complete, commented example including a sample Python script.
See the complete [custom-code configuration example](./cli/examples/configs/custom_code/quantiles.toml) for an annotated configuration and sample Python script.
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,15 @@ git clone https://github.com/quantiles-evals/quantiles.git
cd quantiles
```

For the [`./cli`](./cli) directory, ensure you have Rust installed and use the `mise.toml` file in that directory to run common build/lint/check/test commands. For the [`./python`](./python) directory, use the [`uv`](https://docs.astral.sh/uv/) tool for similar commands. The [`typescript/`](./typescript) directory holds the yet-unreleased, unsupported TypeScript SDK.
For the [CLI directory](./cli), ensure you have Rust installed and use the `mise.toml` file in that directory to run common build/lint/check/test commands. For the [Python SDK directory](./python), use the [`uv`](https://docs.astral.sh/uv/) tool for similar commands. The [TypeScript SDK directory](./typescript) holds the yet-unreleased, unsupported TypeScript SDK.

## Security

Please do not report security vulnerabilities through public GitHub issues. Follow the reporting guidance in [SECURITY.md](./SECURITY.md).
Please do not report security vulnerabilities through public GitHub issues. Follow the security reporting guidance in [SECURITY.md](./SECURITY.md).

## Code of Conduct

By participating in this project, you agree to follow our [CODE_OF_CONDUCT.md](./CODE_OF_CONDUCT.md).
By participating in this project, you agree to follow our [Code of Conduct](./CODE_OF_CONDUCT.md).

## License

Expand Down
26 changes: 13 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,13 @@ Install the CLI:
curl -fsSL https://cli.quantiles.io/install.sh | bash
```

Run the [SimpleQA-verified](https://arxiv.org/abs/2509.07968) built-in benchmark:
Run the [SimpleQA Verified](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) built-in benchmark:

```bash
qt run simpleqa-verified
```

> Important note: this `qt run` command will run the [`simpleqa-verified`](https://arxiv.org/abs/2509.07968) benchmark against a "model" that simply generates random text. This functionality is intended to quickly show you how to run evals with the `qt` tool, without requiring you to set up API keys or spend money on tokens. Do not expect to draw conclusions from the results returned from this command.
> Important note: this `qt run` command will run the [`simpleqa-verified`](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) benchmark against a "model" that simply generates random text. This functionality is intended to quickly show you how to run evals with the `qt` tool, without requiring you to set up API keys or spend money on tokens. Do not expect to draw conclusions from the results returned from this command.

Inspect the recorded run:

Expand Down Expand Up @@ -67,7 +67,7 @@ qt compare <run_id_a> <run_id_b>

> Note: you can pass the `--json` flag to any of the above commands, to output machine and agent-friendly JSON instead of human-formatted output.

To learn more about what you can do with the CLI, see [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli).
See the [CLI reference](https://quantiles.io/documentation/reference/cli) for available commands, options, and usage details.

### Configuration file and customization

Expand Down Expand Up @@ -98,8 +98,8 @@ The CLI will execute the command with `QUANTILES_RUN_ID`, `QUANTILES_WORKFLOW_NA

See the following resources for more details:

- [`CONFIG.md`](./CONFIG.md) - in-depth guides to configuration and reference
- [`./cli/examples/configs`](./cli/examples/configs) - complete examples, including a [custom code benchmark example](./cli/examples/configs/custom_code/quantiles.toml)
- [Configuration guide](./CONFIG.md) - Detailed configuration instructions and reference documentation.
- [Complete configuration examples](./cli/examples/configs) - Complete examples, including a [custom-code benchmark configuration](./cli/examples/configs/custom_code/quantiles.toml)

## Local-First and Offline by Default

Expand All @@ -124,7 +124,7 @@ Built-in benchmarks are ready-to-run evaluations with predefined datasets, scori
| `qt run <benchmark>` | Run a built-in benchmark against the demo model to inspect sample-level inputs and outputs, scoring behavior, workflow steps, and aggregate metrics |
| `qt run <benchmark> --input '{"model":"<model_name>}'` | Run a built-in benchmark against your model |

Quantiles also provides a [benchmark hub](https://quantiles.io/benchmark-hub) for discovering built-in benchmarks, understanding their evaluation setup, and reviewing common metrics used across AI evaluation workflows.
Quantiles also provides a [Benchmark Hub](https://quantiles.io/benchmark-hub) for discovering built-in benchmarks, understanding their evaluation setup, and reviewing common metrics used across AI evaluation workflows.

### Add a built-in benchmark

Expand All @@ -134,7 +134,7 @@ Helpful requests include the benchmark name, source dataset or repository, licen

## Custom Evaluations

A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) program that is run by the `qt` CLI and uses its [local storage](https://quantiles.io/documentation/local-first-offline) and [durable workflow engine](https://quantiles.io/documentation/workflows-and-steps) to run efficiently and reliably. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages [durable steps, step caching, and step resume](https://quantiles.io/documentation/workflows-and-steps), metrics, inputs, outputs, and comparisons.
A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) program that is run by the `qt` CLI and uses its [local storage](https://quantiles.io/documentation/local-first-offline) and [durable workflow engine](https://quantiles.io/documentation/workflows-and-steps) to run efficiently and reliably. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages durable steps, step caching, and step resume, metrics, inputs, outputs, and comparisons.

Custom evaluations are configured in `quantiles.toml` with `type = "custom_code"`:

Expand All @@ -151,21 +151,21 @@ Run the evaluation with `qt run my-eval`. If it fails, resume it later with `qt

Use custom evaluations when you need to measure behavior that is specific to your product, workflow, prompt, dataset, rubric, or release process.

Read more about how to build and run custom evaluations at [quantiles.io/documentation/custom-evaluations](https://quantiles.io/documentation/custom-evaluations).
See the [custom evaluations guide](https://quantiles.io/documentation/custom-evaluations) for instructions on building and running custom evaluations.

### Python SDK

Use the official Quantiles Python SDK to build your custom evaluations with primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python. The SDK integrates tightly with the `qt` CLI’s local API for running, recording, and analyzing benchmarks.

The code for the Python SDK is located in this repository at [`./python/`](./python). Read more about it at [quantiles.io/documentation/reference/python-sdk](https://quantiles.io/documentation/reference/python-sdk).
The [Python SDK source code](./python) is available in this repository. See the [Python SDK reference](https://quantiles.io/documentation/reference/python-sdk) for usage and API documentation.

## Coding Agents

Quantiles is designed to work well with coding agents such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and other agentic development tools. For a concise, public, LLM-readable overview of Quantiles with links to agent guides and related documentation, see [quantiles.io/llms.txt](https://quantiles.io/llms.txt).
Quantiles is designed for use with coding agents such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, and OpenCode. The [Quantiles `llms.txt` file](https://quantiles.io/llms.txt) provides a concise, public, LLM-readable overview with links to agent guides and related documentation.

### `SKILL.md`

The [github.com/quantiles-evals/skill](https://github.com/quantiles-evals/skill) repository includes a [`SKILL.md`](https://github.com/quantiles-evals/skill/blob/main/SKILL.md) file that gives agents complete instructions for running evaluations, inspecting results, comparing runs, and summarizing regressions. To use the skill with your agent, install it with the following prompt:
The [Quantiles agent skill repository](https://github.com/quantiles-evals/skill) provides a [`SKILL.md`](https://github.com/quantiles-evals/skill/blob/main/SKILL.md) instruction file that guides coding agents through creating, running, analyzing, and comparing evaluations. Use the following agent prompt to install it:

```text
Install the Quantiles eval skill at github.com/quantiles-evals/skill
Expand All @@ -183,7 +183,7 @@ The embedded [`AGENTS.md` file](./AGENTS.md) gives agents repository-specific in

## Documentation

Full documentation is available at [quantiles.io/documentation](https://quantiles.io/documentation/).
See the [Quantiles documentation](https://quantiles.io/documentation/) for comprehensive guides and reference documentation.

Start here:

Expand All @@ -199,7 +199,7 @@ Please read our [contributing guide](./CONTRIBUTING.md) to get started.

## Security

Please do not report security vulnerabilities through public GitHub issues. Follow the reporting guidance in [SECURITY.md](./SECURITY.md).
Please do not report security vulnerabilities through public GitHub issues. Follow the security reporting guidance in [SECURITY.md](./SECURITY.md).

## License

Expand Down
8 changes: 4 additions & 4 deletions cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ qt list
qt show 1
```

> See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands.
> See the [CLI reference](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands.

### Custom evaluations

Expand All @@ -49,14 +49,14 @@ qt run my-eval
qt resume <run_id>
```

See [`examples/configs/custom_code/quantiles.toml`](./examples/configs/custom_code/quantiles.toml) for a complete working example.
See the [custom-code configuration example](./examples/configs/custom_code/quantiles.toml) for a complete working configuration.

## Configuration files and customization

You can customize how the CLI executes built-in benchmarks and custom evaluations using a `quantiles.toml` or `.quantiles.toml` configuration file. See the following resources for information and examples:

- [`../CONFIG.md`](../CONFIG.md) for a guide and reference.
- [`./examples/configs`](./examples/configs) for complete working examples.
- [Configuration reference](../CONFIG.md) for configuration guidance and supported options.
- [Configuration examples](./examples/configs) for complete working configurations.

### Built-in benchmarks

Expand Down
2 changes: 1 addition & 1 deletion python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ uv add quantiles

## Usage

To build a custom eval with Python, use the following code. To ensure this eval is runnable with `qt run`, set up a `quantiles.toml` configuration file. See [`../CONFIG.md`](../CONFIG.md) for details.
Use the following code to build a custom evaluation with Python. To run it with `qt run`, configure it in a `quantiles.toml` file as described in the [configuration guide](../CONFIG.md).

```python
import asyncio
Expand Down
4 changes: 2 additions & 2 deletions typescript/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,14 @@ bun install quantiles

## Usage

To build a custom eval with TypeScript, use the following code. To ensure this eval is runnable with `qt run`, set up a `quantiles.toml` configuration file. See [`../CONFIG.md`](../CONFIG.md) for details.
Use the following code to build a custom evaluation with TypeScript. To run it with `qt run`, configure it in a `quantiles.toml` file as described in the [configuration guide](../CONFIG.md).`

```ts
import { QuantilesClient } from "@quantiles/sdk";

const client = new QuantilesClient();
const run = await client.createRun("eval-smoke-test", {
model: "gpt-4.1-mini",
model: "gpt-5.6",
});

const output = await run.step("call-model", { prompt: "hello" }, async () => {
Expand Down
Loading