diff --git a/CONFIG.md b/CONFIG.md index 6ed2a10..ab4e093 100644 --- a/CONFIG.md +++ b/CONFIG.md @@ -73,7 +73,7 @@ You can pass a TOML table instead of such a prefixed string: model = { provider = "openai", model_id = "gpt-5.6" } ``` -Note that models require specific configuration based on the provider. For details, see the `quantiles.toml` file under the provider of your choice in [`cli/examples/configs`](./cli/examples/configs). +Note that models require specific configuration based on the provider. For details, see the `quantiles.toml` file under the provider of your choice in the [provider configuration examples](./cli/examples/configs). ### `custom_code` @@ -177,4 +177,4 @@ input = { greeting = "world" } ### Custom evaluation with failure simulation -See [`cli/examples/configs/custom_code/quantiles.toml`](./cli/examples/configs/custom_code/quantiles.toml) for a complete, commented example including a sample Python script. +See the complete [custom-code configuration example](./cli/examples/configs/custom_code/quantiles.toml) for an annotated configuration and sample Python script. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4e91911..b46385c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -31,15 +31,15 @@ git clone https://github.com/quantiles-evals/quantiles.git cd quantiles ``` -For the [`./cli`](./cli) directory, ensure you have Rust installed and use the `mise.toml` file in that directory to run common build/lint/check/test commands. For the [`./python`](./python) directory, use the [`uv`](https://docs.astral.sh/uv/) tool for similar commands. The [`typescript/`](./typescript) directory holds the yet-unreleased, unsupported TypeScript SDK. +For the [CLI directory](./cli), ensure you have Rust installed and use the `mise.toml` file in that directory to run common build/lint/check/test commands. For the [Python SDK directory](./python), use the [`uv`](https://docs.astral.sh/uv/) tool for similar commands. The [TypeScript SDK directory](./typescript) holds the yet-unreleased, unsupported TypeScript SDK. ## Security -Please do not report security vulnerabilities through public GitHub issues. Follow the reporting guidance in [SECURITY.md](./SECURITY.md). +Please do not report security vulnerabilities through public GitHub issues. Follow the security reporting guidance in [SECURITY.md](./SECURITY.md). ## Code of Conduct -By participating in this project, you agree to follow our [CODE_OF_CONDUCT.md](./CODE_OF_CONDUCT.md). +By participating in this project, you agree to follow our [Code of Conduct](./CODE_OF_CONDUCT.md). ## License diff --git a/README.md b/README.md index 553dd56..5e24ee3 100644 --- a/README.md +++ b/README.md @@ -26,13 +26,13 @@ Install the CLI: curl -fsSL https://cli.quantiles.io/install.sh | bash ``` -Run the [SimpleQA-verified](https://arxiv.org/abs/2509.07968) built-in benchmark: +Run the [SimpleQA Verified](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) built-in benchmark: ```bash qt run simpleqa-verified ``` -> Important note: this `qt run` command will run the [`simpleqa-verified`](https://arxiv.org/abs/2509.07968) benchmark against a "model" that simply generates random text. This functionality is intended to quickly show you how to run evals with the `qt` tool, without requiring you to set up API keys or spend money on tokens. Do not expect to draw conclusions from the results returned from this command. +> Important note: this `qt run` command will run the [`simpleqa-verified`](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) benchmark against a "model" that simply generates random text. This functionality is intended to quickly show you how to run evals with the `qt` tool, without requiring you to set up API keys or spend money on tokens. Do not expect to draw conclusions from the results returned from this command. Inspect the recorded run: @@ -67,7 +67,7 @@ qt compare > Note: you can pass the `--json` flag to any of the above commands, to output machine and agent-friendly JSON instead of human-formatted output. -To learn more about what you can do with the CLI, see [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli). +See the [CLI reference](https://quantiles.io/documentation/reference/cli) for available commands, options, and usage details. ### Configuration file and customization @@ -98,8 +98,8 @@ The CLI will execute the command with `QUANTILES_RUN_ID`, `QUANTILES_WORKFLOW_NA See the following resources for more details: -- [`CONFIG.md`](./CONFIG.md) - in-depth guides to configuration and reference -- [`./cli/examples/configs`](./cli/examples/configs) - complete examples, including a [custom code benchmark example](./cli/examples/configs/custom_code/quantiles.toml) +- [Configuration guide](./CONFIG.md) - Detailed configuration instructions and reference documentation. +- [Complete configuration examples](./cli/examples/configs) - Complete examples, including a [custom-code benchmark configuration](./cli/examples/configs/custom_code/quantiles.toml) ## Local-First and Offline by Default @@ -124,7 +124,7 @@ Built-in benchmarks are ready-to-run evaluations with predefined datasets, scori | `qt run ` | Run a built-in benchmark against the demo model to inspect sample-level inputs and outputs, scoring behavior, workflow steps, and aggregate metrics | | `qt run --input '{"model":"}'` | Run a built-in benchmark against your model | -Quantiles also provides a [benchmark hub](https://quantiles.io/benchmark-hub) for discovering built-in benchmarks, understanding their evaluation setup, and reviewing common metrics used across AI evaluation workflows. +Quantiles also provides a [Benchmark Hub](https://quantiles.io/benchmark-hub) for discovering built-in benchmarks, understanding their evaluation setup, and reviewing common metrics used across AI evaluation workflows. ### Add a built-in benchmark @@ -134,7 +134,7 @@ Helpful requests include the benchmark name, source dataset or repository, licen ## Custom Evaluations -A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) program that is run by the `qt` CLI and uses its [local storage](https://quantiles.io/documentation/local-first-offline) and [durable workflow engine](https://quantiles.io/documentation/workflows-and-steps) to run efficiently and reliably. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages [durable steps, step caching, and step resume](https://quantiles.io/documentation/workflows-and-steps), metrics, inputs, outputs, and comparisons. +A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) program that is run by the `qt` CLI and uses its [local storage](https://quantiles.io/documentation/local-first-offline) and [durable workflow engine](https://quantiles.io/documentation/workflows-and-steps) to run efficiently and reliably. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages durable steps, step caching, and step resume, metrics, inputs, outputs, and comparisons. Custom evaluations are configured in `quantiles.toml` with `type = "custom_code"`: @@ -151,21 +151,21 @@ Run the evaluation with `qt run my-eval`. If it fails, resume it later with `qt Use custom evaluations when you need to measure behavior that is specific to your product, workflow, prompt, dataset, rubric, or release process. -Read more about how to build and run custom evaluations at [quantiles.io/documentation/custom-evaluations](https://quantiles.io/documentation/custom-evaluations). +See the [custom evaluations guide](https://quantiles.io/documentation/custom-evaluations) for instructions on building and running custom evaluations. ### Python SDK Use the official Quantiles Python SDK to build your custom evaluations with primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python. The SDK integrates tightly with the `qt` CLI’s local API for running, recording, and analyzing benchmarks. -The code for the Python SDK is located in this repository at [`./python/`](./python). Read more about it at [quantiles.io/documentation/reference/python-sdk](https://quantiles.io/documentation/reference/python-sdk). +The [Python SDK source code](./python) is available in this repository. See the [Python SDK reference](https://quantiles.io/documentation/reference/python-sdk) for usage and API documentation. ## Coding Agents -Quantiles is designed to work well with coding agents such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and other agentic development tools. For a concise, public, LLM-readable overview of Quantiles with links to agent guides and related documentation, see [quantiles.io/llms.txt](https://quantiles.io/llms.txt). +Quantiles is designed for use with coding agents such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, and OpenCode. The [Quantiles `llms.txt` file](https://quantiles.io/llms.txt) provides a concise, public, LLM-readable overview with links to agent guides and related documentation. ### `SKILL.md` -The [github.com/quantiles-evals/skill](https://github.com/quantiles-evals/skill) repository includes a [`SKILL.md`](https://github.com/quantiles-evals/skill/blob/main/SKILL.md) file that gives agents complete instructions for running evaluations, inspecting results, comparing runs, and summarizing regressions. To use the skill with your agent, install it with the following prompt: +The [Quantiles agent skill repository](https://github.com/quantiles-evals/skill) provides a [`SKILL.md`](https://github.com/quantiles-evals/skill/blob/main/SKILL.md) instruction file that guides coding agents through creating, running, analyzing, and comparing evaluations. Use the following agent prompt to install it: ```text Install the Quantiles eval skill at github.com/quantiles-evals/skill @@ -183,7 +183,7 @@ The embedded [`AGENTS.md` file](./AGENTS.md) gives agents repository-specific in ## Documentation -Full documentation is available at [quantiles.io/documentation](https://quantiles.io/documentation/). +See the [Quantiles documentation](https://quantiles.io/documentation/) for comprehensive guides and reference documentation. Start here: @@ -199,7 +199,7 @@ Please read our [contributing guide](./CONTRIBUTING.md) to get started. ## Security -Please do not report security vulnerabilities through public GitHub issues. Follow the reporting guidance in [SECURITY.md](./SECURITY.md). +Please do not report security vulnerabilities through public GitHub issues. Follow the security reporting guidance in [SECURITY.md](./SECURITY.md). ## License diff --git a/cli/README.md b/cli/README.md index 42747ce..8c75c28 100644 --- a/cli/README.md +++ b/cli/README.md @@ -28,7 +28,7 @@ qt list qt show 1 ``` -> See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. +> See the [CLI reference](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. ### Custom evaluations @@ -49,14 +49,14 @@ qt run my-eval qt resume ``` -See [`examples/configs/custom_code/quantiles.toml`](./examples/configs/custom_code/quantiles.toml) for a complete working example. +See the [custom-code configuration example](./examples/configs/custom_code/quantiles.toml) for a complete working configuration. ## Configuration files and customization You can customize how the CLI executes built-in benchmarks and custom evaluations using a `quantiles.toml` or `.quantiles.toml` configuration file. See the following resources for information and examples: -- [`../CONFIG.md`](../CONFIG.md) for a guide and reference. -- [`./examples/configs`](./examples/configs) for complete working examples. +- [Configuration reference](../CONFIG.md) for configuration guidance and supported options. +- [Configuration examples](./examples/configs) for complete working configurations. ### Built-in benchmarks diff --git a/python/README.md b/python/README.md index 4587eb1..1fc3adc 100644 --- a/python/README.md +++ b/python/README.md @@ -10,7 +10,7 @@ uv add quantiles ## Usage -To build a custom eval with Python, use the following code. To ensure this eval is runnable with `qt run`, set up a `quantiles.toml` configuration file. See [`../CONFIG.md`](../CONFIG.md) for details. +Use the following code to build a custom evaluation with Python. To run it with `qt run`, configure it in a `quantiles.toml` file as described in the [configuration guide](../CONFIG.md). ```python import asyncio diff --git a/typescript/README.md b/typescript/README.md index 2a63a4d..cc416fc 100644 --- a/typescript/README.md +++ b/typescript/README.md @@ -12,14 +12,14 @@ bun install quantiles ## Usage -To build a custom eval with TypeScript, use the following code. To ensure this eval is runnable with `qt run`, set up a `quantiles.toml` configuration file. See [`../CONFIG.md`](../CONFIG.md) for details. +Use the following code to build a custom evaluation with TypeScript. To run it with `qt run`, configure it in a `quantiles.toml` file as described in the [configuration guide](../CONFIG.md).` ```ts import { QuantilesClient } from "@quantiles/sdk"; const client = new QuantilesClient(); const run = await client.createRun("eval-smoke-test", { - model: "gpt-4.1-mini", + model: "gpt-5.6", }); const output = await run.step("call-model", { prompt: "hello" }, async () => {