agent.md and other docs tweaks

.gitignore

@@ -1,3 +1,4 @@
 .envrc
 .DS_Store
 /.quantiles
+node_modules

AGENTS.md

@@ -91,7 +91,7 @@ Start with these files before making public-facing changes:
 - [`SECURITY.md`](./SECURITY.md): supported components and vulnerability reporting process.
 - [`CODE_OF_CONDUCT.md`](./CODE_OF_CONDUCT.md): community participation rules.
 - [`LICENSE`](./LICENSE): Apache 2.0 license text.
-- `CHANGELOG.md`: notable public changes, when present.
+- [`mise.toml`](./mise.toml): task definitions for building, formatting, type-checking, linting, and more, all using the [mise](https://mise.en.dev/) task runner.
 
 ## Agent Instruction Boundaries
 
@@ -122,7 +122,7 @@ Use these terms consistently in public docs:
 - `step`: a durable recorded unit of an evaluation execution.
 - `metric`: a measured value emitted during a run.
 - `event`: recorded observability data from an evaluation.
-- `.quantiles/`: local Quantiles workspace state.
+- `.quantiles/`: local Quantiles workspace state, including SQLite database and metrics Parquet files.
 
 Prefer `local-first` and `offline by default` for open-source behavior.
 
@@ -134,14 +134,13 @@ Use the `qt` CLI as the source of truth for running, listing, inspecting, compar
 
 Prefer CLI output over manually reading `.quantiles/` files. Do not manually edit or delete `.quantiles/` files unless explicitly asked.
 
-Run `qt init` before other `qt` commands if the repository has not been initialized.
+Although `qt init` exists to initialize the local Quantiles database, `qt run` automatically does the same. `qt init` is thus often unnecessary to run explicitly.
 
 Use `--json` for `qt run`, `qt list`, `qt show`, and `qt compare` when producing agent summaries. Inspect selected runs with `qt show <run_id> --json`.
 
 Common commands:
 
 ```bash
-qt init
 qt run <evaluation> --json
 qt list --json
 qt show <run_id> --json
@@ -209,7 +208,6 @@ The `qt` CLI is a local-first Rust CLI for running, recording, inspecting, and c
 It creates local SQLite state under `.quantiles/quantiles.sqlite` and provides commands such as:
 
 ```bash
-qt init
 qt run
 qt list
 qt show
@@ -228,37 +226,25 @@ When editing the Python subdirectory, preserve async behavior, stable JSON paylo
 
 ### TypeScript SDK
 
-The TypeScript SDK is an ESM SDK for authoring local AI workload workflows against the Quantiles local observability server at `http://127.0.0.1:8765` by default.
+The TypeScript SDK is an SDK for authoring local AI workload workflows against the Quantiles local observability server at `http://127.0.0.1:8765` by default.
 
 It exposes workflow primitives, `QuantilesClient`, `QuantilesRun`, stable JSON utilities, and shared SDK types.
 
 When editing the TypeScript subdirectory, preserve strict typing, JSON-serializable public surfaces, ESM behavior, and documented package exports.
 
 ### Agent Skill
 
-The [`skill`](./skill/) subdirectory contains reusable instructions for coding agents that use Quantiles to run, inspect, compare, and summarize evaluation workflows.
+The [`skill`](https://github.com/quantiles-evals/skill) repository contains reusable instructions for coding agents that use Quantiles to run, inspect, compare, and summarize evaluation workflows.
 
 When editing the skill subdirectory, keep instructions operational, command-driven, and safe for public use.
 
-Read [`skill/SKILL.md`](./skill/SKILL.md) for the reusable agent skill instructions.
-
-### Benchmarks
-
-The `benchmarks` subdirectory may contain built-in or example benchmark harnesses, datasets, fixtures, scoring logic, and benchmark documentation.
-
-When editing benchmark content:
-
-- Preserve dataset provenance, scoring behavior, and benchmark limitations.
-- Record benchmark source, version, commit, dataset revision, scoring configuration, and any local patches when relevant.
-- Do not present demo sampler results as model-quality benchmark evidence.
-- Avoid adding large datasets, generated outputs, or cached results unless explicitly required and appropriate for the repository.
-- If benchmark content changes, call out whether the change affects comparability with prior runs.
+Read [github.com/quantiles-evals/skill](https://github.com/quantiles-evals/skill) for the reusable agent skill instructions.
 
 ## Working In This Repository
 
 Keep root-level changes focused on public orientation, documentation, contribution guidance, security guidance, licensing, and agent instructions.
 
-Update docs when any of the following change:
+Update documentation in this repository (e.g. `README.md`, etc...) when any of the following change:
 
 - CLI commands, flags, outputs, or setup steps.
 - SDK APIs, imports, examples, or package names.
@@ -280,7 +266,6 @@ Use the Quantiles CLI as the source of truth for local runs.
 Prefer this flow:
 
 ```bash
-qt init
 qt run <evaluation>
 qt show <run_id> --json
 ```

README.md

@@ -18,6 +18,8 @@ With Quantiles, teams can rely on built-in infrastructure:
 - Inspect and compare runs directly from the same `qt` CLI
 - Resilient execution by default with caching and restartable failed runs
 
+Quantiles borrows concepts from durable workflow execution systems to make evaluation runs resilient to crashes and restarts, while adding a high-throughput execution engine, rich observability, metrics, and eval reproducibility. Use it to run custom eval code or built-in benchmarks, then inspect what changed across runs, all without notebooks, pipelines, manual comparisons, or cloud services.
+
 ## Quickstart
 
 Install the CLI:
@@ -67,6 +69,12 @@ qt compare <run_id_a> <run_id_b>
 
 >Note: you can pass the `--json` flag to any of the above commands, to output machine and agent-friendly JSON instead of human-formatted output.
 
+To learn more detail about what you can do with the CLI, see [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli).
+
+### Customization
+
+You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details.
+
 ## Local-First and Offline by Default
 
 Quantiles is built as a local-first, offline system that keeps benchmark execution, metadata, metrics, and analysis on your computer by default.
@@ -92,23 +100,15 @@ Built-in benchmarks are ready-to-run evaluation harnesses with predefined datase
 
 ## Custom Evaluations
 
-<!--TODO:
-
-- "Quantiles API" should link to docs for API reference
-- "Python" should link to Python SDK docs
-- "Typescript" should link to Typescript SDK docs
-
--->
-
-A custom evaluation is a Python or TypeScript program that is run by the `qt` CLI and uses the Quantiles API to execute an eval. Your code owns the evaluation logic: loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles records the run, durable steps, emitted metrics, events, inputs, outputs, and comparisons.
+A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) or [TypeScript](https://quantiles.io/documentation/reference/typescript-sdk) program that is run by the `qt` CLI and uses the [Quantiles API](https://quantiles.io/documentation/reference/rest-api) to execute an eval. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages [durable steps, step caching, and step resume](https://quantiles.io/documentation/workflows-and-steps), metrics, inputs, outputs, and comparisons.
 
 Use custom evaluations when you need to measure behavior that is specific to your product, workflow, prompt, dataset, rubric, or release process.
 
 Read more about how to build and run custom evaluations at [quantiles.io/documentation/custom-evaluations](https://quantiles.io/documentation/custom-evaluations).
 
-## SDKs
+### SDKs
 
-Use the official Quantiles SDKs to build with higher-level workflow primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python and TypeScript. The SDKs integrate tightly with the `qt` CLI’s local API for running, recording and analyzing benchmarks.
+Use the official Quantiles SDKs to build your custom evaluations with higher-level workflow primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python and TypeScript. The SDKs integrate tightly with the `qt` CLI’s local API for running, recording and analyzing benchmarks.
 
 The Python SDK is located in this repository at [`python/`](./python). Read more about it at [quantiles.io/documentation/reference/python-sdk](https://quantiles.io/documentation/reference/python-sdk). The TypeScript SDK is located in this repository at [`typescript/`](./typescript). Read more about it at [quantiles.io/documentation/reference/typescript-sdk](https://quantiles.io/documentation/reference/typescript-sdk).
 

cli/.gitignore

@@ -22,7 +22,6 @@ target
 /.quantiles
 /.wrangler
 /dist
-/node_modules
 .fastembed_cache
 .envrc
 /qt-linux

cli/AGENTS.md

@@ -9,7 +9,7 @@
 - Prefer focused changes that fit the current CLI and library structure.
 - Keep local-only behavior explicit; do not introduce network or cloud behavior unless the task specifically calls for it. If you do add such behavior, make sure to alert the user about it, and add comments in the code to draw attention to it.
 - Preserve existing SQLite data model assumptions unless the change includes a deliberate schema migration or initialization update.
-- Use idiomatic Rust and keep error handling clear. This project uses `anyhow` for application-level errors.
+- Use idiomatic Rust and keep error handling clear. This project uses the [`anyhow`](https://docs.rs/anyhow) to create and propagate application-level errors.
 - Avoid broad refactors while implementing narrow behavior changes.
 
 ## Validation and Testing