Adding qt resume command

CONFIG.md

@@ -0,0 +1,183 @@
+# Configuration Guide
+
+Quantiles uses a single TOML configuration file in the current working directory to describe how benchmarks and custom evaluations are executed. The CLI looks for `quantiles.toml` or `.quantiles.toml` in the current working directory. If both exist, the CLI exits with an ambiguity error.
+
+## When you need a config file
+
+You don't need a config file to run built-in benchmarks. `qt run pubmedqa` works out of the box. You do, however, need a config file when you want to do one or more of the following:
+
+- Override built-in benchmark defaults (e.g. model, sample limit)
+- Define custom evaluations (`type = "custom_code"`)
+- Resume custom evaluations later with `qt resume <run_id>`
+
+## File location and name
+
+The CLI looks for a configuration file in the current working directory, in this order:
+
+1. `./quantiles.toml`
+2. `./.quantiles.toml`
+
+Only one may exist in a given directory.
+
+## Top-level structure
+
+Every benchmark lives under the `[benchmarks.<name>]` section. The section name is the workflow name you pass to `qt run <name>`.
+
+```toml
+[benchmarks.pubmedqa]
+# builtin fields...
+
+[benchmarks.my-eval]
+type = "custom_code"
+# custom_code fields...
+```
+
+## Benchmark types
+
+Every benchmark section has a `type` field. Valid values are `"builtin"` (default when absent) and `"custom_code"`.
+
+### `builtin`
+
+Built-in benchmarks run natively inside the CLI. Their config sections may contain:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `type` | string | no | Defaults to `"builtin"`. May be omitted for built-in benchmarks. |
+| `samples` | integer | no | Number of dataset rows to evaluate. |
+| `model` | string or table | no | Model sampler. See [model format](#model-format). |
+| `max_workers` | integer | no | Maximum concurrent workers. |
+
+If none of these fields are present, the built-in uses its own defaults and no config JSON is generated.
+
+### `custom_code`
+
+Custom evaluations are external programs run as child processes and generally built with one of the Quantiles SDKs. Their config sections contain the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `type` | string | yes | Must be `"custom_code"`. |
+| `command` | array of strings | yes | Command and arguments to execute. |
+| `input` | table | no | Structured input passed to the child as `QUANTILES_INPUT`. |
+
+The CLI injects these environment variables into the child process:
+
+- `QUANTILES_RUN_ID` — the run ID
+- `QUANTILES_WORKFLOW_NAME` — the benchmark name
+- `QUANTILES_BASE_URL` — local API base URL
+- `QUANTILES_INPUT` — JSON string from the `input` table
+
+## Model format
+
+The `model` field accepts either a provider-prefixed string or a table.
+
+### String form
+
+```toml
+model = "openai:gpt-5.4-nano"
+```
+
+Supported provider prefixes are listed below:
+
+- `openai:`
+- `anthropic:`
+- `gemini:`
+- `cloudflare:`
+
+### Table form
+
+```toml
+model = { provider = "cloudflare", model_id = "@cf/..." }
+```
+
+## Input tables
+
+For `custom_code` benchmarks, `input` is an arbitrary TOML table that becomes a JSON object in the custom eval (e.g. a `dict` in Python and a `Map` in TypeScript):
+
+```toml
+[benchmarks.my-eval]
+type = "custom_code"
+command = ["python", "eval.py"]
+
+[benchmarks.my-eval.input]
+dataset = "my_data.jsonl"
+max_samples = 100
+
+[benchmarks.my-eval.input.nested]
+foo = "bar"
+```
+
+This produces:
+
+```json
+{"dataset":"my_data.jsonl","max_samples":100,"nested":{"foo":"bar"}}
+```
+
+An empty input table (`[benchmarks.my-eval.input]` with no keys) deserializes as an empty JSON object.
+
+## CLI `--input` overrides
+
+You can override or extend config input at runtime:
+
+```bash
+qt run my-eval --input '{"max_samples":50}'
+```
+
+The CLI merges the `--input` JSON object into the config `input` table. If a key exists in both, the CLI value wins and a warning is printed:
+
+```
+Warning: --input overrides config input for keys: max_samples
+```
+
+In `--json` mode, the warning is included in the JSON output under the `warning` key.
+
+## Config validation
+
+The CLI validates benchmark configs before execution:
+
+- `builtin` sections may **not** contain `command` or `input` fields.
+- `custom_code` sections **must** contain a non-empty `command` array.
+- `custom_code` sections may **not** contain builtin-only fields like `samples` or `model`.
+- Unknown `type` values are rejected.
+
+Validation failures produce clear error messages before any run is created.
+
+## Resuming and the config file
+
+When you run `qt resume <run_id>`, the CLI looks up the stored workflow name and input from the database, then re-reads the command from the current config file. This means:
+
+- `qt resume` provides no `--input` flag, and you do not need to re-submit input parameters on resume.
+- If you edited the config file between `qt run` and `qt resume`, the resumed run uses the _updated_ command.
+- If the config section has been removed, resuming a `custom_code` benchmark fails with "no config section found".
+
+## Complete examples
+
+### Built-in with model override
+
+```toml
+[benchmarks.pubmedqa]
+samples = 50
+model = "openai:gpt-5.4-nano"
+max_workers = 100
+```
+
+### Built-in with demo model (no fields needed)
+
+```toml
+[benchmarks.simpleqa-verified]
+samples = 10
+```
+
+### Custom evaluation
+
+```toml
+[benchmarks.hello]
+type = "custom_code"
+command = ["python3", "hello.py"]
+
+[benchmarks.hello.input]
+greeting = "world"
+```
+
+### Custom evaluation with failure simulation
+
+See [`cli/examples/configs/custom_code/quantiles.toml`](./cli/examples/configs/custom_code/quantiles.toml) for a complete, commented example including a sample Python script.

README.md

@@ -69,9 +69,37 @@ qt compare <run_id_a> <run_id_b>
 
 To learn more detail about what you can do with the CLI, see [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli).
 
-### Customization
+### Configuration file and customization
 
-You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details.
+You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory.
+
+For **built-in benchmarks**, configure settings like `samples`, `model`, and `max_workers`:
+
+```toml
+[benchmarks.pubmedqa]
+samples = 50
+model = "openai:gpt-5.4-nano"
+max_workers = 100
+```
+
+For **custom evaluations**, set `type = "custom_code"` and provide the `command` to run:
+
+```toml
+[benchmarks.my-eval]
+type = "custom_code"
+command = ["python", "my_eval.py"]
+
+[benchmarks.my-eval.input]
+foo = "foo_val"
+bar = "bar_val"
+```
+
+The CLI will execute the command with `QUANTILES_RUN_ID`, `QUANTILES_WORKFLOW_NAME`, `QUANTILES_BASE_URL`, and `QUANTILES_INPUT` environment variables injected. If the run fails, you can resume it later with `qt resume <run_id>`.
+
+See the following resources for more details:
+
+- [`CONFIG.md`](./CONFIG.md) - in-depth guides to configuration and reference
+- [`./cli/examples/configs`](./cli/examples/configs) - complete examples, including a [custom code benchmark example](./cli/examples/configs/custom_code/quantiles.toml)
 
 ## Local-First and Offline by Default
 
@@ -102,6 +130,19 @@ Quantiles also provides a [benchmark hub](https://quantiles.io/benchmark-hub) fo
 
 A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) or [TypeScript](https://quantiles.io/documentation/reference/typescript-sdk) program that is run by the `qt` CLI and uses the [Quantiles API](https://quantiles.io/documentation/reference/rest-api) to execute an eval. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages [durable steps, step caching, and step resume](https://quantiles.io/documentation/workflows-and-steps), metrics, inputs, outputs, and comparisons.
 
+Custom evaluations are configured in `quantiles.toml` with `type = "custom_code"`:
+
+```toml
+[benchmarks.my-eval]
+type = "custom_code"
+command = ["python", "my_eval.py"]
+
+[benchmarks.my-eval.input]
+dataset = "my_dataset.jsonl"
+```
+
+Run the evaluation with `qt run my-eval`. If it fails, resume it later with `qt resume <run_id>` — the CLI re-reads the command and stored input automatically.
+
 Use custom evaluations when you need to measure behavior that is specific to your product, workflow, prompt, dataset, rubric, or release process.
 
 Read more about how to build and run custom evaluations at [quantiles.io/documentation/custom-evaluations](https://quantiles.io/documentation/custom-evaluations).

cli/README.md

@@ -16,9 +16,12 @@ A few commands to see `qt` in action:
 # 1. Initialize a workspace
 qt init
 
-# 2. Run an eval — Quantiles auto-starts a local server, records the run, and tears the server
-# down when the command finishes.
-qt run my-eval -- bun run sdk/typescript/examples/run_demo.ts
+# 2. Run a built-in eval
+# 
+# Note that you can also build and run your own custom evals
+# with the `qt` CLI. See the below "Custom evaluations" section
+# for details.
+qt run pubmedqa
 
 # 3. List and inspect what happened
 qt list
@@ -27,16 +30,66 @@ qt show 1
 
 >See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands.
 
+### Custom evaluations
+
+Custom evaluations are denoted in the configuration file with `type = "custom_code"`. The `command` array tells the CLI how to execute your eval, and the optional `input` table is merged with any values passed in the `qt run --input` flag, then passed to your script as `QUANTILES_INPUT`. An example is below
+
+```toml
+[benchmarks.my-eval]
+type = "custom_code"
+command = ["python", "my_eval.py"]
+input = {dataset = "my_dataset.jsonl"}
+```
+
+```bash
+# Run the custom evaluation
+qt run my-eval
+
+# If it fails, resume with only the run ID
+qt resume <run_id>
+```
+
+See [`examples/configs/custom_code/quantiles.toml`](./examples/configs/custom_code/quantiles.toml) for a complete working example.
+
+## Configuration files and customization
+
+You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file.
+
+For **built-in benchmarks**, configure settings like `samples`, `model`, and `max_workers`:
+
+```toml
+[benchmarks.pubmedqa]
+samples = 50
+model = "openai:gpt-5.4-nano"
+max_workers = 100
+```
+
+For **custom evaluations**, set `type = "custom_code"` and provide the `command` to run. The optional `input` table is passed to your script as `QUANTILES_INPUT`.
+
+```toml
+[benchmarks.my-eval]
+type = "custom_code"
+command = ["python", "my_eval.py"]
+
+[benchmarks.my-eval.input]
+foo = "foo_val"
+```
+
+See [`./examples/configs`](./examples/configs) for complete working examples.
+
+>Note: Quantiles is designed for high-throughput execution and may issue many requests in parallel. Depending on your provider, model, and account limits, benchmark runs can quickly hit API rate limits or concurrency quotas. Consider reducing concurrency or using models/providers with higher rate limits if you encounter throttling. Example configurations illustrate how to do so.
+
+
 ### Comparing runs
 
 After iterating on an eval, you can compare two runs to see exactly what changed:
 
 ```bash
 # Run A — baseline
-qt run my-eval -- bun run sdk/typescript/examples/run_demo.ts
+qt run my-eval
 
 # Run B — your latest iteration
-qt run my-eval -- bun run sdk/typescript/examples/run_demo.ts
+qt run my-eval
 
 # See what changed between them
 qt compare 1 2
@@ -81,9 +134,3 @@ The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while `
 - **Client** (your script) owns code execution: the server never runs your logic
   - Note that the CLI itself also has built-in benchmarks, which do not involve your code
 - **CLI** reads the same SQLite database the server writes to
-
-## Customization
-
-You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details.
-
->Note: Quantiles is designed for high-throughput execution and may issue many requests in parallel. Depending on your provider, model, and account limits, benchmark runs can quickly hit API rate limits or concurrency quotas. Consider reducing concurrency or using models/providers with higher rate limits if you encounter throttling. Example configurations illustrate how to do so.

cli/examples/configs/custom_code/my_eval.py

@@ -0,0 +1,28 @@
+"""A minimal custom eval script for the Quantiles CLI example."""
+
+import json
+import os
+import sys
+
+run_id = os.environ.get("QUANTILES_RUN_ID", "unknown")
+workflow = os.environ.get("QUANTILES_WORKFLOW_NAME", "unknown")
+base_url = os.environ.get("QUANTILES_BASE_URL", "unknown")
+input_json = os.environ.get("QUANTILES_INPUT", "{}")
+
+try:
+    parsed = json.loads(input_json)
+except json.JSONDecodeError:
+    parsed = {}
+
+print(
+    {
+        "run_id": run_id,
+        "eval_name": workflow,
+        "base_url": base_url,
+        "parsed_input": parsed,
+    }
+)
+
+if parsed.get("should_fail"):
+    print("Simulated failure triggered by input.", file=sys.stderr)
+    sys.exit(1)

cli/examples/configs/custom_code/quantiles.toml

@@ -0,0 +1,26 @@
+# This config file demonstrates running a custom evaluation with `qt run`.
+#
+# Custom benchmarks use `type = "custom_code"` and must specify a `command`
+# that the CLI will execute. The `input` table is optional and its keys are
+# passed to the child process via the `QUANTILES_INPUT` environment variable
+# as a JSON object.
+#
+# Usage from this directory:
+#
+#   qt run my-eval
+#
+# This will run the command below, injecting `QUANTILES_RUN_ID`,
+# `QUANTILES_WORKFLOW_NAME`, `QUANTILES_BASE_URL`, and `QUANTILES_INPUT`
+# into the child environment.
+#
+# If the run fails, you can resume it later with only the run ID:
+#
+#   qt resume <run_id>
+#
+# The CLI will look up the stored workflow name and input from the database,
+# then re-read the command from this config file.
+
+[benchmarks.my-eval]
+type = "custom_code"
+command = ["python", "my_eval.py"]
+input = {foo = "foo_val", bar = "bar_val"}