quantiles-evals · arschles · Jun 29, 2026 · Jun 29, 2026 · Jun 29, 2026 · Jun 29, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -24,7 +24,7 @@ Common subdirectories include:
 
 - `cli`: Rust `qt` CLI, local server, and local run/metrics storage.
 - `python`: The `quantiles` Python SDK.
-- `typescript`: The `@quantiles` TypeScript SDK.
+- `typescript`: The `@quantiles` TypeScript SDK. (in development and unsupported)
 
 Note there is a separate repository, [github.com/quantiles-evals/skill](https://github.com/quantiles-evals/skill), that contains the agent skill.
 
@@ -85,7 +85,9 @@ When editing the Python SDK subdirectory, preserve `async` behavior, stable JSON
 
 ### `typescript/` directory
 
-The `typescript/` directory contains the Quantiles TypeScript SDK, which allows users to author custom local AI evals using the local Quantiles server and Typescript. It exposes workflow primitives such as `QuantilesClient`, `QuantilesRun`, stable JSON utilities, and shared SDK types.
+>**Not released**. The TypeScript SDK is currently unsupported and unreleased.
+
+The `typescript/` directory contains the Quantiles TypeScript SDK, which allows users to author custom local AI evals with the Quantiles stack using TypeScript. It exposes workflow primitives such as `QuantilesClient`, `QuantilesRun`, stable JSON utilities, and shared SDK types.
 
 When editing the TypeScript SDK subdirectory, preserve strict typing, JSON-serializable public surfaces, ESM behavior, and documented package exports.
 

diff --git a/CONFIG.md b/CONFIG.md
@@ -34,7 +34,7 @@ model = "openai:gpt-5.4-nano"
 # see below for the full reference
 ```
 
-Alternatively, if you want to build your own custom evaluation, using the Quantiles Python or TypeScript SDK, define that custom eval as follows:
+Alternatively, if you want to build your own custom evaluation using the Quantiles Python SDK, define that custom eval as follows:
 
 ```toml
 # This is configuration for a custom code benchmark called my-eval.
@@ -45,8 +45,10 @@ Alternatively, if you want to build your own custom evaluation, using the Quanti
 # This `type` parameter defaults to "builtin". for custom evals, be sure to override
 # the default with "custom_code" here.
 type = "custom_code"
-# This is the command that the CLI will execute to run your custom eval code.
-command = ["python", "my_eval.py"]
+# This is the command that the CLI will execute to run your custom eval code. We recommend
+# using the `uv` tool to configure and manage your Python evals, but this field can be any
+# command, so you're free to use any tools you want.
+command = ["uv", "run", "my_eval.py"]
 # see below for the full reference
 ```
 
@@ -97,7 +99,7 @@ Note that models require specific configuration based on the provider. For detai
 
 ### `custom_code`
 
-Custom evaluations are external programs built with one of the Quantiles SDKs. Their config sections contain the following fields:
+Custom evaluations are external programs built with the Quantiles Python SDK. Their config sections contain the following fields:
 
 | Field | Type | Required | Description |
 |--|--|--|--|
@@ -109,7 +111,7 @@ Note that custom code evaluations can customize the model in code. See the [PubM
 
 #### The `input` table
 
-For `custom_code` benchmarks, `input` is an arbitrary TOML table that becomes a JSON object in the custom eval (e.g. a `dict` in Python and a `Map` in TypeScript):
+For `custom_code` benchmarks, `input` is an arbitrary TOML table that becomes a Python `dict` in your custom eval:
 
 ```toml
 [benchmarks.my-eval]

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -31,9 +31,7 @@ git clone https://github.com/quantiles-evals/quantiles.git
 cd quantiles
 ```
 
-<!--
-TODO: describe different setups for python, typescript and CLI
--->
+For the [`./cli`](./cli) directory, ensure you have Rust installed and use the `mise.toml` file in that directory to run common build/lint/check/test commands. For the [`./python`](./python) directory, use the [`uv`](https://docs.astral.sh/uv/) tool for similar commands. The [`typescript/`](./typescript) directory holds the yet-unreleased, unsupported TypeScript SDK.
 
 ## Security
 

diff --git a/README.md b/README.md
@@ -8,7 +8,7 @@ It includes a CLI (called `qt`), SDKs, built-in benchmarks, local run history, a
 
 Evaluation workflows quickly outgrow one-off scripts once teams need caching, retries, dataset handling, metrics capture, and run comparison. Quantiles gives teams those primitives so they don't have to built them from scratch:
 
-- Write standard Python or TypeScript, with familiar developer patterns
+- Write standard Python, with familiar developer patterns
 - Run workflows locally from the CLI
 - Automatically record runs, steps, metrics, events, inputs, and final outputs
 - Store execution history locally in open data formats
@@ -107,7 +107,7 @@ Quantiles is built as a local-first, offline system that keeps benchmark executi
 
 The Quantiles toolchain, including the `qt` CLI, SDKs, on-disk data formats, and REST API, is optimized to use your local computing power by default instead of relying on cloud or other non-local resources.
 
-The CLI and SDKs (TypeScript or Python) support offline benchmark workflows, including the following local execution and analysis features:
+Both the CLI and Python SDK support offline benchmark workflows, including the following local execution and analysis features:
 
 - Benchmark code runs locally on your machine
 - Measurements are computed locally, except for remote model calls, hosted judges, external tools, and LLM-as-judge evaluations that may call remote providers (e.g. OpenAI, Anthropic, cloud providers, etc.)
@@ -128,7 +128,7 @@ Quantiles also provides a [benchmark hub](https://quantiles.io/benchmark-hub) fo
 
 ## Custom Evaluations
 
-A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) or [TypeScript](https://quantiles.io/documentation/reference/typescript-sdk) program that is run by the `qt` CLI and uses the [Quantiles API](https://quantiles.io/documentation/reference/rest-api) to execute an eval. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages [durable steps, step caching, and step resume](https://quantiles.io/documentation/workflows-and-steps), metrics, inputs, outputs, and comparisons.
+A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) program that is run by the `qt` CLI and uses its [local storage](http://quantiles.io/documentation/local-first-offline) and [durable workflow engine](https://quantiles.io/documentation/workflows-and-steps) to run efficiently and reliably. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages [durable steps, step caching, and step resume](https://quantiles.io/documentation/workflows-and-steps), metrics, inputs, outputs, and comparisons.
 
 Custom evaluations are configured in `quantiles.toml` with `type = "custom_code"`:
 
@@ -147,11 +147,11 @@ Use custom evaluations when you need to measure behavior that is specific to you
 
 Read more about how to build and run custom evaluations at [quantiles.io/documentation/custom-evaluations](https://quantiles.io/documentation/custom-evaluations).
 
-### SDKs
+### Python SDK
 
-Use the official Quantiles SDKs to build your custom evaluations with higher-level workflow primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python and TypeScript. The SDKs integrate tightly with the `qt` CLI’s local API for running, recording and analyzing benchmarks.
+Use the official Quantiles Python SDK to build your custom evaluations with primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python. The SDK integrates tightly with the `qt` CLI’s local API for running, recording and analyzing benchmarks.
 
-The Python SDK is located in this repository at [`python/`](./python). Read more about it at [quantiles.io/documentation/reference/python-sdk](https://quantiles.io/documentation/reference/python-sdk). The TypeScript SDK is located in this repository at [`typescript/`](./typescript). Read more about it at [quantiles.io/documentation/reference/typescript-sdk](https://quantiles.io/documentation/reference/typescript-sdk).
+The code for the Python SDK is located in this repository at [`./python/`](./python). Read more about it at [quantiles.io/documentation/reference/python-sdk](https://quantiles.io/documentation/reference/python-sdk)
 
 ## Coding Agents
 
@@ -184,7 +184,6 @@ Start here:
 - [Quickstart](https://quantiles.io/documentation/quickstart)
 - [Agent Overview](https://quantiles.io/documentation/evals-with-agents)
 - [Python SDK](https://quantiles.io/documentation/reference/python-sdk)
-- [TypeScript SDK](https://quantiles.io/documentation/reference/typescript-sdk)
 
 ## Contributing
 

diff --git a/SECURITY.md b/SECURITY.md
@@ -8,7 +8,6 @@ Quantiles provides security fixes for the latest released versions of actively m
 | --------------------------------------------- | --------------------------------------------- |
 | Latest `qt` CLI release                       | Yes                                           |
 | Latest Python SDK release                     | Yes                                           |
-| Latest TypeScript SDK release                 | Yes                                           |
 | Older releases                                | No, unless explicitly stated in release notes |
 
 ## Reporting a Vulnerability

diff --git a/cli/README.md b/cli/README.md
@@ -105,8 +105,7 @@ The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while `
 
 ```
 +--------------------------------------+
-|            Your Script               |
-|   (TypeScript / Python / Shell)      |
+|   Benchmark / Custom Eval (Python)   |
 +-------------------+------------------+
                     │
                     │  HTTP / JSON

diff --git a/python/examples/prompt_eval.py b/python/examples/prompt_eval.py
@@ -1,113 +1,118 @@
 from typing import Literal, TypedDict, cast
 
 from quantiles import (
-    Statistics,
-    emit,
-    entrypoint,
-    step,
-    workflow,
+  JsonValue,
+  Statistics,
+  emit,
+  entrypoint,
+  step,
+  workflow,
 )
 from quantiles.workflow_context import WorkflowContext
 
 Label = Literal["billing", "bug", "how_to"]
 
 
 class EvalInput(TypedDict):
-    prompt_version: str
+  prompt_version: str
 
 
 class EvalCase(TypedDict):
-    id: str
-    ticket: str
-    expected: Label
+  id: str
+  ticket: str
+  expected: Label
 
 
 class CaseResult(TypedDict):
-    id: str
-    expected: Label
-    prediction: Label
-    correct: bool
-    tokens_used: int
+  id: str
+  expected: Label
+  prediction: Label
+  correct: bool
+  tokens_used: int
 
 
 CASES: list[EvalCase] = [
-    {
-        "id": "double-charge",
-        "ticket": "I was charged twice for my subscription renewal.",
-        "expected": "billing",
-    },
-    {
-        "id": "csv-crash",
-        "ticket": "The app crashes every time I upload a CSV file.",
-        "expected": "bug",
-    },
-    {
-        "id": "invite-teammate",
-        "ticket": "Can you show me how to invite another teammate?",
-        "expected": "how_to",
-    },
+  {
+    "id": "double-charge",
+    "ticket": "I was charged twice for my subscription renewal.",
+    "expected": "billing",
+  },
+  {
+    "id": "csv-crash",
+    "ticket": "The app crashes every time I upload a CSV file.",
+    "expected": "bug",
+  },
+  {
+    "id": "invite-teammate",
+    "ticket": "Can you show me how to invite another teammate?",
+    "expected": "how_to",
+  },
 ]
 
 
 async def evaluate_case(
-    ctx: WorkflowContext,
-    item: EvalCase,
-    prompt_version: str,
+  ctx: WorkflowContext,
+  item: EvalCase,
+  prompt_version: str,
 ) -> CaseResult:
-    prediction = cast(
-        Label,
-        await step(
-            ctx,
-            step_key=f"case:{item['id']}",
-            input_value={
-                "sample_id": item["id"],
-                "ticket": item["ticket"],
-                "expected": item["expected"],
-                "prompt_version": prompt_version,
-            },
-            execute=lambda: call_model(item["ticket"]),
-        ),
-    )
-
-    return {
-        "id": item["id"],
+  prediction = cast(
+    Label,
+    await step(
+      ctx,
+      step_key=f"case:{item['id']}",
+      input_value={
+        "sample_id": item["id"],
+        "ticket": item["ticket"],
         "expected": item["expected"],
-        "prediction": prediction,
-        "correct": prediction == item["expected"],
-        "tokens_used": len(item["ticket"].split()),
-    }
-
-
-async def handler(input_value: EvalInput | None, ctx: WorkflowContext):
-    prompt_version = (input_value or {}).get("prompt_version", "A")
-    results: list[CaseResult] = []
-    for item in CASES:
-        results.append(await evaluate_case(ctx, item, prompt_version))
-
-    correct = sum(1 for result in results if result["correct"])
-    accuracy = Statistics.accuracy(correct, len(results))
-    tokens_used = sum(result["tokens_used"] for result in results)
-
-    await emit(ctx, "accuracy", accuracy)
-    await emit(ctx, "correct_count", float(correct))
-    await emit(ctx, "total_count", float(len(results)))
-    await emit(ctx, "tokens_used", float(tokens_used), "tokens")
-
-    return {
-        "accuracy": accuracy,
-        "correct_count": correct,
-        "total_count": len(results),
-        "results": results,
-    }
+        "prompt_version": prompt_version,
+      },
+      execute=lambda: call_model(item["ticket"]),
+    ),
+  )
+
+  return {
+    "id": item["id"],
+    "expected": item["expected"],
+    "prediction": prediction,
+    "correct": prediction == item["expected"],
+    "tokens_used": len(item["ticket"].split()),
+  }
+
+
+async def handler(input_value: EvalInput | None, ctx: WorkflowContext) -> JsonValue:
+  prompt_version = (input_value or {}).get("prompt_version", "A")
+  results: list[CaseResult] = []
+  for item in CASES:
+    results.append(await evaluate_case(ctx, item, prompt_version))
+
+  correct = sum(1 for result in results if result["correct"])
+  accuracy = Statistics.accuracy(correct, len(results))
+  tokens_used = sum(result["tokens_used"] for result in results)
+
+  await emit(ctx, "accuracy", accuracy)
+  await emit(ctx, "correct_count", float(correct))
+  await emit(ctx, "total_count", float(len(results)))
+  await emit(ctx, "tokens_used", float(tokens_used), "tokens")
+
+  return cast(
+    JsonValue,
+    # the below value is a valid JsonValue
+    {
+      "accuracy": accuracy,
+      "correct_count": correct,
+      "total_count": len(results),
+      "results": results,
+    },
+  )
 
 
 async def call_model(ticket: str) -> Label:
-    text = ticket.lower()
-    if "crash" in text:
-        return "bug"
-    if "invite" in text or "show me" in text:
-        return "how_to"
-    return "billing"
+  text = ticket.lower()
+  if "crash" in text:
+    return "bug"
+  if "invite" in text or "show me" in text:
+    return "how_to"
+  return "billing"
 
 
 support_triage = workflow("support-triage", handler)

diff --git a/typescript/AGENTS.md b/typescript/AGENTS.md
@@ -1,5 +1,7 @@
 # AGENTS.md
 
+>**Note**: This SDK is currently unreleased and unsupported.
+
 ## Scope
 
 These instructions apply to the Quantiles TypeScript SDK. For changes outside the SDK, follow the nearest applicable `AGENTS.md` file.

diff --git a/typescript/README.md b/typescript/README.md
@@ -1,5 +1,7 @@
 # `@quantiles/sdk`
 
+>**Note**: this SDK is currently unsupported and unreleased.
+
 TypeScript client for the local Quantiles eval runner.
 
 ## Installation