diff --git a/.gitignore b/.gitignore index 8f73094..76bb1e4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ .envrc .DS_Store /.quantiles +node_modules diff --git a/AGENTS.md b/AGENTS.md index 1cca915..c477e38 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -91,7 +91,7 @@ Start with these files before making public-facing changes: - [`SECURITY.md`](./SECURITY.md): supported components and vulnerability reporting process. - [`CODE_OF_CONDUCT.md`](./CODE_OF_CONDUCT.md): community participation rules. - [`LICENSE`](./LICENSE): Apache 2.0 license text. -- `CHANGELOG.md`: notable public changes, when present. +- [`mise.toml`](./mise.toml): task definitions for building, formatting, type-checking, linting, and more, all using the [mise](https://mise.en.dev/) task runner. ## Agent Instruction Boundaries @@ -122,7 +122,7 @@ Use these terms consistently in public docs: - `step`: a durable recorded unit of an evaluation execution. - `metric`: a measured value emitted during a run. - `event`: recorded observability data from an evaluation. -- `.quantiles/`: local Quantiles workspace state. +- `.quantiles/`: local Quantiles workspace state, including SQLite database and metrics Parquet files. Prefer `local-first` and `offline by default` for open-source behavior. @@ -134,14 +134,13 @@ Use the `qt` CLI as the source of truth for running, listing, inspecting, compar Prefer CLI output over manually reading `.quantiles/` files. Do not manually edit or delete `.quantiles/` files unless explicitly asked. -Run `qt init` before other `qt` commands if the repository has not been initialized. +Although `qt init` exists to initialize the local Quantiles database, `qt run` automatically does the same. `qt init` is thus often unnecessary to run explicitly. Use `--json` for `qt run`, `qt list`, `qt show`, and `qt compare` when producing agent summaries. Inspect selected runs with `qt show --json`. Common commands: ```bash -qt init qt run --json qt list --json qt show --json @@ -209,7 +208,6 @@ The `qt` CLI is a local-first Rust CLI for running, recording, inspecting, and c It creates local SQLite state under `.quantiles/quantiles.sqlite` and provides commands such as: ```bash -qt init qt run qt list qt show @@ -228,7 +226,7 @@ When editing the Python subdirectory, preserve async behavior, stable JSON paylo ### TypeScript SDK -The TypeScript SDK is an ESM SDK for authoring local AI workload workflows against the Quantiles local observability server at `http://127.0.0.1:8765` by default. +The TypeScript SDK is an SDK for authoring local AI workload workflows against the Quantiles local observability server at `http://127.0.0.1:8765` by default. It exposes workflow primitives, `QuantilesClient`, `QuantilesRun`, stable JSON utilities, and shared SDK types. @@ -236,29 +234,17 @@ When editing the TypeScript subdirectory, preserve strict typing, JSON-serializa ### Agent Skill -The [`skill`](./skill/) subdirectory contains reusable instructions for coding agents that use Quantiles to run, inspect, compare, and summarize evaluation workflows. +The [`skill`](https://github.com/quantiles-evals/skill) repository contains reusable instructions for coding agents that use Quantiles to run, inspect, compare, and summarize evaluation workflows. When editing the skill subdirectory, keep instructions operational, command-driven, and safe for public use. -Read [`skill/SKILL.md`](./skill/SKILL.md) for the reusable agent skill instructions. - -### Benchmarks - -The `benchmarks` subdirectory may contain built-in or example benchmark harnesses, datasets, fixtures, scoring logic, and benchmark documentation. - -When editing benchmark content: - -- Preserve dataset provenance, scoring behavior, and benchmark limitations. -- Record benchmark source, version, commit, dataset revision, scoring configuration, and any local patches when relevant. -- Do not present demo sampler results as model-quality benchmark evidence. -- Avoid adding large datasets, generated outputs, or cached results unless explicitly required and appropriate for the repository. -- If benchmark content changes, call out whether the change affects comparability with prior runs. +Read [github.com/quantiles-evals/skill](https://github.com/quantiles-evals/skill) for the reusable agent skill instructions. ## Working In This Repository Keep root-level changes focused on public orientation, documentation, contribution guidance, security guidance, licensing, and agent instructions. -Update docs when any of the following change: +Update documentation in this repository (e.g. `README.md`, etc...) when any of the following change: - CLI commands, flags, outputs, or setup steps. - SDK APIs, imports, examples, or package names. @@ -280,7 +266,6 @@ Use the Quantiles CLI as the source of truth for local runs. Prefer this flow: ```bash -qt init qt run qt show --json ``` diff --git a/README.md b/README.md index 928e7d7..89028f8 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,8 @@ With Quantiles, teams can rely on built-in infrastructure: - Inspect and compare runs directly from the same `qt` CLI - Resilient execution by default with caching and restartable failed runs +Quantiles borrows concepts from durable workflow execution systems to make evaluation runs resilient to crashes and restarts, while adding a high-throughput execution engine, rich observability, metrics, and eval reproducibility. Use it to run custom eval code or built-in benchmarks, then inspect what changed across runs, all without notebooks, pipelines, manual comparisons, or cloud services. + ## Quickstart Install the CLI: @@ -67,6 +69,12 @@ qt compare >Note: you can pass the `--json` flag to any of the above commands, to output machine and agent-friendly JSON instead of human-formatted output. +To learn more detail about what you can do with the CLI, see [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli). + +### Customization + +You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details. + ## Local-First and Offline by Default Quantiles is built as a local-first, offline system that keeps benchmark execution, metadata, metrics, and analysis on your computer by default. @@ -92,23 +100,15 @@ Built-in benchmarks are ready-to-run evaluation harnesses with predefined datase ## Custom Evaluations - - -A custom evaluation is a Python or TypeScript program that is run by the `qt` CLI and uses the Quantiles API to execute an eval. Your code owns the evaluation logic: loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles records the run, durable steps, emitted metrics, events, inputs, outputs, and comparisons. +A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) or [TypeScript](https://quantiles.io/documentation/reference/typescript-sdk) program that is run by the `qt` CLI and uses the [Quantiles API](https://quantiles.io/documentation/reference/rest-api) to execute an eval. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages [durable steps, step caching, and step resume](https://quantiles.io/documentation/workflows-and-steps), metrics, inputs, outputs, and comparisons. Use custom evaluations when you need to measure behavior that is specific to your product, workflow, prompt, dataset, rubric, or release process. Read more about how to build and run custom evaluations at [quantiles.io/documentation/custom-evaluations](https://quantiles.io/documentation/custom-evaluations). -## SDKs +### SDKs -Use the official Quantiles SDKs to build with higher-level workflow primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python and TypeScript. The SDKs integrate tightly with the `qt` CLI’s local API for running, recording and analyzing benchmarks. +Use the official Quantiles SDKs to build your custom evaluations with higher-level workflow primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python and TypeScript. The SDKs integrate tightly with the `qt` CLI’s local API for running, recording and analyzing benchmarks. The Python SDK is located in this repository at [`python/`](./python). Read more about it at [quantiles.io/documentation/reference/python-sdk](https://quantiles.io/documentation/reference/python-sdk). The TypeScript SDK is located in this repository at [`typescript/`](./typescript). Read more about it at [quantiles.io/documentation/reference/typescript-sdk](https://quantiles.io/documentation/reference/typescript-sdk). diff --git a/cli/.gitignore b/cli/.gitignore index 47b21b7..ff4c21c 100644 --- a/cli/.gitignore +++ b/cli/.gitignore @@ -22,7 +22,6 @@ target /.quantiles /.wrangler /dist -/node_modules .fastembed_cache .envrc /qt-linux diff --git a/cli/AGENTS.md b/cli/AGENTS.md index 3087a22..737a8ab 100644 --- a/cli/AGENTS.md +++ b/cli/AGENTS.md @@ -9,7 +9,7 @@ - Prefer focused changes that fit the current CLI and library structure. - Keep local-only behavior explicit; do not introduce network or cloud behavior unless the task specifically calls for it. If you do add such behavior, make sure to alert the user about it, and add comments in the code to draw attention to it. - Preserve existing SQLite data model assumptions unless the change includes a deliberate schema migration or initialization update. -- Use idiomatic Rust and keep error handling clear. This project uses `anyhow` for application-level errors. +- Use idiomatic Rust and keep error handling clear. This project uses the [`anyhow`](https://docs.rs/anyhow) to create and propagate application-level errors. - Avoid broad refactors while implementing narrow behavior changes. ## Validation and Testing diff --git a/cli/ARCHITECTURE.md b/cli/ARCHITECTURE.md deleted file mode 100644 index 4a1dfe9..0000000 --- a/cli/ARCHITECTURE.md +++ /dev/null @@ -1,32 +0,0 @@ -# `qt`'s Architecture - -The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while `qt` handles durability and observability. - -``` -+--------------------------------------+ -| Your Script | -| (TypeScript / Python / Shell) | -+--------------------+-----------------+ - │ - │ HTTP / JSON - ▼ -+--------------------------------------+ -| Quantiles Server | -+--------------------+-----------------+ - │ - │ SQLite - ▼ -+------------------------------------------------+ -| .quantiles/quantiles.sqlite (local DB) | -+--------------------+---------------------------+ - │ - ▼ -+--------------------------------------+ -| CLI | -| (list, show, compare) | -+--------------------------------------+ -``` - -- **Server** owns durability decisions: step caching, run state, metrics -- **Client** (your script) owns code execution: the server never runs your logic -- **CLI** reads the same SQLite database the server writes to diff --git a/cli/BEST_EVAL_PRACTICES.md b/cli/BEST_EVAL_PRACTICES.md deleted file mode 100644 index e6903d4..0000000 --- a/cli/BEST_EVAL_PRACTICES.md +++ /dev/null @@ -1,79 +0,0 @@ -# Eval Best Practices - -Quantiles provides a framework that makes evals faster to build, more reliable to run, and easier to measure, so you can focusing in figuring out if your AI app is getting better over time. - -This document provides a set of best practices for writing evals using Quantiles. - -## Deterministic Step Inputs - -A _step_ in Quantiles is a discrete action that your eval take, that is a "checkpoint" in the eval script. You generally run code inside a `step` that can fail or is expensive, like calling your model or doing some big database operation. Most often, you'll use the `step(...)` function in the Typescript/Python SDks to mark the processing of a sample in your eval. - -Quantiles makes sure that, if your eval crashes, it doesn't have to start from the beginning - it can just start from the last `step` that succeeded. - -Samples are also useful for observability - they have inputs and outputs, so you can get a report of what happened just before, during, and just after each step. - -For Quantiles to treat samples properly, it's usually helpful to pass data to the `step` function that tells Quantiles whether a given `step` invocation is different from a previous one. Consider the below code: - -```typescript -const cities = ["Tokyo", "Munich", "Nairobi", "San Francisco", "Buenos Aires", "Sydney"]; -const city1 = cities[Math.floor(Math.random() * cities.length)]; -const city2 = cities[Math.floor(Math.random() * cities.length)]; - -step("call-llm", async () => { - return async call_llm(`what is the weather in ${city1}`); -}); - -step("call-llm", async () => { - return async call_llm(`what is the weather in ${city2}`); -}) -``` - -In this code, the two samples have the same name, but they close over different data (`city1` and `city2`, respectively). That means Quantiles can't tell the difference between them, so it will: - -- Assume they're equal - which can lead to problems when comparing two evals -- Not be able to intelligently cache and recover from failed samples - which can lead to problems when, for example, `call_llm` is flaky - -Initially, you won't need to worry about these things, but as you continue to run your evals over time (and we believe that you should!), you will be increasingly likely to encounter them. To solve this problem, you have two options: - -1. Give each step a distinct name - this is the easiest fix, but you might not want to change your step name -2. Pass a cache key to the `step` function - this is the fix if you don't want to change your step name - -```typescript -// - -// The second parameter to `step(...)` tells Quantiles that this -// call-llm step uses the 'city1' data. Quantiles will consider it -// different from any other call-llm step that uses any other -// data. -step("call-llm", {"city": city1}, async () => { - return async call_llm(`what is the weather in ${city1}`); -}); - -// Because the second parameter to `step(...)` has a different -// value than that of the previous `step(...)` call, Quantiles -// considers this step as different from the previous call. -step("call-llm", {"city": city2}, async () => { - return async call_llm(`what is the weather in ${city2}`); -}) -```` - -## Running Samples in Loops - -Many evals start by iterating over a dataset. Below is an example: - -```typescript -interface DatasetRow { - prompt: string; - goldenOutput: string; -} - -const datasetRows: DatasetRow[] = await loadDataset(); -for (let i = 0; i < 100; i++) { - const datasetRow = datasetRows[i]; - await step(`iter_step:${i}`, () => { - doSomething(i) - }); -} -``` - -In this example, Quantiles will be able to differentiate each step, which is great. Over time, though, you might want to alter the dataset over which you're looping. For example: diff --git a/cli/COMMANDS.md b/cli/COMMANDS.md deleted file mode 100644 index 0ce0c80..0000000 --- a/cli/COMMANDS.md +++ /dev/null @@ -1,208 +0,0 @@ -# Quantiles Commands - -This document contains a command reference for the `qt` CLI. - -## `qt init` - -Initialize a local Quantiles workspace in the current directory. - -This creates and configures the local SQLite database (stored in `.quantiles/quantiles.sqlite` by default). - -```bash -qt init -``` - -Example output: - -```text -Initialized Quantiles workspace at ./.quantiles/quantiles.sqlite -``` - -## `qt run` - -Run a command as a durable eval run. - -`qt run` creates a eval run, starts the given command as a subprocess, and records the result. It also injects two environment variables: - -- `QUANTILES_RUN_ID` — the run id so the subprocess can talk back to the API -- `QUANTILES_BASE_URL` — the base URL of the local Quantiles HTTP server - -`qt` automatically starts a local HTTP server on `127.0.0.1:8765` if one is not already running, runs your command, and then tears the server back down when the command finishes. - -```bash -# Simple example -qt run hello-world -- echo "hello from Quantiles" -``` - -You can also pass structured input to the run: - -```bash -qt run eval-smoke-test \ - --input '{"model":"gpt-4.1-mini","dataset":"smoke"}' \ - -- python3 ./my_eval.py -``` - -The `--` separator is required so that flags meant for your command are not parsed by `qt`. - -Example output: - -```text -Created eval run 2 -Executing: echo hello from Quantiles -hello from Quantiles -Run 2 completed successfully -``` - -### Resuming a run - -If a run fails and you want to retry it with the same run id (so that already- -completed samples are reused), pass `--resume`: - -```bash -qt run hello-world --resume 2 -- echo "hello again" -``` - -### TypeScript SDK integration - -The Typescript SDK provides a high-level API that allows users to write evals, and provides seamless integration with the `qt run` command. - -```typescript -import { workflow, step, emit, entrypoint } from "@quantiles/sdk"; - -const runEval = workflow("eval", async (input, ctx) => { - console.log(`Run ${ctx.runId} (${ctx.workflowName})`); - - const result = await step("call-model", async () => { - // expensive work here - return { response: "..." }; - }); - - emit("latency_ms", 120, "ms"); - return result; -}); - -entrypoint(runEval); -``` - -You can register multiple evals and let `qt run ` dispatch to the right one: - -```typescript -const w1 = workflow("eval", async (input, ctx) => { ... }); -const w2 = workflow("benchmark", async (input, ctx) => { ... }); - -entrypoint(w1, w2); -``` - -The callback you pass to `workflow` will receive two arguments: - -- `input` - parsed from `--input` when running via `qt run`, or from a `QUANTILES_INPUT` environment variable if you run this standalone. -- `ctx` — mostly for internal use, but contains the run ID, workflow name (the string you passed as the first parameter to `workflow`, and the raw Quantiles client. All of these are passed in a typescript `interface` like this: `{ runId, workflowName, client }` - -You can also provide explicit input to `step()` for cache invalidation: - -```typescript -const result = await step("call-model", { model: "gpt-4" }, async () => { - return callLLM(model); -}); -``` - -There is also a lower-level `QuantilesClient` API available for users who want finer-grained control over execution. - -See `sdk/typescript/examples/run_demo.ts` for a runnable example. - -## `qt serve` - -Start the local Quantiles HTTP server. - -```bash -qt serve -``` - -Generally, you won't need to run this manually. `qt run` starts a server automatically for the duration of your command. Use `qt serve` only when you want a persistent server (e.g. for development or multiple CLI windows). - -## `qt list` - -Show all eval runs in reverse chronological order. - -```bash -qt list -``` - -Example output: - -```text -ID EVAL STATUS SAMPLES CREATED FINISHED_AT -7 eval-smoke-test completed 2 2026-05-02T18:30:00.000Z 2026-05-02T18:30:05.000Z -3 hello-world failed 1 2026-05-02T18:15:00.000Z 2026-05-02T18:15:02.000Z -``` - -## `qt show` - -Inspect a single eval run, including its samples, metrics, and events. - -```bash -qt show -``` - -Example output: - -```text -Run 7 - eval: eval-smoke-test - status: completed - started_at: 2026-05-02T18:30:00.000Z - finished_at: 2026-05-02T18:30:05.000Z - input: {"model":"gpt-4.1-mini","dataset":"smoke"} - error: - - -Samples - ID KEY STATUS INPUT_HASH CREATED FINISHED_AT - 4 fetch-data completed a1b2c3d4e5f6789 2026-05-02T18:30:00.500Z 2026-05-02T18:30:01.000Z - 5 call-model completed 9f8e7d6c5b4a321 2026-05-02T18:30:01.200Z 2026-05-02T18:30:04.800Z - -Metrics - NAME VALUE UNIT - latency_ms 120 ms - tokens_used 42 - - -Events - ID TYPE CREATED MESSAGE - 10 run.started 2026-05-02T18:30:00.000Z Started eval eval-smoke-test - 11 step.started 2026-05-02T18:30:00.500Z Started step fetch-data - 12 step.completed 2026-05-02T18:30:01.000Z Completed step 4 - ... -``` - -## `qt compare` - -Compare two eval runs side-by-side. - -```bash -qt compare -``` - -Checks samples, outputs, and metrics. Exits with code 0 if the runs are identical -and code 1 if anything differs. - -Example output: - -```text -Run 1: eval-smoke-test (completed) -Run 2: eval-smoke-test (completed) - -Samples - STEP PRESENT INPUT STATUS OUTPUT - fetch-data both same same differs - -Output differences - step: fetch-data - run 1: {"status":200,"body":"hello"} - run 2: {"status":200,"body":"world"} - -Metrics - NAME Run 1 Run 2 - latency_ms 50 150 * - tokens_used 42 89 * - -Runs differ. -``` diff --git a/cli/EVAL_CHALLENGES.md b/cli/EVAL_CHALLENGES.md deleted file mode 100644 index d7acbaa..0000000 --- a/cli/EVAL_CHALLENGES.md +++ /dev/null @@ -1,69 +0,0 @@ -# Why Evals Are Hard (And What to do Instead) - -Evals look simple: - -```typescript -for(const data of dataset) { - const output = await sampleModel(data); - const score = await judge(data, output); - await recordScore(data, output, score); -} -``` - -In practice, they're not. The moment you try to run real experiments, things break down quickly: - -- You need to load and shuffle datasets deterministically -- You need to track which version of your prompt or model you ran -- You need to be able to recover from crashes -- You need to cache expensive samples so you don’t re-run everything -- You need to record the right metrics so you can analyze results later -- You need to compare runs to understand what actually improved - -Instead of writing a loop, you end up building a system before you can even start doing what you came here to do -- figure out if your AI is good or not. - -## What Most People End Up With - -A typical eval looks like this: - -1. Write a script to prepare data -2. Run it -3. Write a script to call the model -4. Run it -5. Write a script to score outputs -6. Run it -7. Dump everything into a database -8. Write SQL or open a dashboard to figure out what happened - -Each step is separate, each run is hard to reproduce, and comparing results is manual. - -## With Quantiles - -Quantiles turns all those samples into a single script: - -```typescript -import { workflow, step, emit } from "quantiles"; -export const evalPrompt = workflow(async ({ promptVersion }) => { - for (const prompt of dataset) { - const output = await step("call-model", {prompt}, async () => { - return callModel(prompt, promptVersion); - }); - const score = await step("judge", {prompt}, async () => { - return judge(output); - }); - emit("score", score); - } -}); -``` - -... which you can then run and analyze with a few commands: - -```bash -# run your eval against two prompts, to figure out -# which performs better -qt run evalPrompt --input '{"promptVersion":"v1"}' -qt run evalPrompt --input '{"promptVersion":"v2"}' -# get a deep analysis of your results -qt compare 1 2 -``` - -You write less code, you get resilience for free, and you spend all your time _actually_ making your AI app better, not building systems. diff --git a/cli/LICENSE b/cli/LICENSE deleted file mode 100644 index 261eeb9..0000000 --- a/cli/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/cli/PROTOCOL.md b/cli/PROTOCOL.md deleted file mode 100644 index 23bdbf9..0000000 --- a/cli/PROTOCOL.md +++ /dev/null @@ -1,314 +0,0 @@ -# Quantiles Eval Protocol - -This document describes the API calls that a successful eval should make to record its execution in Quantiles. It covers the REST API contract and the corresponding TypeScript SDK calls. - -An eval, also called a workflow in the protocol, is a process that does the following: -1. Creates a run (or reuses one assigned by the CLI) -2. Runs samples (durable, memoized units of work) -3. Emits metrics -4. Saves its final output -5. Completes (or fails) - -The exact calls differ slightly depending on whether the eval is started by `qt run` (CLI-driven) or started directly by calling `.run()` on a workflow object (programmatic). - ---- - -## Execution Modes - -### CLI-driven run (`qt run my-eval -- ...`) - -The CLI creates the run, starts the server, and spawns the eval process with these environment variables: - -- `QUANTILES_RUN_ID` — the ID of the run -- `QUANTILES_BASE_URL` — the local server URL -- `QUANTILES_WORKFLOW_NAME` — the eval name -- `QUANTILES_INPUT` — the JSON input (or `{}`) - -In this mode, the eval process **must not** call `POST /runs` to create a new run — the run already exists. It **must** save its output via `POST /runs/{run_id}/output`, but should let the CLI own the final `complete` or `fail` call. - -### Programmatic run (`workflow.run()`) - -There is no `QUANTILES_RUN_ID` environment variable. The SDK creates the run itself, then completes or fails the run directly. - ---- - -## Complete Protocol for a Success - -### 1. Create a run (programmatic only) - -**REST API (not used in CLI-driven mode):** - -```http -POST /runs -Content-Type: application/json - -{ - "eval_name": "my-eval", - "input": "{\"dataset\":\"smoke\"}" -} -``` - -**Response:** - -```json -{ - "run_id": 1 -} -``` - -**TypeScript SDK:** - -```typescript -const run = await client.createRun("my-eval", { dataset: "smoke" }); -// run.id === 1 -``` - ---- - -### 2. Run a step - -Before executing expensive work, call `POST /steps/begin`. If the step was already completed with the same `input_hash`, the server returns a cache hit (`decision: "reuse"`). Otherwise it returns `decision: "run"` and assigns a `step_id`. - -**REST API (begin):** - -```http -POST /steps/begin -Content-Type: application/json - -{ - "run_id": 1, - "step_key": "fetch-data", - "input_hash": "abc123..." -} -``` - -**Response (run):** - -```json -{ - "decision": "run", - "step_id": 3 -} -``` - -**Response (reuse):** - -```json -{ - "decision": "reuse", - "output": "{\"status\":200}" -} -``` - -If the decision is `"run"`, execute the step, then report completion or failure. - -**REST API (complete):** - -```http -POST /steps/complete -Content-Type: application/json - -{ - "step_id": 3, - "output": "{\"status\":200}" -} -``` - -**REST API (fail):** - -```http -POST /steps/fail -Content-Type: application/json - -{ - "step_id": 3, - "error": "network timeout" -} -``` - -**TypeScript SDK:** - -```typescript -const result = await step( - "fetch-data", - { url: "https://example.com" }, - async () => { - return { status: 200 }; - }, -); -``` - ---- - -### 3. Emit a metric - -Metrics are associated with a run. They can be emitted from the eval or from individual samples. - -**REST API:** - -```http -POST /runs/1/metrics -Content-Type: application/json - -{ - "metric_name": "latency_ms", - "metric_value": 50, - "unit": "ms" -} -``` - -**TypeScript SDK:** - -```typescript -await emit("latency_ms", 50, "ms"); -``` - ---- - -### 4. Save the eval output - -At the end of the eval, save the return value. This is a **separate** call from completing the run. - -**CLI-driven mode:** Always call this. The CLI will complete the run later based on the process exit code. - -**Programmatic mode:** Skip this — `completeRun` handles both output and status. - -**REST API:** - -```http -POST /runs/1/output -Content-Type: application/json - -{ - "output": "{\"accuracy\":0.85,\"correct\":17,\"total\":20}" -} -``` - -**TypeScript SDK:** - -```typescript -await client.setRunOutput(1, { accuracy: 0.85, correct: 17, total: 20 }); -``` - ---- - -### 5. Complete the run (programmatic only) - -Marks the run as completed, optionally including the output. Emits a `run.completed` event. - -**CLI-driven mode:** Do not call this. The CLI calls it when the child process exits successfully. - -**Programmatic mode:** Call this (it already includes the output). - -**REST API:** - -```http -POST /runs/1/complete -``` - -**TypeScript SDK:** - -```typescript -await client.completeRun(1, { accuracy: 0.85, correct: 17, total: 20 }); -``` - ---- - -### 6. Fail the run (on error) - -**CLI-driven mode:** Throw an error or exit non-zero. The CLI will call `POST /runs/{run_id}/fail` automatically. - -**Programmatic mode:** Call `failRun`. - -**REST API:** - -```http -POST /runs/1/fail -Content-Type: application/json - -{ - "error": "network failed" -} -``` - -**TypeScript SDK:** - -```typescript -await client.failRun(1, new Error("network failed")); -``` - ---- - -## TypeScript Eval Example (CLI-driven) - -This is the typical pattern used with `qt run`: - -```typescript -import { emit, entrypoint, step, workflow } from "@quantiles/sdk"; - -const eval = workflow("support-triage", async (input) => { - const results = await step("classify", input, async () => { - return await callModel(input); - }); - - emit("accuracy", results.accuracy); - emit("latency_ms", results.latency, "ms"); - - return results; -}); - -entrypoint(eval); -``` - -The SDK handles all API calls automatically. Because `QUANTILES_RUN_ID` is set, it: -1. Reuses the existing run ID -2. Creates/executes/reuses samples -3. Emits metrics -4. Calls `setRunOutput(runId, results)` at the end - -Then the process exits with code 0, and the CLI completes the run. - ---- - -## TypeScript Eval Example (Programmatic) - -Useful for testing or running evals from other scripts: - -```typescript -import { QuantilesClient, workflow, step, emit } from "@quantiles/sdk"; - -const eval = workflow("support-triage", async (input) => { - const results = await step("classify", input, async () => { - return await callModel(input); - }); - - emit("accuracy", results.accuracy); - return results; -}); - -const client = new QuantilesClient({ baseUrl: "http://127.0.0.1:8765" }); -const run = await client.createRun("support-triage", { dataset: "smoke" }); -const output = await eval.run({ dataset: "smoke" }); - -console.log("Run completed:", run.id); -console.log("Output:", output); -``` - -Because `QUANTILES_RUN_ID` is not set, the eval creates its own run and calls `completeRun(runId, output)` automatically. - ---- - -## Endpoints Summary - -| Method | Path | Body | Purpose | -|---|---|---|---| -| `POST` | `/runs` | `{ workflow_name, input }` | Create a new run (programmatic) | -| `GET` | `/runs/{run_id}` | — | Fetch run metadata | -| `POST` | `/runs/{run_id}/output` | `{ output }` | Save eval return value | -| `POST` | `/runs/{run_id}/complete` | `{ output }` | Mark run completed (programmatic) | -| `POST` | `/runs/{run_id}/fail` | `{ error }` | Mark run failed | -| `POST` | `/runs/{run_id}/metrics` | `{ metric_name, metric_value, unit }` | Record a metric | -| `POST` | `/steps/begin` | `{ run_id, step_key, input_hash }` | Reserve/reuse a step | -| `POST` | `/steps/complete` | `{ step_id, output }` | Mark step completed | -| `POST` | `/steps/fail` | `{ step_id, error }` | Mark step failed | -| `GET` | `/health` | — | Server health check | diff --git a/cli/README.md b/cli/README.md index ae49b3f..67f0949 100644 --- a/cli/README.md +++ b/cli/README.md @@ -1,6 +1,6 @@ -# Quantiles +# Quantiles CLI -`qt` is a local-first CLI for running AI evals as experiments, so you can instantly see what improved, what regressed, and why. +This directory holds the source code for the `qt` CLI. It is built with [Rust](https://rust-lang.org/) to help it efficiently use the resources of the local machine, to help ensure safety, and to provide strong lints and type-system invariants for humans and agents to work with. ## Install @@ -8,13 +8,6 @@ curl -fsSL https://cli.quantiles.io/install.sh | bash ``` -It's built for: - -- LLM evals -- Prompt iteration -- Agent workflows -- Debugging AI systems - ## Demo A few commands to see `qt` in action: @@ -23,79 +16,20 @@ A few commands to see `qt` in action: # 1. Initialize a workspace qt init -# 2. Run an eval — Quantiles auto-starts a local server, records the run, -# and tears the server down when the command finishes. -qt run hello-world -- echo "hello from Quantiles" +# 2. Run an eval — Quantiles auto-starts a local server, records the run, and tears the server +# down when the command finishes. +qt run my-eval -- bun run sdk/typescript/examples/run_demo.ts # 3. List and inspect what happened qt list qt show 1 ``` -### Durable step caching + crash resume (TypeScript SDK) - -Use the high-level `workflow()`, `step()`, and `emit()` API for automatic -caching and observability: - -```typescript -import { workflow, step, emit, entrypoint } from "@quantiles/sdk"; - -const runEval = workflow("eval", async (input, ctx) => { - console.log(`Run ${ctx.runId} (${ctx.workflowName})`); - - const data = await step("fetch-data", { url: "https://example.com" }, async () => { - return { status: 200, body: "..." }; - }); - - // Same step key + input hash means cached output, and no re-execution - const cached = await step("fetch-data", { url: "https://example.com" }, async () => { - return { status: 999, body: "this should not execute" }; - }); - - emit("latency_ms", 120, "ms"); - emit("tokens_used", 42); - - return data; -}); - -entrypoint(runEval); -``` - -Register multiple evals and let `qt run ` dispatch to the right one -automatically: - -```typescript -const w1 = workflow("eval", async (input, ctx) => { ... }); -const w2 = workflow("benchmark", async (input, ctx) => { ... }); - -entrypoint(w1, w2); -``` - -Then, when you're ready, run your eval with this command: - -```bash -qt run eval -- bun run sdk/typescript/examples/run_demo.ts -``` - -If the run fails partway through, resume it and only the failed samples rerun: - -```bash -qt run my-eval --resume 1 -- bun run sdk/typescript/examples/run_demo.ts -``` - -### Step Caching - -You can also call `step()` without an input hash when caching by step name alone is fine: +>See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. -```typescript -const modelOutput = await step("call-model", async () => { - return callLLM(prompt); -}); -``` - -### Compare two runs +### Comparing runs -After iterating on an eval, compare two runs to see exactly what changed: +After iterating on an eval, you can compare two runs to see exactly what changed: ```bash # Run A — baseline @@ -108,62 +42,46 @@ qt run my-eval -- bun run sdk/typescript/examples/run_demo.ts qt compare 1 2 ``` -Example output when outputs differ: - -```text -Run 1: my-eval (completed) -Run 2: my-eval (completed) - -Samples - STEP PRESENT INPUT STATUS OUTPUT - fetch-data both same same differs - -Output differences - step: fetch-data - run 1: {"status":200,"body":"hello"} - run 2: {"status":200,"body":"world"} - -Metrics - NAME Run 1 Run 2 - latency_ms 50 150 * - tokens_used 42 89 * - -Runs differ. -``` - `qt compare` exits with code 1 if the runs differ, making it useful in CI scripts. -## Why Quantiles? - -| | Temporal / Trigger.dev | Quantiles | -|---|---|---| -| **Goal** | Run evals reliably | Run evals and **quickly learn what improved, what regressed, and why** | -| **Setup** | Cluster, workers, cloud infra | Single binary + SQLite | -| **Programming Model** | Deterministic constraints | Normal async code | -| **Local Dev** | Usually tied to a service/runtime | Fully offline | -| **Observability** | Logs and traces | Automatically compare runs and surface changes | -| **Iteration** | Run, then inspect | Run, then compare, then improve | - -Quantiles is for the iteration loop before production orchestration. - -Run your code, then instantly see what changed across runs. No notebooks, no pipelines, and no manual comparisons. - -It doesn’t replace Temporal for production orchestration. It’s built for the 90% of iteration that happens before you ever think about production infrastructure. - - -## What Quantiles does - -- Runs your evals locally with durability (step caching + resume) -- Records every run, step, metric, and event -- Lets you inspect and compare runs from the CLI -- Stores everything in a local SQLite database +## Architecture -Quantiles treats evals as experiments in which every run is tracked, comparable, and queryable with no extra work from you. +The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while `qt` handles durability and observability. -## Architecture +``` ++--------------------------------------+ +| Your Script | +| (TypeScript / Python / Shell) | ++-------------------+------------------+ + │ + │ HTTP / JSON + | + ▼ ++--------------------------------------+ +| Quantiles Server | ++-------------------+------------------+ + │ + │ SQLite + | + ▼ ++------------------------------------------------+ +| .quantiles/quantiles.sqlite (local DB) | ++-------------------+----------------------------+ + │ + │ + │ + ▼ ++--------------------------------------+ +| CLI | +| (list, show, compare) | ++--------------------------------------+ +``` -See [ARCHITECTURE.md](./ARCHITECTURE.md) +- **Server** owns durability decisions: step caching, run state, metrics +- **Client** (your script) owns code execution: the server never runs your logic + - Note that the CLI itself also has built-in benchmarks, which do not involve your code +- **CLI** reads the same SQLite database the server writes to -## Command Reference +## Customization -See [COMMANDS.md](./COMMANDS.md) +You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details. diff --git a/cli/examples/configs/anthropic/quantiles.toml b/cli/examples/configs/anthropic/quantiles.toml index d6f56ee..8bf60b1 100644 --- a/cli/examples/configs/anthropic/quantiles.toml +++ b/cli/examples/configs/anthropic/quantiles.toml @@ -1,15 +1,20 @@ +# This config file sets up PubMedQA to use Anthropic's Claude model, and sets up the SimpleQA-verified +# benchmark to use the internal demo model. +# +# The PubMedQA benchmark reads the API key from the `ANTHROPIC_API_KEY` environment +# variable. To get an API key, go to https://platform.claude.com/settings/workspaces/default/keys + [benchmarks.pubmedqa] +# The full expert-labeled PubMedQA benchmark has 1000 samples. Since Anthropic charges for +# usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 -# Use the Anthropic Claude API, reading the API key from the `ANTHROPIC_API_KEY` environment -# variable. To get an API key, go to https://platform.claude.com/settings/workspaces/default/keys -# # The below model is good for demonstrating PubMedQA benchmarks without very high costs, # since it's relatively cheap and fast. Details on the model can be found at: # # https://platform.claude.com/docs/en/about-claude/models/overview#latest-models-comparison # -# This page also has a list of other frontier and near-frontier models that you can use. +# That page also has a list of other Anthropic models that you can use. model = "anthropic:claude-haiku-4-5-20251001" # If you run this or other benchmarks with more samples, you may also need to increase this @@ -22,9 +27,3 @@ model = "anthropic:claude-haiku-4-5-20251001" # # You can also see your usage at https://platform.claude.com/usage max_workers = 5 - -[benchmarks.simpleqa-verified] -samples = 100 - -[benchmarks.financebench] -samples = 50 diff --git a/cli/examples/configs/both_error/.quantiles.toml b/cli/examples/configs/both_error/.quantiles.toml index 26ab1cd..e2a96a1 100644 --- a/cli/examples/configs/both_error/.quantiles.toml +++ b/cli/examples/configs/both_error/.quantiles.toml @@ -1,15 +1,3 @@ -# Example Quantiles workspace configuration. -# -# Place this file in your working directory as either: -# - quantiles.toml -# - .quantiles.toml -# -# If both files exist in the same directory, the CLI will error to avoid -# ambiguity. -# -# Each section under [benchmarks.] maps to a builtin eval. -# The `samples` key limits the number of dataset rows processed. - [benchmarks.pubmedqa] samples = 10 diff --git a/cli/examples/configs/both_error/quantiles.toml b/cli/examples/configs/both_error/quantiles.toml index e69de29..c0d56b3 100644 --- a/cli/examples/configs/both_error/quantiles.toml +++ b/cli/examples/configs/both_error/quantiles.toml @@ -0,0 +1,2 @@ +# The `qt` CLI will refuse to run if both a `quantiles.toml` and `.quantiles.toml` configuration +# file exist in the current directory, even if one of the files is empty. diff --git a/cli/examples/configs/cloudflare/quantiles.toml b/cli/examples/configs/cloudflare/quantiles.toml index 00875a6..9e0dcab 100644 --- a/cli/examples/configs/cloudflare/quantiles.toml +++ b/cli/examples/configs/cloudflare/quantiles.toml @@ -1,13 +1,23 @@ -[benchmarks.pubmedqa] -samples = 50 - -# Use Cloudflare AI Gateway, reading the account ID, gateway ID, and API key -# from environment variables: +# This config file sets up PubMedQA to use the Cloudflare AI gateway, and sets up the SimpleQA-verified +# benchmark to use the internal demo model. +# +# The PubMedQA benchmark reads the Cloudflare account ID, gateway ID, and API key +# from the following environment variables: +# # - CLOUDFLARE_ACCOUNT_ID # - CLOUDFLARE_GATEWAY_ID # - CLOUDFLARE_API_KEY # -# This model is good for demonstrating PubMedQA benchmarks without very high costs, +# To get your account ID, gateway ID and API key, go to your Cloudflare AI gateway dashboard at: +# +# https://dash.cloudflare.com//ai/ai-gateway + +[benchmarks.pubmedqa] +# The full expert-labeled PubMedQA benchmark has 1000 samples. Since Cloudflare charges for +# usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. +samples = 50 + +# The below model is good for demonstrating PubMedQA benchmarks without very high costs, # since it's relatively cheap, fast, and not a reasoning model. Its details can be found at: # # https://developers.cloudflare.com/workers-ai/models/llama-3.3-70b-instruct-fp8-fast/ @@ -16,17 +26,16 @@ samples = 50 # # https://developers.cloudflare.com/workers-ai/models/ model = "cloudflare_ai_gateway:@cf/meta/llama-3.3-70b-instruct-fp8-fast" + +# If you don't want to specify your account ID and gateway ID in environment variables +# (CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_GATEWAY_ID, respectively), you can provide them +# alongside the model in a dictionary as below: # -# Or provide the account ID / gateway ID inline, and still read the API key from -# the CLOUDFLARE_API_KEY env var # model = { # type = "cloudflare_ai_gateway:@cf/meta/llama-3.3-70b-instruct-fp8-fast", # account_id = "...", # gateway_id = "..." # } - -[benchmarks.simpleqa-verified] -samples = 100 - -[benchmarks.financebench] -samples = 50 +# +# Note that if you do this, you will still need to provide the API key in the +# CLOUDFLARE_API_KEY environment variable. diff --git a/cli/examples/configs/gemini/quantiles.toml b/cli/examples/configs/gemini/quantiles.toml index 1e216a8..002b477 100644 --- a/cli/examples/configs/gemini/quantiles.toml +++ b/cli/examples/configs/gemini/quantiles.toml @@ -1,17 +1,22 @@ -[benchmarks.pubmedqa] -samples = 50 - -# Use the Gemini API. The API key is read from the `GEMINI_API_KEY` environment +# This config file sets up PubMedQA to use the Gemini API, and sets up the SimpleQA-verified +# benchmark to use the internal demo model. +# +# The PubMedQA benchmark reads the Gemini API key from the `GEMINI_API_KEY` environment # variable. Learn more about the Gemini API in Google's Gemini AI Studio: # # https://aistudio.google.com/prompts/new_chat -# + +[benchmarks.pubmedqa] +# The full expert-labeled PubMedQA benchmark has 1000 samples. Since Google charges for +# usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. +samples = 50 + # The below model is good for demonstrating PubMedQA benchmarks without very high costs, # since it's relatively cheap and fast. Details on the model can be found at: # # https://ai.google.dev/gemini-api/docs/models/gemini#gemini-3.1-flash-lite # -# This page also has a list of other models that you can use. +# That page also lists other Google models you can use. model = "gemini:gemini-3.1-flash-lite" # If you run this or other benchmarks with more samples, you may also need to increase this @@ -22,9 +27,3 @@ model = "gemini:gemini-3.1-flash-lite" # # https://aistudio.google.com/rate-limit max_workers = 10 - -[benchmarks.simpleqa-verified] -samples = 100 - -[benchmarks.financebench] -samples = 50 diff --git a/cli/examples/configs/openai/quantiles.toml b/cli/examples/configs/openai/quantiles.toml index ad00a94..8f6e164 100644 --- a/cli/examples/configs/openai/quantiles.toml +++ b/cli/examples/configs/openai/quantiles.toml @@ -1,24 +1,32 @@ +# This config file sets up PubMedQA to use the OpenAI API, and sets up the SimpleQA-verified +# benchmark to use the internal demo model. +# +# The PubMedQA benchmark reads the OpenAI API key from the `OPENAI_API_KEY` environment variable. +# +# Learn more about the OpenAI API at: +# +# https://platform.openai.com/home + [benchmarks.pubmedqa] +# The full expert-labeled PubMedQA benchmark has 1000 samples. Since OpenAI charges for +# usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 -# Use the OpenAI API, reading the API key -# from the `OPENAI_API_KEY` environment variable -# # The below model is good for demonstrating PubMedQA benchmarks without very high costs, -# since it's relatively cheap, fast, and not a reasoning model. You will need to ensure the -# project with which your API key is associated with has permissions to use this model. -# Do so by going to the below page, and enabling the model: +# since it's relatively cheap and fast. You may need to ensure the project associated with +# your API key has permissions to use this model. Do so by going to the below page, and +# enabling the model: # # https://platform.openai.com/settings/$PROJECT_ID/limits # -# Details on the model can be found at: +# Details on this specific model can be found at: # -# https://developers.openai.com/api/docs/models/gpt-5.4-nano +# https://developers.openai.com/api/docs/models/gpt-5.5-nano # # And a list of all models can be found at: # # https://developers.openai.com/api/docs/models/all -model = "openai:gpt-5-nano" +model = "openai:gpt-5.4-nano" # If you run this or other benchmarks with more samples, you may also need to increase this # `max_workers` parameter to improve throughput. This might cause you to run into rate @@ -26,9 +34,3 @@ model = "openai:gpt-5-nano" # on the same https://platform.openai.com/settings/$PROJECT_ID/limits page and adjust limits # for the model you're using. max_workers = 100 - -[benchmarks.simpleqa-verified] -samples = 100 - -[benchmarks.financebench] -samples = 50 diff --git a/cli/examples/configs/simple/quantiles.toml b/cli/examples/configs/simple/quantiles.toml index 94fa959..06b287e 100644 --- a/cli/examples/configs/simple/quantiles.toml +++ b/cli/examples/configs/simple/quantiles.toml @@ -1,22 +1,14 @@ -# Example Quantiles workspace configuration. +# This config file sets up the PubMedQA and SimpleQA-verified benchmarks to use the built-in +# demo model. The `qt` CLI automatically detects this file in the current working directory. # -# Place this file in your working directory as either: -# - quantiles.toml -# - .quantiles.toml +# This built-in demo model generates random text and is generally only useful only for smoke +# testing purposes. Do not use results from these runs to determine the quality of any model. # -# If both files exist in the same directory, the CLI will error to avoid -# ambiguity. -# -# Each section under [benchmarks.] maps to a builtin eval. -# The `samples` key limits the number of dataset rows processed. -# The `model` key selects the LLM backend (defaults to the builtin's dummy -# sampler when omitted). +# Each section under [benchmarks.] maps to a builtin eval. The `samples` key limits the +# number of dataset rows processed. [benchmarks.pubmedqa] samples = 10 [benchmarks.simpleqa-verified] samples = 100 - -[benchmarks.financebench] -samples = 50 diff --git a/cli/src/commands/compare/step.rs b/cli/src/commands/compare/step.rs index e5ef230..511414b 100644 --- a/cli/src/commands/compare/step.rs +++ b/cli/src/commands/compare/step.rs @@ -11,6 +11,8 @@ pub(super) struct StepRefs<'a> { } /// Build user-facing step comparisons and borrowed step refs for detailed diffs. +/// +/// TODO: have this return a `Vec<(StepComparison, StepRefs<'a>)>` pub(super) fn build_step_rows<'a>( left_steps: &'a [StepSummary], right_steps: &'a [StepSummary], diff --git a/cli/src/dataset/hf_client.rs b/cli/src/dataset/hf_client.rs index a007e53..017ea41 100644 --- a/cli/src/dataset/hf_client.rs +++ b/cli/src/dataset/hf_client.rs @@ -16,6 +16,7 @@ struct SplitsResponse { splits: Vec, } +/// A single split in an splits response from the HF API #[derive(Debug, Deserialize)] struct SplitItem { #[serde(default)] @@ -68,7 +69,7 @@ struct SplitSize { } impl HuggingFaceClient { - /// Build a new client, injecting `HF_TOKEN` when present. + /// Build a new client, using the `HF_TOKEN` env var when it's present. /// /// # Errors /// diff --git a/cli/src/db/entities/event.rs b/cli/src/db/entities/event.rs index a0454ae..b1269c7 100644 --- a/cli/src/db/entities/event.rs +++ b/cli/src/db/entities/event.rs @@ -3,12 +3,20 @@ use sea_orm::entity::prelude::*; #[derive(Clone, Debug, PartialEq, DeriveEntityModel)] #[sea_orm(table_name = "events")] pub struct Model { + /// Unique ID of the model #[sea_orm(primary_key)] pub id: i64, + /// Unique ID of the run pub run_id: i64, + /// Unique ID of the step, inside the run pub step_id: Option, + /// Type of the event + /// + /// TODO: make this a discriminated union if possible pub event_type: String, + /// Optional explanatory message about the event pub message: Option, + /// Optional additional metadata abou the event pub data: Option, pub created_at: time::OffsetDateTime, } diff --git a/cli/src/db/runs.rs b/cli/src/db/runs.rs index 242fb73..640d6de 100644 --- a/cli/src/db/runs.rs +++ b/cli/src/db/runs.rs @@ -298,7 +298,6 @@ pub async fn get_run(db: &DatabaseConnection, run_id: i64) -> Result = match run.input { Some(ref input) => serde_json::from_str(input.clone().as_str()).ok(), None => None, diff --git a/cli/src/db/steps.rs b/cli/src/db/steps.rs index 325bc52..27d8186 100644 --- a/cli/src/db/steps.rs +++ b/cli/src/db/steps.rs @@ -41,8 +41,8 @@ pub async fn begin_step( }, StepStatus::Running => StepDecision::Run { step_id: step.id }, StepStatus::Failed => { - // Allow retry: reset the step to running and emit a fresh - // started event so the retry is observable. + // Allow retry by resetting the step status to 'running', then emit a fresh + // 'started' event so the retry is apparent and auditable. step::Entity::update_many() .set(step::ActiveModel { status: Set(StepStatus::Running), diff --git a/cli/src/db/workspace.rs b/cli/src/db/workspace.rs index 4f043c6..6acdbfc 100644 --- a/cli/src/db/workspace.rs +++ b/cli/src/db/workspace.rs @@ -9,11 +9,14 @@ use crate::db::{DBUrl, SQLitePathURL, schema}; const DB_DIR: &str = ".quantiles"; const DB_FILE: &str = "quantiles.sqlite"; -/// Walk up from `cwd` looking for a `.quantiles` directory. +/// Look for `.quantiles` directory under the current working directory, then +/// if it's not found, walk up to parents looking for a `.quantiles` directory. /// -/// If one is found, returns the directory that contains `.quantiles`. -/// If `create` is true, ensures the discovered workspace is initialized. -/// Otherwise initializes a new workspace in `cwd` and returns `cwd`. +/// When the first such `.quantiles` directory is found, return its containing +/// dir. +/// +/// If `create` is `true`, ensure the discovered `.quantiles` workspace is initialized. +/// Otherwise, initializes a new workspace in `cwd` and returns `cwd`. /// /// # Errors /// diff --git a/cli/src/llm/cloudflare.rs b/cli/src/llm/cloudflare.rs index b378ab3..f99ddbf 100644 --- a/cli/src/llm/cloudflare.rs +++ b/cli/src/llm/cloudflare.rs @@ -63,6 +63,9 @@ struct WorkersAIMessage { /// tries the most common locations. fn extract_response(raw: &serde_json::Value) -> Option { // Standard text-generation shape: {"result":{"response":"..."}} + // + // TODO: decode result into a strongly-typed struct with serde, rather + // than manual deserialization if let Some(result) = raw.get("result") && let Some(response) = result.get("response") && let Some(text) = response.as_str() @@ -83,7 +86,7 @@ fn extract_response(raw: &serde_json::Value) -> Option { return Some(text.to_string()); } - // Fallback: raw result string + // Extract raw result string as a fallback if let Some(result) = raw.get("result") && let Some(text) = result.as_str() { diff --git a/cli/src/llm/random.rs b/cli/src/llm/random.rs index 90bfc40..bf90379 100644 --- a/cli/src/llm/random.rs +++ b/cli/src/llm/random.rs @@ -6,6 +6,8 @@ use rand::{Rng, thread_rng}; use super::LLMSampler; /// A dummy sampler that returns random alphanumeric characters. +/// +/// Do not use this for actual testing pub struct RandomSampler { pub max_length: usize, } diff --git a/cli/src/llm/random_label.rs b/cli/src/llm/random_label.rs index ff14252..db2ae61 100644 --- a/cli/src/llm/random_label.rs +++ b/cli/src/llm/random_label.rs @@ -7,8 +7,10 @@ use super::LLMSampler; /// Fake sampler that returns one valid label from a list of candidates. /// -/// Commonly used with QA-style benchmarks. For example, `PubMedQA` uses -/// labels "yes", "no" and "maybe" +/// This sampler is commonly used in QA-style benchmarks. +/// +/// For example, `PubMedQA` uses labels "yes", "no" and "maybe". Other QA-style +/// benchmarks can specify other labels dynamically. pub struct RandomLabelSampler { labels: Vec, } diff --git a/cli/src/llm/sampler.rs b/cli/src/llm/sampler.rs index 88fcdcc..dfb4b24 100644 --- a/cli/src/llm/sampler.rs +++ b/cli/src/llm/sampler.rs @@ -48,8 +48,9 @@ impl Sampler { /// /// # Errors /// - /// Returns an error if the required environment variables are missing (e.g. - /// `CLOUDFLARE_API_KEY` for the Cloudflare gateway variant). + /// Returns an error if the required environment variables are missing to convert `self` + /// to an actual `LLMSampler` (e.g. `self` is a `Sampler::CloudflareAIGateway` variant + /// and the `CLOUDFLARE_API_KEY` env var is missing). pub fn resolve(&self) -> Result> { match self { Sampler::Random => Ok(Arc::new(random::RandomSampler::new(80))), diff --git a/cli/src/main.rs b/cli/src/main.rs index 44a4deb..64181ec 100644 --- a/cli/src/main.rs +++ b/cli/src/main.rs @@ -9,6 +9,9 @@ use clap::Parser; fn main() -> Result<()> { let process_start = Instant::now(); + // TODO: allow number of total threads to be configurable, and possibly default + // to something other than the default of 61, which is specified in the implementation + // of new_multi_thread() tokio::runtime::Builder::new_multi_thread() .enable_all() .build()? diff --git a/cli/src/metrics_store.rs b/cli/src/metrics_store.rs index b5a187e..dc4cc22 100644 --- a/cli/src/metrics_store.rs +++ b/cli/src/metrics_store.rs @@ -31,10 +31,17 @@ struct BufferedMetric { /// SQL over the per-run Parquet files. #[derive(Debug, Clone)] pub struct MetricsStore { - // TODO: use a concurrent queue rather than sequential hashmap, - // - // https://docs.rs/crossbeam-queue/latest/crossbeam_queue/struct.SegQueue.html + /// The buffer of not-yet-persisted metrics. + /// + /// TODO: use a concurrent queue rather than sequential hashmap, + /// + /// https://docs.rs/crossbeam-queue/latest/crossbeam_queue/struct.SegQueue.html + /// + /// This is a work in progress at: + /// + /// https://github.com/quantiles-evals/quantiles/pull/7 buffer: Arc>>>, + /// The directory inside which metrics are saved. dir: PathBuf, } diff --git a/cli/src/similarity/vector.rs b/cli/src/similarity/vector.rs index 432e297..4a7878a 100644 --- a/cli/src/similarity/vector.rs +++ b/cli/src/similarity/vector.rs @@ -15,7 +15,7 @@ impl CosineSimilarity { /// Build a new cosine similarity metric backed by the default fastembed model. /// /// Uses a **single** embedder session with all available CPU cores. This is - /// faster than a pool of sessions because ONNX Runtime already parallelises + /// faster than a pool of sessions because ONNX Runtime already parallelizes /// inference internally; multiple sessions just create thread-pool /// oversubscription. /// diff --git a/python/LICENSE b/python/LICENSE deleted file mode 100644 index 261eeb9..0000000 --- a/python/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/python/examples/pubmedqa.py b/python/examples/pubmedqa.py index c5de4a1..b69d753 100644 --- a/python/examples/pubmedqa.py +++ b/python/examples/pubmedqa.py @@ -262,5 +262,6 @@ async def _eval_row(row: PubmedQARow) -> EvalResult: ) -pubmedqa_eval = workflow("pubmedqa", _pubmedqa_handler) -entrypoint(pubmedqa_eval) +if __name__ == "__main__": + pubmedqa_eval = workflow("pubmedqa", _pubmedqa_handler) + entrypoint(pubmedqa_eval) diff --git a/python/pyproject.toml b/python/pyproject.toml index 2c365ae..10c81a8 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -17,7 +17,6 @@ indent-width = 2 src = ["src", "tests"] [tool.ruff.lint] -# TODO: enable all of these select = [ "E", # pycodestyle errors "F", # pyflakes @@ -33,6 +32,7 @@ select = [ "RUF", # Ruff-specific rules "ANN", # annotations "ASYNC", # async correctness + # TODO: enable these. will require code fixes # "BLE", # blind except # "TRY", # exception style ] diff --git a/python/src/quantiles/metrics.py b/python/src/quantiles/metrics.py index d3952b5..4deb5b7 100644 --- a/python/src/quantiles/metrics.py +++ b/python/src/quantiles/metrics.py @@ -1,3 +1,9 @@ +""" +Common, generic metrics utilities for use across benchmarks & evals + +TODO: move into `qt` CLI for use across languages. +""" + import math from collections.abc import Iterable from typing import SupportsFloat, final diff --git a/typescript/.gitignore b/typescript/.gitignore index 471f7b8..74cd03f 100644 --- a/typescript/.gitignore +++ b/typescript/.gitignore @@ -38,7 +38,6 @@ bower_components build/Release # Dependency directories -node_modules/ jspm_packages/ # Snowpack dependency directory (https://snowpack.dev/) diff --git a/typescript/LICENSE b/typescript/LICENSE deleted file mode 100644 index 261eeb9..0000000 --- a/typescript/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/typescript/examples/prompt_eval.ts b/typescript/examples/prompt_eval.ts index 095ae81..527876d 100644 --- a/typescript/examples/prompt_eval.ts +++ b/typescript/examples/prompt_eval.ts @@ -110,7 +110,6 @@ const promptEval = workflow( let tokensUsed = 0; for (const evalCase of evalCases) { - // TODO: change to `task` instead of `step` const result = await step( `case:${evalCase.id}`, { diff --git a/typescript/src/index.ts b/typescript/src/index.ts index 4b204ed..0725e7c 100644 --- a/typescript/src/index.ts +++ b/typescript/src/index.ts @@ -4,6 +4,7 @@ * High-level, durable workflow primitives for the `qt` CLI. * * Quick start: + * * ```ts * import { workflow, step, emit, entrypoint } from "@quantiles/sdk"; *