From 20353c77ad3e0535b1aab75ef5942e1e369a9d64 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 09:48:35 -0700 Subject: [PATCH 01/42] agent.md tweaks --- AGENTS.md | 29 +++++++---------------------- 1 file changed, 7 insertions(+), 22 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 1cca915..c477e38 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -91,7 +91,7 @@ Start with these files before making public-facing changes: - [`SECURITY.md`](./SECURITY.md): supported components and vulnerability reporting process. - [`CODE_OF_CONDUCT.md`](./CODE_OF_CONDUCT.md): community participation rules. - [`LICENSE`](./LICENSE): Apache 2.0 license text. -- `CHANGELOG.md`: notable public changes, when present. +- [`mise.toml`](./mise.toml): task definitions for building, formatting, type-checking, linting, and more, all using the [mise](https://mise.en.dev/) task runner. ## Agent Instruction Boundaries @@ -122,7 +122,7 @@ Use these terms consistently in public docs: - `step`: a durable recorded unit of an evaluation execution. - `metric`: a measured value emitted during a run. - `event`: recorded observability data from an evaluation. -- `.quantiles/`: local Quantiles workspace state. +- `.quantiles/`: local Quantiles workspace state, including SQLite database and metrics Parquet files. Prefer `local-first` and `offline by default` for open-source behavior. @@ -134,14 +134,13 @@ Use the `qt` CLI as the source of truth for running, listing, inspecting, compar Prefer CLI output over manually reading `.quantiles/` files. Do not manually edit or delete `.quantiles/` files unless explicitly asked. -Run `qt init` before other `qt` commands if the repository has not been initialized. +Although `qt init` exists to initialize the local Quantiles database, `qt run` automatically does the same. `qt init` is thus often unnecessary to run explicitly. Use `--json` for `qt run`, `qt list`, `qt show`, and `qt compare` when producing agent summaries. Inspect selected runs with `qt show --json`. Common commands: ```bash -qt init qt run --json qt list --json qt show --json @@ -209,7 +208,6 @@ The `qt` CLI is a local-first Rust CLI for running, recording, inspecting, and c It creates local SQLite state under `.quantiles/quantiles.sqlite` and provides commands such as: ```bash -qt init qt run qt list qt show @@ -228,7 +226,7 @@ When editing the Python subdirectory, preserve async behavior, stable JSON paylo ### TypeScript SDK -The TypeScript SDK is an ESM SDK for authoring local AI workload workflows against the Quantiles local observability server at `http://127.0.0.1:8765` by default. +The TypeScript SDK is an SDK for authoring local AI workload workflows against the Quantiles local observability server at `http://127.0.0.1:8765` by default. It exposes workflow primitives, `QuantilesClient`, `QuantilesRun`, stable JSON utilities, and shared SDK types. @@ -236,29 +234,17 @@ When editing the TypeScript subdirectory, preserve strict typing, JSON-serializa ### Agent Skill -The [`skill`](./skill/) subdirectory contains reusable instructions for coding agents that use Quantiles to run, inspect, compare, and summarize evaluation workflows. +The [`skill`](https://github.com/quantiles-evals/skill) repository contains reusable instructions for coding agents that use Quantiles to run, inspect, compare, and summarize evaluation workflows. When editing the skill subdirectory, keep instructions operational, command-driven, and safe for public use. -Read [`skill/SKILL.md`](./skill/SKILL.md) for the reusable agent skill instructions. - -### Benchmarks - -The `benchmarks` subdirectory may contain built-in or example benchmark harnesses, datasets, fixtures, scoring logic, and benchmark documentation. - -When editing benchmark content: - -- Preserve dataset provenance, scoring behavior, and benchmark limitations. -- Record benchmark source, version, commit, dataset revision, scoring configuration, and any local patches when relevant. -- Do not present demo sampler results as model-quality benchmark evidence. -- Avoid adding large datasets, generated outputs, or cached results unless explicitly required and appropriate for the repository. -- If benchmark content changes, call out whether the change affects comparability with prior runs. +Read [github.com/quantiles-evals/skill](https://github.com/quantiles-evals/skill) for the reusable agent skill instructions. ## Working In This Repository Keep root-level changes focused on public orientation, documentation, contribution guidance, security guidance, licensing, and agent instructions. -Update docs when any of the following change: +Update documentation in this repository (e.g. `README.md`, etc...) when any of the following change: - CLI commands, flags, outputs, or setup steps. - SDK APIs, imports, examples, or package names. @@ -280,7 +266,6 @@ Use the Quantiles CLI as the source of truth for local runs. Prefer this flow: ```bash -qt init qt run qt show --json ``` From c31c74b6efa9835e263defd81798399f35ad014d Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:01:23 -0700 Subject: [PATCH 02/42] rm license --- cli/LICENSE | 201 --------------------------------------------- python/LICENSE | 201 --------------------------------------------- typescript/LICENSE | 201 --------------------------------------------- 3 files changed, 603 deletions(-) delete mode 100644 cli/LICENSE delete mode 100644 python/LICENSE delete mode 100644 typescript/LICENSE diff --git a/cli/LICENSE b/cli/LICENSE deleted file mode 100644 index 261eeb9..0000000 --- a/cli/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/python/LICENSE b/python/LICENSE deleted file mode 100644 index 261eeb9..0000000 --- a/python/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/typescript/LICENSE b/typescript/LICENSE deleted file mode 100644 index 261eeb9..0000000 --- a/typescript/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. From 7edbbe4cbbaa6a3b9415532ce05b18a070d181ee Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:04:19 -0700 Subject: [PATCH 03/42] rm old TODO --- typescript/examples/prompt_eval.ts | 1 - 1 file changed, 1 deletion(-) diff --git a/typescript/examples/prompt_eval.ts b/typescript/examples/prompt_eval.ts index 095ae81..527876d 100644 --- a/typescript/examples/prompt_eval.ts +++ b/typescript/examples/prompt_eval.ts @@ -110,7 +110,6 @@ const promptEval = workflow( let tokensUsed = 0; for (const evalCase of evalCases) { - // TODO: change to `task` instead of `step` const result = await step( `case:${evalCase.id}`, { From 8dd64bc679ed3272e027c79d2cec06b58e15e013 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:09:51 -0700 Subject: [PATCH 04/42] doign TODO in readme --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index 928e7d7..3dcfd02 100644 --- a/README.md +++ b/README.md @@ -95,12 +95,10 @@ Built-in benchmarks are ready-to-run evaluation harnesses with predefined datase -A custom evaluation is a Python or TypeScript program that is run by the `qt` CLI and uses the Quantiles API to execute an eval. Your code owns the evaluation logic: loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles records the run, durable steps, emitted metrics, events, inputs, outputs, and comparisons. +A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) or [TypeScript](https://quantiles.io/documentation/reference/typescript-sdk) program that is run by the `qt` CLI and uses the [Quantiles API](https://quantiles.io/documentation/reference/rest-api) to execute an eval. Your code owns the evaluation logic: loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles records the run, durable steps, emitted metrics, events, inputs, outputs, and comparisons. Use custom evaluations when you need to measure behavior that is specific to your product, workflow, prompt, dataset, rubric, or release process. From 5843702ed6ba69f2f0e6872da135fd5701ef266b Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:09:58 -0700 Subject: [PATCH 05/42] small fmt --- typescript/src/index.ts | 1 + 1 file changed, 1 insertion(+) diff --git a/typescript/src/index.ts b/typescript/src/index.ts index 4b204ed..0725e7c 100644 --- a/typescript/src/index.ts +++ b/typescript/src/index.ts @@ -4,6 +4,7 @@ * High-level, durable workflow primitives for the `qt` CLI. * * Quick start: + * * ```ts * import { workflow, step, emit, entrypoint } from "@quantiles/sdk"; * From 8ec25e09991d18c30fe56244f4c046561e7d9a8e Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:10:59 -0700 Subject: [PATCH 06/42] moving TODO --- python/pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/pyproject.toml b/python/pyproject.toml index 2c365ae..10c81a8 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -17,7 +17,6 @@ indent-width = 2 src = ["src", "tests"] [tool.ruff.lint] -# TODO: enable all of these select = [ "E", # pycodestyle errors "F", # pyflakes @@ -33,6 +32,7 @@ select = [ "RUF", # Ruff-specific rules "ANN", # annotations "ASYNC", # async correctness + # TODO: enable these. will require code fixes # "BLE", # blind except # "TRY", # exception style ] From 80422270fc890d8eb77b96e30d514d73be5df2b5 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:11:43 -0700 Subject: [PATCH 07/42] fixup --- python/examples/pubmedqa.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/python/examples/pubmedqa.py b/python/examples/pubmedqa.py index c5de4a1..b69d753 100644 --- a/python/examples/pubmedqa.py +++ b/python/examples/pubmedqa.py @@ -262,5 +262,6 @@ async def _eval_row(row: PubmedQARow) -> EvalResult: ) -pubmedqa_eval = workflow("pubmedqa", _pubmedqa_handler) -entrypoint(pubmedqa_eval) +if __name__ == "__main__": + pubmedqa_eval = workflow("pubmedqa", _pubmedqa_handler) + entrypoint(pubmedqa_eval) From a9483a4d41593fd0ffd9181b2ad35bb3499d7ad2 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:15:53 -0700 Subject: [PATCH 08/42] doc string and TODO --- python/src/quantiles/metrics.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/python/src/quantiles/metrics.py b/python/src/quantiles/metrics.py index d3952b5..4deb5b7 100644 --- a/python/src/quantiles/metrics.py +++ b/python/src/quantiles/metrics.py @@ -1,3 +1,9 @@ +""" +Common, generic metrics utilities for use across benchmarks & evals + +TODO: move into `qt` CLI for use across languages. +""" + import math from collections.abc import Iterable from typing import SupportsFloat, final From 8fef114bad20c2af93d1b047cb44e08708eb16d2 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:22:28 -0700 Subject: [PATCH 09/42] referencing quantiles.io CLI reference, rather than commands.md --- README.md | 2 + cli/COMMANDS.md | 208 ------------------------------------------------ 2 files changed, 2 insertions(+), 208 deletions(-) delete mode 100644 cli/COMMANDS.md diff --git a/README.md b/README.md index 3dcfd02..f422d95 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,8 @@ qt compare >Note: you can pass the `--json` flag to any of the above commands, to output machine and agent-friendly JSON instead of human-formatted output. +To learn more detail about what you can do with the CLI, see [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli). + ## Local-First and Offline by Default Quantiles is built as a local-first, offline system that keeps benchmark execution, metadata, metrics, and analysis on your computer by default. diff --git a/cli/COMMANDS.md b/cli/COMMANDS.md deleted file mode 100644 index 0ce0c80..0000000 --- a/cli/COMMANDS.md +++ /dev/null @@ -1,208 +0,0 @@ -# Quantiles Commands - -This document contains a command reference for the `qt` CLI. - -## `qt init` - -Initialize a local Quantiles workspace in the current directory. - -This creates and configures the local SQLite database (stored in `.quantiles/quantiles.sqlite` by default). - -```bash -qt init -``` - -Example output: - -```text -Initialized Quantiles workspace at ./.quantiles/quantiles.sqlite -``` - -## `qt run` - -Run a command as a durable eval run. - -`qt run` creates a eval run, starts the given command as a subprocess, and records the result. It also injects two environment variables: - -- `QUANTILES_RUN_ID` — the run id so the subprocess can talk back to the API -- `QUANTILES_BASE_URL` — the base URL of the local Quantiles HTTP server - -`qt` automatically starts a local HTTP server on `127.0.0.1:8765` if one is not already running, runs your command, and then tears the server back down when the command finishes. - -```bash -# Simple example -qt run hello-world -- echo "hello from Quantiles" -``` - -You can also pass structured input to the run: - -```bash -qt run eval-smoke-test \ - --input '{"model":"gpt-4.1-mini","dataset":"smoke"}' \ - -- python3 ./my_eval.py -``` - -The `--` separator is required so that flags meant for your command are not parsed by `qt`. - -Example output: - -```text -Created eval run 2 -Executing: echo hello from Quantiles -hello from Quantiles -Run 2 completed successfully -``` - -### Resuming a run - -If a run fails and you want to retry it with the same run id (so that already- -completed samples are reused), pass `--resume`: - -```bash -qt run hello-world --resume 2 -- echo "hello again" -``` - -### TypeScript SDK integration - -The Typescript SDK provides a high-level API that allows users to write evals, and provides seamless integration with the `qt run` command. - -```typescript -import { workflow, step, emit, entrypoint } from "@quantiles/sdk"; - -const runEval = workflow("eval", async (input, ctx) => { - console.log(`Run ${ctx.runId} (${ctx.workflowName})`); - - const result = await step("call-model", async () => { - // expensive work here - return { response: "..." }; - }); - - emit("latency_ms", 120, "ms"); - return result; -}); - -entrypoint(runEval); -``` - -You can register multiple evals and let `qt run ` dispatch to the right one: - -```typescript -const w1 = workflow("eval", async (input, ctx) => { ... }); -const w2 = workflow("benchmark", async (input, ctx) => { ... }); - -entrypoint(w1, w2); -``` - -The callback you pass to `workflow` will receive two arguments: - -- `input` - parsed from `--input` when running via `qt run`, or from a `QUANTILES_INPUT` environment variable if you run this standalone. -- `ctx` — mostly for internal use, but contains the run ID, workflow name (the string you passed as the first parameter to `workflow`, and the raw Quantiles client. All of these are passed in a typescript `interface` like this: `{ runId, workflowName, client }` - -You can also provide explicit input to `step()` for cache invalidation: - -```typescript -const result = await step("call-model", { model: "gpt-4" }, async () => { - return callLLM(model); -}); -``` - -There is also a lower-level `QuantilesClient` API available for users who want finer-grained control over execution. - -See `sdk/typescript/examples/run_demo.ts` for a runnable example. - -## `qt serve` - -Start the local Quantiles HTTP server. - -```bash -qt serve -``` - -Generally, you won't need to run this manually. `qt run` starts a server automatically for the duration of your command. Use `qt serve` only when you want a persistent server (e.g. for development or multiple CLI windows). - -## `qt list` - -Show all eval runs in reverse chronological order. - -```bash -qt list -``` - -Example output: - -```text -ID EVAL STATUS SAMPLES CREATED FINISHED_AT -7 eval-smoke-test completed 2 2026-05-02T18:30:00.000Z 2026-05-02T18:30:05.000Z -3 hello-world failed 1 2026-05-02T18:15:00.000Z 2026-05-02T18:15:02.000Z -``` - -## `qt show` - -Inspect a single eval run, including its samples, metrics, and events. - -```bash -qt show -``` - -Example output: - -```text -Run 7 - eval: eval-smoke-test - status: completed - started_at: 2026-05-02T18:30:00.000Z - finished_at: 2026-05-02T18:30:05.000Z - input: {"model":"gpt-4.1-mini","dataset":"smoke"} - error: - - -Samples - ID KEY STATUS INPUT_HASH CREATED FINISHED_AT - 4 fetch-data completed a1b2c3d4e5f6789 2026-05-02T18:30:00.500Z 2026-05-02T18:30:01.000Z - 5 call-model completed 9f8e7d6c5b4a321 2026-05-02T18:30:01.200Z 2026-05-02T18:30:04.800Z - -Metrics - NAME VALUE UNIT - latency_ms 120 ms - tokens_used 42 - - -Events - ID TYPE CREATED MESSAGE - 10 run.started 2026-05-02T18:30:00.000Z Started eval eval-smoke-test - 11 step.started 2026-05-02T18:30:00.500Z Started step fetch-data - 12 step.completed 2026-05-02T18:30:01.000Z Completed step 4 - ... -``` - -## `qt compare` - -Compare two eval runs side-by-side. - -```bash -qt compare -``` - -Checks samples, outputs, and metrics. Exits with code 0 if the runs are identical -and code 1 if anything differs. - -Example output: - -```text -Run 1: eval-smoke-test (completed) -Run 2: eval-smoke-test (completed) - -Samples - STEP PRESENT INPUT STATUS OUTPUT - fetch-data both same same differs - -Output differences - step: fetch-data - run 1: {"status":200,"body":"hello"} - run 2: {"status":200,"body":"world"} - -Metrics - NAME Run 1 Run 2 - latency_ms 50 150 * - tokens_used 42 89 * - -Runs differ. -``` From 17a752f2f089253df808687f4187373470e714cf Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:27:03 -0700 Subject: [PATCH 10/42] taking out page --- cli/EVAL_CHALLENGES.md | 69 ------------------------------------------ 1 file changed, 69 deletions(-) delete mode 100644 cli/EVAL_CHALLENGES.md diff --git a/cli/EVAL_CHALLENGES.md b/cli/EVAL_CHALLENGES.md deleted file mode 100644 index d7acbaa..0000000 --- a/cli/EVAL_CHALLENGES.md +++ /dev/null @@ -1,69 +0,0 @@ -# Why Evals Are Hard (And What to do Instead) - -Evals look simple: - -```typescript -for(const data of dataset) { - const output = await sampleModel(data); - const score = await judge(data, output); - await recordScore(data, output, score); -} -``` - -In practice, they're not. The moment you try to run real experiments, things break down quickly: - -- You need to load and shuffle datasets deterministically -- You need to track which version of your prompt or model you ran -- You need to be able to recover from crashes -- You need to cache expensive samples so you don’t re-run everything -- You need to record the right metrics so you can analyze results later -- You need to compare runs to understand what actually improved - -Instead of writing a loop, you end up building a system before you can even start doing what you came here to do -- figure out if your AI is good or not. - -## What Most People End Up With - -A typical eval looks like this: - -1. Write a script to prepare data -2. Run it -3. Write a script to call the model -4. Run it -5. Write a script to score outputs -6. Run it -7. Dump everything into a database -8. Write SQL or open a dashboard to figure out what happened - -Each step is separate, each run is hard to reproduce, and comparing results is manual. - -## With Quantiles - -Quantiles turns all those samples into a single script: - -```typescript -import { workflow, step, emit } from "quantiles"; -export const evalPrompt = workflow(async ({ promptVersion }) => { - for (const prompt of dataset) { - const output = await step("call-model", {prompt}, async () => { - return callModel(prompt, promptVersion); - }); - const score = await step("judge", {prompt}, async () => { - return judge(output); - }); - emit("score", score); - } -}); -``` - -... which you can then run and analyze with a few commands: - -```bash -# run your eval against two prompts, to figure out -# which performs better -qt run evalPrompt --input '{"promptVersion":"v1"}' -qt run evalPrompt --input '{"promptVersion":"v2"}' -# get a deep analysis of your results -qt compare 1 2 -``` - -You write less code, you get resilience for free, and you spend all your time _actually_ making your AI app better, not building systems. From 6fa93a2914c7218a3323beb53afb80ca33610951 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:31:11 -0700 Subject: [PATCH 11/42] minor tweaks --- cli/BEST_EVAL_PRACTICES.md | 79 -------------------------------------- cli/README.md | 4 +- 2 files changed, 3 insertions(+), 80 deletions(-) delete mode 100644 cli/BEST_EVAL_PRACTICES.md diff --git a/cli/BEST_EVAL_PRACTICES.md b/cli/BEST_EVAL_PRACTICES.md deleted file mode 100644 index e6903d4..0000000 --- a/cli/BEST_EVAL_PRACTICES.md +++ /dev/null @@ -1,79 +0,0 @@ -# Eval Best Practices - -Quantiles provides a framework that makes evals faster to build, more reliable to run, and easier to measure, so you can focusing in figuring out if your AI app is getting better over time. - -This document provides a set of best practices for writing evals using Quantiles. - -## Deterministic Step Inputs - -A _step_ in Quantiles is a discrete action that your eval take, that is a "checkpoint" in the eval script. You generally run code inside a `step` that can fail or is expensive, like calling your model or doing some big database operation. Most often, you'll use the `step(...)` function in the Typescript/Python SDks to mark the processing of a sample in your eval. - -Quantiles makes sure that, if your eval crashes, it doesn't have to start from the beginning - it can just start from the last `step` that succeeded. - -Samples are also useful for observability - they have inputs and outputs, so you can get a report of what happened just before, during, and just after each step. - -For Quantiles to treat samples properly, it's usually helpful to pass data to the `step` function that tells Quantiles whether a given `step` invocation is different from a previous one. Consider the below code: - -```typescript -const cities = ["Tokyo", "Munich", "Nairobi", "San Francisco", "Buenos Aires", "Sydney"]; -const city1 = cities[Math.floor(Math.random() * cities.length)]; -const city2 = cities[Math.floor(Math.random() * cities.length)]; - -step("call-llm", async () => { - return async call_llm(`what is the weather in ${city1}`); -}); - -step("call-llm", async () => { - return async call_llm(`what is the weather in ${city2}`); -}) -``` - -In this code, the two samples have the same name, but they close over different data (`city1` and `city2`, respectively). That means Quantiles can't tell the difference between them, so it will: - -- Assume they're equal - which can lead to problems when comparing two evals -- Not be able to intelligently cache and recover from failed samples - which can lead to problems when, for example, `call_llm` is flaky - -Initially, you won't need to worry about these things, but as you continue to run your evals over time (and we believe that you should!), you will be increasingly likely to encounter them. To solve this problem, you have two options: - -1. Give each step a distinct name - this is the easiest fix, but you might not want to change your step name -2. Pass a cache key to the `step` function - this is the fix if you don't want to change your step name - -```typescript -// - -// The second parameter to `step(...)` tells Quantiles that this -// call-llm step uses the 'city1' data. Quantiles will consider it -// different from any other call-llm step that uses any other -// data. -step("call-llm", {"city": city1}, async () => { - return async call_llm(`what is the weather in ${city1}`); -}); - -// Because the second parameter to `step(...)` has a different -// value than that of the previous `step(...)` call, Quantiles -// considers this step as different from the previous call. -step("call-llm", {"city": city2}, async () => { - return async call_llm(`what is the weather in ${city2}`); -}) -```` - -## Running Samples in Loops - -Many evals start by iterating over a dataset. Below is an example: - -```typescript -interface DatasetRow { - prompt: string; - goldenOutput: string; -} - -const datasetRows: DatasetRow[] = await loadDataset(); -for (let i = 0; i < 100; i++) { - const datasetRow = datasetRows[i]; - await step(`iter_step:${i}`, () => { - doSomething(i) - }); -} -``` - -In this example, Quantiles will be able to differentiate each step, which is great. Over time, though, you might want to alter the dataset over which you're looping. For example: diff --git a/cli/README.md b/cli/README.md index ae49b3f..ea9b28f 100644 --- a/cli/README.md +++ b/cli/README.md @@ -93,6 +93,8 @@ const modelOutput = await step("call-model", async () => { }); ``` +See [quantiles.io/documentation/workflows-and-steps](https://quantiles.io/documentation/workflows-and-steps) for more details on how steps and step caching work. + ### Compare two runs After iterating on an eval, compare two runs to see exactly what changed: @@ -166,4 +168,4 @@ See [ARCHITECTURE.md](./ARCHITECTURE.md) ## Command Reference -See [COMMANDS.md](./COMMANDS.md) +See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. From e942f8ff88c662e158a7b950a9554adf0babb6c6 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:33:48 -0700 Subject: [PATCH 12/42] progress on cli readme --- cli/ARCHITECTURE.md | 32 -------------------------------- cli/README.md | 38 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 36 insertions(+), 34 deletions(-) delete mode 100644 cli/ARCHITECTURE.md diff --git a/cli/ARCHITECTURE.md b/cli/ARCHITECTURE.md deleted file mode 100644 index 4a1dfe9..0000000 --- a/cli/ARCHITECTURE.md +++ /dev/null @@ -1,32 +0,0 @@ -# `qt`'s Architecture - -The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while `qt` handles durability and observability. - -``` -+--------------------------------------+ -| Your Script | -| (TypeScript / Python / Shell) | -+--------------------+-----------------+ - │ - │ HTTP / JSON - ▼ -+--------------------------------------+ -| Quantiles Server | -+--------------------+-----------------+ - │ - │ SQLite - ▼ -+------------------------------------------------+ -| .quantiles/quantiles.sqlite (local DB) | -+--------------------+---------------------------+ - │ - ▼ -+--------------------------------------+ -| CLI | -| (list, show, compare) | -+--------------------------------------+ -``` - -- **Server** owns durability decisions: step caching, run state, metrics -- **Client** (your script) owns code execution: the server never runs your logic -- **CLI** reads the same SQLite database the server writes to diff --git a/cli/README.md b/cli/README.md index ea9b28f..89afbda 100644 --- a/cli/README.md +++ b/cli/README.md @@ -1,4 +1,4 @@ -# Quantiles +# Quantiles CLI `qt` is a local-first CLI for running AI evals as experiments, so you can instantly see what improved, what regressed, and why. @@ -164,7 +164,41 @@ Quantiles treats evals as experiments in which every run is tracked, comparable, ## Architecture -See [ARCHITECTURE.md](./ARCHITECTURE.md) +The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while `qt` handles durability and observability. + +``` ++--------------------------------------+ +| Your Script | +| (TypeScript / Python / Shell) | ++--------------------+-----------------+ + │ + │ HTTP / JSON + | + ▼ ++--------------------------------------+ +| Quantiles Server | ++--------------------+-----------------+ + │ + │ SQLite + | + ▼ ++------------------------------------------------+ +| .quantiles/quantiles.sqlite (local DB) | ++--------------------+---------------------------+ + │ + │ + │ + ▼ ++--------------------------------------+ +| CLI | +| (list, show, compare) | ++--------------------------------------+ +``` + +- **Server** owns durability decisions: step caching, run state, metrics +- **Client** (your script) owns code execution: the server never runs your logic + - Note that the CLI itself also has built-in benchmarks, which do not involve your code +- **CLI** reads the same SQLite database the server writes to ## Command Reference From 92a93d08cb95a9fdeed18357fd4be92b65c7fa5c Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:35:17 -0700 Subject: [PATCH 13/42] spacing --- cli/README.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/cli/README.md b/cli/README.md index 89afbda..578c342 100644 --- a/cli/README.md +++ b/cli/README.md @@ -170,25 +170,25 @@ The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while ` +--------------------------------------+ | Your Script | | (TypeScript / Python / Shell) | -+--------------------+-----------------+ - │ - │ HTTP / JSON - | - ▼ ++-------------------+------------------+ + │ + │ HTTP / JSON + | + ▼ +--------------------------------------+ | Quantiles Server | -+--------------------+-----------------+ - │ - │ SQLite - | - ▼ ++-------------------+------------------+ + │ + │ SQLite + | + ▼ +------------------------------------------------+ | .quantiles/quantiles.sqlite (local DB) | -+--------------------+---------------------------+ - │ - │ - │ - ▼ ++-------------------+----------------------------+ + │ + │ + │ + ▼ +--------------------------------------+ | CLI | | (list, show, compare) | From f45c6a73bcd3d4af928bf7d2c4cdcdf4f64b144c Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:35:55 -0700 Subject: [PATCH 14/42] gitignore cleanup --- .gitignore | 1 + cli/.gitignore | 1 - typescript/.gitignore | 1 - 3 files changed, 1 insertion(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 8f73094..76bb1e4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ .envrc .DS_Store /.quantiles +node_modules diff --git a/cli/.gitignore b/cli/.gitignore index 47b21b7..ff4c21c 100644 --- a/cli/.gitignore +++ b/cli/.gitignore @@ -22,7 +22,6 @@ target /.quantiles /.wrangler /dist -/node_modules .fastembed_cache .envrc /qt-linux diff --git a/typescript/.gitignore b/typescript/.gitignore index 471f7b8..74cd03f 100644 --- a/typescript/.gitignore +++ b/typescript/.gitignore @@ -38,7 +38,6 @@ bower_components build/Release # Dependency directories -node_modules/ jspm_packages/ # Snowpack dependency directory (https://snowpack.dev/) From 1b9c439305f02cfaaaa570554659d2293544a0aa Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:41:10 -0700 Subject: [PATCH 15/42] removing protocol.md --- cli/AGENTS.md | 2 +- cli/PROTOCOL.md | 314 ------------------------------------------------ 2 files changed, 1 insertion(+), 315 deletions(-) delete mode 100644 cli/PROTOCOL.md diff --git a/cli/AGENTS.md b/cli/AGENTS.md index 3087a22..737a8ab 100644 --- a/cli/AGENTS.md +++ b/cli/AGENTS.md @@ -9,7 +9,7 @@ - Prefer focused changes that fit the current CLI and library structure. - Keep local-only behavior explicit; do not introduce network or cloud behavior unless the task specifically calls for it. If you do add such behavior, make sure to alert the user about it, and add comments in the code to draw attention to it. - Preserve existing SQLite data model assumptions unless the change includes a deliberate schema migration or initialization update. -- Use idiomatic Rust and keep error handling clear. This project uses `anyhow` for application-level errors. +- Use idiomatic Rust and keep error handling clear. This project uses the [`anyhow`](https://docs.rs/anyhow) to create and propagate application-level errors. - Avoid broad refactors while implementing narrow behavior changes. ## Validation and Testing diff --git a/cli/PROTOCOL.md b/cli/PROTOCOL.md deleted file mode 100644 index 23bdbf9..0000000 --- a/cli/PROTOCOL.md +++ /dev/null @@ -1,314 +0,0 @@ -# Quantiles Eval Protocol - -This document describes the API calls that a successful eval should make to record its execution in Quantiles. It covers the REST API contract and the corresponding TypeScript SDK calls. - -An eval, also called a workflow in the protocol, is a process that does the following: -1. Creates a run (or reuses one assigned by the CLI) -2. Runs samples (durable, memoized units of work) -3. Emits metrics -4. Saves its final output -5. Completes (or fails) - -The exact calls differ slightly depending on whether the eval is started by `qt run` (CLI-driven) or started directly by calling `.run()` on a workflow object (programmatic). - ---- - -## Execution Modes - -### CLI-driven run (`qt run my-eval -- ...`) - -The CLI creates the run, starts the server, and spawns the eval process with these environment variables: - -- `QUANTILES_RUN_ID` — the ID of the run -- `QUANTILES_BASE_URL` — the local server URL -- `QUANTILES_WORKFLOW_NAME` — the eval name -- `QUANTILES_INPUT` — the JSON input (or `{}`) - -In this mode, the eval process **must not** call `POST /runs` to create a new run — the run already exists. It **must** save its output via `POST /runs/{run_id}/output`, but should let the CLI own the final `complete` or `fail` call. - -### Programmatic run (`workflow.run()`) - -There is no `QUANTILES_RUN_ID` environment variable. The SDK creates the run itself, then completes or fails the run directly. - ---- - -## Complete Protocol for a Success - -### 1. Create a run (programmatic only) - -**REST API (not used in CLI-driven mode):** - -```http -POST /runs -Content-Type: application/json - -{ - "eval_name": "my-eval", - "input": "{\"dataset\":\"smoke\"}" -} -``` - -**Response:** - -```json -{ - "run_id": 1 -} -``` - -**TypeScript SDK:** - -```typescript -const run = await client.createRun("my-eval", { dataset: "smoke" }); -// run.id === 1 -``` - ---- - -### 2. Run a step - -Before executing expensive work, call `POST /steps/begin`. If the step was already completed with the same `input_hash`, the server returns a cache hit (`decision: "reuse"`). Otherwise it returns `decision: "run"` and assigns a `step_id`. - -**REST API (begin):** - -```http -POST /steps/begin -Content-Type: application/json - -{ - "run_id": 1, - "step_key": "fetch-data", - "input_hash": "abc123..." -} -``` - -**Response (run):** - -```json -{ - "decision": "run", - "step_id": 3 -} -``` - -**Response (reuse):** - -```json -{ - "decision": "reuse", - "output": "{\"status\":200}" -} -``` - -If the decision is `"run"`, execute the step, then report completion or failure. - -**REST API (complete):** - -```http -POST /steps/complete -Content-Type: application/json - -{ - "step_id": 3, - "output": "{\"status\":200}" -} -``` - -**REST API (fail):** - -```http -POST /steps/fail -Content-Type: application/json - -{ - "step_id": 3, - "error": "network timeout" -} -``` - -**TypeScript SDK:** - -```typescript -const result = await step( - "fetch-data", - { url: "https://example.com" }, - async () => { - return { status: 200 }; - }, -); -``` - ---- - -### 3. Emit a metric - -Metrics are associated with a run. They can be emitted from the eval or from individual samples. - -**REST API:** - -```http -POST /runs/1/metrics -Content-Type: application/json - -{ - "metric_name": "latency_ms", - "metric_value": 50, - "unit": "ms" -} -``` - -**TypeScript SDK:** - -```typescript -await emit("latency_ms", 50, "ms"); -``` - ---- - -### 4. Save the eval output - -At the end of the eval, save the return value. This is a **separate** call from completing the run. - -**CLI-driven mode:** Always call this. The CLI will complete the run later based on the process exit code. - -**Programmatic mode:** Skip this — `completeRun` handles both output and status. - -**REST API:** - -```http -POST /runs/1/output -Content-Type: application/json - -{ - "output": "{\"accuracy\":0.85,\"correct\":17,\"total\":20}" -} -``` - -**TypeScript SDK:** - -```typescript -await client.setRunOutput(1, { accuracy: 0.85, correct: 17, total: 20 }); -``` - ---- - -### 5. Complete the run (programmatic only) - -Marks the run as completed, optionally including the output. Emits a `run.completed` event. - -**CLI-driven mode:** Do not call this. The CLI calls it when the child process exits successfully. - -**Programmatic mode:** Call this (it already includes the output). - -**REST API:** - -```http -POST /runs/1/complete -``` - -**TypeScript SDK:** - -```typescript -await client.completeRun(1, { accuracy: 0.85, correct: 17, total: 20 }); -``` - ---- - -### 6. Fail the run (on error) - -**CLI-driven mode:** Throw an error or exit non-zero. The CLI will call `POST /runs/{run_id}/fail` automatically. - -**Programmatic mode:** Call `failRun`. - -**REST API:** - -```http -POST /runs/1/fail -Content-Type: application/json - -{ - "error": "network failed" -} -``` - -**TypeScript SDK:** - -```typescript -await client.failRun(1, new Error("network failed")); -``` - ---- - -## TypeScript Eval Example (CLI-driven) - -This is the typical pattern used with `qt run`: - -```typescript -import { emit, entrypoint, step, workflow } from "@quantiles/sdk"; - -const eval = workflow("support-triage", async (input) => { - const results = await step("classify", input, async () => { - return await callModel(input); - }); - - emit("accuracy", results.accuracy); - emit("latency_ms", results.latency, "ms"); - - return results; -}); - -entrypoint(eval); -``` - -The SDK handles all API calls automatically. Because `QUANTILES_RUN_ID` is set, it: -1. Reuses the existing run ID -2. Creates/executes/reuses samples -3. Emits metrics -4. Calls `setRunOutput(runId, results)` at the end - -Then the process exits with code 0, and the CLI completes the run. - ---- - -## TypeScript Eval Example (Programmatic) - -Useful for testing or running evals from other scripts: - -```typescript -import { QuantilesClient, workflow, step, emit } from "@quantiles/sdk"; - -const eval = workflow("support-triage", async (input) => { - const results = await step("classify", input, async () => { - return await callModel(input); - }); - - emit("accuracy", results.accuracy); - return results; -}); - -const client = new QuantilesClient({ baseUrl: "http://127.0.0.1:8765" }); -const run = await client.createRun("support-triage", { dataset: "smoke" }); -const output = await eval.run({ dataset: "smoke" }); - -console.log("Run completed:", run.id); -console.log("Output:", output); -``` - -Because `QUANTILES_RUN_ID` is not set, the eval creates its own run and calls `completeRun(runId, output)` automatically. - ---- - -## Endpoints Summary - -| Method | Path | Body | Purpose | -|---|---|---|---| -| `POST` | `/runs` | `{ workflow_name, input }` | Create a new run (programmatic) | -| `GET` | `/runs/{run_id}` | — | Fetch run metadata | -| `POST` | `/runs/{run_id}/output` | `{ output }` | Save eval return value | -| `POST` | `/runs/{run_id}/complete` | `{ output }` | Mark run completed (programmatic) | -| `POST` | `/runs/{run_id}/fail` | `{ error }` | Mark run failed | -| `POST` | `/runs/{run_id}/metrics` | `{ metric_name, metric_value, unit }` | Record a metric | -| `POST` | `/steps/begin` | `{ run_id, step_key, input_hash }` | Reserve/reuse a step | -| `POST` | `/steps/complete` | `{ step_id, output }` | Mark step completed | -| `POST` | `/steps/fail` | `{ step_id, error }` | Mark step failed | -| `GET` | `/health` | — | Server health check | From c8660879e2e2eac5833c340ef495951f44cd1f6b Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:49:20 -0700 Subject: [PATCH 16/42] cleaning up anthropic demo config --- cli/examples/configs/anthropic/quantiles.toml | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/cli/examples/configs/anthropic/quantiles.toml b/cli/examples/configs/anthropic/quantiles.toml index d6f56ee..dc432f5 100644 --- a/cli/examples/configs/anthropic/quantiles.toml +++ b/cli/examples/configs/anthropic/quantiles.toml @@ -1,15 +1,20 @@ +# This config file sets up PubMedQA to use Anthropic's Claude model, and sets up the SimpleQA-verified +# benchmark to use the internal demo model. +# +# The PubMedQA benchmark reads the API key from the `ANTHROPIC_API_KEY` environment +# variable. To get an API key, go to https://platform.claude.com/settings/workspaces/default/keys + [benchmarks.pubmedqa] +# Since Anthropic charges per input and output token, we are limiting the number of samples +# in this demo to keep costs low. samples = 50 -# Use the Anthropic Claude API, reading the API key from the `ANTHROPIC_API_KEY` environment -# variable. To get an API key, go to https://platform.claude.com/settings/workspaces/default/keys -# # The below model is good for demonstrating PubMedQA benchmarks without very high costs, # since it's relatively cheap and fast. Details on the model can be found at: # # https://platform.claude.com/docs/en/about-claude/models/overview#latest-models-comparison # -# This page also has a list of other frontier and near-frontier models that you can use. +# That page also has a list of other Anthropic models that you can use. model = "anthropic:claude-haiku-4-5-20251001" # If you run this or other benchmarks with more samples, you may also need to increase this @@ -23,8 +28,8 @@ model = "anthropic:claude-haiku-4-5-20251001" # You can also see your usage at https://platform.claude.com/usage max_workers = 5 +# If you don't want to use a hosted/paid model, you can omit model configuration and +# use the built-in demo model. Note, however, that this built-in model emits random text, and +# should only be used for testing purposes. [benchmarks.simpleqa-verified] samples = 100 - -[benchmarks.financebench] -samples = 50 From d42917ce14a61aa89985924c80b9890b7b9d89fc Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:49:41 -0700 Subject: [PATCH 17/42] progress --- cli/examples/configs/both_error/.quantiles.toml | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/cli/examples/configs/both_error/.quantiles.toml b/cli/examples/configs/both_error/.quantiles.toml index 26ab1cd..e2a96a1 100644 --- a/cli/examples/configs/both_error/.quantiles.toml +++ b/cli/examples/configs/both_error/.quantiles.toml @@ -1,15 +1,3 @@ -# Example Quantiles workspace configuration. -# -# Place this file in your working directory as either: -# - quantiles.toml -# - .quantiles.toml -# -# If both files exist in the same directory, the CLI will error to avoid -# ambiguity. -# -# Each section under [benchmarks.] maps to a builtin eval. -# The `samples` key limits the number of dataset rows processed. - [benchmarks.pubmedqa] samples = 10 From 4d77c5454c87d7664e9711b9ec82bb930a2b9a3e Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:54:57 -0700 Subject: [PATCH 18/42] progress on config docs --- cli/examples/configs/anthropic/quantiles.toml | 2 +- .../configs/cloudflare/quantiles.toml | 37 +++++++++++++------ 2 files changed, 27 insertions(+), 12 deletions(-) diff --git a/cli/examples/configs/anthropic/quantiles.toml b/cli/examples/configs/anthropic/quantiles.toml index dc432f5..fc36c60 100644 --- a/cli/examples/configs/anthropic/quantiles.toml +++ b/cli/examples/configs/anthropic/quantiles.toml @@ -6,7 +6,7 @@ [benchmarks.pubmedqa] # Since Anthropic charges per input and output token, we are limiting the number of samples -# in this demo to keep costs low. +# in this demo to keep costs lower. samples = 50 # The below model is good for demonstrating PubMedQA benchmarks without very high costs, diff --git a/cli/examples/configs/cloudflare/quantiles.toml b/cli/examples/configs/cloudflare/quantiles.toml index 00875a6..0122144 100644 --- a/cli/examples/configs/cloudflare/quantiles.toml +++ b/cli/examples/configs/cloudflare/quantiles.toml @@ -1,13 +1,23 @@ -[benchmarks.pubmedqa] -samples = 50 - -# Use Cloudflare AI Gateway, reading the account ID, gateway ID, and API key -# from environment variables: +# This config file sets up PubMedQA to use the Cloudflare AI gateway, and sets up the SimpleQA-verified +# benchmark to use the internal demo model. +# +# The PubMedQA benchmark reads the Cloudflare account ID, gateway ID, and API key +# from the following environment variables: +# # - CLOUDFLARE_ACCOUNT_ID # - CLOUDFLARE_GATEWAY_ID # - CLOUDFLARE_API_KEY # -# This model is good for demonstrating PubMedQA benchmarks without very high costs, +# To get your account ID, gateway ID and API key, go to your Cloudflare AI gateway dashboard at: +# +# https://dash.cloudflare.com//ai/ai-gateway + +[benchmarks.pubmedqa] +# Since CLoudflare charges for usage, we are limiting the number of samples in this demo to +# keep costs lower. +samples = 50 + +# The below model is good for demonstrating PubMedQA benchmarks without very high costs, # since it's relatively cheap, fast, and not a reasoning model. Its details can be found at: # # https://developers.cloudflare.com/workers-ai/models/llama-3.3-70b-instruct-fp8-fast/ @@ -16,17 +26,22 @@ samples = 50 # # https://developers.cloudflare.com/workers-ai/models/ model = "cloudflare_ai_gateway:@cf/meta/llama-3.3-70b-instruct-fp8-fast" + +# If you don't want to specify your account ID and gateway ID in environment variables +# (CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_GATEWAY_ID, respectively), you can provide them +# alongside the model in a dictionary as below: # -# Or provide the account ID / gateway ID inline, and still read the API key from -# the CLOUDFLARE_API_KEY env var # model = { # type = "cloudflare_ai_gateway:@cf/meta/llama-3.3-70b-instruct-fp8-fast", # account_id = "...", # gateway_id = "..." # } +# +# Note that if you do this, you will still need to provide the API key in the +# CLOUDFLARE_API_KEY environment variable. +# If you don't want to use a hosted/paid model, you can omit model configuration and +# use the built-in demo model. Note, however, that this built-in model emits random text, and +# should only be used for testing purposes. [benchmarks.simpleqa-verified] samples = 100 - -[benchmarks.financebench] -samples = 50 From 2e6dc03775b995ca9d482cadb5faf96de26c5ec3 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 10:55:06 -0700 Subject: [PATCH 19/42] typo --- cli/examples/configs/cloudflare/quantiles.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cli/examples/configs/cloudflare/quantiles.toml b/cli/examples/configs/cloudflare/quantiles.toml index 0122144..fff63c0 100644 --- a/cli/examples/configs/cloudflare/quantiles.toml +++ b/cli/examples/configs/cloudflare/quantiles.toml @@ -13,7 +13,7 @@ # https://dash.cloudflare.com//ai/ai-gateway [benchmarks.pubmedqa] -# Since CLoudflare charges for usage, we are limiting the number of samples in this demo to +# Since Cloudflare charges for usage, we are limiting the number of samples in this demo to # keep costs lower. samples = 50 From 6c1224620472b1c1ed73d871cbfb6b133d35f2e8 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 11:05:01 -0700 Subject: [PATCH 20/42] customization note --- README.md | 4 ++++ cli/README.md | 6 ++++++ cli/examples/configs/simple/quantiles.toml | 1 + 3 files changed, 11 insertions(+) diff --git a/README.md b/README.md index f422d95..eddeb9e 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,10 @@ qt compare To learn more detail about what you can do with the CLI, see [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli). +### Customization + +You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details. + ## Local-First and Offline by Default Quantiles is built as a local-first, offline system that keeps benchmark execution, metadata, metrics, and analysis on your computer by default. diff --git a/cli/README.md b/cli/README.md index 578c342..05b1a22 100644 --- a/cli/README.md +++ b/cli/README.md @@ -32,6 +32,12 @@ qt list qt show 1 ``` +## Customization + +### Customization + +You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details. + ### Durable step caching + crash resume (TypeScript SDK) Use the high-level `workflow()`, `step()`, and `emit()` API for automatic diff --git a/cli/examples/configs/simple/quantiles.toml b/cli/examples/configs/simple/quantiles.toml index 94fa959..1199405 100644 --- a/cli/examples/configs/simple/quantiles.toml +++ b/cli/examples/configs/simple/quantiles.toml @@ -1,6 +1,7 @@ # Example Quantiles workspace configuration. # # Place this file in your working directory as either: +# # - quantiles.toml # - .quantiles.toml # From d11b82054243972826cf6104b99b24dd1187677a Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 11:07:04 -0700 Subject: [PATCH 21/42] gemini cleanups --- cli/examples/configs/gemini/quantiles.toml | 24 ++++++++++++++-------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/cli/examples/configs/gemini/quantiles.toml b/cli/examples/configs/gemini/quantiles.toml index 1e216a8..106edb9 100644 --- a/cli/examples/configs/gemini/quantiles.toml +++ b/cli/examples/configs/gemini/quantiles.toml @@ -1,17 +1,23 @@ -[benchmarks.pubmedqa] -samples = 50 - -# Use the Gemini API. The API key is read from the `GEMINI_API_KEY` environment +# This config file sets up PubMedQA to use the Gemini API, and sets up the SimpleQA-verified +# benchmark to use the internal demo model. +# +# The PubMedQA benchmark reads the Gemini API key from the `GEMINI_API_KEY` environment # variable. Learn more about the Gemini API in Google's Gemini AI Studio: # # https://aistudio.google.com/prompts/new_chat -# + + +[benchmarks.pubmedqa] +# Since Google charges per input and output token, we are limiting the number of samples +# in this demo to keep costs lower. +samples = 50 + # The below model is good for demonstrating PubMedQA benchmarks without very high costs, # since it's relatively cheap and fast. Details on the model can be found at: # # https://ai.google.dev/gemini-api/docs/models/gemini#gemini-3.1-flash-lite # -# This page also has a list of other models that you can use. +# That page also lists other Google models you can use. model = "gemini:gemini-3.1-flash-lite" # If you run this or other benchmarks with more samples, you may also need to increase this @@ -23,8 +29,8 @@ model = "gemini:gemini-3.1-flash-lite" # https://aistudio.google.com/rate-limit max_workers = 10 +# If you don't want to use a hosted/paid model, you can omit model configuration and +# use the built-in demo model. Note, however, that this built-in model emits random text, and +# should only be used for testing purposes. [benchmarks.simpleqa-verified] samples = 100 - -[benchmarks.financebench] -samples = 50 From f49f28599c8f8a46ca5d6b4418e8633a68c47b24 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 11:13:35 -0700 Subject: [PATCH 22/42] more notes --- cli/examples/configs/anthropic/quantiles.toml | 4 ++-- cli/examples/configs/both_error/quantiles.toml | 2 ++ cli/examples/configs/cloudflare/quantiles.toml | 4 ++-- cli/examples/configs/gemini/quantiles.toml | 4 ++-- cli/examples/configs/openai/quantiles.toml | 2 ++ 5 files changed, 10 insertions(+), 6 deletions(-) diff --git a/cli/examples/configs/anthropic/quantiles.toml b/cli/examples/configs/anthropic/quantiles.toml index fc36c60..73616fb 100644 --- a/cli/examples/configs/anthropic/quantiles.toml +++ b/cli/examples/configs/anthropic/quantiles.toml @@ -5,8 +5,8 @@ # variable. To get an API key, go to https://platform.claude.com/settings/workspaces/default/keys [benchmarks.pubmedqa] -# Since Anthropic charges per input and output token, we are limiting the number of samples -# in this demo to keep costs lower. +# The full expert-labeled PubMedQA benchmark has 1000 samples. Since Anthropic charges for +# usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 # The below model is good for demonstrating PubMedQA benchmarks without very high costs, diff --git a/cli/examples/configs/both_error/quantiles.toml b/cli/examples/configs/both_error/quantiles.toml index e69de29..c0d56b3 100644 --- a/cli/examples/configs/both_error/quantiles.toml +++ b/cli/examples/configs/both_error/quantiles.toml @@ -0,0 +1,2 @@ +# The `qt` CLI will refuse to run if both a `quantiles.toml` and `.quantiles.toml` configuration +# file exist in the current directory, even if one of the files is empty. diff --git a/cli/examples/configs/cloudflare/quantiles.toml b/cli/examples/configs/cloudflare/quantiles.toml index fff63c0..8d03529 100644 --- a/cli/examples/configs/cloudflare/quantiles.toml +++ b/cli/examples/configs/cloudflare/quantiles.toml @@ -13,8 +13,8 @@ # https://dash.cloudflare.com//ai/ai-gateway [benchmarks.pubmedqa] -# Since Cloudflare charges for usage, we are limiting the number of samples in this demo to -# keep costs lower. +# The full expert-labeled PubMedQA benchmark has 1000 samples. Since Cloudflare charges for +# usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 # The below model is good for demonstrating PubMedQA benchmarks without very high costs, diff --git a/cli/examples/configs/gemini/quantiles.toml b/cli/examples/configs/gemini/quantiles.toml index 106edb9..16b175b 100644 --- a/cli/examples/configs/gemini/quantiles.toml +++ b/cli/examples/configs/gemini/quantiles.toml @@ -8,8 +8,8 @@ [benchmarks.pubmedqa] -# Since Google charges per input and output token, we are limiting the number of samples -# in this demo to keep costs lower. +# The full expert-labeled PubMedQA benchmark has 1000 samples. Since Google charges for +# usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 # The below model is good for demonstrating PubMedQA benchmarks without very high costs, diff --git a/cli/examples/configs/openai/quantiles.toml b/cli/examples/configs/openai/quantiles.toml index ad00a94..7286554 100644 --- a/cli/examples/configs/openai/quantiles.toml +++ b/cli/examples/configs/openai/quantiles.toml @@ -1,4 +1,6 @@ [benchmarks.pubmedqa] +# The full expert-labeled PubMedQA benchmark has 1000 samples. Since OpenAI charges for +# usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 # Use the OpenAI API, reading the API key From 088a12f75ae1dec8cffad339ff87233d03539b93 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 11:15:57 -0700 Subject: [PATCH 23/42] openai progress --- cli/examples/configs/openai/quantiles.toml | 28 +++++++++++++--------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/cli/examples/configs/openai/quantiles.toml b/cli/examples/configs/openai/quantiles.toml index 7286554..bb3c3d8 100644 --- a/cli/examples/configs/openai/quantiles.toml +++ b/cli/examples/configs/openai/quantiles.toml @@ -1,21 +1,27 @@ +# This config file sets up PubMedQA to use the OpenAI API, and sets up the SimpleQA-verified +# benchmark to use the internal demo model. +# +# The PubMedQA benchmark reads the OpenAI API key from the `OPENAI_API_KEY` environment variable. +# +# Learn more about the OpenAI API at: +# +# https://platform.openai.com/home + [benchmarks.pubmedqa] # The full expert-labeled PubMedQA benchmark has 1000 samples. Since OpenAI charges for # usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 -# Use the OpenAI API, reading the API key -# from the `OPENAI_API_KEY` environment variable -# # The below model is good for demonstrating PubMedQA benchmarks without very high costs, -# since it's relatively cheap, fast, and not a reasoning model. You will need to ensure the -# project with which your API key is associated with has permissions to use this model. -# Do so by going to the below page, and enabling the model: +# since it's relatively cheap, fast. You may need to ensure the project associated with +# your API key has permissions to use this model. Do so by going to the below page, and +# enabling the model: # # https://platform.openai.com/settings/$PROJECT_ID/limits # -# Details on the model can be found at: +# Details on this specific model can be found at: # -# https://developers.openai.com/api/docs/models/gpt-5.4-nano +# https://developers.openai.com/api/docs/models/gpt-5-nano # # And a list of all models can be found at: # @@ -29,8 +35,8 @@ model = "openai:gpt-5-nano" # for the model you're using. max_workers = 100 +# If you don't want to use a hosted/paid model, you can omit model configuration and +# use the built-in demo model. Note, however, that this built-in model emits random text, and +# should only be used for testing purposes. [benchmarks.simpleqa-verified] samples = 100 - -[benchmarks.financebench] -samples = 50 From e43d3566f179cab573377242f69a262c265d8f25 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 11:19:31 -0700 Subject: [PATCH 24/42] tweaks --- cli/examples/configs/openai/quantiles.toml | 6 +++--- cli/examples/configs/simple/quantiles.toml | 21 ++++++--------------- 2 files changed, 9 insertions(+), 18 deletions(-) diff --git a/cli/examples/configs/openai/quantiles.toml b/cli/examples/configs/openai/quantiles.toml index bb3c3d8..4372168 100644 --- a/cli/examples/configs/openai/quantiles.toml +++ b/cli/examples/configs/openai/quantiles.toml @@ -13,7 +13,7 @@ samples = 50 # The below model is good for demonstrating PubMedQA benchmarks without very high costs, -# since it's relatively cheap, fast. You may need to ensure the project associated with +# since it's relatively cheap and fast. You may need to ensure the project associated with # your API key has permissions to use this model. Do so by going to the below page, and # enabling the model: # @@ -21,12 +21,12 @@ samples = 50 # # Details on this specific model can be found at: # -# https://developers.openai.com/api/docs/models/gpt-5-nano +# https://developers.openai.com/api/docs/models/gpt-5.5-nano # # And a list of all models can be found at: # # https://developers.openai.com/api/docs/models/all -model = "openai:gpt-5-nano" +model = "openai:gpt-5.4-nano" # If you run this or other benchmarks with more samples, you may also need to increase this # `max_workers` parameter to improve throughput. This might cause you to run into rate diff --git a/cli/examples/configs/simple/quantiles.toml b/cli/examples/configs/simple/quantiles.toml index 1199405..06b287e 100644 --- a/cli/examples/configs/simple/quantiles.toml +++ b/cli/examples/configs/simple/quantiles.toml @@ -1,23 +1,14 @@ -# Example Quantiles workspace configuration. +# This config file sets up the PubMedQA and SimpleQA-verified benchmarks to use the built-in +# demo model. The `qt` CLI automatically detects this file in the current working directory. # -# Place this file in your working directory as either: +# This built-in demo model generates random text and is generally only useful only for smoke +# testing purposes. Do not use results from these runs to determine the quality of any model. # -# - quantiles.toml -# - .quantiles.toml -# -# If both files exist in the same directory, the CLI will error to avoid -# ambiguity. -# -# Each section under [benchmarks.] maps to a builtin eval. -# The `samples` key limits the number of dataset rows processed. -# The `model` key selects the LLM backend (defaults to the builtin's dummy -# sampler when omitted). +# Each section under [benchmarks.] maps to a builtin eval. The `samples` key limits the +# number of dataset rows processed. [benchmarks.pubmedqa] samples = 10 [benchmarks.simpleqa-verified] samples = 100 - -[benchmarks.financebench] -samples = 50 From 58283d489d9745843abbdb76a648fd9d37ebcbc4 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 11:22:29 -0700 Subject: [PATCH 25/42] deleting demo model section --- cli/examples/configs/anthropic/quantiles.toml | 6 ------ cli/examples/configs/cloudflare/quantiles.toml | 6 ------ cli/examples/configs/gemini/quantiles.toml | 6 ------ cli/examples/configs/openai/quantiles.toml | 6 ------ 4 files changed, 24 deletions(-) diff --git a/cli/examples/configs/anthropic/quantiles.toml b/cli/examples/configs/anthropic/quantiles.toml index 73616fb..8bf60b1 100644 --- a/cli/examples/configs/anthropic/quantiles.toml +++ b/cli/examples/configs/anthropic/quantiles.toml @@ -27,9 +27,3 @@ model = "anthropic:claude-haiku-4-5-20251001" # # You can also see your usage at https://platform.claude.com/usage max_workers = 5 - -# If you don't want to use a hosted/paid model, you can omit model configuration and -# use the built-in demo model. Note, however, that this built-in model emits random text, and -# should only be used for testing purposes. -[benchmarks.simpleqa-verified] -samples = 100 diff --git a/cli/examples/configs/cloudflare/quantiles.toml b/cli/examples/configs/cloudflare/quantiles.toml index 8d03529..9e0dcab 100644 --- a/cli/examples/configs/cloudflare/quantiles.toml +++ b/cli/examples/configs/cloudflare/quantiles.toml @@ -39,9 +39,3 @@ model = "cloudflare_ai_gateway:@cf/meta/llama-3.3-70b-instruct-fp8-fast" # # Note that if you do this, you will still need to provide the API key in the # CLOUDFLARE_API_KEY environment variable. - -# If you don't want to use a hosted/paid model, you can omit model configuration and -# use the built-in demo model. Note, however, that this built-in model emits random text, and -# should only be used for testing purposes. -[benchmarks.simpleqa-verified] -samples = 100 diff --git a/cli/examples/configs/gemini/quantiles.toml b/cli/examples/configs/gemini/quantiles.toml index 16b175b..b586d8c 100644 --- a/cli/examples/configs/gemini/quantiles.toml +++ b/cli/examples/configs/gemini/quantiles.toml @@ -28,9 +28,3 @@ model = "gemini:gemini-3.1-flash-lite" # # https://aistudio.google.com/rate-limit max_workers = 10 - -# If you don't want to use a hosted/paid model, you can omit model configuration and -# use the built-in demo model. Note, however, that this built-in model emits random text, and -# should only be used for testing purposes. -[benchmarks.simpleqa-verified] -samples = 100 diff --git a/cli/examples/configs/openai/quantiles.toml b/cli/examples/configs/openai/quantiles.toml index 4372168..8f6e164 100644 --- a/cli/examples/configs/openai/quantiles.toml +++ b/cli/examples/configs/openai/quantiles.toml @@ -34,9 +34,3 @@ model = "openai:gpt-5.4-nano" # on the same https://platform.openai.com/settings/$PROJECT_ID/limits page and adjust limits # for the model you're using. max_workers = 100 - -# If you don't want to use a hosted/paid model, you can omit model configuration and -# use the built-in demo model. Note, however, that this built-in model emits random text, and -# should only be used for testing purposes. -[benchmarks.simpleqa-verified] -samples = 100 From 251d8246e5b38499a98141c4d314c30f98e66eca Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 11:23:17 -0700 Subject: [PATCH 26/42] spacing --- cli/examples/configs/gemini/quantiles.toml | 1 - 1 file changed, 1 deletion(-) diff --git a/cli/examples/configs/gemini/quantiles.toml b/cli/examples/configs/gemini/quantiles.toml index b586d8c..002b477 100644 --- a/cli/examples/configs/gemini/quantiles.toml +++ b/cli/examples/configs/gemini/quantiles.toml @@ -6,7 +6,6 @@ # # https://aistudio.google.com/prompts/new_chat - [benchmarks.pubmedqa] # The full expert-labeled PubMedQA benchmark has 1000 samples. Since Google charges for # usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. From f456822c93749d0197bcc719a62d3ecc4525116d Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 11:33:05 -0700 Subject: [PATCH 27/42] adding TODO --- cli/src/commands/compare/step.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cli/src/commands/compare/step.rs b/cli/src/commands/compare/step.rs index e5ef230..511414b 100644 --- a/cli/src/commands/compare/step.rs +++ b/cli/src/commands/compare/step.rs @@ -11,6 +11,8 @@ pub(super) struct StepRefs<'a> { } /// Build user-facing step comparisons and borrowed step refs for detailed diffs. +/// +/// TODO: have this return a `Vec<(StepComparison, StepRefs<'a>)>` pub(super) fn build_step_rows<'a>( left_steps: &'a [StepSummary], right_steps: &'a [StepSummary], From 450ccf58423e52735584c79bf19aa5f7281e49c3 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 11:57:08 -0700 Subject: [PATCH 28/42] more comments --- cli/src/dataset/hf_client.rs | 3 ++- cli/src/db/runs.rs | 1 - cli/src/db/steps.rs | 4 ++-- cli/src/db/workspace.rs | 11 +++++++---- 4 files changed, 11 insertions(+), 8 deletions(-) diff --git a/cli/src/dataset/hf_client.rs b/cli/src/dataset/hf_client.rs index a007e53..017ea41 100644 --- a/cli/src/dataset/hf_client.rs +++ b/cli/src/dataset/hf_client.rs @@ -16,6 +16,7 @@ struct SplitsResponse { splits: Vec, } +/// A single split in an splits response from the HF API #[derive(Debug, Deserialize)] struct SplitItem { #[serde(default)] @@ -68,7 +69,7 @@ struct SplitSize { } impl HuggingFaceClient { - /// Build a new client, injecting `HF_TOKEN` when present. + /// Build a new client, using the `HF_TOKEN` env var when it's present. /// /// # Errors /// diff --git a/cli/src/db/runs.rs b/cli/src/db/runs.rs index 242fb73..640d6de 100644 --- a/cli/src/db/runs.rs +++ b/cli/src/db/runs.rs @@ -298,7 +298,6 @@ pub async fn get_run(db: &DatabaseConnection, run_id: i64) -> Result = match run.input { Some(ref input) => serde_json::from_str(input.clone().as_str()).ok(), None => None, diff --git a/cli/src/db/steps.rs b/cli/src/db/steps.rs index 325bc52..27d8186 100644 --- a/cli/src/db/steps.rs +++ b/cli/src/db/steps.rs @@ -41,8 +41,8 @@ pub async fn begin_step( }, StepStatus::Running => StepDecision::Run { step_id: step.id }, StepStatus::Failed => { - // Allow retry: reset the step to running and emit a fresh - // started event so the retry is observable. + // Allow retry by resetting the step status to 'running', then emit a fresh + // 'started' event so the retry is apparent and auditable. step::Entity::update_many() .set(step::ActiveModel { status: Set(StepStatus::Running), diff --git a/cli/src/db/workspace.rs b/cli/src/db/workspace.rs index 4f043c6..6acdbfc 100644 --- a/cli/src/db/workspace.rs +++ b/cli/src/db/workspace.rs @@ -9,11 +9,14 @@ use crate::db::{DBUrl, SQLitePathURL, schema}; const DB_DIR: &str = ".quantiles"; const DB_FILE: &str = "quantiles.sqlite"; -/// Walk up from `cwd` looking for a `.quantiles` directory. +/// Look for `.quantiles` directory under the current working directory, then +/// if it's not found, walk up to parents looking for a `.quantiles` directory. /// -/// If one is found, returns the directory that contains `.quantiles`. -/// If `create` is true, ensures the discovered workspace is initialized. -/// Otherwise initializes a new workspace in `cwd` and returns `cwd`. +/// When the first such `.quantiles` directory is found, return its containing +/// dir. +/// +/// If `create` is `true`, ensure the discovered `.quantiles` workspace is initialized. +/// Otherwise, initializes a new workspace in `cwd` and returns `cwd`. /// /// # Errors /// From d8bce53e5b2fbb9061cbba855492d12858803764 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:08:17 -0700 Subject: [PATCH 29/42] progress on commentary --- cli/src/db/entities/event.rs | 8 ++++++++ cli/src/llm/cloudflare.rs | 5 ++++- cli/src/llm/random.rs | 2 ++ cli/src/llm/random_label.rs | 6 ++++-- 4 files changed, 18 insertions(+), 3 deletions(-) diff --git a/cli/src/db/entities/event.rs b/cli/src/db/entities/event.rs index a0454ae..b1269c7 100644 --- a/cli/src/db/entities/event.rs +++ b/cli/src/db/entities/event.rs @@ -3,12 +3,20 @@ use sea_orm::entity::prelude::*; #[derive(Clone, Debug, PartialEq, DeriveEntityModel)] #[sea_orm(table_name = "events")] pub struct Model { + /// Unique ID of the model #[sea_orm(primary_key)] pub id: i64, + /// Unique ID of the run pub run_id: i64, + /// Unique ID of the step, inside the run pub step_id: Option, + /// Type of the event + /// + /// TODO: make this a discriminated union if possible pub event_type: String, + /// Optional explanatory message about the event pub message: Option, + /// Optional additional metadata abou the event pub data: Option, pub created_at: time::OffsetDateTime, } diff --git a/cli/src/llm/cloudflare.rs b/cli/src/llm/cloudflare.rs index b378ab3..f99ddbf 100644 --- a/cli/src/llm/cloudflare.rs +++ b/cli/src/llm/cloudflare.rs @@ -63,6 +63,9 @@ struct WorkersAIMessage { /// tries the most common locations. fn extract_response(raw: &serde_json::Value) -> Option { // Standard text-generation shape: {"result":{"response":"..."}} + // + // TODO: decode result into a strongly-typed struct with serde, rather + // than manual deserialization if let Some(result) = raw.get("result") && let Some(response) = result.get("response") && let Some(text) = response.as_str() @@ -83,7 +86,7 @@ fn extract_response(raw: &serde_json::Value) -> Option { return Some(text.to_string()); } - // Fallback: raw result string + // Extract raw result string as a fallback if let Some(result) = raw.get("result") && let Some(text) = result.as_str() { diff --git a/cli/src/llm/random.rs b/cli/src/llm/random.rs index 90bfc40..bf90379 100644 --- a/cli/src/llm/random.rs +++ b/cli/src/llm/random.rs @@ -6,6 +6,8 @@ use rand::{Rng, thread_rng}; use super::LLMSampler; /// A dummy sampler that returns random alphanumeric characters. +/// +/// Do not use this for actual testing pub struct RandomSampler { pub max_length: usize, } diff --git a/cli/src/llm/random_label.rs b/cli/src/llm/random_label.rs index ff14252..ad1125a 100644 --- a/cli/src/llm/random_label.rs +++ b/cli/src/llm/random_label.rs @@ -7,8 +7,10 @@ use super::LLMSampler; /// Fake sampler that returns one valid label from a list of candidates. /// -/// Commonly used with QA-style benchmarks. For example, `PubMedQA` uses -/// labels "yes", "no" and "maybe" +/// Commonly used with QA-style benchmarks. +/// +/// For example, `PubMedQA` uses labels "yes", "no" and "maybe". Other QA-style +/// benchmarks can specify other labels dynamically. pub struct RandomLabelSampler { labels: Vec, } From 89f6fb4be2d83c71a4ec95834ac5817ea1aab8f9 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:08:41 -0700 Subject: [PATCH 30/42] tweak --- cli/src/llm/random_label.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cli/src/llm/random_label.rs b/cli/src/llm/random_label.rs index ad1125a..db2ae61 100644 --- a/cli/src/llm/random_label.rs +++ b/cli/src/llm/random_label.rs @@ -7,7 +7,7 @@ use super::LLMSampler; /// Fake sampler that returns one valid label from a list of candidates. /// -/// Commonly used with QA-style benchmarks. +/// This sampler is commonly used in QA-style benchmarks. /// /// For example, `PubMedQA` uses labels "yes", "no" and "maybe". Other QA-style /// benchmarks can specify other labels dynamically. From adfd660eb074d44d756928419a6de155f04f4f8a Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:10:11 -0700 Subject: [PATCH 31/42] doc --- cli/src/llm/sampler.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/cli/src/llm/sampler.rs b/cli/src/llm/sampler.rs index 88fcdcc..dfb4b24 100644 --- a/cli/src/llm/sampler.rs +++ b/cli/src/llm/sampler.rs @@ -48,8 +48,9 @@ impl Sampler { /// /// # Errors /// - /// Returns an error if the required environment variables are missing (e.g. - /// `CLOUDFLARE_API_KEY` for the Cloudflare gateway variant). + /// Returns an error if the required environment variables are missing to convert `self` + /// to an actual `LLMSampler` (e.g. `self` is a `Sampler::CloudflareAIGateway` variant + /// and the `CLOUDFLARE_API_KEY` env var is missing). pub fn resolve(&self) -> Result> { match self { Sampler::Random => Ok(Arc::new(random::RandomSampler::new(80))), From 677d3a427ddba4b6df12584bfc6fc6ef442899e1 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:13:08 -0700 Subject: [PATCH 32/42] fixup --- cli/src/similarity/vector.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cli/src/similarity/vector.rs b/cli/src/similarity/vector.rs index 432e297..4a7878a 100644 --- a/cli/src/similarity/vector.rs +++ b/cli/src/similarity/vector.rs @@ -15,7 +15,7 @@ impl CosineSimilarity { /// Build a new cosine similarity metric backed by the default fastembed model. /// /// Uses a **single** embedder session with all available CPU cores. This is - /// faster than a pool of sessions because ONNX Runtime already parallelises + /// faster than a pool of sessions because ONNX Runtime already parallelizes /// inference internally; multiple sessions just create thread-pool /// oversubscription. /// From 3988744e4085987fd8a74b02787df4175add3e01 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:18:07 -0700 Subject: [PATCH 33/42] comments --- cli/src/main.rs | 3 +++ cli/src/metrics_store.rs | 13 ++++++++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/cli/src/main.rs b/cli/src/main.rs index 44a4deb..64181ec 100644 --- a/cli/src/main.rs +++ b/cli/src/main.rs @@ -9,6 +9,9 @@ use clap::Parser; fn main() -> Result<()> { let process_start = Instant::now(); + // TODO: allow number of total threads to be configurable, and possibly default + // to something other than the default of 61, which is specified in the implementation + // of new_multi_thread() tokio::runtime::Builder::new_multi_thread() .enable_all() .build()? diff --git a/cli/src/metrics_store.rs b/cli/src/metrics_store.rs index b5a187e..dc4cc22 100644 --- a/cli/src/metrics_store.rs +++ b/cli/src/metrics_store.rs @@ -31,10 +31,17 @@ struct BufferedMetric { /// SQL over the per-run Parquet files. #[derive(Debug, Clone)] pub struct MetricsStore { - // TODO: use a concurrent queue rather than sequential hashmap, - // - // https://docs.rs/crossbeam-queue/latest/crossbeam_queue/struct.SegQueue.html + /// The buffer of not-yet-persisted metrics. + /// + /// TODO: use a concurrent queue rather than sequential hashmap, + /// + /// https://docs.rs/crossbeam-queue/latest/crossbeam_queue/struct.SegQueue.html + /// + /// This is a work in progress at: + /// + /// https://github.com/quantiles-evals/quantiles/pull/7 buffer: Arc>>>, + /// The directory inside which metrics are saved. dir: PathBuf, } From 5ec63d61ba8f53f0d4db666a4ff6152ad2774928 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:18:41 -0700 Subject: [PATCH 34/42] progress --- cli/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/cli/README.md b/cli/README.md index 05b1a22..2223062 100644 --- a/cli/README.md +++ b/cli/README.md @@ -34,8 +34,6 @@ qt show 1 ## Customization -### Customization - You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details. ### Durable step caching + crash resume (TypeScript SDK) From 7ac6d7ffb97e59528562edc9cff25de7b1b12eca Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:24:54 -0700 Subject: [PATCH 35/42] changes --- cli/README.md | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/cli/README.md b/cli/README.md index 2223062..58a9686 100644 --- a/cli/README.md +++ b/cli/README.md @@ -1,6 +1,6 @@ # Quantiles CLI -`qt` is a local-first CLI for running AI evals as experiments, so you can instantly see what improved, what regressed, and why. +This directory holds the source code for the `qt` CLI. It is built with [Rust](https://rust-lang.org/) to help it efficiently use the resources of the local machine, to help ensure safety, and to provide strong lints and type-system invariants for Humans and Agents to work with. ## Install @@ -8,13 +8,6 @@ curl -fsSL https://cli.quantiles.io/install.sh | bash ``` -It's built for: - -- LLM evals -- Prompt iteration -- Agent workflows -- Debugging AI systems - ## Demo A few commands to see `qt` in action: @@ -36,7 +29,7 @@ qt show 1 You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details. -### Durable step caching + crash resume (TypeScript SDK) +## Durable step caching + crash resume (TypeScript SDK) Use the high-level `workflow()`, `step()`, and `emit()` API for automatic caching and observability: @@ -87,7 +80,7 @@ If the run fails partway through, resume it and only the failed samples rerun: qt run my-eval --resume 1 -- bun run sdk/typescript/examples/run_demo.ts ``` -### Step Caching +## Step Caching You can also call `step()` without an input hash when caching by step name alone is fine: @@ -99,7 +92,7 @@ const modelOutput = await step("call-model", async () => { See [quantiles.io/documentation/workflows-and-steps](https://quantiles.io/documentation/workflows-and-steps) for more details on how steps and step caching work. -### Compare two runs +## Comparing runs After iterating on an eval, compare two runs to see exactly what changed: @@ -139,7 +132,7 @@ Runs differ. `qt compare` exits with code 1 if the runs differ, making it useful in CI scripts. -## Why Quantiles? +## Comparisons to other systems | | Temporal / Trigger.dev | Quantiles | |---|---|---| From 5046952741e5334b35693dd0783490c2bf4582cd Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:26:39 -0700 Subject: [PATCH 36/42] arch move --- cli/README.md | 86 +++++++++++++++++++++++---------------------------- 1 file changed, 38 insertions(+), 48 deletions(-) diff --git a/cli/README.md b/cli/README.md index 58a9686..82b0689 100644 --- a/cli/README.md +++ b/cli/README.md @@ -25,6 +25,44 @@ qt list qt show 1 ``` +## Architecture + +The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while `qt` handles durability and observability. + +``` ++--------------------------------------+ +| Your Script | +| (TypeScript / Python / Shell) | ++-------------------+------------------+ + │ + │ HTTP / JSON + | + ▼ ++--------------------------------------+ +| Quantiles Server | ++-------------------+------------------+ + │ + │ SQLite + | + ▼ ++------------------------------------------------+ +| .quantiles/quantiles.sqlite (local DB) | ++-------------------+----------------------------+ + │ + │ + │ + ▼ ++--------------------------------------+ +| CLI | +| (list, show, compare) | ++--------------------------------------+ +``` + +- **Server** owns durability decisions: step caching, run state, metrics +- **Client** (your script) owns code execution: the server never runs your logic + - Note that the CLI itself also has built-in benchmarks, which do not involve your code +- **CLI** reads the same SQLite database the server writes to + ## Customization You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details. @@ -149,54 +187,6 @@ Run your code, then instantly see what changed across runs. No notebooks, no pip It doesn’t replace Temporal for production orchestration. It’s built for the 90% of iteration that happens before you ever think about production infrastructure. - -## What Quantiles does - -- Runs your evals locally with durability (step caching + resume) -- Records every run, step, metric, and event -- Lets you inspect and compare runs from the CLI -- Stores everything in a local SQLite database - -Quantiles treats evals as experiments in which every run is tracked, comparable, and queryable with no extra work from you. - -## Architecture - -The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while `qt` handles durability and observability. - -``` -+--------------------------------------+ -| Your Script | -| (TypeScript / Python / Shell) | -+-------------------+------------------+ - │ - │ HTTP / JSON - | - ▼ -+--------------------------------------+ -| Quantiles Server | -+-------------------+------------------+ - │ - │ SQLite - | - ▼ -+------------------------------------------------+ -| .quantiles/quantiles.sqlite (local DB) | -+-------------------+----------------------------+ - │ - │ - │ - ▼ -+--------------------------------------+ -| CLI | -| (list, show, compare) | -+--------------------------------------+ -``` - -- **Server** owns durability decisions: step caching, run state, metrics -- **Client** (your script) owns code execution: the server never runs your logic - - Note that the CLI itself also has built-in benchmarks, which do not involve your code -- **CLI** reads the same SQLite database the server writes to - ## Command Reference See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. From 97e6748527adeea5c1b0aa980eb452b74f785594 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:27:39 -0700 Subject: [PATCH 37/42] command reference --- cli/README.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/cli/README.md b/cli/README.md index 82b0689..46f8b05 100644 --- a/cli/README.md +++ b/cli/README.md @@ -1,6 +1,6 @@ # Quantiles CLI -This directory holds the source code for the `qt` CLI. It is built with [Rust](https://rust-lang.org/) to help it efficiently use the resources of the local machine, to help ensure safety, and to provide strong lints and type-system invariants for Humans and Agents to work with. +This directory holds the source code for the `qt` CLI. It is built with [Rust](https://rust-lang.org/) to help it efficiently use the resources of the local machine, to help ensure safety, and to provide strong lints and type-system invariants for humans and agents to work with. ## Install @@ -25,6 +25,8 @@ qt list qt show 1 ``` +See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. + ## Architecture The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while `qt` handles durability and observability. @@ -186,7 +188,3 @@ Quantiles is for the iteration loop before production orchestration. Run your code, then instantly see what changed across runs. No notebooks, no pipelines, and no manual comparisons. It doesn’t replace Temporal for production orchestration. It’s built for the 90% of iteration that happens before you ever think about production infrastructure. - -## Command Reference - -See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. From 3181e7f372b44c7b9d09a6df2f334ee6c5cc50af Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:31:53 -0700 Subject: [PATCH 38/42] re-arrange --- README.md | 8 +------ cli/README.md | 63 --------------------------------------------------- 2 files changed, 1 insertion(+), 70 deletions(-) diff --git a/README.md b/README.md index eddeb9e..7f06189 100644 --- a/README.md +++ b/README.md @@ -98,13 +98,7 @@ Built-in benchmarks are ready-to-run evaluation harnesses with predefined datase ## Custom Evaluations - - -A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) or [TypeScript](https://quantiles.io/documentation/reference/typescript-sdk) program that is run by the `qt` CLI and uses the [Quantiles API](https://quantiles.io/documentation/reference/rest-api) to execute an eval. Your code owns the evaluation logic: loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles records the run, durable steps, emitted metrics, events, inputs, outputs, and comparisons. +A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) or [TypeScript](https://quantiles.io/documentation/reference/typescript-sdk) program that is run by the `qt` CLI and uses the [Quantiles API](https://quantiles.io/documentation/reference/rest-api) to execute an eval. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages [durable steps, step caching, and step resume](https://quantiles.io/documentation/workflows-and-steps), metrics, inputs, outputs, and comparisons. Use custom evaluations when you need to measure behavior that is specific to your product, workflow, prompt, dataset, rubric, or release process. diff --git a/cli/README.md b/cli/README.md index 46f8b05..64b5096 100644 --- a/cli/README.md +++ b/cli/README.md @@ -69,69 +69,6 @@ The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while ` You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details. -## Durable step caching + crash resume (TypeScript SDK) - -Use the high-level `workflow()`, `step()`, and `emit()` API for automatic -caching and observability: - -```typescript -import { workflow, step, emit, entrypoint } from "@quantiles/sdk"; - -const runEval = workflow("eval", async (input, ctx) => { - console.log(`Run ${ctx.runId} (${ctx.workflowName})`); - - const data = await step("fetch-data", { url: "https://example.com" }, async () => { - return { status: 200, body: "..." }; - }); - - // Same step key + input hash means cached output, and no re-execution - const cached = await step("fetch-data", { url: "https://example.com" }, async () => { - return { status: 999, body: "this should not execute" }; - }); - - emit("latency_ms", 120, "ms"); - emit("tokens_used", 42); - - return data; -}); - -entrypoint(runEval); -``` - -Register multiple evals and let `qt run ` dispatch to the right one -automatically: - -```typescript -const w1 = workflow("eval", async (input, ctx) => { ... }); -const w2 = workflow("benchmark", async (input, ctx) => { ... }); - -entrypoint(w1, w2); -``` - -Then, when you're ready, run your eval with this command: - -```bash -qt run eval -- bun run sdk/typescript/examples/run_demo.ts -``` - -If the run fails partway through, resume it and only the failed samples rerun: - -```bash -qt run my-eval --resume 1 -- bun run sdk/typescript/examples/run_demo.ts -``` - -## Step Caching - -You can also call `step()` without an input hash when caching by step name alone is fine: - -```typescript -const modelOutput = await step("call-model", async () => { - return callLLM(prompt); -}); -``` - -See [quantiles.io/documentation/workflows-and-steps](https://quantiles.io/documentation/workflows-and-steps) for more details on how steps and step caching work. - ## Comparing runs After iterating on an eval, compare two runs to see exactly what changed: From 5a7b7a861b66b0298c3a267948a26e1350daa657 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:38:59 -0700 Subject: [PATCH 39/42] changes --- README.md | 2 ++ cli/README.md | 17 ----------------- 2 files changed, 2 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 7f06189..eaf6bb9 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,8 @@ With Quantiles, teams can rely on built-in infrastructure: - Inspect and compare runs directly from the same `qt` CLI - Resilient execution by default with caching and restartable failed runs +Quantiles borrows concepts from durable workflow execution systems to ensure evals are resilient to crashes and restarts, but also adds rich observability and metrics on top. You can use it to run your custom eval code or run built-in benchmarks, then instantly see what changed across runs, all without notebooks, pipelines, manual comparisons, or cloud services. + ## Quickstart Install the CLI: diff --git a/cli/README.md b/cli/README.md index 64b5096..a97c89d 100644 --- a/cli/README.md +++ b/cli/README.md @@ -108,20 +108,3 @@ Runs differ. ``` `qt compare` exits with code 1 if the runs differ, making it useful in CI scripts. - -## Comparisons to other systems - -| | Temporal / Trigger.dev | Quantiles | -|---|---|---| -| **Goal** | Run evals reliably | Run evals and **quickly learn what improved, what regressed, and why** | -| **Setup** | Cluster, workers, cloud infra | Single binary + SQLite | -| **Programming Model** | Deterministic constraints | Normal async code | -| **Local Dev** | Usually tied to a service/runtime | Fully offline | -| **Observability** | Logs and traces | Automatically compare runs and surface changes | -| **Iteration** | Run, then inspect | Run, then compare, then improve | - -Quantiles is for the iteration loop before production orchestration. - -Run your code, then instantly see what changed across runs. No notebooks, no pipelines, and no manual comparisons. - -It doesn’t replace Temporal for production orchestration. It’s built for the 90% of iteration that happens before you ever think about production infrastructure. From 836238b0c5ff696011a42f0e805f38c89d94824d Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:44:45 -0700 Subject: [PATCH 40/42] fixup --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index eaf6bb9..583d48a 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ With Quantiles, teams can rely on built-in infrastructure: - Inspect and compare runs directly from the same `qt` CLI - Resilient execution by default with caching and restartable failed runs -Quantiles borrows concepts from durable workflow execution systems to ensure evals are resilient to crashes and restarts, but also adds rich observability and metrics on top. You can use it to run your custom eval code or run built-in benchmarks, then instantly see what changed across runs, all without notebooks, pipelines, manual comparisons, or cloud services. +Quantiles borrows concepts from durable workflow execution systems to make evaluation runs resilient to crashes and restarts, while adding a high-throughput execution engine, rich observability, metrics, and eval reproducibility. Use it to run custom eval code or built-in benchmarks, then inspect what changed across runs, all without notebooks, pipelines, manual comparisons, or cloud services. ## Quickstart From 7572d83f40b61da62c661e65e9a6bdff3e1a11a5 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:47:14 -0700 Subject: [PATCH 41/42] fixup --- cli/README.md | 65 +++++++++++++++++---------------------------------- 1 file changed, 21 insertions(+), 44 deletions(-) diff --git a/cli/README.md b/cli/README.md index a97c89d..67f0949 100644 --- a/cli/README.md +++ b/cli/README.md @@ -16,16 +16,33 @@ A few commands to see `qt` in action: # 1. Initialize a workspace qt init -# 2. Run an eval — Quantiles auto-starts a local server, records the run, -# and tears the server down when the command finishes. -qt run hello-world -- echo "hello from Quantiles" +# 2. Run an eval — Quantiles auto-starts a local server, records the run, and tears the server +# down when the command finishes. +qt run my-eval -- bun run sdk/typescript/examples/run_demo.ts # 3. List and inspect what happened qt list qt show 1 ``` -See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. +>See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. + +### Comparing runs + +After iterating on an eval, you can compare two runs to see exactly what changed: + +```bash +# Run A — baseline +qt run my-eval -- bun run sdk/typescript/examples/run_demo.ts + +# Run B — your latest iteration +qt run my-eval -- bun run sdk/typescript/examples/run_demo.ts + +# See what changed between them +qt compare 1 2 +``` + +`qt compare` exits with code 1 if the runs differ, making it useful in CI scripts. ## Architecture @@ -68,43 +85,3 @@ The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while ` ## Customization You can customize how the CLI executes benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. This file can be used to control benchmark execution behavior as well as customize the models, providers, and other settings used during eval runs. See [`./cli/examples/configs`](./cli/examples/configs) for examples and more details. - -## Comparing runs - -After iterating on an eval, compare two runs to see exactly what changed: - -```bash -# Run A — baseline -qt run my-eval -- bun run sdk/typescript/examples/run_demo.ts - -# Run B — your latest iteration -qt run my-eval -- bun run sdk/typescript/examples/run_demo.ts - -# See what changed between them -qt compare 1 2 -``` - -Example output when outputs differ: - -```text -Run 1: my-eval (completed) -Run 2: my-eval (completed) - -Samples - STEP PRESENT INPUT STATUS OUTPUT - fetch-data both same same differs - -Output differences - step: fetch-data - run 1: {"status":200,"body":"hello"} - run 2: {"status":200,"body":"world"} - -Metrics - NAME Run 1 Run 2 - latency_ms 50 150 * - tokens_used 42 89 * - -Runs differ. -``` - -`qt compare` exits with code 1 if the runs differ, making it useful in CI scripts. From c97f2abe8cba75b9a03fdf3b3b99fe3282571db5 Mon Sep 17 00:00:00 2001 From: Aaron Schlesinger <70865+arschles@users.noreply.github.com> Date: Thu, 18 Jun 2026 12:49:10 -0700 Subject: [PATCH 42/42] progress --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 583d48a..89028f8 100644 --- a/README.md +++ b/README.md @@ -106,9 +106,9 @@ Use custom evaluations when you need to measure behavior that is specific to you Read more about how to build and run custom evaluations at [quantiles.io/documentation/custom-evaluations](https://quantiles.io/documentation/custom-evaluations). -## SDKs +### SDKs -Use the official Quantiles SDKs to build with higher-level workflow primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python and TypeScript. The SDKs integrate tightly with the `qt` CLI’s local API for running, recording and analyzing benchmarks. +Use the official Quantiles SDKs to build your custom evaluations with higher-level workflow primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python and TypeScript. The SDKs integrate tightly with the `qt` CLI’s local API for running, recording and analyzing benchmarks. The Python SDK is located in this repository at [`python/`](./python). Read more about it at [quantiles.io/documentation/reference/python-sdk](https://quantiles.io/documentation/reference/python-sdk). The TypeScript SDK is located in this repository at [`typescript/`](./typescript). Read more about it at [quantiles.io/documentation/reference/typescript-sdk](https://quantiles.io/documentation/reference/typescript-sdk).