Adding support for multiple-choice custom_nocode benchmarks

CONFIG.md

@@ -7,8 +7,8 @@ Quantiles uses a single TOML configuration file in the current working directory
 You need a config file when you want to do one or more of the following:
 
 - Override built-in benchmark defaults (e.g. model, sample limit)
-- Define custom evaluations (`type = "custom_code"`)
-- Define no-code QA benchmarks (`type = "custom_nocode"`, `style = "qa"`)
+- Build custom evaluations without writing or maintaining code (`type = "custom_nocode"` in the `quantiles.toml` configuration file)
+- Build custom evaluations with code (`type = "custom_code"` in the `quantiles.toml` configuration file)
 - Resume custom evaluations later with `qt resume <run_id>`
 
 You don't, however, need any configuration when you want to run built-in benchmarks. `qt run pubmedqa`, for example, works out of the box.
@@ -100,17 +100,17 @@ Note that models require specific configuration based on the provider. For detai
 
 ### `custom_nocode`
 
-No-code benchmarks are configured in TOML and run natively inside the CLI, without a custom Python or TypeScript evaluation program. The initial supported style is `qa`, which renders a prompt from a dataset row, calls a model, and scores the response with exact-match accuracy against the configured golden answer column.
+No-code evals are configured in TOML and run natively inside the CLI, without any custom Python code. These evals are defined with `type = "custom_nocode"` and a `style` parameter. The `style = "exact_match"` configuration creates an eval that scores an open answer or label against a golden answer column. The `style = "multiple_choice"` configuration normalizes choices, extracts the selected label from the response, and scores it against a configured label, index, or correct-choice column.
+
+When you use a `"random"` model (`model = "random"`) with an `exact_match` eval, your eval will use the built-in "model" that generates random text, so you'll likely to get very low accuracy numbers. When you configure a `multiple_choice` eval with the `"random"` model, the built-in "model" will uniformly sample from one of the the configured `style.choice_labels`, so you can expect higher accuracies than with `exact_match`. In both cases, this model is intended for testing your benchmark. We do not recommend relying on it for any real ML evaluation work.
 
 ```toml
 [benchmarks.nocode_custom]
 type = "custom_nocode"
-style = "qa"
-dataset = "quantiles/simpleqa-verified"
+style = { type = "exact_match", golden_column = "answer" }
+dataset = { name = "quantiles/simpleqa-verified" }
 model = "random"
 prompt_template_file = "prompts/qa.txt"
-prompt_column = "problem"
-golden_column = "answer"
 limit = 10
 ```
 
@@ -125,15 +125,62 @@ For a complete minimal example, see [`custom-nocode-examples/quantiles.toml`](./
 | Field | Type | Required | Description |
 |--|--|--|--|
 | `type` | string | yes | Must be `"custom_nocode"`. |
-| `style` | string | yes | Must be `"qa"`. |
-| `dataset` | string | yes | Dataset identifier, for example `"quantiles/simpleqa-verified"`. |
+| `style` | table | yes | Scoring style and its style-specific configuration. |
+| `style.type` | string | yes | `"exact_match"` for open-answer or label exact match, or `"multiple_choice"` for choice-based benchmarks. |
+| `dataset` | table | yes | Hugging Face dataset coordinates. |
+| `dataset.name` | string | yes | Dataset identifier, for example `"quantiles/simpleqa-verified"`. |
+| `dataset.config_name` | string | no | Hugging Face dataset configuration or subset. |
+| `dataset.split` | string | no | Dataset split. When omitted, Quantiles selects a standard evaluation split. |
+| `dataset.revision` | string | no | Dataset revision. |
 | `model` | string or table | no | Model sampler. Defaults to the demo random sampler. See [model naming](#model-naming). |
-| `prompt_template_file` | string | yes | Path to a Jinja prompt template file. The template receives `prompt` from the configured prompt column. |
-| `prompt_column` | string | yes | Dataset column containing the prompt text. |
-| `golden_column` | string | yes | Dataset column containing the golden answer. |
+| `prompt_template_file` | string | yes | Path to a Jinja prompt template file. The template receives the complete dataset `row` and, for multiple-choice benchmarks, normalized `choices`. |
+| `style.golden_column` | string | conditional | Dataset column containing the golden answer. Required for `exact_match`. |
+| `style.choices` | table | conditional | Choice source. Required for `multiple_choice`. Configure exactly one of `style.choices.column` or `style.choices.columns`. |
+| `style.choices.column` | string | conditional | Dataset column containing choices as an array or label-keyed object. |
+| `style.choices.columns` | array of strings | conditional | Dataset columns containing choices in their original order. |
+| `style.answer` | table | conditional | Correct-answer source. Required for `multiple_choice`. Configure exactly one answer-source form. |
+| `style.answer.label_column` | string | conditional | Dataset column containing the golden choice label. |
+| `style.answer.index_column` | string | conditional | Dataset column containing the golden choice index. |
+| `style.answer.index_base` | integer | no | Index base for `style.answer.index_column`. Defaults to `0`. |
+| `style.answer.correct_choice_column` | string | conditional | Member of `style.choices.columns` known to contain the correct answer. |
+| `style.choice_labels` | array of strings | conditional | Labels assigned to choices in order. Required for multiple choice; array-backed rows may use a prefix of the configured labels. |
+| `style.shuffle` | table | no | Enables deterministic choice shuffling for `multiple_choice`. |
+| `style.shuffle.seed_column` | string | conditional | Stable row identifier used to seed deterministic shuffling. Required when `style.shuffle` is present. |
 | `limit` | integer | no | Number of dataset rows to evaluate. |
 | `max_workers` | integer | no | Maximum concurrent workers. |
 
+Each sample emits `is_correct`, `response_parsed`, and `latency_ms`. For exact-match benchmarks, every response is considered parsed; for multiple-choice benchmarks, `response_parsed` is `0` when the response cannot be parsed as a configured choice label.
+
+Each run emits these aggregate metrics:
+
+- `accuracy`, `correct_count`, `incorrect_count`, and `total_count`
+- `parsed_response_count`, `unparsed_response_count`, and `parse_rate`
+- `mean_latency_ms`, `median_latency_ms`, `p95_latency_ms`, `p99_latency_ms`, `min_latency_ms`, and `max_latency_ms`
+
+Multiple-choice configuration keeps its choice and answer sources inside `style`:
+
+```toml
+[benchmarks.medqa]
+type = "custom_nocode"
+style = { type = "multiple_choice", choices = { column = "options" }, choice_labels = ["A", "B", "C", "D"], answer = { label_column = "answer_idx" } }
+dataset = { name = "quantiles/MedQA-USMLE-4-options", config_name = "default", split = "test" }
+model = "random"
+prompt_template_file = "prompts/medqa.txt"
+limit = 10
+```
+
+Templates access dataset fields directly. A multiple-choice template can iterate the normalized choices:
+
+```jinja
+{{ row.question }}
+
+{% for choice in choices %}
+{{ choice.label }}. {{ choice.text }}
+{% endfor %}
+```
+
+See [`custom-nocode-examples/quantiles.toml`](./custom-nocode-examples/quantiles.toml) for runnable MedQA, MedMCQA, MMLU-Pro, and GPQA configurations.
+
 ### `custom_code`
 
 Custom evaluations are external programs built with the Quantiles Python SDK. Their config sections contain the following fields:

README.md

@@ -123,7 +123,7 @@ Both the CLI and Python SDK support offline benchmark workflows, including the f
 
 Built-in benchmarks are ready-to-run evaluations with predefined datasets, scoring methodologies, and metrics. Use them when you want a standardized evaluation that provides a common reference point, a repeatable baseline, or a well-defined implementation of an industry benchmark.
 
-For dataset-backed QA checks that do not need custom Python or TypeScript code, use a `custom_nocode` benchmark in `quantiles.toml`. A `custom_nocode` QA benchmark points to a dataset, a Jinja prompt template, the dataset column containing the prompt, and the dataset column containing the golden answer. See [`custom-nocode-examples/quantiles.toml`](./custom-nocode-examples/quantiles.toml) for a minimal example.
+For dataset-backed QA checks that do not need custom Python or TypeScript code, use a `custom_nocode` benchmark in `quantiles.toml`. A `custom_nocode` QA benchmark points to a dataset, renders a Jinja prompt from the complete dataset row, and supports exact-answer and multiple-choice scoring. See [`custom-nocode-examples/quantiles.toml`](./custom-nocode-examples/quantiles.toml) for runnable SimpleQA Verified, MedQA, MedMCQA, MMLU-Pro, and GPQA configurations.
 
 | Code | When to use |
 | --- | --- |

cli/README.md

@@ -73,17 +73,19 @@ max_workers = 100
 
 ### No-code QA benchmarks
 
-For dataset-backed QA checks that do not need custom evaluation code, set `type = "custom_nocode"` and `style = "qa"`. The benchmark runs inside the CLI, renders each prompt with the configured Jinja template, calls the configured model, and scores each row with exact-match accuracy against the golden answer column.
+For dataset-backed QA checks that do not need custom evaluation code, set `type = "custom_nocode"`. Set `style.type` to `"exact_match"` for benchmarks that test an exact match to a golden answer, or `"multiple_choice"` for choice-based benchmarks. Style-specific dataset columns belong inside the `style` table. The benchmark runs inside the CLI, renders each prompt with the configured Jinja template, calls the configured model, and scores each row against the configured answer source.
+
+With `model = "random"`, exact-match benchmarks generate demo random text and multiple-choice benchmarks uniformly select from `style.choice_labels`.
+
+Samples emit correctness, response-parsing, and latency metrics. Runs aggregate accuracy and counts, parse success, and mean, median, p95, p99, minimum, and maximum latency.
 
 ```toml
 [benchmarks.nocode_custom]
 type = "custom_nocode"
-style = "qa"
-dataset = "quantiles/simpleqa-verified"
+style = { type = "exact_match", golden_column = "answer" }
+dataset = { name = "quantiles/simpleqa-verified" }
 model = "random"
 prompt_template_file = "prompts/qa.txt"
-prompt_column = "problem"
-golden_column = "answer"
 limit = 10
 ```
 

cli/src/builtins/common.rs

@@ -326,44 +326,51 @@ mod tests {
     }
 
     #[rstest]
-    fn test_resolve_sampler_uses_default_when_none() {
-        let result = resolve_sampler(None, || Arc::new(crate::llm::random::RandomSampler::new(80)))
-            .unwrap();
+    #[tokio::test]
+    async fn test_resolve_sampler_uses_default_when_none() {
+        let result = resolve_sampler(None, || {
+            Arc::new(crate::llm::random::RandomSampler::new(80))
+        })
+        .unwrap();
         // Should get the default sampler, not panic or error.
-        assert!(!result.sample("test").block_on().is_empty());
+        assert!(!result.sample("test").await.unwrap().is_empty());
     }
 
     #[rstest]
-    fn test_resolve_sampler_resolves_configured_sampler() {
+    #[tokio::test]
+    async fn test_resolve_sampler_resolves_configured_sampler() {
         let sampler = crate::llm::Sampler::Random {};
         let result = resolve_sampler(Some(&sampler), || {
             panic!("default should not be called when model is Some")
         })
         .unwrap();
         // Should get a resolved sampler, not the default.
-        assert!(!result.sample("test").block_on().is_empty());
+        assert!(!result.sample("test").await.unwrap().is_empty());
     }
 
     #[rstest]
     fn test_emit_accuracy_metrics_empty() {
         let tmpdir = tempfile::tempdir().unwrap();
-        let metrics_store = crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf())
-            .unwrap();
+        let metrics_store =
+            crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf()).unwrap();
 
         let rt = tokio::runtime::Runtime::new().unwrap();
         rt.block_on(async {
             emit_accuracy_metrics(&metrics_store, 1, [false; 0]).await;
             metrics_store.flush(1).await.unwrap();
             let agg = metrics_store.list_aggregate_for_run(1).await.unwrap();
-            assert!(agg.is_empty(), "no metrics should be emitted for empty results");
+            assert!(
+                agg.is_empty(),
+                "no metrics should be emitted for empty results"
+            );
         });
     }
 
     #[rstest]
     fn test_emit_accuracy_metrics_all_correct() {
         let tmpdir = tempfile::tempdir().unwrap();
-        let metrics_store = crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf())
-            .unwrap();
+        let metrics_store =
+            crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf()).unwrap();
 
         let rt = tokio::runtime::Runtime::new().unwrap();
         rt.block_on(async {
@@ -374,19 +381,22 @@ mod tests {
             let accuracy = agg.iter().find(|m| m.metric_name == "accuracy").unwrap();
             assert!((accuracy.metric_value - 1.0).abs() < 1e-10);
 
-            let correct = agg.iter().find(|m| m.metric_name == "correct_count").unwrap();
-            assert_eq!(correct.metric_value as i64, 3);
+            let correct = agg
+                .iter()
+                .find(|m| m.metric_name == "correct_count")
+                .unwrap();
+            assert!((correct.metric_value - 3.0).abs() < f64::EPSILON);
 
             let total = agg.iter().find(|m| m.metric_name == "total_count").unwrap();
-            assert_eq!(total.metric_value as i64, 3);
+            assert!((total.metric_value - 3.0).abs() < f64::EPSILON);
         });
     }
 
     #[rstest]
     fn test_emit_accuracy_metrics_mixed() {
         let tmpdir = tempfile::tempdir().unwrap();
-        let metrics_store = crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf())
-            .unwrap();
+        let metrics_store =
+            crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf()).unwrap();
 
         let rt = tokio::runtime::Runtime::new().unwrap();
         rt.block_on(async {
@@ -397,11 +407,14 @@ mod tests {
             let accuracy = agg.iter().find(|m| m.metric_name == "accuracy").unwrap();
             assert!((accuracy.metric_value - 0.5).abs() < 1e-10);
 
-            let correct = agg.iter().find(|m| m.metric_name == "correct_count").unwrap();
-            assert_eq!(correct.metric_value as i64, 2);
+            let correct = agg
+                .iter()
+                .find(|m| m.metric_name == "correct_count")
+                .unwrap();
+            assert!((correct.metric_value - 2.0).abs() < f64::EPSILON);
 
             let total = agg.iter().find(|m| m.metric_name == "total_count").unwrap();
-            assert_eq!(total.metric_value as i64, 4);
+            assert!((total.metric_value - 4.0).abs() < f64::EPSILON);
         });
     }
 }