Skip to content

feat(cli): filter tests by vars #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RGBuilds wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(cli): filter tests by vars #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions site/docs/configuration/test-cases.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,18 @@ promptfoo eval --filter-metadata tags=ai
promptfoo eval --filter-metadata category=math --filter-metadata difficulty=easy
```

Filter regular test variables with exact matching:

```bash
promptfoo eval --filter-vars language=en

# Multiple filters use AND logic
promptfoo eval --filter-vars language=en --filter-vars category=safety

# Quote structured values as JSON
promptfoo eval --filter-vars 'config={"mode":"strict"}'
```

### JSON in CSV

Include structured data:
Expand Down
93 changes: 47 additions & 46 deletions site/docs/usage/command-line.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,51 +99,52 @@ All specified files must exist or an error is thrown.

By default the `eval` command will read the `promptfooconfig.yaml` configuration file in your current directory. But, if you're looking to override certain parameters you can supply optional arguments:

| Option | Description |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------- |
| `-a, --assertions <path>` | Path to assertions file |
| `-c, --config <paths...>` | Path to configuration file(s). Automatically loads promptfooconfig.yaml |
| `--delay <number>` | Delay between each test (in milliseconds) |
| `--description <description>` | Description of the eval run |
| `--filter-failing <path or id>` | Filter tests that failed in a previous eval (by file path or eval ID) |
| `--filter-failing-only <path or id>` | Filter tests that had assertion failures in a previous eval, excluding errors |
| `--filter-errors-only <path or id>` | Filter tests that resulted in errors in a previous eval |
| `-n, --filter-first-n <number>` | Only run the first N tests |
| `--filter-range <start:end>` | Only run tests whose zero-based index is in the range. The end index is exclusive. |
| `--filter-sample <number>` | Only run a random sample of N tests |
| `--filter-sample-seed <number>` | Numeric seed used to make `--filter-sample` select the same tests on repeated runs |
| `--filter-metadata <key=value>` | Only run tests whose metadata matches the key=value pair. Can be specified multiple times for AND logic. |
| `--filter-pattern <pattern>` | Only run tests whose description matches the regex pattern |
| `--filter-prompts <pattern>` | Only run tests with prompts whose id or label matches the regex pattern |
| `--filter-providers <providers>` | Only run tests with these providers (regex match on provider `id` or `label`) |
| `--filter-targets <targets>` | Only run tests with these targets (alias for --filter-providers) |
| `--grader <provider>` | Model that will grade outputs |
| `-j, --max-concurrency <number>` | Maximum number of concurrent API calls |
| `--model-outputs <path>` | Path to JSON containing list of LLM output strings |
| `--no-cache` | Do not read or write results to disk cache |
| `--no-progress-bar` | Do not show progress bar |
| `--no-table` | Do not output table in CLI |
| `--no-write` | Do not write results to promptfoo directory |
| `--resume [evalId]` | Resume a paused/incomplete eval. If `evalId` is omitted, resumes latest |
| `--retry-errors` | Retry all ERROR results from the latest eval |
| `-o, --output <paths...>` | Path(s) to output file (csv, txt, json, jsonl, yaml, yml, html, xml, junit.xml) |
| `-p, --prompts <paths...>` | Paths to prompt files (.txt) |
| `--prompt-prefix <path>` | Prefix prepended to every prompt |
| `--prompt-suffix <path>` | Suffix appended to every prompt |
| `-r, --providers <name or path...>` | Provider names or paths to custom API caller modules |
| `--remote` | Force remote inference wherever possible (used for red teams) |
| `--repeat <number>` | Number of times to run each test |
| `--share` | Create a shareable URL |
| `--no-share` | Do not create a shareable URL, this overrides the config file |
| `--suggest-prompts <number>` | Generate N new prompts and append them to the prompt list |
| `--tag <key=value>` | Set an eval tag. Can be specified multiple times; CLI tags override config tags. |
| `--table` | Output table in CLI |
| `--table-cell-max-length <number>` | Truncate console table cells to this length |
| `-t, --tests <path>` | Path to CSV with test cases |
| `--var <key=value>` | Set a variable in key=value format |
| `-v, --vars <path>` | Path to CSV with test cases (alias for --tests) |
| `-w, --watch` | Watch for changes in config and re-run |
| `-x, --extension <paths...>` | Extension hooks to run, such as `file://handler.js:afterAll` |
| Option | Description |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| `-a, --assertions <path>` | Path to assertions file |
| `-c, --config <paths...>` | Path to configuration file(s). Automatically loads promptfooconfig.yaml |
| `--delay <number>` | Delay between each test (in milliseconds) |
| `--description <description>` | Description of the eval run |
| `--filter-failing <path or id>` | Filter tests that failed in a previous eval (by file path or eval ID) |
| `--filter-failing-only <path or id>` | Filter tests that had assertion failures in a previous eval, excluding errors |
| `--filter-errors-only <path or id>` | Filter tests that resulted in errors in a previous eval |
| `-n, --filter-first-n <number>` | Only run the first N tests |
| `--filter-range <start:end>` | Only run tests whose zero-based index is in the range. The end index is exclusive. |
| `--filter-sample <number>` | Only run a random sample of N tests |
| `--filter-sample-seed <number>` | Numeric seed used to make `--filter-sample` select the same tests on repeated runs |
| `--filter-metadata <key=value>` | Only run tests whose metadata matches the key=value pair. Can be specified multiple times for AND logic. |
| `--filter-pattern <pattern>` | Only run tests whose description matches the regex pattern |
| `--filter-prompts <pattern>` | Only run tests with prompts whose id or label matches the regex pattern |
| `--filter-providers <providers>` | Only run tests with these providers (regex match on provider `id` or `label`) |
| `--filter-vars <key=value>` | Only run tests whose vars exactly match the key=value pair. Can be specified multiple times for AND logic. |
| `--filter-targets <targets>` | Only run tests with these targets (alias for --filter-providers) |
| `--grader <provider>` | Model that will grade outputs |
| `-j, --max-concurrency <number>` | Maximum number of concurrent API calls |
| `--model-outputs <path>` | Path to JSON containing list of LLM output strings |
| `--no-cache` | Do not read or write results to disk cache |
| `--no-progress-bar` | Do not show progress bar |
| `--no-table` | Do not output table in CLI |
| `--no-write` | Do not write results to promptfoo directory |
| `--resume [evalId]` | Resume a paused/incomplete eval. If `evalId` is omitted, resumes latest |
| `--retry-errors` | Retry all ERROR results from the latest eval |
| `-o, --output <paths...>` | Path(s) to output file (csv, txt, json, jsonl, yaml, yml, html, xml, junit.xml) |
| `-p, --prompts <paths...>` | Paths to prompt files (.txt) |
| `--prompt-prefix <path>` | Prefix prepended to every prompt |
| `--prompt-suffix <path>` | Suffix appended to every prompt |
| `-r, --providers <name or path...>` | Provider names or paths to custom API caller modules |
| `--remote` | Force remote inference wherever possible (used for red teams) |
| `--repeat <number>` | Number of times to run each test |
| `--share` | Create a shareable URL |
| `--no-share` | Do not create a shareable URL, this overrides the config file |
| `--suggest-prompts <number>` | Generate N new prompts and append them to the prompt list |
| `--tag <key=value>` | Set an eval tag. Can be specified multiple times; CLI tags override config tags. |
| `--table` | Output table in CLI |
| `--table-cell-max-length <number>` | Truncate console table cells to this length |
| `-t, --tests <path>` | Path to CSV with test cases |
| `--var <key=value>` | Set a variable in key=value format |
| `-v, --vars <path>` | Path to CSV with test cases (alias for --tests) |
| `-w, --watch` | Watch for changes in config and re-run |
| `-x, --extension <paths...>` | Extension hooks to run, such as `file://handler.js:afterAll` |

Use `--tag` for run-specific eval tags that should not change `promptfooconfig.yaml`:

Expand All @@ -162,7 +163,7 @@ promptfoo eval --filter-range 200: # tests 200 through the end
promptfoo eval --filter-range :50 # first 50 tests
```

Range is applied before `--repeat` expansion, so `--filter-range 0:5 --repeat 3` runs 15 evaluations across the same 5 tests. When combined with other filters (`--filter-pattern`, `--filter-metadata`, etc.), range slices the post-filter list.
Range is applied before `--repeat` expansion, so `--filter-range 0:5 --repeat 3` runs 15 evaluations across the same 5 tests. When combined with other filters (`--filter-pattern`, `--filter-metadata`, `--filter-vars`, etc.), range slices the post-filter list.

When resuming an eval, promptfoo reuses the range saved with the original run so test indices stay stable. A `--filter-range` flag passed on resume is ignored (with a warning) and other transient filters from the original run are not restored, so resume is most predictable when range was the only selection filter.

Expand Down
7 changes: 7 additions & 0 deletions src/commands/eval.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,13 @@ export function evalCommand(
return previous ? [...previous, value] : [value];
},
)
.option(
'--filter-vars <key=value>',
'Only run tests whose vars exactly match the key=value pair. Can be specified multiple times for AND logic (e.g. --filter-vars language=en --filter-vars category=safety)',
(value: string, previous: string[] | undefined) => {
return previous ? [...previous, value] : [value];
},
)

// Output configuration
.option(
Expand Down
2 changes: 2 additions & 0 deletions src/node/doEval.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -609,6 +609,7 @@ export async function doEval(
cmdObj.filterFirstN !== undefined ||
cmdObj.filterMetadata !== undefined ||
cmdObj.filterPattern !== undefined ||
cmdObj.filterVars !== undefined ||
filterSample !== undefined;
const shouldApplyFiltersToImplicitDefaultTest =
hasActiveTestFilter && canSynthesizeImplicitDefaultTest && !testSuite.tests?.length;
Expand All @@ -630,6 +631,7 @@ export async function doEval(
range: hasScenarios ? undefined : filterRange,
sample: filterSample,
sampleSeed: filterSampleSeed,
vars: cmdObj.filterVars,
};
testSuite.tests = await filterTests(testSuite, filterOptions);
const shouldSuppressImplicitDefaultTest =
Expand Down
1 change: 1 addition & 0 deletions src/types/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ export const CommandLineOptionsSchema = z.object({
filterSample: z.coerce.number().int().positive().optional(),
filterSampleSeed: z.coerce.number().int().safe().optional(),
filterTargets: z.string().optional(),
filterVars: z.union([z.string(), z.array(z.string())]).optional(),
var: z.record(z.string(), z.string()).optional(),
tags: z.record(z.string(), z.string()).optional(),

Expand Down
Loading