diff --git a/site/docs/configuration/test-cases.md b/site/docs/configuration/test-cases.md index 4160d5a44c72..fbe38ad0921f 100644 --- a/site/docs/configuration/test-cases.md +++ b/site/docs/configuration/test-cases.md @@ -394,6 +394,18 @@ promptfoo eval --filter-metadata tags=ai promptfoo eval --filter-metadata category=math --filter-metadata difficulty=easy ``` +Filter regular test variables with exact matching: + +```bash +promptfoo eval --filter-vars language=en + +# Multiple filters use AND logic +promptfoo eval --filter-vars language=en --filter-vars category=safety + +# Quote structured values as JSON +promptfoo eval --filter-vars 'config={"mode":"strict"}' +``` + ### JSON in CSV Include structured data: diff --git a/site/docs/usage/command-line.md b/site/docs/usage/command-line.md index a4a6f709c0b1..3f0609e675bf 100644 --- a/site/docs/usage/command-line.md +++ b/site/docs/usage/command-line.md @@ -99,51 +99,52 @@ All specified files must exist or an error is thrown. By default the `eval` command will read the `promptfooconfig.yaml` configuration file in your current directory. But, if you're looking to override certain parameters you can supply optional arguments: -| Option | Description | -| ------------------------------------ | -------------------------------------------------------------------------------------------------------- | -| `-a, --assertions ` | Path to assertions file | -| `-c, --config ` | Path to configuration file(s). Automatically loads promptfooconfig.yaml | -| `--delay ` | Delay between each test (in milliseconds) | -| `--description ` | Description of the eval run | -| `--filter-failing ` | Filter tests that failed in a previous eval (by file path or eval ID) | -| `--filter-failing-only ` | Filter tests that had assertion failures in a previous eval, excluding errors | -| `--filter-errors-only ` | Filter tests that resulted in errors in a previous eval | -| `-n, --filter-first-n ` | Only run the first N tests | -| `--filter-range ` | Only run tests whose zero-based index is in the range. The end index is exclusive. | -| `--filter-sample ` | Only run a random sample of N tests | -| `--filter-sample-seed ` | Numeric seed used to make `--filter-sample` select the same tests on repeated runs | -| `--filter-metadata ` | Only run tests whose metadata matches the key=value pair. Can be specified multiple times for AND logic. | -| `--filter-pattern ` | Only run tests whose description matches the regex pattern | -| `--filter-prompts ` | Only run tests with prompts whose id or label matches the regex pattern | -| `--filter-providers ` | Only run tests with these providers (regex match on provider `id` or `label`) | -| `--filter-targets ` | Only run tests with these targets (alias for --filter-providers) | -| `--grader ` | Model that will grade outputs | -| `-j, --max-concurrency ` | Maximum number of concurrent API calls | -| `--model-outputs ` | Path to JSON containing list of LLM output strings | -| `--no-cache` | Do not read or write results to disk cache | -| `--no-progress-bar` | Do not show progress bar | -| `--no-table` | Do not output table in CLI | -| `--no-write` | Do not write results to promptfoo directory | -| `--resume [evalId]` | Resume a paused/incomplete eval. If `evalId` is omitted, resumes latest | -| `--retry-errors` | Retry all ERROR results from the latest eval | -| `-o, --output ` | Path(s) to output file (csv, txt, json, jsonl, yaml, yml, html, xml, junit.xml) | -| `-p, --prompts ` | Paths to prompt files (.txt) | -| `--prompt-prefix ` | Prefix prepended to every prompt | -| `--prompt-suffix ` | Suffix appended to every prompt | -| `-r, --providers ` | Provider names or paths to custom API caller modules | -| `--remote` | Force remote inference wherever possible (used for red teams) | -| `--repeat ` | Number of times to run each test | -| `--share` | Create a shareable URL | -| `--no-share` | Do not create a shareable URL, this overrides the config file | -| `--suggest-prompts ` | Generate N new prompts and append them to the prompt list | -| `--tag ` | Set an eval tag. Can be specified multiple times; CLI tags override config tags. | -| `--table` | Output table in CLI | -| `--table-cell-max-length ` | Truncate console table cells to this length | -| `-t, --tests ` | Path to CSV with test cases | -| `--var ` | Set a variable in key=value format | -| `-v, --vars ` | Path to CSV with test cases (alias for --tests) | -| `-w, --watch` | Watch for changes in config and re-run | -| `-x, --extension ` | Extension hooks to run, such as `file://handler.js:afterAll` | +| Option | Description | +| ------------------------------------ | ---------------------------------------------------------------------------------------------------------- | +| `-a, --assertions ` | Path to assertions file | +| `-c, --config ` | Path to configuration file(s). Automatically loads promptfooconfig.yaml | +| `--delay ` | Delay between each test (in milliseconds) | +| `--description ` | Description of the eval run | +| `--filter-failing ` | Filter tests that failed in a previous eval (by file path or eval ID) | +| `--filter-failing-only ` | Filter tests that had assertion failures in a previous eval, excluding errors | +| `--filter-errors-only ` | Filter tests that resulted in errors in a previous eval | +| `-n, --filter-first-n ` | Only run the first N tests | +| `--filter-range ` | Only run tests whose zero-based index is in the range. The end index is exclusive. | +| `--filter-sample ` | Only run a random sample of N tests | +| `--filter-sample-seed ` | Numeric seed used to make `--filter-sample` select the same tests on repeated runs | +| `--filter-metadata ` | Only run tests whose metadata matches the key=value pair. Can be specified multiple times for AND logic. | +| `--filter-pattern ` | Only run tests whose description matches the regex pattern | +| `--filter-prompts ` | Only run tests with prompts whose id or label matches the regex pattern | +| `--filter-providers ` | Only run tests with these providers (regex match on provider `id` or `label`) | +| `--filter-vars ` | Only run tests whose vars exactly match the key=value pair. Can be specified multiple times for AND logic. | +| `--filter-targets ` | Only run tests with these targets (alias for --filter-providers) | +| `--grader ` | Model that will grade outputs | +| `-j, --max-concurrency ` | Maximum number of concurrent API calls | +| `--model-outputs ` | Path to JSON containing list of LLM output strings | +| `--no-cache` | Do not read or write results to disk cache | +| `--no-progress-bar` | Do not show progress bar | +| `--no-table` | Do not output table in CLI | +| `--no-write` | Do not write results to promptfoo directory | +| `--resume [evalId]` | Resume a paused/incomplete eval. If `evalId` is omitted, resumes latest | +| `--retry-errors` | Retry all ERROR results from the latest eval | +| `-o, --output ` | Path(s) to output file (csv, txt, json, jsonl, yaml, yml, html, xml, junit.xml) | +| `-p, --prompts ` | Paths to prompt files (.txt) | +| `--prompt-prefix ` | Prefix prepended to every prompt | +| `--prompt-suffix ` | Suffix appended to every prompt | +| `-r, --providers ` | Provider names or paths to custom API caller modules | +| `--remote` | Force remote inference wherever possible (used for red teams) | +| `--repeat ` | Number of times to run each test | +| `--share` | Create a shareable URL | +| `--no-share` | Do not create a shareable URL, this overrides the config file | +| `--suggest-prompts ` | Generate N new prompts and append them to the prompt list | +| `--tag ` | Set an eval tag. Can be specified multiple times; CLI tags override config tags. | +| `--table` | Output table in CLI | +| `--table-cell-max-length ` | Truncate console table cells to this length | +| `-t, --tests ` | Path to CSV with test cases | +| `--var ` | Set a variable in key=value format | +| `-v, --vars ` | Path to CSV with test cases (alias for --tests) | +| `-w, --watch` | Watch for changes in config and re-run | +| `-x, --extension ` | Extension hooks to run, such as `file://handler.js:afterAll` | Use `--tag` for run-specific eval tags that should not change `promptfooconfig.yaml`: @@ -162,7 +163,7 @@ promptfoo eval --filter-range 200: # tests 200 through the end promptfoo eval --filter-range :50 # first 50 tests ``` -Range is applied before `--repeat` expansion, so `--filter-range 0:5 --repeat 3` runs 15 evaluations across the same 5 tests. When combined with other filters (`--filter-pattern`, `--filter-metadata`, etc.), range slices the post-filter list. +Range is applied before `--repeat` expansion, so `--filter-range 0:5 --repeat 3` runs 15 evaluations across the same 5 tests. When combined with other filters (`--filter-pattern`, `--filter-metadata`, `--filter-vars`, etc.), range slices the post-filter list. When resuming an eval, promptfoo reuses the range saved with the original run so test indices stay stable. A `--filter-range` flag passed on resume is ignored (with a warning) and other transient filters from the original run are not restored, so resume is most predictable when range was the only selection filter. diff --git a/src/commands/eval.ts b/src/commands/eval.ts index 12e429927f04..ae56c3bdf50e 100644 --- a/src/commands/eval.ts +++ b/src/commands/eval.ts @@ -137,6 +137,13 @@ export function evalCommand( return previous ? [...previous, value] : [value]; }, ) + .option( + '--filter-vars ', + 'Only run tests whose vars exactly match the key=value pair. Can be specified multiple times for AND logic (e.g. --filter-vars language=en --filter-vars category=safety)', + (value: string, previous: string[] | undefined) => { + return previous ? [...previous, value] : [value]; + }, + ) // Output configuration .option( diff --git a/src/node/doEval.ts b/src/node/doEval.ts index 3ff89cf5f507..e698ab404e2c 100644 --- a/src/node/doEval.ts +++ b/src/node/doEval.ts @@ -609,6 +609,7 @@ export async function doEval( cmdObj.filterFirstN !== undefined || cmdObj.filterMetadata !== undefined || cmdObj.filterPattern !== undefined || + cmdObj.filterVars !== undefined || filterSample !== undefined; const shouldApplyFiltersToImplicitDefaultTest = hasActiveTestFilter && canSynthesizeImplicitDefaultTest && !testSuite.tests?.length; @@ -630,6 +631,7 @@ export async function doEval( range: hasScenarios ? undefined : filterRange, sample: filterSample, sampleSeed: filterSampleSeed, + vars: cmdObj.filterVars, }; testSuite.tests = await filterTests(testSuite, filterOptions); const shouldSuppressImplicitDefaultTest = diff --git a/src/types/index.ts b/src/types/index.ts index 189573029d09..1c8f55026ba1 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -130,6 +130,7 @@ export const CommandLineOptionsSchema = z.object({ filterSample: z.coerce.number().int().positive().optional(), filterSampleSeed: z.coerce.number().int().safe().optional(), filterTargets: z.string().optional(), + filterVars: z.union([z.string(), z.array(z.string())]).optional(), var: z.record(z.string(), z.string()).optional(), tags: z.record(z.string(), z.string()).optional(), diff --git a/src/util/eval/filterTests.ts b/src/util/eval/filterTests.ts index 4de537e251ca..a06d2f329dd9 100644 --- a/src/util/eval/filterTests.ts +++ b/src/util/eval/filterTests.ts @@ -15,9 +15,11 @@ * @module util/eval/filterTests */ +import { isDeepStrictEqual } from 'node:util'; + import logger from '../../logger'; import { ResultFailureReason } from '../../types/index'; -import { getTestCaseDeduplicationKey } from '../../util/comparison'; +import { filterRuntimeVars, getTestCaseDeduplicationKey } from '../../util/comparison'; import { filterByRange } from '../../util/filterRange'; import { warnEmptyFilterRange } from '../../util/filterRangeWarn'; import { filterTestsByResults } from './filterTestsUtil'; @@ -59,11 +61,97 @@ export interface FilterOptions { sample?: number | string; /** Seed used to make random sampling repeatable */ sampleSeed?: number; + /** Key-value pair(s) (format: "key=value") to filter tests by vars. Multiple values use AND logic. */ + vars?: string | string[]; } type Tests = NonNullable; type TestFilterFn = (test: TestCase) => boolean; +function parseKeyValueFilters( + filters: string | string[], + flagName: '--filter-metadata' | '--filter-vars', +): Array<{ key: string; value: string }> { + const normalizedFilters = Array.isArray(filters) ? filters : [filters]; + return normalizedFilters.map((filter) => { + const [key, ...valueParts] = filter.split('='); + const value = valueParts.join('='); + if (!key || value === '') { + throw new Error(`${flagName} must be specified in key=value format`); + } + return { key, value }; + }); +} + +function matchesVarValue(testValue: unknown, expectedValue: string): boolean { + if (typeof testValue === 'string') { + return testValue === expectedValue; + } + if (typeof testValue === 'number' || typeof testValue === 'boolean') { + return String(testValue) === expectedValue; + } + if (testValue !== undefined) { + try { + return isDeepStrictEqual(testValue, JSON.parse(expectedValue)); + } catch { + return false; + } + } + return false; +} + +function createVarsFilter(testSuite: TestSuite, filters: string | string[]): TestFilterFn { + const parsedFilters = parseKeyValueFilters(filters, '--filter-vars'); + const defaultVars = + testSuite.defaultTest && typeof testSuite.defaultTest === 'object' + ? testSuite.defaultTest.vars + : undefined; + + logger.debug(`Filtering for ${parsedFilters.length} vars condition(s) using AND logic`); + + return (test) => { + const effectiveVars = filterRuntimeVars({ + ...defaultVars, + ...test.vars, + }); + return Boolean( + effectiveVars && + parsedFilters.every(({ key, value }) => matchesVarValue(effectiveVars[key], value)), + ); + }; +} + +function createMetadataFilter(filters: string | string[]): TestFilterFn { + const parsedFilters = parseKeyValueFilters(filters, '--filter-metadata'); + + logger.debug( + `Filtering for metadata conditions (AND logic): ${parsedFilters.map((filter) => filter.key).join(', ')}`, + ); + + return (test) => { + if (!test.metadata) { + logger.debug(`Test has no metadata: ${test.description || 'unnamed test'}`); + return false; + } + + for (const { key, value } of parsedFilters) { + const testValue = test.metadata[key]; + const matches = Array.isArray(testValue) + ? testValue.some((item) => item.toString().includes(value)) + : testValue !== undefined && testValue.toString().includes(value); + + if (!matches) { + logger.debug( + `Test "${test.description || 'unnamed test'}" metadata does not match key "${key}"`, + ); + return false; + } + } + + return true; + }; +} + function createSeededRandom(seed: number): () => number { const stringSeed = String(seed); let state = 2166136261; @@ -142,24 +230,26 @@ async function filterErrorTests( /** * Applies multiple filters to a test suite based on the provided options. * Filters are applied in the following order: - * 1. Metadata filter - * 2. Failing tests filter - * 3. Error tests filter - * 4. Pattern filter - * 5. Range filter - * 6. First N filter - * 7. Random sample filter + * 1. Vars filter + * 2. Metadata filter + * 3. Failing tests filter + * 4. Error tests filter + * 5. Pattern filter + * 6. Range filter + * 7. First N filter + * 8. Random sample filter * * @param testSuite - The test suite containing all tests * @param options - Configuration options for filtering * @returns A filtered array of tests that match all the specified criteria - * @throws {Error} If metadata filter format is invalid or if numeric filters contain non-numeric values + * @throws {Error} If a key-value filter format is invalid or if numeric filters contain non-numeric values */ export async function filterTests(testSuite: TestSuite, options: FilterOptions): Promise { let tests = testSuite.tests || []; + let varsFilter: TestFilterFn | undefined; let metadataFilter: TestFilterFn | undefined; - logger.debug(`Starting filterTests with options: ${JSON.stringify(options)}`); + logger.debug(`Starting filterTests with option keys: ${Object.keys(options).join(', ')}`); logger.debug(`Initial test count: ${tests.length}`); if (Object.keys(options).length === 0) { @@ -167,61 +257,30 @@ export async function filterTests(testSuite: TestSuite, options: FilterOptions): return tests; } - if (options.metadata) { - // Normalize to array for consistent handling - const metadataFilters = Array.isArray(options.metadata) ? options.metadata : [options.metadata]; - - // Validate all filters first - const parsedFilters: Array<{ key: string; value: string }> = []; - for (const filter of metadataFilters) { - const [key, ...valueParts] = filter.split('='); - const value = valueParts.join('='); // Rejoin in case value contains '=' - if (!key || value === undefined || value === '') { - throw new Error('--filter-metadata must be specified in key=value format'); - } - parsedFilters.push({ key, value }); - } + if (options.vars) { + logger.debug(`Before vars filter: ${tests.length} tests`); - logger.debug( - `Filtering for metadata conditions (AND logic): ${parsedFilters.map((f) => `${f.key}=${f.value}`).join(', ')}`, - ); - logger.debug(`Before metadata filter: ${tests.length} tests`); + varsFilter = createVarsFilter(testSuite, options.vars); + tests = tests.filter(varsFilter); - metadataFilter = (test) => { - if (!test.metadata) { - logger.debug(`Test has no metadata: ${test.description || 'unnamed test'}`); - return false; - } + logger.debug(`After vars filter: ${tests.length} tests remain`); + } - // ALL conditions must match (AND logic) - for (const { key, value } of parsedFilters) { - const testValue = test.metadata[key]; - let matches = false; - - if (Array.isArray(testValue)) { - // For array metadata, check if any value includes the search term - matches = testValue.some((v) => v.toString().includes(value)); - } else if (testValue !== undefined) { - // For single value metadata, check if it includes the search term - matches = testValue.toString().includes(value); - } - - if (!matches) { - logger.debug( - `Test "${test.description || 'unnamed test'}" metadata doesn't match. Expected ${key} to include ${value}, got ${JSON.stringify(test.metadata)}`, - ); - return false; - } - } + if (options.metadata) { + logger.debug(`Before metadata filter: ${tests.length} tests`); - return true; - }; + metadataFilter = createMetadataFilter(options.metadata); tests = tests.filter(metadataFilter); logger.debug(`After metadata filter: ${tests.length} tests remain`); } - const resultFilterTestSuite = metadataFilter ? { ...testSuite, tests } : testSuite; + const extractedTestFilter = + varsFilter || metadataFilter + ? (test: TestCase) => + (!varsFilter || varsFilter(test)) && (!metadataFilter || metadataFilter(test)) + : undefined; + const resultFilterTestSuite = extractedTestFilter ? { ...testSuite, tests } : testSuite; // Handle failing, failingOnly, and errorsOnly filters // - failing: all non-successful results (failures + errors) @@ -235,12 +294,12 @@ export async function filterTests(testSuite: TestSuite, options: FilterOptions): const failingOnlyTests = await filterFailingOnlyTests( resultFilterTestSuite, options.failingOnly, - metadataFilter, + extractedTestFilter, ); const errorTests = await filterErrorTests( resultFilterTestSuite, options.errorsOnly, - metadataFilter, + extractedTestFilter, ); // Create a union of both sets, deduplicating by test identity @@ -266,7 +325,7 @@ export async function filterTests(testSuite: TestSuite, options: FilterOptions): } } else if (options.failing) { // --filter-failing includes both failures and errors - tests = await filterFailingTests(resultFilterTestSuite, options.failing, metadataFilter); + tests = await filterFailingTests(resultFilterTestSuite, options.failing, extractedTestFilter); if (tests.length === 0) { logNoTestsWarning('filter-failing', options.failing, 'no failures/errors'); } @@ -275,7 +334,7 @@ export async function filterTests(testSuite: TestSuite, options: FilterOptions): tests = await filterFailingOnlyTests( resultFilterTestSuite, options.failingOnly, - metadataFilter, + extractedTestFilter, ); if (tests.length === 0) { logNoTestsWarning( @@ -285,7 +344,7 @@ export async function filterTests(testSuite: TestSuite, options: FilterOptions): ); } } else if (options.errorsOnly) { - tests = await filterErrorTests(resultFilterTestSuite, options.errorsOnly, metadataFilter); + tests = await filterErrorTests(resultFilterTestSuite, options.errorsOnly, extractedTestFilter); if (tests.length === 0) { logNoTestsWarning('filter-errors-only', options.errorsOnly, 'no errors'); } diff --git a/test/commands/eval/evaluateOptions.test.ts b/test/commands/eval/evaluateOptions.test.ts index 46a260c15b55..a8b9dfccdfe2 100644 --- a/test/commands/eval/evaluateOptions.test.ts +++ b/test/commands/eval/evaluateOptions.test.ts @@ -754,11 +754,13 @@ describe('evaluateOptions behavior', () => { it.each([ ['--filter-pattern', { filterPattern: 'no match' }], ['--filter-metadata', { filterMetadata: 'category=drop' }], + ['--filter-vars', { filterVars: 'language=drop' }], ])('should apply %s to the implicit default test', async (_filterName, filterOptions) => { const tempConfig = writeTempConfig(tmpDir, 'test-filter-implicit-default.yaml', { providers: ['echo'], prompts: ['Hello'], defaultTest: { + vars: { language: 'keep' }, metadata: { category: 'keep' }, assert: [{ type: 'contains', value: 'Hello' }], }, @@ -811,6 +813,34 @@ describe('evaluateOptions behavior', () => { expect(testSuite.scenarios).toBeUndefined(); }); + it('should filter an implicit default test by inherited vars', async () => { + const tempConfig = writeTempConfig(tmpDir, 'test-filter-implicit-default-vars.yaml', { + providers: ['echo'], + prompts: ['Hello {{language}}'], + defaultTest: { + vars: { language: 'en' }, + assert: [{ type: 'contains', value: 'en' }], + }, + }); + + await doEval( + { + table: false, + write: false, + config: [tempConfig], + filterVars: 'language=en', + }, + {}, + undefined, + {}, + ); + + expect(evaluateMock).toHaveBeenCalled(); + const testSuite = evaluateMock.mock.calls.at(-1)?.[0] as TestSuite; + expect(testSuite.tests).toHaveLength(1); + expect(testSuite.scenarios).toBeUndefined(); + }); + it('should apply filterRange to the implicit default test', async () => { const tempConfig = writeTempConfig(tmpDir, 'test-filter-range-implicit-default.yaml', { providers: ['echo'], diff --git a/test/types/index.test.ts b/test/types/index.test.ts index a356322661de..1c680fba2534 100644 --- a/test/types/index.test.ts +++ b/test/types/index.test.ts @@ -708,6 +708,7 @@ describe('CommandLineOptionsSchema', () => { filterFirstN: 5, filterMetadata: 'meta', filterRange: '1:3', + filterVars: ['language=en', 'category=safety'], }; expect(() => CommandLineOptionsSchema.parse(options)).not.toThrow( 'Invalid command line options', @@ -739,6 +740,7 @@ describe('CommandLineOptionsSchema', () => { filterSample: 5, filterSampleSeed: 42, filterTargets: 'target1', + filterVars: 'language=en', }; expect(() => CommandLineOptionsSchema.parse(options)).not.toThrow( 'Invalid command line options', diff --git a/test/util/eval/filterTests.test.ts b/test/util/eval/filterTests.test.ts index 2afcb531cd19..7ec90b37ebcd 100644 --- a/test/util/eval/filterTests.test.ts +++ b/test/util/eval/filterTests.test.ts @@ -106,6 +106,100 @@ describe('filterTests', () => { expect(result).toEqual([]); }); + describe('vars filter', () => { + const varsTestSuite: TestSuite = { + prompts: [], + providers: [], + tests: [ + { + description: 'english safety', + vars: { + language: 'en', + category: 'safety', + enabled: true, + attempts: 2, + config: { mode: 'strict', tags: ['a', 'b'] }, + }, + }, + { + description: 'english quality', + vars: { language: 'en-US', category: 'quality', enabled: false, attempts: 1 }, + }, + { + description: 'german safety', + vars: { language: 'de', category: 'safety', enabled: true, attempts: 1 }, + }, + ], + }; + + it('should filter string vars using exact matching', async () => { + const result = await filterTests(varsTestSuite, { vars: 'language=en' }); + + expect(result.map((test) => test.description)).toEqual(['english safety']); + }); + + it('should combine multiple vars filters using AND logic', async () => { + const result = await filterTests(varsTestSuite, { + vars: ['category=safety', 'language=de'], + }); + + expect(result.map((test) => test.description)).toEqual(['german safety']); + }); + + it('should match number and boolean vars', async () => { + const result = await filterTests(varsTestSuite, { + vars: ['enabled=true', 'attempts=2'], + }); + + expect(result.map((test) => test.description)).toEqual(['english safety']); + }); + + it('should match structured vars using JSON', async () => { + const result = await filterTests(varsTestSuite, { + vars: 'config={"tags":["a","b"],"mode":"strict"}', + }); + + expect(result.map((test) => test.description)).toEqual(['english safety']); + }); + + it('should handle string values containing equals signs', async () => { + const result = await filterTests( + { + prompts: [], + providers: [], + tests: [{ vars: { query: 'a=1&b=2' } }], + }, + { vars: 'query=a=1&b=2' }, + ); + + expect(result).toHaveLength(1); + }); + + it('should include inherited defaultTest vars', async () => { + const result = await filterTests( + { + prompts: [], + providers: [], + defaultTest: { vars: { language: 'en', tier: 'default' } }, + tests: [{ description: 'inherits defaults' }, { vars: { language: 'de' } }], + }, + { vars: ['language=en', 'tier=default'] }, + ); + + expect(result.map((test) => test.description)).toEqual(['inherits defaults']); + }); + + it.each([ + 'invalid', + '=value', + 'key=', + ])('should reject invalid filter format: %s', async (vars) => { + await expect(filterTests(varsTestSuite, { vars })).rejects.toThrow( + '--filter-vars must be specified in key=value format', + ); + }); + }); + describe('metadata filter', () => { it('should filter tests by metadata key-value pair', async () => { const result = await filterTests(mockTestSuite, { metadata: 'type=unit' });