diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index d29878d3e..928bf1ed0 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -1392,6 +1392,7 @@ For the AI agent search-rule template, see [AI Integration](USER_GUIDE.md#ai-int | Bounded high-volume responses | `search`, `definition`, `find`, `status`, `hotspots`, `references`, `callers`, `callees`, `symbols`, `files`, `languages`, `impact`, and `map` accept shared bounded-response controls where their schema exposes them. Newly emitted opaque `--cursor ` values bind the offset to the command/query/filter selection and index generation; legacy `response:v1::` cursors remain accepted for transition. Reuse with changed selection or generation fails with restart-required guidance. `search --format compact`, `symbols --format compact`, and `files --format compact` auto-select the bounded contract, while `search --json=array --json-envelope` provides the opt-in array envelope and `languages --json` selects it when paging or `--max-json-bytes` is requested. Existing compact roots and location rows remain compatible while adding shared metadata. Metadata reports `returned_count`, authoritative `total_count` where available, `omitted_count`, `remaining_count`, `cursor_offset`, `page_limit`, `has_more`, `next_cursor`, `result_stable_at`, `pagination_window_limit`, and `pagination_window_exhausted`. The safety window is 10,000 rows; exhaustion suppresses `next_cursor` rather than returning a cursor that the next request would reject. Pageable commands pass the cursor offset into their database/scan layer instead of serializing an `offset + limit` prefix. `find --all` partial scans encode the next path/line in the opaque cursor so replay continues after the last scanned line. `hotspots` and `impact` page their active primary nested collection as `results`, identify it with `metadata.primary_collection`, and retain scalar/container evidence in `metadata.response_context`; dotted fields such as `callers.path,callers.depth` select that collection and project its rows. The final newline is included in `--max-json-bytes`, and trailing whole rows are removed until the complete envelope fits. `definition` remains metadata-only by default; explicit `--body` content is retained for `body`, `body_content`, or `all`, and suppressed when the projection excludes it. `map --sections` remains its section-level projection, while dotted bounded fields page a selected array section with section-specific totals and scalar projections skip unused ranked arrays. | | Bounded response edge cases | `impact` applies the cursor offset only to the selected nested collection so definition pages do not repeat or alter caller/fallback mode. Plain `map --compact` preserves its established section arrays and truncation payload; a collection projection is rejected when `--summary-only` or an excluding `--sections` filter would remove it. Explicit definition body fields override compact defaults. Profile and verbose records are moved into `metadata.stream_control_records`, and parser/capture failures emit an error envelope only when it fits the active hard byte cap. | | `--count --json` envelope | Count-only JSON for `search`, `definition`, `references`, `callers`, `callees`, `symbols`, `files`, `find`, `impact`, and `unused` is a single automation-oriented object. It always includes `count`, applied `query_context`, freshness metadata (`indexed_file_count`, `indexed_at`, `freshness_available`), and trust flags `degraded` / `authoritative_count`; commands with matched-file totals also include `files` and the older `file_count` compatibility alias. `file_count` carries the same value as `files`, remains for compatibility, and is not scheduled for removal before the next major release. `unused --count --json` also includes `returned_bucket_counts`, `returned_contract_domain_counts`, and `summary.by_bucket` / `summary.by_confidence` / `summary.by_contract_domain`. `authoritative_count=false` means a readiness or graph/exact trust signal made the count non-authoritative, while the freshness fields describe the indexed snapshot used for the count. | +| Recipe row selection | Row-producing recipe paths share `ApplySearchOutputSelection`: `--first-per-file` and deterministic `--sample` run before the effective per-query / remaining total limit. Sample fetch envelopes are sized from at least the requested sample target. Aggregate and compact query DTOs expose nullable `selection_reason` / `selection_omitted_count`; issue-draft source DTOs preserve them and recipe NDJSON forwards their aggregate to the terminal record. Selection-only omission updates matched/omitted lower bounds without setting `truncated`, `has_more`, or `next_cursor`. When a later limit truncates selected rows, limit truncation remains visible but `next_cursor` is suppressed because raw database cursors cannot preserve selector state; incoming `--cursor` values are rejected with either selector for the same reason. Generated compact and issue-draft replay commands retain the selector. Count, aggregation, and summary-only compact shapes reject `--first-per-file` / `--sample`, while every recipe shape rejects grouped-only `--per-file-limit`. | | Recipe SARIF | `search --recipe --format sarif` emits one result per bounded recipe result. Rule IDs use `recipe/query`; standard `fingerprints.cdidx/v1` values are derived from the normalized source location; result properties preserve recipe/query identity, severity, confidence, and per-query truncation; run properties preserve scope, applied result limits, aggregate counts, and conservative omitted-result metadata. Bound SARIF with `--limit` / `--total-limit`; row selectors such as `--sample`, `--first-per-file`, and `--per-file-limit` are rejected instead of being silently ignored. Recipe severity maps `critical` / `high` to `error`, `medium` to `warning`, and `low` / `info` to `note`. | | Recipe classifier output | Recipe run JSON may add `audit_classifications` to individual `CompactSearchResult` rows when a recipe classifier can classify the hit, and query/count payloads may add `classifier_counts` when classified rows are present. These fields are additive; use them to separate triage domains such as DTO/result-wrapper `.Result` properties versus Task/ValueTask blocking waits without changing the raw search query. | | NDJSON terminal records | Default NDJSON for `search`, `symbols`, and `files` appends one final `terminal_record` after result rows; search also emits it for zero-result responses, while raw `symbols` and `files` keep zero-result NDJSON empty. Recipe/audit search row streams share the same writer. Terminals report returned and observed total counts, `total_count_authoritative` / `total_count_lower_bound`, selection or interruption reason, applied limits, omitted rows, and recovery guidance. `--max-json-bytes` covers the complete stdout stream, including newlines and this terminal record; a cap that cannot fit the terminal fails before stdout. Capped output rejects `--profile`, `--verbose`, and `--json-envelope`. Byte-cap partial output exits with `CommandExitCodes.PartialResult` (`11`) unless `--allow-partial` explicitly opts into exit `0`. `--results-only` is the explicit terminal-record opt-out for these NDJSON row streams and is rejected with array, compact, summary, or count output. | @@ -4422,6 +4423,7 @@ AI エージェント向け検索ルールのテンプレートについては | 高ボリューム応答の bounded 契約 | `search`、`definition`、`find`、`status`、`hotspots`、`references`、`callers`、`callees`、`symbols`、`files`、`languages`、`impact`、`map` は、それぞれの schema が公開する共通 bounded-response control に対応します。新しく発行する opaque な `--cursor ` は offset を command / query / filter と index generation に束縛し、移行用に legacy の `response:v1::` も受理します。選択条件または generation を変えて再利用すると restart-required の案内付きで失敗します。`search --format compact`、`symbols --format compact`、`files --format compact` は bounded 契約を自動選択し、`search --json=array --json-envelope` は opt-in の array envelope、`languages --json` は paging または `--max-json-bytes` 指定時に同じ契約を使います。既存 compact の root と location row は維持したまま共通 metadata を追加します。metadata は `returned_count`、取得可能な場合は authoritative な `total_count`、`omitted_count`、`remaining_count`、`cursor_offset`、`page_limit`、`has_more`、`next_cursor`、`result_stable_at`、`pagination_window_limit`、`pagination_window_exhausted` を返します。safety window は 10,000 row で、上限到達時は次の request が拒否する cursor を返さず `next_cursor` を抑止します。pageable command は `offset + limit` 件を serialize せず、cursor offset を database / scan layer へ渡します。`find --all` の partial scan cursor は次の path / line を保持し、再利用時は最後に scan した line の次から継続します。`hotspots` と `impact` は active な主要 nested collection を `results` としてページングし、`metadata.primary_collection` でその名前を示し、scalar / container evidence は `metadata.response_context` に保持します。`callers.path,callers.depth` のような dotted field で collection と row field を同時に選べます。`--max-json-bytes` は最後の改行を含み、完全な envelope が収まるまで末尾の完全な row を省略します。`definition` は既定で metadata-only のままで、明示的な `--body` は `body`、`body_content`、`all` で保持し、それ以外の projection では materialize 前に抑止します。`map --sections` は section-level projection として残り、dotted な bounded field は選択した array section を section 固有の総件数付きでページングし、scalar projection は不要な ranking array を構築しません。 | | bounded 応答の edge case | `impact` は選択された nested collection だけに cursor offset を適用するため、definition page の重複や caller / fallback mode の変化を防ぎます。通常の `map --compact` は既存の section array と truncation payload を維持し、collection projection が `--summary-only` または除外する `--sections` filter で失われる組み合わせは拒否します。明示的な definition body field は compact default より優先します。profile / verbose record は `metadata.stream_control_records` へ移し、parser / capture failure の error envelope は active な hard byte cap に収まる場合だけ出力します。 | | `--count --json` envelope | `search`、`definition`、`references`、`callers`、`callees`、`symbols`、`files`、`find`、`impact`、`unused` の count-only JSON は単一の自動化向け object です。常に `count`、適用済み `query_context`、freshness metadata（`indexed_file_count`、`indexed_at`、`freshness_available`）、trust flag の `degraded` / `authoritative_count` を含みます。matched-file total を持つ command は `files` と古い互換 alias の `file_count` も含みます。`file_count` は `files` と同じ値を持つ互換用 field として残り、少なくとも次の major release までは削除予定はありません。`unused --count --json` は `returned_bucket_counts`、`returned_contract_domain_counts`、`summary.by_bucket` / `summary.by_confidence` / `summary.by_contract_domain` も含みます。`authoritative_count=false` は readiness または graph/exact trust signal により count が authoritative ではないことを示し、freshness field は count に使った index snapshot を説明します。 | +| recipe row selection | row を返す recipe path は `ApplySearchOutputSelection` を共有し、`--first-per-file` と決定的な `--sample` を、有効な query ごとの limit / 残り total limit より先に適用します。sample 用 fetch envelope は少なくとも要求 sample 数を基準に sizing します。aggregate / compact の query DTO は nullable な `selection_reason` / `selection_omitted_count` を公開し、issue-draft の source DTO もこれらを保持し、recipe NDJSON は集計値を terminal record へ渡します。selection だけによる省略は matched / omitted の lower bound を更新しますが、`truncated`、`has_more`、`next_cursor` は設定しません。後続の limit が選択済み row を truncate する場合、limit truncation は表示しますが raw database cursor は selector state を保持できないため `next_cursor` を抑止し、同じ理由で selector と受け取った `--cursor` の併用も拒否します。compact / issue-draft の生成 replay command は selector を保持します。count、aggregation、summary-only compact の shape は `--first-per-file` / `--sample` を拒否し、すべての recipe shape は grouped 専用の `--per-file-limit` を拒否します。 | | Recipe SARIF | `search --recipe --format sarif` は、上限付き recipe result ごとに result を1件出力します。rule ID は `recipe/query` を使い、標準の `fingerprints.cdidx/v1` は正規化済み source location から導出します。result properties は recipe/query identity、severity、confidence、query ごとの truncation を保持し、run properties は scope、適用済み result limit、集計 count、保守的な omitted-result metadata を保持します。SARIF の上限には `--limit` / `--total-limit` を使い、`--sample`、`--first-per-file`、`--per-file-limit` のような row selector は黙って無視せず拒否します。recipe severity は `critical` / `high` を `error`、`medium` を `warning`、`low` / `info` を `note` に対応付けます。 | | Recipe classifier output | recipe classifier が hit を分類できる場合、recipe run JSON は個別の `CompactSearchResult` row に `audit_classifications` を追加することがあり、分類済み row がある query / count payload は `classifier_counts` を追加することがあります。これらは additive field です。raw search query を変えずに、DTO / result-wrapper の `.Result` property と Task / ValueTask の blocking wait などの triage domain を分離するために使います。 | | NDJSON terminal record | `search`、`symbols`、`files` の既定 NDJSON は result row の後に最後の `terminal_record` を 1 件追加します。`search` は 0 件応答にも終端を出力しますが、raw `symbols` / `files` の 0 件 NDJSON は空のままです。recipe / audit search の row stream も同じ writer を使います。終端は返却件数と観測済み総件数、`total_count_authoritative` / `total_count_lower_bound`、selection または中断理由、適用上限、省略行数、復旧案内を報告します。`--max-json-bytes` は改行と終端レコードを含む stdout stream 全体を対象にし、終端自体が収まらない cap は stdout 出力前に失敗します。上限付き出力は `--profile`、`--verbose`、`--json-envelope` を拒否します。byte cap による部分出力は、`--allow-partial` で終了コード `0` を明示許可しない限り `CommandExitCodes.PartialResult`（`11`）を返します。`--results-only` はこれらの NDJSON row stream から終端レコードを明示的に除外するための option であり、array / compact / summary / count 出力との組み合わせは拒否されます。 | diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index bdfb462fe..bca2aa67e 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -194,6 +194,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding Validate JSON, compact, count, and SARIF pagination/severity coverage shares one mixed informational/actionable fixture so authoritative totals, limited rows, SARIF levels, and actionability metadata cannot drift across formats; keep missing-`file_issues` degradation coverage in a separate legacy-schema fixture because availability is a distinct mutable state. Command-specific output format coverage uses a command/format matrix that checks both parser acceptance and the matching usage line; recognized shared formats without a command implementation need a separate usage-error assertion. Recipe SARIF coverage must assert bounded result counts, `recipe/query` rule identity, source locations, severity mapping, confidence, conservative truncation metadata, and stable `fingerprints.cdidx/v1` values across identical runs. + Recipe row-selection coverage reuses one multi-file, multi-chunk fixture across aggregate JSON, compact JSON, NDJSON, and issue-draft source metadata. Assert emitted/matched/omitted counts, `selection_reason` / `selection_omitted_count`, first-per-file path uniqueness, selector-preserving replay commands, suppressed raw cursors when a later limit truncates selected rows, and rejection of incoming cursors with either selector. A separate candidate-window fixture must exceed the default low-limit fetch envelope and prove that `--sample ` observes at least its requested candidate target; validate rejected selectors for non-row recipe shapes without opening a database. Unused default-suppression row, JSON count, summary-only, and text count envelopes, including the `--all` count control, share one unused-symbol fixture. Unused default-suppressed and `--all` JSON cursor pagination share one unused-symbol fixture. Unused full and compact `--by-bucket` JSON envelopes share one taxonomy fixture. @@ -1045,6 +1046,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" validate の JSON、compact、count、SARIF における pagination / severity coverage は、informational finding と actionable finding が混在する1つの fixture を共有し、authoritative な総件数、limited row、SARIF level、actionability metadata が format 間で drift しないことを検証してください。`file_issues` 欠落時の degradation coverage は availability が独立した mutable state なので、別の legacy-schema fixture に分けてください。 コマンド別の出力形式 coverage は command / format matrix で parser の受理と対応する usage line の両方を検証してください。共通 parser が認識してもコマンド側に実装がない形式には、別途 usage error の assertion が必要です。 Recipe SARIF coverage では、上限付き result count、`recipe/query` rule identity、source location、severity mapping、confidence、保守的な truncation metadata、同一 run 間で安定する `fingerprints.cdidx/v1` を検証してください。 + recipe row-selection coverage は aggregate JSON、compact JSON、NDJSON、issue-draft の source metadata で1つの multi-file / multi-chunk fixture を共有します。emitted / matched / omitted count、`selection_reason` / `selection_omitted_count`、first-per-file の path uniqueness、selector を保持する replay command、後続 limit が選択済み row を truncate する場合の raw cursor 抑止、両 selector と受け取った cursor の併用拒否を検証してください。別の candidate-window fixture では既定の low-limit fetch envelope を超え、`--sample ` が少なくとも要求 candidate 数を観測することを証明し、row を持たない recipe shape での selector 拒否は database を開かずに確認してください。 unused default-suppressionのrow、JSON count、summary-only、text count envelopeは、`--all` count controlも含めて1つのunused-symbol fixtureを共有してください。 unusedのdefault-suppressed JSON cursor paginationと`--all` JSON cursor paginationは1つのunused-symbol fixtureを共有してください。 unusedのfull JSONとcompact `--by-bucket` JSON envelopeは1つのtaxonomy fixtureを共有してください。 diff --git a/USER_GUIDE.md b/USER_GUIDE.md index 982a3a3bf..c3fbfd65b 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -1342,6 +1342,22 @@ names, aggregate counts, per-query counts, and query freshness. Recipe count aggregations support `--count-by path|file|symbol|origin|return-type|subsystem`, `--group-by file|symbol|origin|return-type|subsystem --count`, and `--unique path|file|symbol|origin|return-type|subsystem`. +Row-producing recipe modes (text, aggregate JSON, compact JSON, NDJSON, and +issue drafts) apply `--first-per-file` and deterministic `--sample ` before +the effective per-query `--limit` / cross-query `--total-limit`. Aggregate JSON +and compact query objects report `selection_reason` and +`selection_omitted_count` when selection removes rows; issue-draft `source` +objects and NDJSON terminal records report the same fields. Selection-only +omission contributes to matched and omitted counts but does not set `truncated`, +`has_more`, or `next_cursor`. If a later limit also omits selected rows, +`truncated` / `has_more` are set but `next_cursor` is suppressed because a raw +cursor cannot preserve row-selection state; increase the applicable limit and +rerun instead. For the same reason, recipe row selectors reject an incoming +`--cursor`. Generated compact and issue-draft replay commands retain the active +selector. Count, aggregation, +and summary-only compact recipe output reject row-selection controls because +they cannot represent selected rows, and recipe execution rejects +`--per-file-limit` because it does not produce grouped search output. Recipe SARIF emits one bounded finding per returned recipe result. Its rule IDs use `recipe/query`, result fingerprints are stable for the recipe/query/source location, and result/run properties preserve severity, confidence, scope, @@ -1349,7 +1365,8 @@ applied result limits, and conservative truncation metadata. Other search export formats and `--json=array` are rejected for recipe modes because recipe output is grouped by query or list metadata. Recipe JSON and compact output apply `--limit` per query, include a `summary` -with emitted/truncated counts, and mark truncated child queries with + with emitted/truncated counts, and mark truncated child queries that do not + use a row selector with `next_cursor`; rerun a single child query as `--recipe / --cursor ` to page the next result set. The MCP `search` tool exposes the same recipe surface with @@ -4565,14 +4582,28 @@ child query 全体の emitted row 数を制限でき、NDJSON では `--max-json recipe count output は `--format count --summary-only --max-json-bytes ` により、recipe / scope 名、 aggregate count、query ごとの count、query freshness だけを出力できます。recipe の count aggregation は `--count-by path|file|symbol|origin|return-type|subsystem`、 `--group-by file|symbol|origin|return-type|subsystem --count`、`--unique path|file|symbol|origin|return-type|subsystem` に対応します。 +row を返す recipe mode（text、aggregate JSON、compact JSON、NDJSON、issue draft）は、 +`--first-per-file` と決定的な `--sample ` を、有効な query ごとの `--limit` / +query 全体の `--total-limit` より先に適用します。aggregate JSON / compact の query object は +selection で row が省略された場合に `selection_reason` と `selection_omitted_count` を返し、 +issue-draft の `source` object と NDJSON terminal record も同じ field を返します。 +selection だけによる省略は matched / omitted count に含まれますが、`truncated`、 +`has_more`、`next_cursor` は設定しません。後続の limit でも選択済み row が省略される場合は +`truncated` / `has_more` を設定しますが、raw cursor では row-selection state を保持できないため +`next_cursor` は抑止します。この場合は該当 limit を増やして再実行してください。同じ理由で、 +recipe の row selector は受け取った `--cursor` も拒否します。compact / issue-draft が生成する +replay command は有効な selector を保持します。count、aggregation、summary-only compact の +recipe output は選択済み row を表現できないため row-selection control を拒否し、recipe +execution は grouped search output を生成しないため `--per-file-limit` を拒否します。 recipe SARIF は返却された recipe result ごとに上限付き finding を1件出力します。rule ID は `recipe/query` を使い、result fingerprint は recipe / query / source location に対して安定し、 result / run properties は severity、confidence、scope、適用済み result limit、 保守的な truncation metadata を保持します。 その他の search export format と `--json=array` は、recipe output が query または list metadata ごとに grouped されるため usage error で拒否します。 -recipe の JSON/compact output は `--limit` を query ごとに適用し、emitted/truncated -count を含む `summary` を出力し、truncated な child query には `next_cursor` を付けます。 + recipe の JSON/compact output は `--limit` を query ごとに適用し、emitted/truncated + count を含む `summary` を出力し、row selector を使っていない truncated な child query には + `next_cursor` を付けます。 次の result set を取得するには、単一 child query を `--recipe / --cursor ` として再実行してください。 MCP `search` tool では `{"listRecipes":true}` で recipe を発見し、 diff --git a/changelog.d/unreleased/4759.fixed.md b/changelog.d/unreleased/4759.fixed.md new file mode 100644 index 000000000..6596b82be --- /dev/null +++ b/changelog.d/unreleased/4759.fixed.md @@ -0,0 +1,25 @@ +--- +category: fixed +issues: + - 4759 +affected: + - src/CodeIndex/Cli/CliFlagSchema.cs + - src/CodeIndex/Cli/ConsoleUi.cs + - src/CodeIndex/Cli/QueryCommandRunner.Search.cs + - src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs + - src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs + - src/CodeIndex/Cli/SearchAuditRecipes.cs + - tests/CodeIndex.Tests/ConsoleUiTests.cs + - tests/CodeIndex.Tests/QueryCommandRunnerSearchTests.cs + - USER_GUIDE.md + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **Audit recipe row selection now matches regular search behavior (#4759)** — Row-producing recipe output applies `--first-per-file` and deterministic `--sample` before result limits, sizes sample candidate windows from the requested sample, preserves selectors in generated replay commands, reports selection metadata without exposing or accepting unsafe raw continuation cursors, and rejects selectors for recipe shapes that cannot represent them. + +## 日本語 + +- **audit recipe の row selection を通常の search と同じ挙動に修正しました (#4759)** — row を返す recipe output は `--first-per-file` と決定的な `--sample` を result limit より先に適用し、要求 sample 数を基準に candidate window を確保し、生成 replay command に selector を保持し、安全でない raw continuation cursor を公開も受理もせず selection metadata を返し、表現できない recipe shape では selector を拒否します。 diff --git a/src/CodeIndex/Cli/CliFlagSchema.cs b/src/CodeIndex/Cli/CliFlagSchema.cs index 3bc66a27a..8991e87d0 100644 --- a/src/CodeIndex/Cli/CliFlagSchema.cs +++ b/src/CodeIndex/Cli/CliFlagSchema.cs @@ -369,9 +369,9 @@ private static IReadOnlyList BuildAll() new() { Name = "--search-fields", ValuePlaceholder = "", Description = "Search/Audit: project JSON/NDJSON result fields for audit pipelines", PrimaryCommands = Set("search", "audit") }, new() { Name = "--outline-fields", ValuePlaceholder = "", Description = "Outline JSON: project symbol fields for audit pipelines", PrimaryCommands = Set("outline") }, new() { Name = "--results-only", Description = "Search/Symbols/Files/Audit: emit result-only NDJSON without stream terminal records", PrimaryCommands = Set("search", "symbols", "files", "audit") }, - new() { Name = "--first-per-file", Description = "Search: keep the first returned match for each file", PrimaryCommands = Set("search") }, - new() { Name = "--sample", ValuePlaceholder = "", Description = "Search: deterministically sample returned rows down to n results", PrimaryCommands = Set("search") }, - new() { Name = "--per-file-limit", ValuePlaceholder = "", Description = "Search/Audit grouped output: representative matches per file", PrimaryCommands = Set("search", "audit") }, + new() { Name = "--first-per-file", Description = "Search/Audit row output: keep the first returned match for each file", PrimaryCommands = Set("search", "audit") }, + new() { Name = "--sample", ValuePlaceholder = "", Description = "Search/Audit row output: deterministically sample returned rows down to n results", PrimaryCommands = Set("search", "audit") }, + new() { Name = "--per-file-limit", ValuePlaceholder = "", Description = "Search grouped output: representative matches per file", PrimaryCommands = Set("search") }, new() { Name = "--total-limit", ValuePlaceholder = "", Description = "Search/Audit recipes: cap emitted rows across all child queries", PrimaryCommands = Set("search", "audit") }, new() { Name = "--env-inventory", Description = "Doctor: include a compact environment-variable summary; use --env-inventory=full for the full inventory", PrimaryCommands = Set("doctor") }, new() { Name = "--env-domain", ValuePlaceholder = "", Description = "Doctor full environment inventory: filter by exact domain", PrimaryCommands = Set("doctor") }, diff --git a/src/CodeIndex/Cli/ConsoleUi.cs b/src/CodeIndex/Cli/ConsoleUi.cs index 4ccd7e55c..51d59afc2 100644 --- a/src/CodeIndex/Cli/ConsoleUi.cs +++ b/src/CodeIndex/Cli/ConsoleUi.cs @@ -86,7 +86,7 @@ private static readonly (string Command, string Usage)[] CommandUsageLines = ("search", "cdidx search |--query |-- |--named-query = [--named-query = ...]|--recipe [--include-query ] [--exclude-query ]|--list-recipes [--query ] [--names|--summary-only] [--cursor ] [--audit-scope ] [--source-only] [--show-excluded] [--db ] [--json[=ndjson|array]] [--pretty] [--format ] [--open-issues ] [--repo ] [--duplicate-confidence |--duplicate-threshold ] [--issue-title ] [--issue-label <label>] [--verbose] [--limit <n>|--top <n>|--max-results <n>] [--total-limit <n>] [--lang <lang>] [--path <glob>] [--exclude-path <glob>] [--exclude-tests] [--exclude-comments] [--exclude-strings] [--exclude-fixtures] [--snippet-lines <n>] [--snippet-focus <leftmost|quality|proximity>] [--max-line-width <n>] [--fts] [--exact|--exact-substring|--token-boundary] [--prefix] [--count] [--group-by <file|symbol|origin|return-type|subsystem>] [--since <datetime>] [--no-dedup] [--no-visibility-rank] [--require-before <query>] [--require-after <query>] [--reject-before <query>] [--reject-after <query>] [--guard-window <n>] [--guard-scope <window|same-line>] [--unique <path|file|symbol|origin|return-type|subsystem>] [--count-by <path|file|symbol|origin|return-type|subsystem>] [--origin <origin>] [--match-origin <origin>] [--exclude-origin <origin>] [--result-kind <kind>] [--search-fields <csv>] [--results-only] [--first-per-file] [--sample <n>] [--per-file-limit <n>] [--max-json-bytes <n>] [--allow-partial] [--next-steps]"), ("recipes", "cdidx recipes [list] [--query <filter>] [--names|--summary-only] [--json] [--pretty] [--format <text|json|compact>] [--max-json-bytes <n>]"), ("recipes-list", "cdidx recipes list [--query <filter>] [--names|--summary-only] [--json] [--pretty] [--format <text|json|compact>] [--max-json-bytes <n>]"), - ("audit", "cdidx audit <recipe|recipe/query> [search filters] [--json[=ndjson]] [--format <text|json|count|compact|issue-drafts>] [--summary-only] [--limit <n>] [--total-limit <n>] [--results-only] [--search-fields <csv>] [--per-file-limit <n>] [--max-json-bytes <n>] [--snippet-lines <n>]"), + ("audit", "cdidx audit <recipe|recipe/query> [search filters] [--json[=ndjson]] [--format <text|json|count|compact|issue-drafts>] [--summary-only] [--limit <n>] [--total-limit <n>] [--results-only] [--search-fields <csv>] [--first-per-file] [--sample <n>] [--max-json-bytes <n>] [--snippet-lines <n>]"), ("definition", "cdidx definition <query>|--query <query>|-- <query> [--db <path>] [--json] [--format <text|json|count|compact|csv|tsv|lsp|qf|sarif>] [--fields <csv>] [--cursor <next_cursor>] [--max-json-bytes <n>] [--verbose] [--limit <n>|--top <n>] [--lang <lang>] [--kind <kind>] [--visibility <v[,v]>] [--exclude-visibility <v[,v]>] [--path <glob>] [--exclude-path <glob>] [--exclude-tests] [--body] [--exact|--exact-name] [--count] [--group-partials] [--since <datetime>]"), ("goto", "cdidx goto <query>|--query <query>|-- <query> [--db <path>] [--json] [--limit <n>|--top <n>] [--lang <lang>] [--kind <kind>] [--path <glob>] [--exclude-path <glob>] [--exclude-tests] [--exact|--exact-name] [--all]"), ("references", "cdidx references <query>|--query <query>|-- <query> [--db <path>] [--json] [--format <text|json|count|compact|csv|tsv|lsp|qf|sarif>] [--fields <csv>] [--cursor <next_cursor>] [--max-json-bytes <n>] [--verbose] [--limit <n>|--top <n>] [--lang <lang>] [--kind <kind>] [--path <glob>] [--exclude-path <glob>] [--exclude-tests] [--body] [--snippet-lines <n>] [--max-line-width <n>] [--exact|--exact-name] [--count]"), diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Search.cs b/src/CodeIndex/Cli/QueryCommandRunner.Search.cs index d761aba8a..a8918bc98 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Search.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Search.cs @@ -404,6 +404,36 @@ private static int RunSearchCore( "Use `--recipe <name> --format sarif` with result filters and `--limit` / `--total-limit`, or choose the JSON/count output shape instead."); return CommandExitCodes.UsageError; } + if (options.GroupedPerFileLimitExplicit) + { + WriteUsageError( + "--per-file-limit is not supported with --recipe because recipe execution does not produce grouped search output.", + GetUsageLineOrThrow("search"), + "Use --first-per-file for one selected recipe row per file, or remove --recipe and use grouped ad hoc search output."); + return CommandExitCodes.UsageError; + } + if ((options.FirstPerFile || options.SampleSize.HasValue) + && options.SearchCursor.HasValue) + { + WriteUsageError( + "recipe row-selection controls cannot be combined with --cursor because raw recipe cursors cannot preserve selector state.", + GetUsageLineOrThrow("search"), + "Remove --cursor and rerun selection from the beginning, or remove --first-per-file / --sample to resume from the cursor."); + return CommandExitCodes.UsageError; + } + if ((options.FirstPerFile || options.SampleSize.HasValue) + && (options.CountOnly + || options.GroupBy != null + || options.CountBy != null + || options.UniqueBy != null + || (options.SummaryOnly && (options.Compact || options.OutputFormat == OutputFormatCompact)))) + { + WriteUsageError( + "recipe row-selection controls cannot be combined with count, aggregation, or summary-only compact output.", + GetUsageLineOrThrow("search"), + "Remove --first-per-file / --sample to keep the non-row output, or choose text, JSON, compact, NDJSON, or issue-drafts row output."); + return CommandExitCodes.UsageError; + } if (options.MaxJsonBytes.HasValue && !SupportsSearchJsonByteLimit(options)) { WriteUsageError( diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs index d422ce496..d2c81d354 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs @@ -515,7 +515,6 @@ private static int RunSearchRecipe(QueryCommandOptions options, JsonSerializerOp recipe.Name, rowQueryResults, rowMinimumMatchedTotal, - rowQueryResults.Any(query => query.Truncated), options, GetCompactJsonOptions(jsonOptions)); ndjsonTerminalLine = stream.TerminalLine; @@ -975,6 +974,7 @@ private static void AddSearchRecipeCompactReplayOptions(List<string> args, Query AddReplayValueOption(args, "--result-kind", kind); if (options.TotalLimit.HasValue) AddReplayValueOption(args, "--total-limit", options.TotalLimit.Value.ToString(CultureInfo.InvariantCulture)); + AddSearchRecipeRowSelectionReplayOptions(args, options); if (options.MaxJsonBytes.HasValue) AddReplayValueOption(args, "--max-json-bytes", options.MaxJsonBytes.Value.ToString(CultureInfo.InvariantCulture)); if (options.ShowExcluded) @@ -1069,7 +1069,6 @@ private static NdjsonStreamWriteResult WriteRecipeSearchResultRows( string recipeName, IReadOnlyList<SearchRecipeQueryResultJsonResult> queryResults, int totalCount, - bool limitTruncated, QueryCommandOptions options, JsonSerializerOptions ndjsonOptions) { @@ -1086,6 +1085,13 @@ private static NdjsonStreamWriteResult WriteRecipeSearchResultRows( } } + var limitTruncated = queryResults.Any(query => query.Truncated); + var selectionReason = queryResults + .Select(query => query.SelectionReason) + .FirstOrDefault(reason => reason != null); + var selectionOmittedCount = queryResults + .Where(query => query.SelectionOmittedCount.HasValue) + .Sum(query => query.SelectionOmittedCount!.Value); return WriteNdjsonStream( records, totalCount, @@ -1096,7 +1102,9 @@ private static NdjsonStreamWriteResult WriteRecipeSearchResultRows( limitTruncated, "Increase --limit or --total-limit, select one recipe query, or narrow the recipe scope.", totalCountAuthoritative: false, - truncationReason: limitTruncated ? "limit" : null); + truncationReason: limitTruncated ? "limit" : null, + selectionReason: selectionReason, + selectionOmittedCount: selectionReason != null ? selectionOmittedCount : null); } private static JsonObject BuildRecipeSearchResultRow( @@ -1324,6 +1332,8 @@ private static int RunSearchIssueDrafts( rows.Count, rows.Count, 0, + null, + null, options.Limit, 0, BuildSearchRecipeTopFiles(rows), @@ -1381,7 +1391,7 @@ private static List<SearchRecipeQueryResultJsonResult> CollectSearchRecipeQueryR var guardFilters = BuildSearchRecipeGuardFilters(options, recipeQuery); var results = reader.Search( recipeQuery.Query, - FetchLimitForSearchEnvelope(resultLimit), + GetSearchRecipeFetchLimit(options, resultLimit), options.Lang, false, queryScope.PathPatterns, @@ -1400,12 +1410,13 @@ private static List<SearchRecipeQueryResultJsonResult> CollectSearchRecipeQueryR resultRanking: GetSearchRecipeResultRanking(recipeQuery.ResultRanking, resultLimit)); results = ApplySearchRecipeFileRejectQueries(reader, results, options, recipeQuery); var rows = BuildSearchDisplayRows(results, options, exact, recipeQuery.Query, rawFtsOverride: false, recipeQuery: recipeQuery); - var availableCount = rows.Count; - var truncated = TrimSearchRowsToRequestedLimit(rows, resultLimit); + var outputSelection = ApplySearchOutputSelection(rows, options, resultLimit); + rows = outputSelection.Rows; ApplySearchRecipeAuditClassifications(recipeQuery, rows); - var minimumOmitted = truncated ? Math.Max(1, availableCount - rows.Count) : 0; + var minimumOmitted = Math.Max(0, outputSelection.OriginalCount - rows.Count); + var selectionReason = GetSearchRecipeSelectionReason(outputSelection); total += rows.Count; - minimumMatchedTotal += rows.Count + minimumOmitted; + minimumMatchedTotal += outputSelection.OriginalCount; queryResults.Add(new SearchRecipeQueryResultJsonResult( recipeQuery.Name, recipeQuery.Query, @@ -1428,13 +1439,20 @@ private static List<SearchRecipeQueryResultJsonResult> CollectSearchRecipeQueryR BuildSearchRecipeClassifierCounts(rows), rows.Count, rows.Count, - rows.Count + minimumOmitted, + outputSelection.OriginalCount, minimumOmitted, + selectionReason, + selectionReason != null ? outputSelection.SelectionOmittedCount : null, resultLimit, minimumOmitted, BuildSearchRecipeTopFiles(rows), - truncated, - truncated && rows.Count > 0 ? FormatSearchCursor(rows[^1].Result) : null, + outputSelection.LimitTruncated, + outputSelection.LimitTruncated + && !options.FirstPerFile + && !options.SampleSize.HasValue + && rows.Count > 0 + ? FormatSearchCursor(rows[^1].Result) + : null, rows.Select(row => row.Compact).ToList())); } @@ -1459,7 +1477,7 @@ private static List<SearchRecipeCompactQueryResultJsonResult> CollectSearchRecip var guardFilters = BuildSearchRecipeGuardFilters(options, recipeQuery); var results = reader.Search( recipeQuery.Query, - FetchLimitForSearchEnvelope(resultLimit), + GetSearchRecipeFetchLimit(options, resultLimit), options.Lang, false, queryScope.PathPatterns, @@ -1478,10 +1496,11 @@ private static List<SearchRecipeCompactQueryResultJsonResult> CollectSearchRecip resultRanking: GetSearchRecipeResultRanking(recipeQuery.ResultRanking, resultLimit)); results = ApplySearchRecipeFileRejectQueries(reader, results, options, recipeQuery); var rows = BuildSearchDisplayRows(results, options, exact, recipeQuery.Query, recipeQuery: recipeQuery); - var availableCount = rows.Count; - var truncated = TrimSearchRowsToRequestedLimit(rows, resultLimit); + var outputSelection = ApplySearchOutputSelection(rows, options, resultLimit); + rows = outputSelection.Rows; ApplySearchRecipeAuditClassifications(recipeQuery, rows); - var minimumOmitted = truncated ? Math.Max(1, availableCount - rows.Count) : 0; + var minimumOmitted = Math.Max(0, outputSelection.OriginalCount - rows.Count); + var selectionReason = GetSearchRecipeSelectionReason(outputSelection); total += rows.Count; queryResults.Add(new SearchRecipeCompactQueryResultJsonResult( recipeQuery.Name, @@ -1501,13 +1520,20 @@ private static List<SearchRecipeCompactQueryResultJsonResult> CollectSearchRecip BuildSearchRecipeClassifierCounts(rows), rows.Count, rows.Count, - rows.Count + minimumOmitted, + outputSelection.OriginalCount, minimumOmitted, + selectionReason, + selectionReason != null ? outputSelection.SelectionOmittedCount : null, resultLimit, minimumOmitted, BuildSearchRecipeTopFiles(rows), - truncated, - truncated && rows.Count > 0 ? FormatSearchCursor(rows[^1].Result) : null, + outputSelection.LimitTruncated, + outputSelection.LimitTruncated + && !options.FirstPerFile + && !options.SampleSize.HasValue + && rows.Count > 0 + ? FormatSearchCursor(rows[^1].Result) + : null, rows.Select(row => new SearchRecipeCompactResultJsonResult( row.Result.Path, row.Result.Lang, @@ -1523,6 +1549,20 @@ private static List<SearchRecipeCompactQueryResultJsonResult> CollectSearchRecip return queryResults; } + private static string? GetSearchRecipeSelectionReason(SearchOutputSelection selection) + => selection.SelectionOmittedCount > 0 + && selection.TruncationReason is "first_per_file" or "sample" + ? selection.TruncationReason + : null; + + private static int GetSearchRecipeFetchLimit(QueryCommandOptions options, int resultLimit) + { + var selectionTarget = resultLimit > 0 && options.SampleSize.HasValue + ? Math.Max(resultLimit, options.SampleSize.Value) + : resultLimit; + return FetchLimitForSearchEnvelope(selectionTarget); + } + private static List<SearchRecipeCountQueryJsonResult> CountSearchRecipeQueryResults( DbReader reader, IReadOnlyList<SearchAuditRecipeQuery> recipeQueries, @@ -1893,7 +1933,9 @@ private static SearchRecipeRunSummaryJsonResult BuildSearchRecipeRunSummary( queryResults.Sum(query => query.MinimumOmittedResultCount), BuildSearchRecipeQueryFreshness(queryResults), queryResults.Any(query => query.Truncated && !string.IsNullOrWhiteSpace(query.NextCursor)), - "When a query is truncated, rerun a single child query with --recipe <recipe>/<query> --cursor <next_cursor> to page the next result set."); + BuildSearchRecipeCursoringHint( + queryResults.Any(query => query.Truncated), + queryResults.Any(query => query.Truncated && !string.IsNullOrWhiteSpace(query.NextCursor)))); private static SearchRecipeRunSummaryJsonResult BuildSearchRecipeRunSummary( IReadOnlyList<SearchRecipeCompactQueryResultJsonResult> queryResults, @@ -1908,7 +1950,16 @@ private static SearchRecipeRunSummaryJsonResult BuildSearchRecipeRunSummary( queryResults.Sum(query => query.MinimumOmittedResultCount), BuildSearchRecipeQueryFreshness(queryResults), queryResults.Any(query => query.Truncated && !string.IsNullOrWhiteSpace(query.NextCursor)), - "When a query is truncated, rerun a single child query with --recipe <recipe>/<query> --cursor <next_cursor> to page the next result set."); + BuildSearchRecipeCursoringHint( + queryResults.Any(query => query.Truncated), + queryResults.Any(query => query.Truncated && !string.IsNullOrWhiteSpace(query.NextCursor)))); + + private static string BuildSearchRecipeCursoringHint(bool hasTruncatedQuery, bool cursoringAvailable) + => cursoringAvailable + ? "When a query is truncated, rerun a single child query with --recipe <recipe>/<query> --cursor <next_cursor> to page the next result set." + : hasTruncatedQuery + ? "Continuation cursors are unavailable for the selected rows; increase --limit or --total-limit and rerun." + : "No query is truncated, so no continuation cursor is needed."; private static SearchRecipeQueryFreshnessJsonResult BuildSearchRecipeQueryFreshness(IReadOnlyList<SearchRecipeQueryResultJsonResult> queryResults) => BuildSearchRecipeQueryFreshness(queryResults.Select(query => (query.Name, query.MinimumMatchedCount))); @@ -2274,6 +2325,8 @@ private static SearchIssueDraftJsonResult ToSearchIssueDraft( queryResult.Count, queryResult.ResultLimit, queryResult.OmittedCount, + queryResult.SelectionReason, + queryResult.SelectionOmittedCount, queryResult.MinimumOmittedResultCount, queryResult.Truncated, queryResult.NextCursor), @@ -2329,6 +2382,8 @@ private static SearchIssueDraftJsonResult ToAdHocSearchIssueDraft( queryResult.Count, queryResult.ResultLimit, queryResult.OmittedCount, + null, + null, queryResult.MinimumOmittedResultCount, queryResult.Truncated, queryResult.NextCursor), @@ -2561,6 +2616,11 @@ private static string BuildSearchIssueDraftBody( sb.AppendLine($"- result_count: `{queryResult.Count}`"); sb.AppendLine($"- result_limit: `{queryResult.ResultLimit}`"); sb.AppendLine($"- omitted_count: `{queryResult.OmittedCount}`"); + if (!string.IsNullOrWhiteSpace(queryResult.SelectionReason)) + { + sb.AppendLine($"- selection_reason: `{queryResult.SelectionReason}`"); + sb.AppendLine($"- selection_omitted_count: `{queryResult.SelectionOmittedCount.GetValueOrDefault()}`"); + } sb.AppendLine($"- minimum_omitted_result_count: `{queryResult.MinimumOmittedResultCount}`"); sb.AppendLine($"- exact_substring: `{queryResult.ExactSubstring.ToString().ToLowerInvariant()}`"); return sb.ToString().TrimEnd(); @@ -2674,6 +2734,7 @@ private static string BuildSearchRecipeReplayCommand(SearchAuditRecipe recipe, Q args.Add("--exact-substring"); if (options.TokenBoundary) args.Add("--token-boundary"); + AddSearchRecipeRowSelectionReplayOptions(args, options); foreach (var guardFilter in options.GuardFilters) AddReplayValueOption(args, BuildSearchGuardReplayOptionName(guardFilter), guardFilter.Query); if (options.GuardFilters.Count > 0 && options.GuardWindow != DbReader.DefaultSearchGuardWindow) @@ -2709,6 +2770,14 @@ private static string BuildSearchRecipeReplayCommand(SearchAuditRecipe recipe, Q return string.Join(" ", args.Select(QuoteReplayShellArg)); } + private static void AddSearchRecipeRowSelectionReplayOptions(List<string> args, QueryCommandOptions options) + { + if (options.FirstPerFile) + args.Add("--first-per-file"); + if (options.SampleSize.HasValue) + AddReplayValueOption(args, "--sample", options.SampleSize.Value.ToString(CultureInfo.InvariantCulture)); + } + private static void AddReplayValueOption(List<string> args, string optionName, string? value) { if (string.IsNullOrWhiteSpace(value)) diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs index 3dd1c9585..74010a5f6 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs @@ -369,6 +369,12 @@ private static bool IsSupportedSearchAggregationValue(string value) => NormalizeSearchAggregationKey(value) is "file" or "symbol" or "origin" or "return_type" or "subsystem"; private static SearchOutputSelection ApplySearchOutputSelection(List<SearchDisplayRow> rows, QueryCommandOptions options) + => ApplySearchOutputSelection(rows, options, options.Limit); + + private static SearchOutputSelection ApplySearchOutputSelection( + List<SearchDisplayRow> rows, + QueryCommandOptions options, + int limit) { var originalCount = rows.Count; rows = ApplySearchPostSelectors( @@ -377,9 +383,9 @@ private static SearchOutputSelection ApplySearchOutputSelection(List<SearchDispl out var firstPerFileTruncated, out var sampleTruncated); var postSelectionCount = rows.Count; - var limitTruncated = rows.Count > options.Limit; + var limitTruncated = rows.Count > limit; if (limitTruncated) - rows = rows.Take(options.Limit).ToList(); + rows = rows.Take(limit).ToList(); var truncationReason = firstPerFileTruncated ? "first_per_file" diff --git a/src/CodeIndex/Cli/SearchAuditRecipes.cs b/src/CodeIndex/Cli/SearchAuditRecipes.cs index 64396a583..e3b914b54 100644 --- a/src/CodeIndex/Cli/SearchAuditRecipes.cs +++ b/src/CodeIndex/Cli/SearchAuditRecipes.cs @@ -4390,6 +4390,12 @@ internal sealed record SearchRecipeQueryResultJsonResult( [property: JsonPropertyName("emitted_count")] int EmittedCount, [property: JsonPropertyName("minimum_matched_count")] int MinimumMatchedCount, [property: JsonPropertyName("omitted_count")] int OmittedCount, + [property: JsonPropertyName("selection_reason")] + [property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + string? SelectionReason, + [property: JsonPropertyName("selection_omitted_count")] + [property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + int? SelectionOmittedCount, [property: JsonPropertyName("result_limit")] int ResultLimit, [property: JsonPropertyName("minimum_omitted_result_count")] int MinimumOmittedResultCount, [property: JsonPropertyName("top_files")] List<SearchRecipeTopFileJsonResult> TopFiles, @@ -4511,6 +4517,12 @@ internal sealed record SearchRecipeCompactQueryResultJsonResult( [property: JsonPropertyName("emitted_count")] int EmittedCount, [property: JsonPropertyName("minimum_matched_count")] int MinimumMatchedCount, [property: JsonPropertyName("omitted_count")] int OmittedCount, + [property: JsonPropertyName("selection_reason")] + [property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + string? SelectionReason, + [property: JsonPropertyName("selection_omitted_count")] + [property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + int? SelectionOmittedCount, [property: JsonPropertyName("result_limit")] int ResultLimit, [property: JsonPropertyName("minimum_omitted_result_count")] int MinimumOmittedResultCount, [property: JsonPropertyName("top_files")] List<SearchRecipeTopFileJsonResult> TopFiles, @@ -4580,6 +4592,12 @@ internal sealed record SearchIssueDraftSourceJsonResult( [property: JsonPropertyName("result_count")] int ResultCount, [property: JsonPropertyName("result_limit")] int ResultLimit, [property: JsonPropertyName("omitted_count")] int OmittedCount, + [property: JsonPropertyName("selection_reason")] + [property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + string? SelectionReason, + [property: JsonPropertyName("selection_omitted_count")] + [property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + int? SelectionOmittedCount, [property: JsonPropertyName("minimum_omitted_result_count")] int MinimumOmittedResultCount, [property: JsonPropertyName("truncated")] bool Truncated, [property: JsonPropertyName("next_cursor")] diff --git a/tests/CodeIndex.Tests/ConsoleUiTests.cs b/tests/CodeIndex.Tests/ConsoleUiTests.cs index 935cd123d..2c566eff2 100644 --- a/tests/CodeIndex.Tests/ConsoleUiTests.cs +++ b/tests/CodeIndex.Tests/ConsoleUiTests.cs @@ -253,7 +253,7 @@ public void PrintCommandUsage_SearchIncludesAuditOutputControlFlags_Issue3893() } [Fact] - public void PrintCommandUsage_AuditAndRecipesExposeLowOutputControls_Issues4064_4118() + public void PrintCommandUsage_AuditAndRecipesExposeLowOutputControls_Issues4064_4118_4759() { var (_, auditOutput, _) = ConsoleCapture.Capture(() => { @@ -270,7 +270,8 @@ public void PrintCommandUsage_AuditAndRecipesExposeLowOutputControls_Issues4064_ { "--results-only", "--search-fields", - "--per-file-limit", + "--first-per-file", + "--sample", "--total-limit", "--summary-only", "--max-json-bytes", diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerSearchTests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerSearchTests.cs index 9b5a11af8..7f1ba29f7 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerSearchTests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerSearchTests.cs @@ -7131,7 +7131,7 @@ public void RunSearch_RecipeCursorSkipsDedupedEnvelopeRows_Issue3392() { var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); TestProjectHelper.InsertIndexedFile(dbPath, "src/overlap.cs", "csharp", "Console.WriteLine(ex.Message);\n"); - ReplaceChunks( + ReplaceIndexedChunks( dbPath, "src/overlap.cs", new ChunkRecord { ChunkIndex = 0, StartLine = 1, EndLine = 20, Content = "Console.WriteLine(ex.Message);\n" }, @@ -7168,19 +7168,209 @@ public void RunSearch_RecipeCursorSkipsDedupedEnvelopeRows_Issue3392() { TestProjectHelper.DeleteDirectory(projectRoot); } + } - static void ReplaceChunks(string dbPath, string path, params ChunkRecord[] chunks) + [Fact] + public void RunSearch_RecipeRowSelectionAppliesToJsonCompactAndNdjson_Issue4759() + { + var projectRoot = TestProjectHelper.CreateTempProject("cdidx_search_recipe_selection"); + try { - using var db = new DbContext(DbOpenIntent.WriteIndex, dbPath); - using var cmd = db.Connection.CreateCommand(); - cmd.CommandText = "SELECT id FROM files WHERE path = @path"; - cmd.Parameters.AddWithValue("@path", path); - var fileId = (long)(cmd.ExecuteScalar() ?? throw new InvalidOperationException($"Missing indexed file {path}.")); - var writer = new DbWriter(db.Connection); - writer.DeleteFileData(fileId); - foreach (var chunk in chunks) - chunk.FileId = fileId; - writer.InsertChunks(chunks); + var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); + TestProjectHelper.InsertIndexedFile(dbPath, "src/selection-a.cs", "csharp", "Console.WriteLine(ex.Message);\n"); + TestProjectHelper.InsertIndexedFile(dbPath, "src/selection-b.cs", "csharp", "Console.WriteLine(ex.Message);\n"); + ReplaceIndexedChunks( + dbPath, + "src/selection-a.cs", + new ChunkRecord { ChunkIndex = 0, StartLine = 1, EndLine = 5, Content = "Console.WriteLine(ex.Message); // a1\n" }, + new ChunkRecord { ChunkIndex = 1, StartLine = 10, EndLine = 15, Content = "Console.WriteLine(ex.Message); // a2\n" }, + new ChunkRecord { ChunkIndex = 2, StartLine = 20, EndLine = 25, Content = "Console.WriteLine(ex.Message); // a3\n" }); + ReplaceIndexedChunks( + dbPath, + "src/selection-b.cs", + new ChunkRecord { ChunkIndex = 0, StartLine = 1, EndLine = 5, Content = "Console.WriteLine(ex.Message); // b1\n" }, + new ChunkRecord { ChunkIndex = 1, StartLine = 10, EndLine = 15, Content = "Console.WriteLine(ex.Message); // b2\n" }); + + var (sampleExitCode, sampleStdout, sampleStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( + ["--recipe", "risky-code/raw-diagnostic-echo", "--db", dbPath, "--json", "--sample", "1", "--limit", "20"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, sampleExitCode); + Assert.Equal(string.Empty, sampleStderr); + using var sampleDocument = ParseJsonOutput(sampleStdout); + Assert.Equal(1, sampleDocument.RootElement.GetProperty("result_count").GetInt32()); + var sampleSummary = sampleDocument.RootElement.GetProperty("summary"); + Assert.Equal(1, sampleSummary.GetProperty("emitted_result_count").GetInt32()); + Assert.Equal(0, sampleSummary.GetProperty("truncated_query_count").GetInt32()); + Assert.Equal(4, sampleSummary.GetProperty("minimum_omitted_result_count").GetInt32()); + Assert.False(sampleSummary.GetProperty("cursoring_available").GetBoolean()); + var sampleQuery = Assert.Single(sampleDocument.RootElement.GetProperty("queries").EnumerateArray()); + Assert.Equal(1, sampleQuery.GetProperty("count").GetInt32()); + Assert.Equal(1, sampleQuery.GetProperty("emitted_count").GetInt32()); + Assert.Equal(5, sampleQuery.GetProperty("minimum_matched_count").GetInt32()); + Assert.Equal(4, sampleQuery.GetProperty("omitted_count").GetInt32()); + Assert.Equal("sample", sampleQuery.GetProperty("selection_reason").GetString()); + Assert.Equal(4, sampleQuery.GetProperty("selection_omitted_count").GetInt32()); + Assert.False(sampleQuery.GetProperty("truncated").GetBoolean()); + Assert.Equal(JsonValueKind.Null, sampleQuery.GetProperty("next_cursor").ValueKind); + Assert.Single(sampleQuery.GetProperty("results").EnumerateArray()); + + var (firstPerFileExitCode, firstPerFileStdout, firstPerFileStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( + ["--recipe", "risky-code/raw-diagnostic-echo", "--db", dbPath, "--json", "--first-per-file", "--limit", "20"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, firstPerFileExitCode); + Assert.Equal(string.Empty, firstPerFileStderr); + using var firstPerFileDocument = ParseJsonOutput(firstPerFileStdout); + var firstPerFileQuery = Assert.Single(firstPerFileDocument.RootElement.GetProperty("queries").EnumerateArray()); + Assert.Equal(2, firstPerFileQuery.GetProperty("count").GetInt32()); + Assert.Equal(5, firstPerFileQuery.GetProperty("minimum_matched_count").GetInt32()); + Assert.Equal(3, firstPerFileQuery.GetProperty("omitted_count").GetInt32()); + Assert.Equal("first_per_file", firstPerFileQuery.GetProperty("selection_reason").GetString()); + Assert.Equal(3, firstPerFileQuery.GetProperty("selection_omitted_count").GetInt32()); + Assert.False(firstPerFileQuery.GetProperty("truncated").GetBoolean()); + Assert.Equal( + 2, + firstPerFileQuery.GetProperty("results") + .EnumerateArray() + .Select(result => result.GetProperty("path").GetString()) + .Distinct(StringComparer.Ordinal) + .Count()); + + var (compactExitCode, compactStdout, compactStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( + ["--recipe", "risky-code/raw-diagnostic-echo", "--db", dbPath, "--format", "compact", "--sample", "1", "--limit", "20"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, compactExitCode); + Assert.Equal(string.Empty, compactStderr); + using var compactDocument = ParseJsonOutput(compactStdout); + var compactQuery = Assert.Single(compactDocument.RootElement.GetProperty("queries").EnumerateArray()); + Assert.Equal(1, compactQuery.GetProperty("count").GetInt32()); + Assert.Equal(5, compactQuery.GetProperty("minimum_matched_count").GetInt32()); + Assert.Equal("sample", compactQuery.GetProperty("selection_reason").GetString()); + Assert.Equal(4, compactQuery.GetProperty("selection_omitted_count").GetInt32()); + Assert.False(compactQuery.GetProperty("truncated").GetBoolean()); + var compactNextCommand = Assert.Single(compactDocument.RootElement.GetProperty("next_commands").EnumerateArray()).GetString()!; + Assert.Contains("--sample 1", compactNextCommand, StringComparison.Ordinal); + + var (pagedExitCode, pagedStdout, pagedStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( + ["--recipe", "risky-code/raw-diagnostic-echo", "--db", dbPath, "--format", "compact", "--first-per-file", "--limit", "1"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, pagedExitCode); + Assert.Equal(string.Empty, pagedStderr); + using var pagedDocument = ParseJsonOutput(pagedStdout); + var pagedQuery = Assert.Single(pagedDocument.RootElement.GetProperty("queries").EnumerateArray()); + Assert.True(pagedQuery.GetProperty("truncated").GetBoolean()); + Assert.Equal(JsonValueKind.Null, pagedQuery.GetProperty("next_cursor").ValueKind); + Assert.False(pagedDocument.RootElement.GetProperty("summary").GetProperty("cursoring_available").GetBoolean()); + Assert.Contains( + "increase --limit or --total-limit", + pagedDocument.RootElement.GetProperty("summary").GetProperty("cursoring_hint").GetString(), + StringComparison.Ordinal); + var pagedNextCommand = Assert.Single(pagedDocument.RootElement.GetProperty("next_commands").EnumerateArray()).GetString()!; + Assert.Contains("--first-per-file", pagedNextCommand, StringComparison.Ordinal); + Assert.DoesNotContain("--cursor", pagedNextCommand, StringComparison.Ordinal); + + var openIssuesPath = Path.Combine(projectRoot, "open-issues.json"); + File.WriteAllText(openIssuesPath, "[]"); + var (draftExitCode, draftStdout, draftStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( + ["--recipe", "risky-code/raw-diagnostic-echo", "--db", dbPath, "--format", "issue-drafts", "--sample", "1", "--limit", "20", "--open-issues", openIssuesPath], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, draftExitCode); + Assert.Equal(string.Empty, draftStderr); + using var draftDocument = ParseJsonOutput(draftStdout); + var draft = Assert.Single(draftDocument.RootElement.GetProperty("drafts").EnumerateArray()); + Assert.Contains("--sample 1", draft.GetProperty("body").GetString(), StringComparison.Ordinal); + Assert.Equal("sample", draft.GetProperty("source").GetProperty("selection_reason").GetString()); + Assert.Equal(4, draft.GetProperty("source").GetProperty("selection_omitted_count").GetInt32()); + + var (ndjsonExitCode, ndjsonStdout, ndjsonStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( + ["--recipe", "risky-code/raw-diagnostic-echo", "--db", dbPath, "--json=ndjson", "--sample", "1", "--limit", "20"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, ndjsonExitCode); + Assert.Equal(string.Empty, ndjsonStderr); + var ndjsonLines = ndjsonStdout.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); + Assert.Equal(2, ndjsonLines.Length); + using var ndjsonRow = JsonDocument.Parse(ndjsonLines[0]); + using var ndjsonTerminal = JsonDocument.Parse(ndjsonLines[1]); + Assert.Equal("src/selection-a.cs", ndjsonRow.RootElement.GetProperty("path").GetString()); + Assert.Equal(1, ndjsonTerminal.RootElement.GetProperty("count").GetInt32()); + Assert.Equal(5, ndjsonTerminal.RootElement.GetProperty("total_count_lower_bound").GetInt32()); + Assert.Equal("sample", ndjsonTerminal.RootElement.GetProperty("selection_reason").GetString()); + Assert.Equal(4, ndjsonTerminal.RootElement.GetProperty("selection_omitted_count").GetInt32()); + Assert.False(ndjsonTerminal.RootElement.GetProperty("truncated").GetBoolean()); + Assert.False(ndjsonTerminal.RootElement.GetProperty("has_more").GetBoolean()); + } + finally + { + TestProjectHelper.DeleteDirectory(projectRoot); + } + } + + [Fact] + public void RunSearch_RecipeSampleFetchesEnoughCandidatesBeforeLimit_Issue4759() + { + var projectRoot = TestProjectHelper.CreateTempProject("cdidx_search_recipe_sample_candidates"); + try + { + var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); + TestProjectHelper.InsertIndexedFile(dbPath, "src/sample.cs", "csharp", "Console.WriteLine(ex.Message);\n"); + ReplaceIndexedChunks( + dbPath, + "src/sample.cs", + [.. Enumerable.Range(0, 201).Select(index => new ChunkRecord + { + ChunkIndex = index, + StartLine = (index * 10) + 1, + EndLine = (index * 10) + 5, + Content = $"Console.WriteLine(ex.Message); // sample {index}\n" + })]); + + var (exitCode, stdout, stderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( + ["--recipe", "risky-code/raw-diagnostic-echo", "--db", dbPath, "--json", "--sample", "200", "--limit", "2"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, exitCode); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var query = Assert.Single(document.RootElement.GetProperty("queries").EnumerateArray()); + Assert.Equal(2, query.GetProperty("count").GetInt32()); + Assert.Equal(201, query.GetProperty("minimum_matched_count").GetInt32()); + Assert.Equal("sample", query.GetProperty("selection_reason").GetString()); + Assert.Equal(1, query.GetProperty("selection_omitted_count").GetInt32()); + Assert.Equal(199, query.GetProperty("omitted_count").GetInt32()); + Assert.True(query.GetProperty("truncated").GetBoolean()); + Assert.Equal(JsonValueKind.Null, query.GetProperty("next_cursor").ValueKind); + } + finally + { + TestProjectHelper.DeleteDirectory(projectRoot); + } + } + + [Fact] + public void RunSearch_RecipeRejectsRowSelectionForNonRowOutputs_Issue4759() + { + var cases = new[] + { + (Args: new[] { "--recipe", "risky-code/raw-diagnostic-echo", "--format", "count", "--sample", "1" }, Expected: "recipe row-selection controls"), + (Args: new[] { "--recipe", "risky-code/raw-diagnostic-echo", "--count-by", "path", "--first-per-file" }, Expected: "recipe row-selection controls"), + (Args: new[] { "--recipe", "risky-code/raw-diagnostic-echo", "--format", "compact", "--summary-only", "--sample", "1" }, Expected: "recipe row-selection controls"), + (Args: new[] { "--recipe", "risky-code/raw-diagnostic-echo", "--per-file-limit", "1" }, Expected: "--per-file-limit is not supported with --recipe"), + (Args: new[] { "--recipe", "risky-code/raw-diagnostic-echo", "--cursor", "0:1:1", "--sample", "1" }, Expected: "cannot be combined with --cursor"), + (Args: new[] { "--recipe", "risky-code/raw-diagnostic-echo", "--cursor", "0:1:1", "--first-per-file" }, Expected: "cannot be combined with --cursor"), + }; + + foreach (var testCase in cases) + { + var (exitCode, stdout, stderr) = CaptureConsole(() => QueryCommandRunner.RunSearch(testCase.Args, _jsonOptions)); + + Assert.Equal(CommandExitCodes.UsageError, exitCode); + Assert.Equal(string.Empty, stdout); + Assert.Contains(testCase.Expected, stderr); } } @@ -12672,6 +12862,20 @@ private static string BuildOverlappingChunkContent(int startLine, int endLine, s .Select(line => line == 75 ? targetLine : $"// filler {line}")); } + private static void ReplaceIndexedChunks(string dbPath, string path, params ChunkRecord[] chunks) + { + using var db = new DbContext(DbOpenIntent.WriteIndex, dbPath); + using var cmd = db.Connection.CreateCommand(); + cmd.CommandText = "SELECT id FROM files WHERE path = @path"; + cmd.Parameters.AddWithValue("@path", path); + var fileId = (long)(cmd.ExecuteScalar() ?? throw new InvalidOperationException($"Missing indexed file {path}.")); + var writer = new DbWriter(db.Connection); + writer.DeleteFileData(fileId); + foreach (var chunk in chunks) + chunk.FileId = fileId; + writer.InsertChunks(chunks); + } + private sealed class IssueDraftRepositoryLabelsHandler : HttpMessageHandler { internal List<string> RequestUris { get; } = [];