diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 3bff3b8c5..b24c47251 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -359,6 +359,20 @@ previews are sanitized and bounded. JSON failures write the envelope to stdout and leave stderr empty. Human failures write the matching coded `Error`, `Hint`, and `Usage` lines to stderr and leave stdout empty. +Response-budget preflight failures use `E028_RESPONSE_BUDGET_TOO_SMALL` / +`response_budget` through `CommandErrorWriter.WriteResponseBudgetError`. +They add `requested_bytes`, `effective_bytes`, `minimum_required_bytes`, +`minimum_required_bytes_known`, stable unavailable / uncertainty reason fields, +and a machine-readable `retry` object. `--max-json-bytes` caps normal payloads, +not this diagnostic: object and NDJSON preflights must emit one complete +parseable error object on stdout with empty stderr even when the diagnostic +exceeds the requested cap. A materialized stable payload reports a retryable +minimum; runtime-dependent envelopes mark the minimum uncertain and recommend +headroom. `retry.action=increase_max_json_bytes` includes `recommended_bytes`. +When the minimum exceeds the effective 16 MiB ceiling, +`retry.action=reduce_response_size` instead leaves `option` and +`recommended_bytes` null and reports `maximum_effective_bytes`. + Unsupported `search --regex` / `search --all` diagnostics add a command-specific, display-only `find` recovery contract. Preserve exact normalized argument semantics in typed `alternative_command.argv`, render both POSIX-shell and @@ -381,6 +395,7 @@ Other commands load config before those consumers run; malformed config in JSON | Invalid configuration | 1 | `E024_CONFIG_INVALID` | `configuration` | | Hook platform or filesystem failure | 9 | `E025_HOOK_OPERATION_FAILED` | `platform` | | Hooks outside a Git repository | 2 | `E026_NOT_GIT_REPOSITORY` | `not_found` | +| JSON response budget too small | 1 or underlying command-specific | `E028_RESPONSE_BUDGET_TOO_SMALL` | `response_budget` | | Other recoverable command failure | command-specific | `E023_COMMAND_FAILED` | stable writer classification | ### Process launch policy @@ -1738,10 +1753,10 @@ access. | Compact location envelope | CLI `--format compact` location output uses a versioned envelope with `api_version`, returned `count`, conservative limit-based `truncated` / `truncation` metadata, applied `query_context`, and lightweight `results` rows. | | Grouped search totals | `search --format grouped` derives `total_matches` / `matched_count`, `total_groups`, and `total_files` from the complete bounded query rather than the displayed page. `grouped_match_count` counts rows supplied to returned groups, `emitted_match_count` counts rows left after per-file grouping limits, and `omitted_match_count`, `truncated`, `has_more`, and `continuation_action` describe incomplete output. | | Bounded high-volume responses | `search`, `definition`, `find`, `status`, `hotspots`, `references`, `callers`, `callees`, `symbols`, `files`, `languages`, `impact`, and `map` accept shared bounded-response controls where their schema exposes them. Newly emitted opaque `--cursor ` values bind the offset to the command/query/filter selection and index generation; legacy `response:v1::` cursors remain accepted for transition. Reuse with changed selection or generation fails with restart-required guidance. `search --format compact`, `symbols --format compact`, and `files --format compact` auto-select the bounded contract, while `search --json=array --json-envelope` provides the opt-in array envelope and `languages --json` selects it when paging or `--max-json-bytes` is requested. Existing compact roots and location rows remain compatible while adding shared metadata. Metadata reports `returned_count`, authoritative `total_count` where available, `omitted_count`, `remaining_count`, `cursor_offset`, `page_limit`, `has_more`, `next_cursor`, `result_stable_at`, `pagination_window_limit`, and `pagination_window_exhausted`. The safety window is 10,000 rows; exhaustion suppresses `next_cursor` rather than returning a cursor that the next request would reject. Pageable commands pass the cursor offset into their database/scan layer instead of serializing an `offset + limit` prefix. `find --all` partial scans encode the next path/line in the opaque cursor so replay continues after the last scanned line. `hotspots` and `impact` page their active primary nested collection as `results`, identify it with `metadata.primary_collection`, and retain scalar/container evidence in `metadata.response_context`; dotted fields such as `callers.path,callers.depth` select that collection and project its rows. The final newline is included in `--max-json-bytes`, and trailing whole rows are removed until the complete envelope fits. `definition` remains metadata-only by default; explicit `--body` content is retained for `body`, `body_content`, or `all`, and suppressed when the projection excludes it. `map --sections` remains its section-level projection, while dotted bounded fields page a selected array section with section-specific totals and scalar projections skip unused ranked arrays. | -| Bounded outline responses | `outline` opts into the shared bounded-response contract only when `--max-json-bytes` is present. The wrapper extracts complete projected symbol rows, preserves hierarchy and deterministic order, reports authoritative returned / total / omitted counts, includes the final newline in its UTF-8 measurement, and emits a bound `response:v2` continuation cursor. An undersized minimum envelope produces typed `E010_USAGE_ERROR` diagnostics with empty stdout. Uncapped outline JSON retains its existing root shape and outline cursor contract. | -| Bounded unused responses | `unused` opts into the shared bounded-response contract only when `--max-json-bytes` is present. The wrapper extracts the canonical `symbols` rows, applies the cursor offset in the unused query layer, recomputes returned bucket / confidence / contract-domain counts after byte trimming, and includes an optional `by_bucket` view in the same whole-response UTF-8 budget. Compact mode projects smaller audit rows. Continuation cursors bind the effective audit filters, bucket mode, ordering, and index generation; an undersized one-row envelope returns typed `E010_USAGE_ERROR` diagnostics with empty stdout. Uncapped JSON, compact summaries, and the legacy unused cursor remain unchanged. | +| Bounded outline responses | `outline` opts into the shared bounded-response contract only when `--max-json-bytes` is present. The wrapper extracts complete projected symbol rows, preserves hierarchy and deterministic order, reports authoritative returned / total / omitted counts, includes the final newline in its UTF-8 measurement, and emits a bound `response:v2` continuation cursor. An undersized minimum envelope produces one typed `E028_RESPONSE_BUDGET_TOO_SMALL` object on stdout with empty stderr and actionable byte fields. Uncapped outline JSON retains its existing root shape and outline cursor contract. | +| Bounded unused responses | `unused` opts into the shared bounded-response contract only when `--max-json-bytes` is present. The wrapper extracts the canonical `symbols` rows, applies the cursor offset in the unused query layer, recomputes returned bucket / confidence / contract-domain counts after byte trimming, and includes an optional `by_bucket` view in the same whole-response UTF-8 budget. Compact mode projects smaller audit rows. Continuation cursors bind the effective audit filters, bucket mode, ordering, and index generation; an undersized one-row envelope returns one typed `E028_RESPONSE_BUDGET_TOO_SMALL` object on stdout with empty stderr and actionable byte fields. Uncapped JSON, compact summaries, and the legacy unused cursor remain unchanged. | | MCP outline pages | MCP `outline` routes `fields`, `sort`, `limit`, and `cursor` through `QueryCommandRunner.BuildOutlinePage`, so its projection aliases, derived sort fields, stable tie-breakers, `page:v1` query fingerprint, and generation validation remain the CLI outline contract rather than a second MCP-specific implementation. The default page is 100 rows and the MCP-wide maximum is 200. `maxBytes` measures the fully enriched serialized `structuredContent`; a binary search rebuilds the page with fewer complete rows and therefore regenerates `next_cursor` from the actual returned count. A budget that cannot hold metadata plus one row fails instead of returning a zero-progress cursor. Default MCP symbol serialization remains backward-compatible, while explicit projection fields use the CLI snake_case names. | -| Bounded response edge cases | `impact` applies the cursor offset only to the selected nested collection so definition pages do not repeat or alter caller/fallback mode. Plain `map --compact` preserves its established section arrays and truncation payload; a collection projection is rejected when `--summary-only` or an excluding `--sections` filter would remove it. Explicit definition body fields override compact defaults. Profile and verbose records are moved into `metadata.stream_control_records`, and parser/capture failures emit an error envelope only when it fits the active hard byte cap. | +| Bounded response edge cases | `impact` applies the cursor offset only to the selected nested collection so definition pages do not repeat or alter caller/fallback mode. Plain `map --compact` preserves its established section arrays and truncation payload; a collection projection is rejected when `--summary-only` or an excluding `--sections` filter would remove it. Explicit definition body fields override compact defaults. Profile and verbose records are moved into `metadata.stream_control_records`. Parser/capture error envelopes use the normal cap when they fit; otherwise a complete `E028_RESPONSE_BUDGET_TOO_SMALL` diagnostic replaces them so machine output never becomes empty or malformed. | | `--count --json` envelope | Count-only JSON for `search`, `definition`, `references`, `callers`, `callees`, `symbols`, `files`, `find`, `impact`, and `unused` is a single automation-oriented object. It always includes `count`, applied `query_context`, freshness metadata (`indexed_file_count`, `indexed_at`, `freshness_available`), and trust flags `degraded` / `authoritative_count`; commands with matched-file totals also include `files` and the older `file_count` compatibility alias. `file_count` carries the same value as `files`, remains for compatibility, and is not scheduled for removal before the next major release. `unused --count --json` also includes `returned_bucket_counts`, `returned_contract_domain_counts`, and `summary.by_bucket` / `summary.by_confidence` / `summary.by_contract_domain`. `authoritative_count=false` means a readiness or graph/exact trust signal made the count non-authoritative, while the freshness fields describe the indexed snapshot used for the count. | | Search row selection | Row-producing plain-search and recipe paths share `ApplySearchOutputSelection`: `--first-per-file` and fixed-seed deterministic `--sample` run before the effective per-query / remaining total limit. Sample fetch envelopes are sized from at least the requested sample target. Aggregate/compact query DTOs, plain compact roots, run summaries, issue-draft source DTOs, NDJSON terminals, and bounded array-envelope stream terminals expose `source_total`, `selected_total`, `returned`, `selector_omitted_count`, and `limit_omitted_count`; `source_total_authoritative` / `source_total_lower_bound` distinguish complete populations from bounded observations. Guard filters, origin/facet post-filters, exhausted candidate windows, and recipe file-reject post-filters force lower-bound authority. Their ordered `selectors` entries preserve each stage's input/output/omission counts plus sample size, mode, and seed, while nullable `selection_reason` / `selection_omitted_count` remain compatibility summaries. Bounded plain-search selection is computed once and its selected page is reused by compact/envelope serialization. Search `query_context.row_selectors` records the applied selector configuration. Selection-only omission updates matched/omitted lower bounds without setting `truncated`, `has_more`, or `next_cursor`. When a later limit truncates selected rows, limit truncation remains visible but `next_cursor` is suppressed because raw database cursors cannot preserve selector state; incoming `--cursor` values are rejected with either selector for the same reason. Generated compact and issue-draft replay commands retain the selector. Count, aggregation, named-query, recipe-list, results-only, metadata-free array, unsupported formatted, and summary-only compact shapes reject `--first-per-file` / `--sample`, while every recipe shape rejects grouped-only `--per-file-limit`. | | Search selection edge cases | Issue-draft roots independently retain per-query `selection_accounting`, including zero-draft and exhausted-total-limit queries. Byte-bounded compact and array envelopes rewrite `returned` to the emitted row count while preserving logical `limit_omitted_count`; hard-cap omissions remain separate in `metadata.byte_limit_omitted_count`. | @@ -3806,6 +3821,20 @@ sanitization し、上限を適用してから merge します。JSON の失敗 出し、stderr を空に保ちます。human の失敗は対応する code 付き `Error`、`Hint`、 `Usage` を stderr に出し、stdout を空に保ちます。 +response-budget preflight failure は +`CommandErrorWriter.WriteResponseBudgetError` を通して +`E028_RESPONSE_BUDGET_TOO_SMALL` / `response_budget` を返します。 +`requested_bytes`、`effective_bytes`、`minimum_required_bytes`、 +`minimum_required_bytes_known`、安定した算出不能 / 不確実性理由 field、 +機械可読な `retry` object を追加します。 +`--max-json-bytes` が制限するのは通常 payload であり、この diagnostic ではありません。 +object と NDJSON の preflight は diagnostic が要求 cap を超えても、解析可能な完全な +error object 1 件を stdout に出し、stderr を空に保つ必要があります。実体化済みで安定した payload は +再試行可能な最小値を返し、runtime 依存 envelope は最小値を不確実と明示して余裕を持つ +推奨値を返します。`retry.action=increase_max_json_bytes` は `recommended_bytes` を返します。 +最小値が有効な 16 MiB 上限を超える場合は、`retry.action=reduce_response_size` が +`option` と `recommended_bytes` を null にし、`maximum_effective_bytes` を返します。 + 未対応の `search --regex` / `search --all` diagnostic は、command 固有で表示専用の `find` recovery 契約を追加します。正規化した引数の semantics を型付き `alternative_command.argv` に正確に保持し、その argv から POSIX shell と PowerShell @@ -3828,6 +3857,7 @@ JSON mode の不正 config は共通の command-error envelope で | 不正な設定 | 1 | `E024_CONFIG_INVALID` | `configuration` | | hook の platform / filesystem failure | 9 | `E025_HOOK_OPERATION_FAILED` | `platform` | | Git repository 外での hooks 実行 | 2 | `E026_NOT_GIT_REPOSITORY` | `not_found` | +| JSON response budget が小さすぎる | 1 または基となる command 固有値 | `E028_RESPONSE_BUDGET_TOO_SMALL` | `response_budget` | | その他の回復可能な command failure | command ごと | `E023_COMMAND_FAILED` | writer による安定した分類 | ### プロセス起動ポリシー @@ -5197,10 +5227,10 @@ help はすべてこのレジストリを参照します。field 名は大文字 | compact location envelope | CLI の `--format compact` location output は、`api_version`、返却 `count`、limit 到達を基準にした保守的な `truncated` / `truncation` metadata、適用済み `query_context`、軽量な `results` row を持つ versioned envelope です。 | | grouped search の総数 | `search --format grouped` の `total_matches` / `matched_count`、`total_groups`、`total_files` は、表示 page ではなく上限適用前の query 全体から算出します。`grouped_match_count` は返却 group に渡した row 数、`emitted_match_count` は file ごとの上限適用後に残った row 数を表し、`omitted_match_count`、`truncated`、`has_more`、`continuation_action` が未完了出力を示します。 | | 高ボリューム応答の bounded 契約 | `search`、`definition`、`find`、`status`、`hotspots`、`references`、`callers`、`callees`、`symbols`、`files`、`languages`、`impact`、`map` は、それぞれの schema が公開する共通 bounded-response control に対応します。新しく発行する opaque な `--cursor ` は offset を command / query / filter と index generation に束縛し、移行用に legacy の `response:v1::` も受理します。選択条件または generation を変えて再利用すると restart-required の案内付きで失敗します。`search --format compact`、`symbols --format compact`、`files --format compact` は bounded 契約を自動選択し、`search --json=array --json-envelope` は opt-in の array envelope、`languages --json` は paging または `--max-json-bytes` 指定時に同じ契約を使います。既存 compact の root と location row は維持したまま共通 metadata を追加します。metadata は `returned_count`、取得可能な場合は authoritative な `total_count`、`omitted_count`、`remaining_count`、`cursor_offset`、`page_limit`、`has_more`、`next_cursor`、`result_stable_at`、`pagination_window_limit`、`pagination_window_exhausted` を返します。safety window は 10,000 row で、上限到達時は次の request が拒否する cursor を返さず `next_cursor` を抑止します。pageable command は `offset + limit` 件を serialize せず、cursor offset を database / scan layer へ渡します。`find --all` の partial scan cursor は次の path / line を保持し、再利用時は最後に scan した line の次から継続します。`hotspots` と `impact` は active な主要 nested collection を `results` としてページングし、`metadata.primary_collection` でその名前を示し、scalar / container evidence は `metadata.response_context` に保持します。`callers.path,callers.depth` のような dotted field で collection と row field を同時に選べます。`--max-json-bytes` は最後の改行を含み、完全な envelope が収まるまで末尾の完全な row を省略します。`definition` は既定で metadata-only のままで、明示的な `--body` は `body`、`body_content`、`all` で保持し、それ以外の projection では materialize 前に抑止します。`map --sections` は section-level projection として残り、dotted な bounded field は選択した array section を section 固有の総件数付きでページングし、scalar projection は不要な ranking array を構築しません。 | -| bounded outline 応答 | `outline` は `--max-json-bytes` がある場合だけ共通 bounded-response 契約を選択します。wrapper は projection 済みの完全な symbol row を抽出し、階層と決定的な順序を維持し、authoritative な返却 / 総 / 省略件数を報告します。UTF-8 計測には最後の改行を含め、束縛済みの `response:v2` continuation cursor を発行します。最小 envelope が収まらない場合は stdout を空に保ち、型付きの `E010_USAGE_ERROR` diagnostic を返します。上限なしの outline JSON は既存の root shape と outline cursor 契約を維持します。 | -| bounded unused 応答 | `unused` は `--max-json-bytes` がある場合だけ共通 bounded-response 契約を選択します。wrapper は canonical な `symbols` row を抽出し、unused query layer で cursor offset を適用し、byte trimming 後の返却 bucket / confidence / contract-domain 件数を再計算します。任意の `by_bucket` view も同じ応答全体の UTF-8 budget に含め、compact mode はより小さな audit row へ projection します。continuation cursor は有効な audit filter、bucket mode、ordering、index generation に束縛されます。1 row を含む最小 envelope が収まらない場合は stdout を空にして、型付きの `E010_USAGE_ERROR` diagnostic を返します。上限なしの JSON、compact summary、legacy unused cursor は既存契約を維持します。 | +| bounded outline 応答 | `outline` は `--max-json-bytes` がある場合だけ共通 bounded-response 契約を選択します。wrapper は projection 済みの完全な symbol row を抽出し、階層と決定的な順序を維持し、authoritative な返却 / 総 / 省略件数を報告します。UTF-8 計測には最後の改行を含め、束縛済みの `response:v2` continuation cursor を発行します。最小 envelope が収まらない場合は、実行可能な byte field を持つ型付きの `E028_RESPONSE_BUDGET_TOO_SMALL` object 1 件を stdout に出し、stderr を空に保ちます。上限なしの outline JSON は既存の root shape と outline cursor 契約を維持します。 | +| bounded unused 応答 | `unused` は `--max-json-bytes` がある場合だけ共通 bounded-response 契約を選択します。wrapper は canonical な `symbols` row を抽出し、unused query layer で cursor offset を適用し、byte trimming 後の返却 bucket / confidence / contract-domain 件数を再計算します。任意の `by_bucket` view も同じ応答全体の UTF-8 budget に含め、compact mode はより小さな audit row へ projection します。continuation cursor は有効な audit filter、bucket mode、ordering、index generation に束縛されます。1 row を含む最小 envelope が収まらない場合は、実行可能な byte field を持つ型付きの `E028_RESPONSE_BUDGET_TOO_SMALL` object 1 件を stdout に出し、stderr を空に保ちます。上限なしの JSON、compact summary、legacy unused cursor は既存契約を維持します。 | | MCP outline page | MCP `outline` は `fields`、`sort`、`limit`、`cursor` を `QueryCommandRunner.BuildOutlinePage` へ渡すため、projection alias、派生 sort field、安定した tie-breaker、`page:v1` query fingerprint、generation validation は第 2 の MCP 固有実装ではなく CLI outline 契約のままです。既定 page は 100 row、MCP 共通の上限は 200 row です。`maxBytes` は enrichment 済みの `structuredContent` 全体を serialize した byte 数で計測し、binary search で完全な row 数を減らして page を再構築するため、`next_cursor` も実際の返却件数から再生成されます。metadata と 1 row が収まらない budget は、進捗しない cursor を返さず失敗します。既定の MCP symbol serialization は後方互換を維持し、明示的な projection field は CLI の snake_case 名を使います。 | -| bounded 応答の edge case | `impact` は選択された nested collection だけに cursor offset を適用するため、definition page の重複や caller / fallback mode の変化を防ぎます。通常の `map --compact` は既存の section array と truncation payload を維持し、collection projection が `--summary-only` または除外する `--sections` filter で失われる組み合わせは拒否します。明示的な definition body field は compact default より優先します。profile / verbose record は `metadata.stream_control_records` へ移し、parser / capture failure の error envelope は active な hard byte cap に収まる場合だけ出力します。 | +| bounded 応答の edge case | `impact` は選択された nested collection だけに cursor offset を適用するため、definition page の重複や caller / fallback mode の変化を防ぎます。通常の `map --compact` は既存の section array と truncation payload を維持し、collection projection が `--summary-only` または除外する `--sections` filter で失われる組み合わせは拒否します。明示的な definition body field は compact default より優先します。profile / verbose record は `metadata.stream_control_records` へ移します。parser / capture failure の error envelope が通常 cap に収まらない場合は完全な `E028_RESPONSE_BUDGET_TOO_SMALL` diagnostic で置き換え、machine output が空または不正 JSON にならないようにします。 | | `--count --json` envelope | `search`、`definition`、`references`、`callers`、`callees`、`symbols`、`files`、`find`、`impact`、`unused` の count-only JSON は単一の自動化向け object です。常に `count`、適用済み `query_context`、freshness metadata(`indexed_file_count`、`indexed_at`、`freshness_available`)、trust flag の `degraded` / `authoritative_count` を含みます。matched-file total を持つ command は `files` と古い互換 alias の `file_count` も含みます。`file_count` は `files` と同じ値を持つ互換用 field として残り、少なくとも次の major release までは削除予定はありません。`unused --count --json` は `returned_bucket_counts`、`returned_contract_domain_counts`、`summary.by_bucket` / `summary.by_confidence` / `summary.by_contract_domain` も含みます。`authoritative_count=false` は readiness または graph/exact trust signal により count が authoritative ではないことを示し、freshness field は count に使った index snapshot を説明します。 | | search row selection | row を返す plain search / recipe path は `ApplySearchOutputSelection` を共有し、`--first-per-file` と固定 seed の決定的な `--sample` を、有効な query ごとの limit / 残り total limit より先に適用します。sample 用 fetch envelope は少なくとも要求 sample 数を基準に sizing します。aggregate / compact の query DTO、plain compact root、run summary、issue-draft の source DTO、NDJSON terminal、bounded array envelope の stream terminal は `source_total`、`selected_total`、`returned`、`selector_omitted_count`、`limit_omitted_count` を公開し、`source_total_authoritative` / `source_total_lower_bound` で完全な population と bounded な観測を区別します。guard filter、origin / facet の後段 filter、candidate window の枯渇、recipe の file-reject 後段 filter は lower-bound authority にします。適用順の `selectors` entry は各段階の input / output / omission count と sample の size / mode / seed を保持し、nullable な `selection_reason` / `selection_omitted_count` は互換用 summary として維持します。bounded plain-search selection は一度だけ計算し、その selected page を compact / envelope serialize で再利用します。search の `query_context.row_selectors` は適用済み selector 設定を記録します。selection だけによる省略は matched / omitted の lower bound を更新しますが、`truncated`、`has_more`、`next_cursor` は設定しません。後続の limit が選択済み row を truncate する場合、limit truncation は表示しますが raw database cursor は selector state を保持できないため `next_cursor` を抑止し、同じ理由で selector と受け取った `--cursor` の併用も拒否します。compact / issue-draft の生成 replay command は selector を保持します。count、aggregation、named-query、recipe-list、results-only、metadata を持たない array、非対応 formatted、summary-only compact の shape は `--first-per-file` / `--sample` を拒否し、すべての recipe shape は grouped 専用の `--per-file-limit` を拒否します。 | | search selection の edge case | issue-draft の root は query ごとの `selection_accounting` を独立して保持するため、draft が 0 件の場合や total limit を使い切った query も accounting を失いません。byte 上限付き compact / array envelope は `returned` を実際の出力 row 数へ更新し、論理的な `limit_omitted_count` を保持します。hard cap による省略は `metadata.byte_limit_omitted_count` で別に報告します。 | diff --git a/README.md b/README.md index 37e7ec9f9..c55ebb1b0 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,7 @@ When the byte cap omits rows, these commands return partial-result exit code `11 High-volume `definition`, `find`, `status`, `hotspots`, `references`, `callers`, `callees`, `impact`, `map`, and `outline` responses also support an opt-in bounded envelope through `--fields`, `--cursor`, compact output where advertised, and a total `--max-json-bytes` budget. Its metadata reports returned/total/omitted counts and an opaque `next_cursor`; replay that cursor with the same query, filters, and sort arguments. The response also exposes its 10,000-row safety window and reports when that window is exhausted instead of emitting an unusable cursor. Existing compact location responses retain their top-level keys and lightweight `file` / `line` rows while adding the shared metadata; `refs` / `stats` aliases and matching read-only `batch` children use the same envelope and hard cap. `hotspots` and `impact` page their active primary collection, while dotted projections such as `callers.path,callers.depth` select nested rows and report that collection's total. `map --sections` selects whole response sections; a bounded projection such as `--fields top_files.path` instead pages that section's rows and avoids building unrelated ranked sections. For `definition --body`, `body`, `body_content`, and `all` retain the explicit body; projections that exclude it avoid materializing body text. `outline --max-json-bytes` pages complete UTF-8 symbol rows, including the final newline, and keeps hierarchy and ordering stable while honoring `--outline-fields`; uncapped outline JSON retains its established shape and cursor contract. -Bounded `map --compact` keeps the established top-level section arrays and compact truncation data while adding shared metadata. A map collection projection is rejected when `--summary-only` or an excluding `--sections` selection would remove that collection. Diagnostic `--profile` / `--verbose` objects are retained as metadata control records rather than projected rows, and every hard byte cap also applies to parser or capture error output; stdout stays empty when even the bounded error envelope cannot fit. +Bounded `map --compact` keeps the established top-level section arrays and compact truncation data while adding shared metadata. A map collection projection is rejected when `--summary-only` or an excluding `--sections` selection would remove that collection. Diagnostic `--profile` / `--verbose` objects are retained as metadata control records rather than projected rows. `--max-json-bytes` is a hard cap for normal JSON payloads; if the minimum complete payload, envelope, NDJSON terminal, or first results-only NDJSON row cannot fit, JSON mode instead writes one complete `E028_RESPONSE_BUDGET_TOO_SMALL` object to stdout and leaves stderr empty, even when that error object is larger than the requested normal-payload cap. The error reports requested/effective bytes, a computable minimum (or a stable unavailability reason), any retry uncertainty, and machine-readable retry guidance. `retry.action=increase_max_json_bytes` supplies `recommended_bytes` when a larger supported budget can work; if the minimum exceeds the effective 16 MiB ceiling, `retry.action=reduce_response_size` leaves the byte recommendation null and reports `maximum_effective_bytes`. `find --all --json` also makes bounded scans explicit. Repository-wide case-insensitive ASCII literals of at least three characters use the trigram index to select candidate files before the existing line matcher verifies every result; regex, `--exact`, short, non-ASCII, legacy, unsynchronized, and actively rebuilding trigram-index queries use an explicit line-scan fallback. `search_strategy` and optional `search_fallback_reason` report which path ran. Default streaming JSON rows end with a terminal record containing `scan_complete`, `authoritative_rows`, verified file/line counts, active caps, truncation reason, and recovery guidance; count JSON carries the same scan state in its single result object through `authoritative_count`. Row formats that cannot carry this metadata, including JSON array and location-only formats, are rejected with `--all`; use text, NDJSON, or count output. A candidate-file or line-scan cap returns partial-result exit code `11` unless `--allow-partial` is set. Ordinary result-limit early stops remain exit `0` but report `scan_complete=false` and `result_limit_reached=true`. @@ -655,7 +655,7 @@ byte cap により行を省略した場合、これらのコマンドは partial 高ボリュームな `definition`、`find`、`status`、`hotspots`、`references`、`callers`、`callees`、`impact`、`map`、`outline` の応答は、`--fields`、`--cursor`、対応 command の compact 出力、応答全体に対する `--max-json-bytes` により opt-in の bounded envelope も利用できます。metadata は返却 / 総 / 省略件数と opaque な `next_cursor` を返します。次ページでは同じ query、filter、sort 引数とともにその cursor を再利用してください。応答は 10,000 row の safety window も公開し、上限到達時には利用不能な cursor を返さず、window の消費完了を報告します。既存の compact location 応答はトップレベル key と軽量な `file` / `line` row を維持したまま共通 metadata を追加し、`refs` / `stats` alias と対応する read-only `batch` 子 command にも同じ envelope と hard cap を適用します。`hotspots` と `impact` は active な主要 collection をページングし、`callers.path,callers.depth` のような dotted projection は nested row とその collection の総件数を返します。`map --sections` は section 全体を選びますが、`--fields top_files.path` のような bounded projection はその section の row をページングし、無関係な ranking section を構築しません。`definition --body` では `body`、`body_content`、`all` が明示的な body を保持し、body を除外する projection では本文を取得しません。`outline --max-json-bytes` は最後の改行を含む完全な UTF-8 symbol row 単位でページングし、`--outline-fields` を尊重しながら階層と順序を維持します。上限なしの outline JSON は従来の形状と cursor 契約を維持します。 -bounded な `map --compact` は、共通 metadata を追加しながら既存のトップレベル section array と compact truncation data を維持します。map collection projection と `--summary-only`、またはその collection を除外する `--sections` の組み合わせは拒否します。`--profile` / `--verbose` の diagnostic object は projected row ではなく metadata の control record として保持し、parser / capture error 出力にも hard byte cap を適用します。bounded error envelope 自体が収まらない場合、stdout は空のままです。 +bounded な `map --compact` は、共通 metadata を追加しながら既存のトップレベル section array と compact truncation data を維持します。map collection projection と `--summary-only`、またはその collection を除外する `--sections` の組み合わせは拒否します。`--profile` / `--verbose` の diagnostic object は projected row ではなく metadata の control record として保持します。`--max-json-bytes` は通常の JSON payload に対する hard cap です。最小の完全な payload、envelope、NDJSON terminal、または results-only NDJSON の先頭 row が収まらない場合、JSON mode は要求された通常 payload の cap より大きくなっても、完全な `E028_RESPONSE_BUDGET_TOO_SMALL` object を stdout に1件出力し、stderr を空に保ちます。この error は requested / effective byte、算出可能な最小値(または安定した算出不能理由)、再試行値の不確実性、機械可読な再試行案内を返します。より大きい対応 budget が有効な場合は `retry.action=increase_max_json_bytes` と `recommended_bytes` を返し、最小値が有効な 16 MiB 上限を超える場合は `retry.action=reduce_response_size`、null の byte 推奨値、`maximum_effective_bytes` を返します。 `find --all --json` も上限付き scan を明示します。3 文字以上の大文字小文字を区別しない ASCII literal は、trigram index で候補 file を選んだ後、既存の行 matcher ですべての結果を検証します。regex、`--exact`、短い query、非 ASCII query、旧 trigram index、同期 trigger が欠けた index、再構築中の index は明示的な line-scan fallback を使います。実行経路は `search_strategy` と任意の `search_fallback_reason` で確認できます。既定の streaming JSON row は `scan_complete`、`authoritative_rows`、検証済み file / line 数、有効な cap、切り詰め理由、復旧案内を含む終端レコードで終了します。count JSON は単一 result object の `authoritative_count` と同じ scan 状態を返します。この metadata を表現できない JSON array や location-only 形式は `--all` との組み合わせを拒否するため、text、NDJSON、count 出力を使ってください。candidate-file cap または line-scan cap に達した場合は、`--allow-partial` を指定しない限り partial-result 終了コード `11` を返します。通常の result limit による早期停止は終了コード `0` のままですが、`scan_complete=false` と `result_limit_reached=true` を報告します。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index b10b91aff..7c6a16fbb 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -332,6 +332,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding Multi named-query output coverage reuses one indexed fixture for compact projection, rich JSON compatibility, per-query limits/truncation, and UTF-8 byte caps so the serializer modes stay directly comparable. Shared bounded-response coverage reuses one graph-ready database across definition, find, status, hotspots, references, callers/callees, impact, and map; keep cursor and UTF-8 byte-budget boundary cases in a separate minimal multi-row fixture so family parity does not multiply indexing setup. Outline keeps one focused deep-hierarchy fixture with long signatures and Unicode to verify exact newline-inclusive byte boundaries, full cursor walks without gaps or duplicates, minimum-budget diagnostics, and unchanged uncapped output. Regression coverage must also exercise aliases and read-only batch dispatch, explicit definition body projections, inactive impact collections, and row-wise map-section pagination with authoritative totals. Adversarial bounded-response coverage must also lock parser-failure byte caps, impact definition-page offsets, legacy map compact sections, conflicting map shape controls, compact explicit bodies, and profile/verbose control-record extraction. + Response-budget preflight coverage must assert parseable stdout and empty stderr for zero and tiny budgets, duplicate and multi-error option parsing, NDJSON terminal and first-results-only-row preflight, exact-minimum retry for stable map/recipe payloads, explicit uncertainty plus recommended headroom for runtime envelopes, size-reduction guidance above the effective maximum, empty and non-empty rows, Unicode/escaping, and the invariant that no normal payload exceeds its requested UTF-8 cap. Search alias variants for JavaScript extensions, YAML, batch, and SQL dialects each reuse one language fixture and iterate casing/spelling forms in a fact. Raw FTS syntax coverage reuses one indexed source for a valid control query and all invalid query/hint variants. Literal and raw FTS complexity bounds reuse one indexed source across length, token-count, NEAR-count, and lowercase-operator controls. @@ -1288,6 +1289,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" 複数 named-query の output coverage は、compact projection、rich JSON 互換性、query ごとの limit / truncation、UTF-8 byte cap に1つの indexed fixture を再利用し、serializer mode を直接比較できるようにしてください。 共通 bounded-response coverage は、definition、find、status、hotspots、references、callers / callees、impact、map 全体で1つの graph-ready databaseを再利用してください。cursor と UTF-8 byte-budget の境界 case は別の最小 multi-row fixture にまとめ、family parity のために indexing setup を重複させないでください。outline は、長い signature と Unicode を含む深い階層の focused fixture 1つを使い、最後の改行を含む正確な byte 境界、欠落や重複のない cursor 全 page 走査、最小 budget の diagnostic、上限なし出力の非変更を確認してください。regression coverage では alias と read-only batch dispatch、明示的な definition body projection、inactive な impact collection、authoritative な総件数を持つ map section の row 単位 pagination も確認してください。 adversarial な bounded-response coverage では、parser failure の byte cap、impact definition page の offset、既存 map compact section、map shape control の競合、compact と明示 body の組み合わせ、profile / verbose control record の抽出も固定してください。 + response-budget preflight coverage では、0 / tiny budget で stdout が解析可能かつ stderr が空であること、重複 option と複数 error の parse、NDJSON terminal と results-only の先頭 row の preflight、安定した map / recipe payload の exact-minimum retry、runtime envelope の明示的な不確実性と余裕を持つ推奨値、有効な最大値を超える場合の size-reduction 案内、空 / 非空 row、Unicode / escape、通常 payload が要求 UTF-8 cap を超えないことを検証してください。 JavaScript extension、YAML、batch、SQL dialectのsearch alias variantは、それぞれ1つのlanguage fixtureを再利用し、casing/spelling形式をfact内で反復してください。 raw FTS syntax coverage はvalid control queryと全invalid query/hint variantで1つのindexed sourceを再利用してください。 literalとraw FTSのcomplexity boundはlength、token count、NEAR count、lowercase operator control全体で1つのindexed sourceを再利用してください。 diff --git a/USER_GUIDE.md b/USER_GUIDE.md index 0f1de59fd..56951bd13 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -2209,7 +2209,7 @@ If a query itself begins with `-`, pass it as `--query ` or `-- `. ### Error codes -For scripts and AI agents that need to classify failures without substring-matching the human prose, every CLI error carries a stable machine-readable code. Human stderr prefixes the code in brackets (`Error [E001_DB_NOT_FOUND]: database not found at …`) and CLI `--json` envelopes add an optional `error_code` field (omitted when not applicable, so existing JSON consumers see no schema break). Recoverable non-database failures from `outline`, `hooks`, `doctor`, and `validate-config` always use the versioned error envelope with `error_code`, `category`, `command`, `exit_code`, `hint`, and `usage`, plus sanitized optional context. In JSON mode, missing-query validation for `search` / `find`, incompatible `status --config` modes, `definition` / `goto` misses, and missing or out-of-range `excerpt` coordinates are emitted as one versioned `{ "status": "error", ... }` object on stdout instead of plain text or an empty stream. A `definition` miss uses `E018_QUERY_NOT_FOUND` and exit code `2`; bounded-envelope controls retain the error under `metadata.error` with an empty `results` array, while an explicitly impossible `--max-json-bytes` cap instead produces a usage error on stderr before any oversized stdout is written. MCP tool errors usually surface as `isError: true` text content, while newer failure modes can also expose stable fields under `structuredContent`; the bracketed CLI constant is not guaranteed to appear in MCP message text. See [Troubleshooting](#troubleshooting) for the MCP message text and structured fields each failure mode expects clients to match. Codes never get renamed or reused once published — retired codes simply stop being emitted. +For scripts and AI agents that need to classify failures without substring-matching the human prose, every CLI error carries a stable machine-readable code. Human stderr prefixes the code in brackets (`Error [E001_DB_NOT_FOUND]: database not found at …`) and CLI `--json` envelopes add an optional `error_code` field (omitted when not applicable, so existing JSON consumers see no schema break). Recoverable non-database failures from `outline`, `hooks`, `doctor`, and `validate-config` always use the versioned error envelope with `error_code`, `category`, `command`, `exit_code`, `hint`, and `usage`, plus sanitized optional context. In JSON mode, missing-query validation for `search` / `find`, incompatible `status --config` modes, `definition` / `goto` misses, and missing or out-of-range `excerpt` coordinates are emitted as one versioned `{ "status": "error", ... }` object on stdout instead of plain text or an empty stream. A `definition` miss uses `E018_QUERY_NOT_FOUND` and exit code `2`; bounded-envelope controls retain the error under `metadata.error` with an empty `results` array. An impossible `--max-json-bytes` request—including an NDJSON terminal or first results-only row that cannot fit—uses `E028_RESPONSE_BUDGET_TOO_SMALL` / `response_budget` and writes one complete error object to stdout with empty stderr, even when the diagnostic exceeds the cap that applies to normal payloads. Its requested/effective/minimum and known/uncertain fields accompany a machine-readable `retry` action. `increase_max_json_bytes` provides `recommended_bytes`; when the minimum exceeds the effective 16 MiB ceiling, `reduce_response_size` instead returns a null recommendation and `maximum_effective_bytes`. MCP tool errors usually surface as `isError: true` text content, while newer failure modes can also expose stable fields under `structuredContent`; the bracketed CLI constant is not guaranteed to appear in MCP message text. See [Troubleshooting](#troubleshooting) for the MCP message text and structured fields each failure mode expects clients to match. Codes never get renamed or reused once published — retired codes simply stop being emitted. | Code | When emitted | |---|---| @@ -2240,6 +2240,7 @@ For scripts and AI agents that need to classify failures without substring-match | `E025_HOOK_OPERATION_FAILED` | A Git hook operation failed at a platform or filesystem boundary | | `E026_NOT_GIT_REPOSITORY` | `hooks` was run outside a Git worktree and no valid `--project` was supplied | | `E027_DB_NOT_DATABASE` | SQLite rejected the target as not being a database, or maintenance validation rejected it as not being a CodeIndex database | +| `E028_RESPONSE_BUDGET_TOO_SMALL` | The requested JSON byte budget cannot fit the minimum complete normal payload or envelope | ### Debugging reader errors @@ -5551,7 +5552,7 @@ raw match density を正確に測る、といった理由で全 raw chunk hit ### エラーコード -スクリプトや AI エージェントが人間向け文言の部分一致なしで失敗を分類できるよう、CLI のエラーには安定した機械可読コードが付与されます。人間向け stderr ではコードを角括弧で前置し(`Error [E001_DB_NOT_FOUND]: database not found at …`)、CLI `--json` エンベロープには任意フィールド `error_code` を追加します(該当しない場合は省略されるので、既存 JSON 利用者にスキーマ破壊なし)。`outline`、`hooks`、`doctor`、`validate-config` の回復可能な非データベース系失敗は、`error_code`、`category`、`command`、`exit_code`、`hint`、`usage` と sanitization 済みの任意 context を持つバージョン付き error envelope を必ず使います。JSON モードでは、`search` / `find` の query 欠落、`status --config` の mode 競合、`definition` / `goto` の未検出、`excerpt` の file 未検出・行範囲外を plain text や空ストリームではなく、version 付きの `{ "status": "error", ... }` オブジェクト 1 件として stdout に出力します。`definition` の未検出は `E018_QUERY_NOT_FOUND` と終了コード `2` を使い、bounded-envelope control の使用時も空の `results` array と `metadata.error` に error を維持します。明示した `--max-json-bytes` が object を格納できない場合は、上限超過の stdout を書く前に stderr の usage error で終了します。MCP ツールエラーは通常 `isError: true` のテキストコンテンツとして返りますが、新しい失敗モードでは `structuredContent` に安定フィールドを持つこともあります。本文に CLI 側の角括弧付き定数が必ず含まれる保証はありません。MCP クライアントが照合すべき各失敗モードの MCP メッセージ本文と構造化フィールドは [トラブルシューティング](#トラブルシューティング) を参照してください。一度公開したコードは renaming / 使い回しをせず、廃止する場合も新規 emission を止めるだけです。 +スクリプトや AI エージェントが人間向け文言の部分一致なしで失敗を分類できるよう、CLI のエラーには安定した機械可読コードが付与されます。人間向け stderr ではコードを角括弧で前置し(`Error [E001_DB_NOT_FOUND]: database not found at …`)、CLI `--json` エンベロープには任意フィールド `error_code` を追加します(該当しない場合は省略されるので、既存 JSON 利用者にスキーマ破壊なし)。`outline`、`hooks`、`doctor`、`validate-config` の回復可能な非データベース系失敗は、`error_code`、`category`、`command`、`exit_code`、`hint`、`usage` と sanitization 済みの任意 context を持つバージョン付き error envelope を必ず使います。JSON モードでは、`search` / `find` の query 欠落、`status --config` の mode 競合、`definition` / `goto` の未検出、`excerpt` の file 未検出・行範囲外を plain text や空ストリームではなく、version 付きの `{ "status": "error", ... }` オブジェクト 1 件として stdout に出力します。`definition` の未検出は `E018_QUERY_NOT_FOUND` と終了コード `2` を使い、bounded-envelope control の使用時も空の `results` array と `metadata.error` に error を維持します。NDJSON terminal または results-only の先頭 row が収まらない場合を含む不可能な `--max-json-bytes` 要求は、`E028_RESPONSE_BUDGET_TOO_SMALL` / `response_budget` を使い、通常 payload に適用する cap より diagnostic が大きくなっても、完全な error object 1 件を stdout に出力して stderr を空に保ちます。requested / effective / minimum と既知 / 不確実性 field に加え、機械可読な `retry` action を返します。`increase_max_json_bytes` は `recommended_bytes` を返し、最小値が有効な 16 MiB 上限を超える場合は `reduce_response_size` が null の推奨値と `maximum_effective_bytes` を返します。MCP ツールエラーは通常 `isError: true` のテキストコンテンツとして返りますが、新しい失敗モードでは `structuredContent` に安定フィールドを持つこともあります。本文に CLI 側の角括弧付き定数が必ず含まれる保証はありません。MCP クライアントが照合すべき各失敗モードの MCP メッセージ本文と構造化フィールドは [トラブルシューティング](#トラブルシューティング) を参照してください。一度公開したコードは renaming / 使い回しをせず、廃止する場合も新規 emission を止めるだけです。 | コード | 発行条件 | |---|---| @@ -5582,6 +5583,7 @@ raw match density を正確に測る、といった理由で全 raw chunk hit | `E025_HOOK_OPERATION_FAILED` | Git hook 操作が platform または filesystem boundary で失敗した | | `E026_NOT_GIT_REPOSITORY` | Git worktree 外で `hooks` を実行し、有効な `--project` も指定されていなかった | | `E027_DB_NOT_DATABASE` | SQLite が対象を database ではないと拒否した、または maintenance validation が CodeIndex database ではないと判定した | +| `E028_RESPONSE_BUDGET_TOO_SMALL` | 要求した JSON byte budget に最小の完全な通常 payload または envelope が収まらない | ### reader エラーのデバッグ diff --git a/changelog.d/unreleased/4909.fixed.md b/changelog.d/unreleased/4909.fixed.md new file mode 100644 index 000000000..6e6cb55df --- /dev/null +++ b/changelog.d/unreleased/4909.fixed.md @@ -0,0 +1,41 @@ +--- +category: fixed +issues: + - 4909 +affected: + - src/CodeIndex/Cli/CliContractManifest.cs + - src/CodeIndex/Cli/CommandErrorCodes.cs + - src/CodeIndex/Cli/CommandErrorWriter.cs + - src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs + - src/CodeIndex/Cli/QueryCommandRunner.cs + - src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.cs + - src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.Results.cs + - src/CodeIndex/Cli/QueryCommandRunner.Map.cs + - src/CodeIndex/Cli/QueryCommandRunner.Ndjson.cs + - src/CodeIndex/Cli/QueryCommandRunner.OptionValidation.cs + - src/CodeIndex/Cli/QueryCommandRunner.Search.cs + - src/CodeIndex/Cli/QueryCommandRunner.SearchConstants.cs + - src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs + - tests/CodeIndex.Tests/JsonEnvelopeWrapperIssue4585Tests.cs + - tests/CodeIndex.Tests/ProgramRunnerTests.cs + - tests/CodeIndex.Tests/ProjectionFieldRegistryIssue4836Tests.cs + - tests/CodeIndex.Tests/QueryCommandRunnerGraphTests.cs + - tests/CodeIndex.Tests/QueryCommandRunnerInspectTests.cs + - tests/CodeIndex.Tests/QueryCommandRunnerMapTests.cs + - tests/CodeIndex.Tests/QueryCommandRunnerOutlineIssue4880Tests.cs + - tests/CodeIndex.Tests/QueryCommandRunnerSearchTests.cs + - tests/CodeIndex.Tests/QueryCommandRunnerSymbolTests.cs + - tests/CodeIndex.Tests/QueryCommandRunnerUnusedIssue4905Tests.cs + - README.md + - USER_GUIDE.md + - DEVELOPER_GUIDE.md + - TESTING_GUIDE.md +--- + +## English + +- **JSON response-budget preflight failures are now parseable and actionable (#4909)** — `map`, `recipes`, NDJSON streams, and shared bounded-response users emit a complete `E028_RESPONSE_BUDGET_TOO_SMALL` object on stdout with requested, effective, minimum, uncertainty, and retry guidance. Stable payloads report a retryable fixed-point minimum, oversized responses beyond the effective ceiling request size reduction, and normal payloads never exceed their requested UTF-8 cap. + +## 日本語 + +- **JSON response-budget の preflight failure が解析可能で実行可能になりました (#4909)** — `map`、`recipes`、NDJSON stream、共通 bounded-response 利用 command は、requested / effective / minimum / uncertainty / retry 案内を持つ完全な `E028_RESPONSE_BUDGET_TOO_SMALL` object を stdout に出力します。安定した payload は再試行可能な固定点の最小値を返し、有効な上限を超える巨大 response には size reduction を案内し、通常 payload が要求 UTF-8 cap を超えることはありません。 diff --git a/src/CodeIndex/Cli/CliContractManifest.cs b/src/CodeIndex/Cli/CliContractManifest.cs index 8be4d68e1..e8533a19e 100644 --- a/src/CodeIndex/Cli/CliContractManifest.cs +++ b/src/CodeIndex/Cli/CliContractManifest.cs @@ -64,6 +64,7 @@ internal static class CliContractManifest new(nameof(CommandErrorCodes.HookOperationFailed), CommandErrorCodes.HookOperationFailed, null), new(nameof(CommandErrorCodes.NotGitRepository), CommandErrorCodes.NotGitRepository, null), new(nameof(CommandErrorCodes.DbNotDatabase), CommandErrorCodes.DbNotDatabase, CommandExitCodes.DatabaseError), + new(nameof(CommandErrorCodes.ResponseBudgetTooSmall), CommandErrorCodes.ResponseBudgetTooSmall, null), ]; public static IReadOnlyList CliJsonRootTypes { get; } = LoadCliJsonRootTypes(); diff --git a/src/CodeIndex/Cli/CommandErrorCodes.cs b/src/CodeIndex/Cli/CommandErrorCodes.cs index 8d072ddd9..8a93e3653 100644 --- a/src/CodeIndex/Cli/CommandErrorCodes.cs +++ b/src/CodeIndex/Cli/CommandErrorCodes.cs @@ -95,4 +95,7 @@ internal static class CommandErrorCodes /// SQLite rejected the file as not being a database, or CodeIndex validation rejected its format. public const string DbNotDatabase = "E027_DB_NOT_DATABASE"; + + /// A requested JSON response byte budget cannot fit the minimum complete payload or envelope. + public const string ResponseBudgetTooSmall = "E028_RESPONSE_BUDGET_TOO_SMALL"; } diff --git a/src/CodeIndex/Cli/CommandErrorWriter.cs b/src/CodeIndex/Cli/CommandErrorWriter.cs index 8c7f85f1d..842d59d9e 100644 --- a/src/CodeIndex/Cli/CommandErrorWriter.cs +++ b/src/CodeIndex/Cli/CommandErrorWriter.cs @@ -7,6 +7,10 @@ namespace CodeIndex.Cli; internal static class CommandErrorWriter { internal const string DefaultHint = "Run ' --help' for usage information."; + internal const string ResponseBudgetCategory = "response_budget"; + internal const string MinimumResponseBytesUnavailableBeforeMaterialization = "normal_payload_not_materialized"; + internal const string MinimumResponseBytesUncertainRuntimeEnvelope = "runtime_metadata_or_embedded_budget_varies_between_invocations"; + internal const string MinimumResponseBytesUncertainCapturedValidation = "captured_validation_output_may_vary_between_invocations"; private const int SanitizedExceptionTypeNameLimit = 120; internal static void WriteStdout(string message = "") @@ -115,6 +119,82 @@ internal static JsonObject BuildJsonPayload( return payload; } + internal static int WriteResponseBudgetError( + bool json, + JsonSerializerOptions jsonOptions, + string command, + string message, + string hint, + long? requestedBytes, + long? effectiveBytes, + long? minimumRequiredBytes, + string? minimumRequiredBytesUnavailableReason = null, + string? minimumRequiredBytesUncertaintyReason = null, + long? recommendedBytes = null, + string? usage = null, + int exitCode = CommandExitCodes.UsageError, + bool retryByIncreasingBudget = true, + long? maximumEffectiveBytes = null, + JsonObject? additionalJsonProperties = null) + { + var minimumKnown = minimumRequiredBytes.HasValue; + if (!minimumKnown && string.IsNullOrWhiteSpace(minimumRequiredBytesUnavailableReason)) + throw new ArgumentException( + "An unavailable minimum response size requires a stable reason.", + nameof(minimumRequiredBytesUnavailableReason)); + + long? retryBytes = retryByIncreasingBudget + ? recommendedBytes ?? minimumRequiredBytes ?? 1 + : null; + var minimumUncertain = !string.IsNullOrWhiteSpace(minimumRequiredBytesUncertaintyReason); + var responseBudgetProperties = new JsonObject + { + ["requested_bytes"] = requestedBytes, + ["effective_bytes"] = effectiveBytes, + ["minimum_required_bytes"] = minimumRequiredBytes, + ["minimum_required_bytes_known"] = minimumKnown, + ["minimum_required_bytes_unavailable_reason"] = minimumKnown + ? null + : minimumRequiredBytesUnavailableReason, + ["minimum_required_bytes_uncertain"] = minimumUncertain, + ["minimum_required_bytes_uncertainty_reason"] = minimumUncertain + ? minimumRequiredBytesUncertaintyReason + : null, + ["retry"] = new JsonObject + { + ["action"] = retryByIncreasingBudget + ? "increase_max_json_bytes" + : "reduce_response_size", + ["option"] = retryByIncreasingBudget ? "--max-json-bytes" : null, + ["recommended_bytes"] = retryBytes, + ["maximum_effective_bytes"] = retryByIncreasingBudget + ? null + : maximumEffectiveBytes, + ["command"] = command, + }, + }; + if (additionalJsonProperties != null) + { + foreach (var property in additionalJsonProperties) + { + if (!responseBudgetProperties.ContainsKey(property.Key)) + responseBudgetProperties[property.Key] = property.Value?.DeepClone(); + } + } + + return WriteJsonOrHuman( + json, + jsonOptions, + message, + exitCode, + hint, + usage, + errorCode: CommandErrorCodes.ResponseBudgetTooSmall, + category: ResponseBudgetCategory, + command: command, + additionalJsonProperties: responseBudgetProperties); + } + internal static (string ErrorCode, string Category) ResolveMachineContract( int exitCode, string? errorCode = null, diff --git a/src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs b/src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs index 8f6077bf3..de53350a1 100644 --- a/src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs +++ b/src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs @@ -18,6 +18,7 @@ namespace CodeIndex.Cli; internal static partial class JsonEnvelopeWrapper { private const int DefaultPageLimit = 20; + private const int ResponseBudgetRetryHeadroomBytes = 1024; private const int MaxPageWindow = MaxRawJsonItems; private const string LegacyResponseCursorPrefix = "response:v1:"; private const string ResponseCursorPrefix = "response:v2:"; @@ -195,8 +196,27 @@ private static int RunBoundedResponse( JsonSerializerOptions jsonOptions, Func runInner) { + if (TryReadRequestedMaxJsonBytes(args, out var requestedBytes) + && requestedBytes <= 0) + { + return CommandErrorWriter.WriteResponseBudgetError( + json: true, + jsonOptions, + command, + "--max-json-bytes requires a positive integer.", + "Use a positive --max-json-bytes value; retry with at least 1 byte to begin response sizing.", + requestedBytes, + effectiveBytes: null, + minimumRequiredBytes: null, + minimumRequiredBytesUnavailableReason: + CommandErrorWriter.MinimumResponseBytesUnavailableBeforeMaterialization, + usage: GetBoundedResponseUsage(command)); + } + if (!TryParseBoundedResponseControls(command, args, out var controls, out var controlError)) + { return WriteBoundedResponseUsageError(controlError!, "Use the command help to pass positive --limit/--max-json-bytes values and a next_cursor returned by the same query."); + } if (HasUnsupportedStandaloneBoundedControl(command, args) || command == "unused" && HasArgument(args, "--summary-only")) { @@ -204,15 +224,18 @@ private static int RunBoundedResponse( command, args, controls.MaxJsonBytes!.Value, + jsonOptions, runInner); } if (ProjectionFieldRegistry.IsDiscoveryRequest(controls.Fields)) { var discoveryJson = ProjectionFieldRegistry.CreateDiscoveryDocument(command).ToJsonString(jsonOptions); return WriteProjectionRegistryResponse( + command, discoveryJson, CommandExitCodes.Success, - controls.MaxJsonBytes); + controls.MaxJsonBytes, + jsonOptions); } if (!ProjectionFieldRegistry.TryValidate(command, controls.Fields, out var fieldError)) { @@ -225,9 +248,11 @@ private static int RunBoundedResponse( Category: "usage"), CliJsonSerializerContextFactory.Create(jsonOptions).CommandErrorJsonResult); return WriteProjectionRegistryResponse( + command, errorJson, CommandExitCodes.UsageError, - controls.MaxJsonBytes); + controls.MaxJsonBytes, + jsonOptions); } var bodyProjected = HasExplicitBodyProjection(controls.Fields); var bodyOutputHidden = !bodyProjected @@ -270,6 +295,7 @@ private static int RunBoundedResponse( var innerArgs = PrepareBoundedInnerArgs(command, args, controls); using var captured = new BoundedStringWriter(MaxCapturedOutputChars); + using var capturedError = new BoundedStringWriter(MaxCapturedOutputChars); var stopwatch = System.Diagnostics.Stopwatch.StartNew(); int exitCode; JsonEnvelopeCaptureLimitExceededException? captureLimitExceeded = null; @@ -277,6 +303,7 @@ private static int RunBoundedResponse( try { using var outputScope = ScopedConsoleOutput.Redirect(captured); + using var errorScope = ScopedConsoleError.Redirect(capturedError); executionContext = new BoundedExecutionContext( command, controls.Offset, @@ -335,7 +362,14 @@ private static int RunBoundedResponse( return WriteBoundedParseError(command, queryNormalized, resolvedDbPath, dbPathExplicit, appVersion, stopwatch.Elapsed.TotalMilliseconds, jsonOptions, $"Bounded response raw JSON {ex.BudgetName} exceeded {ex.MaxValue}.", "Reduce --limit or narrow the query.", ex.JsonPropertyName, ex.MaxValue, controls.MaxJsonBytes, suppressRuntimeMetadata); } + var capturedErrorText = capturedError.ToString(); var commandError = TakeCommandError(rawResults, exitCode); + if (commandError is null + && exitCode != CommandExitCodes.Success + && !string.IsNullOrWhiteSpace(capturedErrorText)) + { + commandError = BuildCapturedCommandError(command, capturedErrorText, exitCode); + } PromoteEmptyLegacyCompactPayload(command, controls, rawResults, streamControlRecords); var extraction = ExtractResponseItems(command, rawResults, controls); var availableItems = extraction.Items; @@ -390,7 +424,8 @@ private static int RunBoundedResponse( suppressRuntimeMetadata, jsonOptions, out var emittedJson, - out var emittedCount); + out var emittedCount, + out var minimumRequiredBytes); if (envelope is null) { @@ -400,11 +435,28 @@ private static int RunBoundedResponse( "unused" => "add --compact", _ => "choose fewer --fields", }; - return WriteBoundedResponseUsageError( - $"--max-json-bytes {controls.MaxJsonBytes} is too small for the bounded response metadata and one projected row.", - $"Increase --max-json-bytes or {selectionHint}."); - } - + var minimumDescription = commandError is not null + ? "the complete bounded error envelope" + : pageItems.Count > 0 + ? "bounded response metadata and one projected row" + : "the complete empty bounded response envelope"; + return CommandErrorWriter.WriteResponseBudgetError( + json: true, + jsonOptions, + command, + $"--max-json-bytes {controls.MaxJsonBytes} is too small for {minimumDescription}.", + $"Increase --max-json-bytes or {selectionHint}.", + requestedBytes: controls.MaxJsonBytes, + effectiveBytes: controls.MaxJsonBytes, + minimumRequiredBytes: minimumRequiredBytes, + minimumRequiredBytesUncertaintyReason: + CommandErrorWriter.MinimumResponseBytesUncertainRuntimeEnvelope, + recommendedBytes: minimumRequiredBytes + ResponseBudgetRetryHeadroomBytes, + usage: GetBoundedResponseUsage(command)); + } + + if (exitCode == CommandExitCodes.Success && capturedErrorText.Length > 0) + Console.Error.Write(capturedErrorText); Console.WriteLine(emittedJson); return exitCode; } @@ -413,6 +465,7 @@ private static int RunStandaloneValidationWithinBudget( string command, string[] args, int maxJsonBytes, + JsonSerializerOptions jsonOptions, Func runInner) { using var captured = new BoundedStringWriter(MaxCapturedOutputChars); @@ -442,9 +495,19 @@ private static int RunStandaloneValidationWithinBudget( // Fall back to a bounded generic diagnostic if validation emitted malformed JSON. } - return WriteBoundedResponseUsageError( + return CommandErrorWriter.WriteResponseBudgetError( + json: true, + jsonOptions, + command, message ?? $"{command} output-selector validation failed or its summary document exceeds the byte budget.", - $"{hint ?? $"Use only options shown in `{command} --help`."} Increase --max-json-bytes to receive the structured response."); + $"{hint ?? $"Use only options shown in `{command} --help`."} Increase --max-json-bytes to receive the structured response.", + requestedBytes: maxJsonBytes, + effectiveBytes: maxJsonBytes, + minimumRequiredBytes: Encoding.UTF8.GetByteCount(output), + minimumRequiredBytesUncertaintyReason: + CommandErrorWriter.MinimumResponseBytesUncertainCapturedValidation, + recommendedBytes: (long)Encoding.UTF8.GetByteCount(output) + ResponseBudgetRetryHeadroomBytes, + usage: GetBoundedResponseUsage(command)); } private static JsonObject? BuildBoundedEnvelopeWithinBudget( @@ -468,7 +531,8 @@ private static int RunStandaloneValidationWithinBudget( bool suppressRuntimeMetadata, JsonSerializerOptions jsonOptions, out string emittedJson, - out int emittedCount) + out int emittedCount, + out long minimumRequiredBytes) { emittedJson = string.Empty; emittedCount = 0; @@ -613,6 +677,9 @@ JsonObject BuildCandidate(int count) } var requestedCount = pageItems.Count; + var minimumCandidate = BuildCandidate(requestedCount > 0 ? 1 : 0); + var minimumJson = SerializeBoundedEnvelope(minimumCandidate, jsonOptions); + minimumRequiredBytes = GetJsonResponseByteCount(minimumJson); var candidate = BuildCandidate(requestedCount); var candidateJson = SerializeBoundedEnvelope(candidate, jsonOptions); if (!controls.MaxJsonBytes.HasValue || JsonFitsResponseBudget(candidateJson, controls.MaxJsonBytes.Value)) @@ -666,18 +733,30 @@ private static string SerializeBoundedEnvelope(JsonNode node, JsonSerializerOpti } internal static bool JsonFitsResponseBudget(string json, int maxJsonBytes) - => Encoding.UTF8.GetByteCount(json) + Encoding.UTF8.GetByteCount(Environment.NewLine) <= maxJsonBytes; + => GetJsonResponseByteCount(json) <= maxJsonBytes; + + private static long GetJsonResponseByteCount(string json) + => (long)Encoding.UTF8.GetByteCount(json) + Encoding.UTF8.GetByteCount(Environment.NewLine); private static int WriteProjectionRegistryResponse( + string command, string json, int exitCode, - int? maxJsonBytes) + int? maxJsonBytes, + JsonSerializerOptions jsonOptions) { if (maxJsonBytes.HasValue && !JsonFitsResponseBudget(json, maxJsonBytes.Value)) { - return WriteBoundedResponseUsageError( + return CommandErrorWriter.WriteResponseBudgetError( + json: true, + jsonOptions, + command, $"--max-json-bytes {maxJsonBytes.Value} is too small for the projection-field response.", - "Increase --max-json-bytes and rerun the same --fields request."); + "Increase --max-json-bytes and rerun the same --fields request.", + requestedBytes: maxJsonBytes.Value, + effectiveBytes: maxJsonBytes.Value, + minimumRequiredBytes: GetJsonResponseByteCount(json), + usage: GetBoundedResponseUsage(command)); } Console.WriteLine(json); @@ -704,6 +783,46 @@ private static int WriteProjectionRegistryResponse( return error; } + private static JsonObject BuildCapturedCommandError( + string command, + string stderr, + int exitCode) + { + var lines = stderr + .ReplaceLineEndings("\n") + .Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); + var firstLine = lines.FirstOrDefault() ?? "Command validation failed."; + var (errorCode, category) = CommandErrorWriter.ResolveMachineContract(exitCode); + string message; + if (firstLine.StartsWith("Error [", StringComparison.Ordinal) + && firstLine.IndexOf("]: ", StringComparison.Ordinal) is var separator + && separator > "Error [".Length) + { + errorCode = firstLine["Error [".Length..separator]; + message = firstLine[(separator + 3)..]; + } + else if (firstLine.StartsWith("Error: ", StringComparison.Ordinal)) + { + message = firstLine["Error: ".Length..]; + } + else + { + message = firstLine; + } + + var hint = lines + .FirstOrDefault(line => line.StartsWith("Hint: ", StringComparison.Ordinal)); + return new JsonObject + { + ["message"] = message, + ["hint"] = hint is null ? null : hint["Hint: ".Length..], + ["error_code"] = errorCode, + ["category"] = category, + ["command"] = command, + ["usage"] = GetBoundedResponseUsage(command), + }; + } + private static void PromoteEmptyLegacyCompactPayload( string command, BoundedResponseControls controls, @@ -1543,6 +1662,37 @@ private static bool TryReadPositiveIntControl( return true; } + private static bool TryReadRequestedMaxJsonBytes(string[] args, out long requestedBytes) + { + requestedBytes = 0; + for (var i = 0; i < args.Length; i++) + { + const string option = "--max-json-bytes"; + var arg = args[i]; + string? raw = null; + if (arg.StartsWith(option + "=", StringComparison.Ordinal)) + raw = arg[(option.Length + 1)..]; + else if (string.Equals(arg, option, StringComparison.Ordinal) && i + 1 < args.Length) + raw = args[i + 1]; + if (raw is null) + continue; + if (!long.TryParse( + raw, + NumberStyles.Integer, + CultureInfo.InvariantCulture, + out requestedBytes)) + { + return false; + } + if (requestedBytes <= 0) + return true; + if (requestedBytes > int.MaxValue) + return false; + } + + return false; + } + private static string BuildResponseFingerprint(string command, string[] args) { var scanMode = command == "find" @@ -1731,6 +1881,9 @@ private static int WriteBoundedResponseUsageError(string message, string hint) return CommandExitCodes.UsageError; } + private static string GetBoundedResponseUsage(string command) + => ConsoleUi.GetUsageLine(command) ?? $"cdidx {command} --help"; + private static int WriteBoundedCaptureError( string command, string? queryNormalized, @@ -1778,8 +1931,6 @@ private static int WriteBoundedErrorEnvelope( int? maxJsonBytes, bool suppressRuntimeMetadata) { - CommandErrorWriter.WriteStderr($"Error [{CommandErrorCodes.UsageError}]: {message}"); - CommandErrorWriter.WriteStderr($"Hint: {hint}"); var error = new JsonObject { ["message"] = message, @@ -1800,8 +1951,25 @@ private static int WriteBoundedErrorEnvelope( error, suppressRuntimeMetadata: suppressRuntimeMetadata); var json = envelope.ToJsonString(jsonOptions); - if (!maxJsonBytes.HasValue || JsonFitsResponseBudget(json, maxJsonBytes.Value)) - Console.WriteLine(json); + if (maxJsonBytes.HasValue && !JsonFitsResponseBudget(json, maxJsonBytes.Value)) + { + return CommandErrorWriter.WriteResponseBudgetError( + json: true, + jsonOptions, + command, + $"--max-json-bytes {maxJsonBytes.Value} is too small for the complete error envelope.", + "Increase --max-json-bytes and retry the same command.", + requestedBytes: maxJsonBytes.Value, + effectiveBytes: maxJsonBytes.Value, + minimumRequiredBytes: GetJsonResponseByteCount(json), + minimumRequiredBytesUncertaintyReason: + CommandErrorWriter.MinimumResponseBytesUncertainRuntimeEnvelope, + recommendedBytes: GetJsonResponseByteCount(json) + ResponseBudgetRetryHeadroomBytes, + usage: GetBoundedResponseUsage(command), + exitCode: exitCode); + } + + Console.WriteLine(json); return exitCode; } diff --git a/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.Results.cs b/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.Results.cs index 9ae3da075..b1ca96012 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.Results.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.Results.cs @@ -225,6 +225,7 @@ private bool TryParseResultOption(string normalizedArg, string currentArg, strin else if (TryParsePositiveInt(maxJsonBytesValue!, "--max-json-bytes", out var parsedMaxJsonBytes, out var maxJsonBytesError)) { WarnIfDuplicateSingleValueOption("--max-json-bytes", maxJsonBytesValue!); + requestedMaxJsonBytes = parsedMaxJsonBytes; maxJsonBytes = Math.Min(parsedMaxJsonBytes, MaxSearchJsonByteLimit); } else diff --git a/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.cs b/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.cs index 19130b8e6..5ef9f5bf4 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.ArgumentParser.cs @@ -113,6 +113,7 @@ private sealed partial class QueryArgumentParser private int groupedPerFileLimit = DefaultSearchGroupedPerFileLimit; private bool groupedPerFileLimitExplicit; private int? sampleSize; + private int? requestedMaxJsonBytes; private int? maxJsonBytes; private bool rawBytes; private bool rawKinds; @@ -505,6 +506,7 @@ private QueryCommandOptions BuildOptions(DbPathResolution dbResolution, string r GroupedPerFileLimit = groupedPerFileLimit, GroupedPerFileLimitExplicit = groupedPerFileLimitExplicit, SampleSize = sampleSize, + RequestedMaxJsonBytes = requestedMaxJsonBytes, MaxJsonBytes = maxJsonBytes, RawBytes = rawBytes, RawKinds = rawKinds, diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Batch.cs b/src/CodeIndex/Cli/QueryCommandRunner.Batch.cs index 504be07b8..642fe2103 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Batch.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Batch.cs @@ -896,6 +896,7 @@ private static BatchParallelCommandResult RunBatchParallelCommand( using var stdout = new BatchBoundedStringWriter(BatchMaxCapturedOutputChars, "stdout"); using var stderr = new BatchBoundedStringWriter(BatchMaxCapturedOutputChars, "stderr"); using var stdoutRouterRegistration = ScopedConsoleOutput.Register(stdoutRouter); + using var stderrRouterRegistration = ScopedConsoleError.Register(stderrRouter); using var stdoutScope = stdoutRouter.Push(stdout); using var stderrScope = stderrRouter.Push(stderr); var exitCode = CommandExitCodes.DatabaseError; diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Definition.cs b/src/CodeIndex/Cli/QueryCommandRunner.Definition.cs index d35fb8c94..d6c8f7fee 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Definition.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Definition.cs @@ -199,6 +199,7 @@ public static int RunDefinition(string[] cmdArgs, JsonSerializerOptions jsonOpti options, "definition not-found response", "Increase --max-json-bytes to allow the structured not-found response.", + jsonOptions, "definition"); return writeExitCode == CommandExitCodes.Success ? CommandExitCodes.NotFound diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Discovery.cs b/src/CodeIndex/Cli/QueryCommandRunner.Discovery.cs index b7a112b81..0a6c46009 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Discovery.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Discovery.cs @@ -69,6 +69,7 @@ public static int RunSymbols(string[] cmdArgs, JsonSerializerOptions jsonOptions options, "symbols count", "Narrow the query or increase --max-json-bytes.", + jsonOptions, "symbols"); } @@ -814,6 +815,7 @@ private static int WriteBoundedDiscoveryJsonPayload( options, $"{commandName} compact", "Use --summary-only, reduce --limit, or increase --max-json-bytes.", + jsonOptions, commandName); string BuildBoundedDiscoveryJson(int emittedRows) @@ -859,6 +861,7 @@ private static int WriteBoundedDiscoveryFlatJsonPayload( options, $"{commandName} JSON array", "Increase --max-json-bytes.", + jsonOptions, commandName); } @@ -904,6 +907,7 @@ private static int WriteBoundedDiscoveryFlatJsonPayload( options, $"{commandName} JSON row", "Reduce --limit or increase --max-json-bytes.", + jsonOptions, commandName); } break; diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Map.cs b/src/CodeIndex/Cli/QueryCommandRunner.Map.cs index 2f77c91be..87d084baa 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Map.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Map.cs @@ -26,7 +26,15 @@ public static int RunMap(string[] cmdArgs, JsonSerializerOptions jsonOptions) options.Lang); if (TryWriteUnsupportedOptionError("map", cmdArgs, CliFlagSchema.GetAcceptedFlagNamesForCommand("map"))) return CommandExitCodes.UsageError; - if (TryWriteParseError(options, "map", options.LanguageValidationError ? jsonOptions : null)) + if (TryWriteParseError( + options, + "map", + options.LanguageValidationError + || options.Json + && options.ParseError is not null + && TryExtractNonPositiveMaxJsonBytes(options.ParseError, out _, out _, out _) + ? jsonOptions + : null)) return CommandExitCodes.UsageError; if (TryWriteUnsupportedOutputFormat("map", options, RepoMapOutputFormats, "Use `--format json`, `--format compact`, or `--format issue-drafts` for map output; use `cdidx files --count` when you need only a file count.")) return CommandExitCodes.UsageError; @@ -108,6 +116,7 @@ public static int RunMap(string[] cmdArgs, JsonSerializerOptions jsonOptions) options, "map issue-draft", "Reduce --limit, narrow --path/--lang filters, or increase --max-json-bytes.", + jsonOptions, "map"); return issueDraftsExitCode != CommandExitCodes.Success ? issueDraftsExitCode @@ -130,6 +139,7 @@ public static int RunMap(string[] cmdArgs, JsonSerializerOptions jsonOptions) options, "map", "Use `--summary-only`, narrow --sections/--path/--lang filters, switch to --compact, or increase --max-json-bytes.", + jsonOptions, "map"); if (zeroJsonExitCode != CommandExitCodes.Success) return zeroJsonExitCode; @@ -156,6 +166,7 @@ public static int RunMap(string[] cmdArgs, JsonSerializerOptions jsonOptions) options, "map", "Use `--summary-only`, narrow --sections/--path/--lang filters, switch to --compact, or increase --max-json-bytes.", + jsonOptions, "map"); } else diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Ndjson.cs b/src/CodeIndex/Cli/QueryCommandRunner.Ndjson.cs index 4b619ffde..ae36279e6 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Ndjson.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Ndjson.cs @@ -6,6 +6,8 @@ namespace CodeIndex.Cli; public static partial class QueryCommandRunner { + private const int NdjsonResponseBudgetRetryHeadroomBytes = 1024; + private sealed record NdjsonOutputRecord(string Line, bool CountsAsResult = true); private sealed record NdjsonStreamWriteResult( @@ -51,7 +53,7 @@ private static NdjsonStreamWriteResult WriteNdjsonStream( List? selectors = null) { if (options.ResultsOnly) - return WriteResultOnlyNdjson(records, options); + return WriteResultOnlyNdjson(records, options, jsonOptions, commandName); var emittedRecords = records.Count; string? terminalLine = null; @@ -154,12 +156,16 @@ string BuildTerminal( "Increase --max-json-bytes so the bounded NDJSON terminal record fits before streaming begins.", includeSelectionAccounting: false); } - WriteUsageError( - $"{commandName} NDJSON terminal record is {JsonLineBytes(requiredTerminal)} bytes and exceeds --max-json-bytes {options.MaxJsonBytes.Value}.", + var requiredTerminalBytes = JsonLineBytes(requiredTerminal); + var budgetExitCode = WriteNdjsonResponseBudgetError( options, + jsonOptions, commandName, - "Increase --max-json-bytes; the hard cap includes both result records and the terminal record."); - return new(0, false, null, null, CommandExitCodes.UsageError); + $"{commandName} NDJSON terminal record is {requiredTerminalBytes} bytes and exceeds --max-json-bytes {options.MaxJsonBytes.Value}.", + "Increase --max-json-bytes; the hard cap includes both result records and the terminal record.", + requiredTerminalBytes, + minimumUncertain: true); + return new(0, false, null, null, budgetExitCode); } } else @@ -194,8 +200,27 @@ string BuildTerminal( private static NdjsonStreamWriteResult WriteResultOnlyNdjson( IReadOnlyList records, - QueryCommandOptions options) + QueryCommandOptions options, + JsonSerializerOptions jsonOptions, + string commandName) { + if (options.MaxJsonBytes.HasValue && records.Count > 0) + { + var firstRecordBytes = JsonLineBytes(records[0].Line); + if (firstRecordBytes > options.MaxJsonBytes.Value) + { + var exitCode = WriteNdjsonResponseBudgetError( + options, + jsonOptions, + commandName, + $"{commandName} first complete NDJSON result record is {firstRecordBytes} bytes and exceeds --max-json-bytes {options.MaxJsonBytes.Value}.", + "Reduce projected fields or increase --max-json-bytes before streaming begins.", + firstRecordBytes, + minimumUncertain: false); + return new(0, false, firstRecordBytes, null, exitCode); + } + } + var emittedRecords = 0; var bytesWritten = 0; foreach (var record in records) @@ -217,6 +242,42 @@ private static NdjsonStreamWriteResult WriteResultOnlyNdjson( interrupted && !options.AllowPartial ? CommandExitCodes.PartialResult : CommandExitCodes.Success); } + private static int WriteNdjsonResponseBudgetError( + QueryCommandOptions options, + JsonSerializerOptions jsonOptions, + string commandName, + string message, + string hint, + long minimumRequiredBytes, + bool minimumUncertain) + { + var uncertainRecommendedBytes = minimumRequiredBytes + NdjsonResponseBudgetRetryHeadroomBytes; + var retryByIncreasingBudget = minimumRequiredBytes <= MaxSearchJsonByteLimit + && (!minimumUncertain + || uncertainRecommendedBytes <= MaxSearchJsonByteLimit); + var effectiveHint = retryByIncreasingBudget + ? hint + : $"{hint} The response minimum exceeds the usable maximum effective --max-json-bytes budget; reduce the response size before retrying."; + return CommandErrorWriter.WriteResponseBudgetError( + json: true, + jsonOptions, + commandName, + message, + effectiveHint, + requestedBytes: options.RequestedMaxJsonBytes ?? options.MaxJsonBytes, + effectiveBytes: options.MaxJsonBytes, + minimumRequiredBytes, + minimumRequiredBytesUncertaintyReason: minimumUncertain + ? CommandErrorWriter.MinimumResponseBytesUncertainRuntimeEnvelope + : null, + recommendedBytes: retryByIncreasingBudget + ? minimumUncertain ? uncertainRecommendedBytes : minimumRequiredBytes + : null, + usage: GetUsageLineOrThrow(commandName), + retryByIncreasingBudget: retryByIncreasingBudget, + maximumEffectiveBytes: MaxSearchJsonByteLimit); + } + private static int CountResults(IReadOnlyList records, int count) { var resultCount = 0; diff --git a/src/CodeIndex/Cli/QueryCommandRunner.OptionValidation.cs b/src/CodeIndex/Cli/QueryCommandRunner.OptionValidation.cs index 30c09020d..470925294 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.OptionValidation.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.OptionValidation.cs @@ -1,5 +1,6 @@ using System.Globalization; using System.Text.Json; +using System.Text.Json.Nodes; using CodeIndex.Indexer; namespace CodeIndex.Cli; @@ -54,10 +55,17 @@ private static bool TryWriteParseError( : primaryError == inspectCursorScopeError && options.ParseError == null && dbPathError == null ? "Pass this cursor back to the unchanged `cdidx inspect` query that returned it." : "fix the invalid or missing option value, then rerun with the command shape below."; - var machineErrorOutput = options.Json - && jsonOptions != null - && (!invocationContext.StructuredMachineUsageErrors - || options.InvocationMachineErrorOutputRequested); + var responseBudgetMachineError = options.Json + && TryExtractNonPositiveMaxJsonBytes( + primaryError, + out _, + out _, + out _); + var machineErrorOutput = responseBudgetMachineError + || options.Json + && jsonOptions != null + && (!invocationContext.StructuredMachineUsageErrors + || options.InvocationMachineErrorOutputRequested); WriteParseError(primaryError, primaryHint, invocationContext, options, jsonOptions); if (options.ParseError != null && dbPathError != null @@ -136,6 +144,29 @@ private static void WriteParseError( QueryCommandOptions options, JsonSerializerOptions? jsonOptions) { + if (options.Json + && TryExtractNonPositiveMaxJsonBytes( + error, + out var requestedBytes, + out var responseBudgetError, + out var additionalJsonProperties)) + { + CommandErrorWriter.WriteResponseBudgetError( + json: true, + jsonOptions ?? options.InvocationJsonOptions ?? ProgramRunner.CreateDefaultJsonOptions(), + invocationContext.CommandName, + StripInlineHint(StripErrorPrefix(responseBudgetError)), + "Use a positive --max-json-bytes value; retry with at least 1 byte to begin response sizing.", + requestedBytes, + effectiveBytes: null, + minimumRequiredBytes: null, + minimumRequiredBytesUnavailableReason: + CommandErrorWriter.MinimumResponseBytesUnavailableBeforeMaterialization, + usage: invocationContext.UsageLine, + additionalJsonProperties: additionalJsonProperties); + return; + } + if (options.Json && jsonOptions != null && (!invocationContext.StructuredMachineUsageErrors @@ -174,6 +205,66 @@ private static void WriteParseError( : null)); } + private static bool TryExtractNonPositiveMaxJsonBytes( + string error, + out long requestedBytes, + out string matchingError, + out JsonObject? additionalJsonProperties) + { + requestedBytes = 0; + matchingError = string.Empty; + additionalJsonProperties = null; + const string marker = "--max-json-bytes requires a positive integer, got '"; + var errors = error + .Split('\n', StringSplitOptions.RemoveEmptyEntries) + .Select(line => line.TrimEnd('\r')) + .ToArray(); + for (var index = 0; index < errors.Length; index++) + { + var candidate = errors[index]; + var valueStart = candidate.IndexOf(marker, StringComparison.Ordinal); + if (valueStart < 0) + continue; + valueStart += marker.Length; + var valueEnd = candidate.IndexOf('\'', valueStart); + if (valueEnd <= valueStart + || !long.TryParse( + candidate.AsSpan(valueStart, valueEnd - valueStart), + NumberStyles.Integer, + CultureInfo.InvariantCulture, + out requestedBytes) + || requestedBytes > 0) + { + continue; + } + + matchingError = candidate; + var otherErrors = new JsonArray(); + for (var otherIndex = 0; otherIndex < errors.Length; otherIndex++) + { + if (otherIndex == index) + continue; + otherErrors.Add(StripInlineHint(StripErrorPrefix(errors[otherIndex]))); + } + if (otherErrors.Count > 0) + { + additionalJsonProperties = new JsonObject + { + ["other_validation_errors"] = otherErrors, + }; + } + return true; + } + + return false; + } + + private static string StripInlineHint(string message) + { + var hintStart = message.IndexOf(" Hint:", StringComparison.Ordinal); + return hintStart < 0 ? message : message[..hintStart].TrimEnd(); + } + private static string? BuildExplicitDbPathParseError(QueryCommandOptions options) { if (options.StatusConfig) diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Search.cs b/src/CodeIndex/Cli/QueryCommandRunner.Search.cs index 7d2a4d196..9c346725a 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Search.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Search.cs @@ -81,7 +81,13 @@ private static int RunSearchCore( if (TryWriteParseError( options, invocationContext, - options.LanguageValidationError || invocationContext.StructuredMachineUsageErrors ? jsonOptions : null)) + options.LanguageValidationError + || invocationContext.StructuredMachineUsageErrors + || options.Json + && options.ParseError is not null + && TryExtractNonPositiveMaxJsonBytes(options.ParseError, out _, out _, out _) + ? jsonOptions + : null)) return CommandExitCodes.UsageError; if (!TryResolveSearchExactMode(options, out var exact, out var exactError, out var exactHint)) { @@ -783,7 +789,8 @@ and not OutputFormatCompact exactSubstringHint: exactSubstringHint).ToJsonString(jsonOptions), options, "search count", - "Narrow the query or increase --max-json-bytes."); + "Narrow the query or increase --max-json-bytes.", + jsonOptions); } else { @@ -807,7 +814,8 @@ and not OutputFormatCompact exactSubstringHint: exactSubstringHint).ToJsonString(jsonOptions), options, "search count", - "Narrow the query or increase --max-json-bytes."); + "Narrow the query or increase --max-json-bytes.", + jsonOptions); } else { @@ -865,7 +873,8 @@ and not OutputFormatCompact CliJsonSerializerContextFactory.Create(jsonOptions).CompactSearchResultArray), options, "search result array", - "Increase --max-json-bytes or remove the byte cap."); + "Increase --max-json-bytes or remove the byte cap.", + jsonOptions); } else { @@ -996,7 +1005,8 @@ selection.TruncationReason is "sample" or "first_per_file" CliJsonSerializerContextFactory.Create(jsonOptions).CompactSearchResultArray), options, "search result array", - "Reduce --limit, --snippet-lines, or use `--json=ndjson --max-json-bytes` for streaming output."); + "Reduce --limit, --snippet-lines, or use `--json=ndjson --max-json-bytes` for streaming output.", + jsonOptions); } else { diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchConstants.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchConstants.cs index fa6811065..97f5d7a48 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.SearchConstants.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchConstants.cs @@ -8,7 +8,7 @@ public static partial class QueryCommandRunner private const int DefaultSearchGroupedPerFileLimit = 3; private const int MaxSearchGroupedPerFileLimit = 20; private const int MaxSearchNextStepLimit = 10; - private const int MaxSearchJsonByteLimit = 16 * 1024 * 1024; + internal const int MaxSearchJsonByteLimit = 16 * 1024 * 1024; private const int MaxIssueDraftEvidenceItems = 5; private const int MaxIssueDraftEvidenceSnippetLength = 512; private const string BareTokenAuthAuditHint = "Bare `token` searches are intentionally broad. For credential/auth-token review, run `cdidx search --recipe auth-token-audit`; use `cdidx search --recipe broad-token-audit` only when parser, LSP, or cancellation token domains are intentional."; diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs index 38e7194d3..419c80ba0 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs @@ -51,6 +51,7 @@ private static int WriteSearchRecipeList( options, "recipe-name list", "Use a larger --max-json-bytes value or remove recipe filters.", + jsonOptions, usageCommandName); } @@ -71,6 +72,7 @@ private static int WriteSearchRecipeList( options, "recipe summary", "Use `cdidx recipes --names --json` for the smallest recipe-list JSON.", + jsonOptions, usageCommandName); } if (options.SummaryOnly) @@ -89,6 +91,7 @@ private static int WriteSearchRecipeList( options, "recipe list", "Use `cdidx recipes --names --json` or `cdidx recipes --summary-only --json` for smaller output.", + jsonOptions, usageCommandName); } @@ -122,25 +125,42 @@ private static int WriteSearchRecipeList( return CommandExitCodes.Success; } - private static int WriteJsonObjectWithOptionalByteLimit( + internal static int WriteJsonObjectWithOptionalByteLimit( string json, QueryCommandOptions options, string outputDescription, string hint, + JsonSerializerOptions jsonOptions, string commandName = "search") { json = AddActiveSqliteDiagnostics(json); if (options.MaxJsonBytes.HasValue) { - var byteCount = Encoding.UTF8.GetByteCount(json) + Environment.NewLine.Length; + var byteCount = Encoding.UTF8.GetByteCount(json) + + Encoding.UTF8.GetByteCount(Environment.NewLine); if (byteCount > options.MaxJsonBytes.Value) { - WriteUsageError( - $"{outputDescription} JSON output is {byteCount.ToString(CultureInfo.InvariantCulture)} bytes and exceeds --max-json-bytes {options.MaxJsonBytes.Value.ToString(CultureInfo.InvariantCulture)}.", - options, + var minimumRequiredBytes = ComputeRetryableMinimumJsonBytes( + json, + options.MaxJsonBytes.Value, + jsonOptions); + var retryByIncreasingBudget = minimumRequiredBytes <= MaxSearchJsonByteLimit; + var effectiveHint = retryByIncreasingBudget + ? hint + : $"{hint} The response minimum exceeds the maximum effective --max-json-bytes value of {MaxSearchJsonByteLimit.ToString(CultureInfo.InvariantCulture)}; reduce the response size before retrying."; + return CommandErrorWriter.WriteResponseBudgetError( + json: true, + jsonOptions, commandName, - hint); - return CommandExitCodes.UsageError; + $"{outputDescription} JSON output is {byteCount.ToString(CultureInfo.InvariantCulture)} bytes and exceeds --max-json-bytes {options.MaxJsonBytes.Value.ToString(CultureInfo.InvariantCulture)}.", + effectiveHint, + requestedBytes: options.RequestedMaxJsonBytes ?? options.MaxJsonBytes.Value, + effectiveBytes: options.MaxJsonBytes.Value, + minimumRequiredBytes: minimumRequiredBytes, + recommendedBytes: retryByIncreasingBudget ? minimumRequiredBytes : null, + usage: GetUsageLineOrThrow(commandName), + retryByIncreasingBudget: retryByIncreasingBudget, + maximumEffectiveBytes: MaxSearchJsonByteLimit); } } @@ -148,6 +168,91 @@ private static int WriteJsonObjectWithOptionalByteLimit( return CommandExitCodes.Success; } + private static long ComputeRetryableMinimumJsonBytes( + string json, + int requestedBytes, + JsonSerializerOptions jsonOptions) + { + var minimumRequiredBytes = (long)Encoding.UTF8.GetByteCount(json) + + Encoding.UTF8.GetByteCount(Environment.NewLine); + var payload = JsonNode.Parse(json); + if (payload is null) + return minimumRequiredBytes; + + for (var iteration = 0; iteration < 8; iteration++) + { + RewriteEmbeddedJsonByteLimit(payload, requestedBytes, minimumRequiredBytes); + requestedBytes = checked((int)Math.Min(minimumRequiredBytes, int.MaxValue)); + var candidateJson = payload.ToJsonString(EnsureJsonNodeSerializerOptions(jsonOptions)); + var candidateBytes = (long)Encoding.UTF8.GetByteCount(candidateJson) + + Encoding.UTF8.GetByteCount(Environment.NewLine); + if (candidateBytes <= minimumRequiredBytes) + return minimumRequiredBytes; + minimumRequiredBytes = candidateBytes; + } + + return minimumRequiredBytes; + } + + private static void RewriteEmbeddedJsonByteLimit( + JsonNode node, + int previousBytes, + long nextBytes) + { + if (node is JsonObject obj) + { + foreach (var property in obj.ToList()) + { + if (property.Key is "output_byte_limit" or "max_json_bytes" + && property.Value is JsonValue) + { + obj[property.Key] = nextBytes; + continue; + } + + if (property.Value is JsonValue value + && value.TryGetValue(out var text)) + { + obj[property.Key] = RewriteEmbeddedMaxJsonBytesArgument( + text, + previousBytes, + nextBytes); + } + else if (property.Value is not null) + { + RewriteEmbeddedJsonByteLimit(property.Value, previousBytes, nextBytes); + } + } + return; + } + + if (node is not JsonArray array) + return; + for (var index = 0; index < array.Count; index++) + { + if (array[index] is JsonValue value && value.TryGetValue(out var text)) + { + array[index] = RewriteEmbeddedMaxJsonBytesArgument(text, previousBytes, nextBytes); + } + else if (array[index] is not null) + { + RewriteEmbeddedJsonByteLimit(array[index]!, previousBytes, nextBytes); + } + } + } + + private static string RewriteEmbeddedMaxJsonBytesArgument( + string value, + int previousBytes, + long nextBytes) + { + var previousText = previousBytes.ToString(CultureInfo.InvariantCulture); + var nextText = nextBytes.ToString(CultureInfo.InvariantCulture); + return value + .Replace($"--max-json-bytes {previousText}", $"--max-json-bytes {nextText}", StringComparison.Ordinal) + .Replace($"--max-json-bytes={previousText}", $"--max-json-bytes={nextText}", StringComparison.Ordinal); + } + private static int WriteJsonPayloadWithOptionalByteLimit( JsonObject payload, QueryCommandOptions options, @@ -160,6 +265,7 @@ private static int WriteJsonPayloadWithOptionalByteLimit( options, outputDescription, hint, + jsonOptions, commandName); private static JsonSerializerOptions EnsureJsonNodeSerializerOptions(JsonSerializerOptions jsonOptions) @@ -721,10 +827,11 @@ private static int RunSearchRecipe(QueryCommandOptions options, JsonSerializerOp var compactJson = compactPayload.ToJsonString(GetJsonNodeSerializationOptions(jsonOptions)); return CompleteSearchRecipeOutput( WriteJsonObjectWithOptionalByteLimit( - compactJson, - options, - "recipe compact", - $"Reduce --limit or --total-limit, select one child query with {options.InvocationContext.RecipeCursorSelectorSyntax}, stream rows with --json=ndjson, or increase --max-json-bytes."), + compactJson, + options, + "recipe compact", + $"Reduce --limit or --total-limit, select one child query with {options.InvocationContext.RecipeCursorSelectorSyntax}, stream rows with --json=ndjson, or increase --max-json-bytes.", + jsonOptions), compactHasFailures); } @@ -783,7 +890,8 @@ private static int RunSearchRecipe(QueryCommandOptions options, JsonSerializerOp json, options, "recipe search", - "Reduce --limit, use --snippet-lines 0, or increase --max-json-bytes."), + "Reduce --limit, use --snippet-lines 0, or increase --max-json-bytes.", + jsonOptions), hasFailures); } @@ -1577,7 +1685,8 @@ private static int RunSearchRecipeAggregation(QueryCommandOptions options, JsonS json, options, "recipe aggregation", - "Reduce --limit or increase --max-json-bytes."); + "Reduce --limit or increase --max-json-bytes.", + jsonOptions); } else { @@ -1769,7 +1878,8 @@ private static int RunSearchRecipeIssueDrafts( json, options, "issue-draft", - "Reduce --limit, use --snippet-lines 0, or increase --max-json-bytes."), + "Reduce --limit, use --snippet-lines 0, or increase --max-json-bytes.", + jsonOptions), hasFailures); }); } @@ -1833,7 +1943,8 @@ private static int RunSearchRecipeCount(QueryCommandOptions options, JsonSeriali summaryJson, options, "recipe count summary", - "Use a larger --max-json-bytes value or narrow the recipe/query selection."), + "Use a larger --max-json-bytes value or narrow the recipe/query selection.", + jsonOptions), hasFailures); } @@ -1851,7 +1962,8 @@ private static int RunSearchRecipeCount(QueryCommandOptions options, JsonSeriali json, options, "recipe count", - "Use `--summary-only` to omit recipe metadata from count output."); + "Use `--summary-only` to omit recipe metadata from count output.", + jsonOptions); } else { @@ -1985,7 +2097,8 @@ private static int RunSearchIssueDrafts( json, options, "issue-draft", - "Reduce --limit, use --snippet-lines 0, or increase --max-json-bytes."); + "Reduce --limit, use --snippet-lines 0, or increase --max-json-bytes.", + jsonOptions); }); } diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs index 3c417cd56..b06023891 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchResults.cs @@ -50,7 +50,8 @@ private static int RunGroupedSearchCount(DbReader reader, QueryCommandOptions op json, options, "grouped search count", - "Reduce --limit or increase --max-json-bytes."); + "Reduce --limit or increase --max-json-bytes.", + jsonOptions); } else { @@ -87,7 +88,8 @@ private static int RunGroupedSearchCount(DbReader reader, QueryCommandOptions op json, options, "grouped search count", - "Reduce --limit or increase --max-json-bytes."); + "Reduce --limit or increase --max-json-bytes.", + jsonOptions); } else { @@ -330,7 +332,8 @@ private static int RunSearchAggregation(DbReader reader, QueryCommandOptions opt json, options, "search aggregation", - "Reduce --limit or increase --max-json-bytes."); + "Reduce --limit or increase --max-json-bytes.", + jsonOptions); } else { @@ -527,7 +530,8 @@ private static int WriteGroupedSearchResults( json, options, "grouped search results", - "Reduce --limit, --per-file-limit, or increase --max-json-bytes."); + "Reduce --limit, --per-file-limit, or increase --max-json-bytes.", + jsonOptions); } private static QueryCountResult CountSearchMatches(DbReader reader, QueryCommandOptions options, bool exact) @@ -655,7 +659,8 @@ private static int WriteProjectedSearchResults( writer.ToString().TrimEnd('\r', '\n'), options, "projected search result array", - "Reduce --limit, --search-fields, or use `--json=ndjson --max-json-bytes` for streaming output."); + "Reduce --limit, --search-fields, or use `--json=ndjson --max-json-bytes` for streaming output.", + jsonOptions); } var records = new List(projected.Length); @@ -910,7 +915,8 @@ private static bool TryWriteEmptySearchJsonWithOptionalByteLimit(QueryCommandOpt BuildCompactLocationsPayload([], options, jsonOptions).ToJsonString(jsonOptions), options, "compact search results", - "Increase --max-json-bytes or remove the byte cap."); + "Increase --max-json-bytes or remove the byte cap.", + jsonOptions); return true; } @@ -924,7 +930,8 @@ private static bool TryWriteEmptySearchJsonWithOptionalByteLimit(QueryCommandOpt }.ToJsonString(jsonOptions), options, "search count", - "Increase --max-json-bytes or remove the byte cap."); + "Increase --max-json-bytes or remove the byte cap.", + jsonOptions); return true; } @@ -970,7 +977,8 @@ private static int RunSearchNamedBatchCount(QueryCommandOptions options, JsonSer json, options, "named-query count summary", - "Use a larger --max-json-bytes value or narrow the named-query selection."); + "Use a larger --max-json-bytes value or narrow the named-query selection.", + jsonOptions); if (writeExitCode != CommandExitCodes.Success || !hasFailures) return writeExitCode; @@ -1011,7 +1019,8 @@ private static int RunSearchNamedBatch(QueryCommandOptions options, JsonSerializ json, options, "named-query search", - "Reduce --limit, use --snippet-lines 0, or increase --max-json-bytes."); + "Reduce --limit, use --snippet-lines 0, or increase --max-json-bytes.", + jsonOptions); } Console.WriteLine("Named search batch"); @@ -1804,7 +1813,8 @@ private static int WriteCompactSearchResults( writer.ToString().TrimEnd('\r', '\n'), options, "compact search results", - "Reduce --limit, --snippet-lines, or use `--json=ndjson --max-json-bytes` for streaming output."); + "Reduce --limit, --snippet-lines, or use `--json=ndjson --max-json-bytes` for streaming output.", + jsonOptions); } private static void WriteCompactSearchResults( diff --git a/src/CodeIndex/Cli/QueryCommandRunner.cs b/src/CodeIndex/Cli/QueryCommandRunner.cs index 6f0794522..b87b36282 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.cs @@ -179,6 +179,7 @@ public sealed class QueryCommandOptions public int GroupedPerFileLimit { get; init; } = 3; public bool GroupedPerFileLimitExplicit { get; init; } public int? SampleSize { get; init; } + public int? RequestedMaxJsonBytes { get; init; } public int? MaxJsonBytes { get; init; } public bool RawBytes { get; init; } public bool RawKinds { get; init; } diff --git a/src/CodeIndex/Cli/ScopedConsoleOutput.cs b/src/CodeIndex/Cli/ScopedConsoleOutput.cs index d75fbce21..6ddc87d8b 100644 --- a/src/CodeIndex/Cli/ScopedConsoleOutput.cs +++ b/src/CodeIndex/Cli/ScopedConsoleOutput.cs @@ -58,3 +58,57 @@ public void Dispose() } } } + +internal static class ScopedConsoleError +{ + private static readonly AsyncLocal s_activeRouter = new(); + + internal static IDisposable Register(IScopedConsoleOutputRouter router) + { + var previous = s_activeRouter.Value; + s_activeRouter.Value = router; + return new DelegateScope(() => s_activeRouter.Value = previous); + } + + internal static IDisposable Redirect(TextWriter target) + { + if (s_activeRouter.Value is { } router) + return router.Push(target); + + var ownership = ConsoleStreamOwnership.Enter(); + var original = Console.Error; + try + { + Console.SetError(target); + return new DelegateScope(() => + { + try + { + ConsoleStreamOwnership.RestoreError(original); + } + finally + { + ownership.Dispose(); + } + }); + } + catch + { + ownership.Dispose(); + throw; + } + } + + private sealed class DelegateScope(Action dispose) : IDisposable + { + private bool _disposed; + + public void Dispose() + { + if (_disposed) + return; + _disposed = true; + dispose(); + } + } +} diff --git a/tests/CodeIndex.Tests/JsonEnvelopeWrapperIssue4585Tests.cs b/tests/CodeIndex.Tests/JsonEnvelopeWrapperIssue4585Tests.cs index 5784dabf8..a47c2babc 100644 --- a/tests/CodeIndex.Tests/JsonEnvelopeWrapperIssue4585Tests.cs +++ b/tests/CodeIndex.Tests/JsonEnvelopeWrapperIssue4585Tests.cs @@ -467,7 +467,7 @@ public void Map_SelectedSectionPagesRowsAndScalarProjectionOmitsSections_Issue45 } [Fact] - public void BoundedParserFailure_RespectsHardByteCapByLeavingStdoutEmpty_Issue4585() + public void BoundedParserFailure_EmitsResponseBudgetError_Issues4585_4909() { var projectRoot = TestProjectHelper.CreateTempProject("bounded_parser_cap_4585"); try @@ -482,8 +482,19 @@ public void BoundedParserFailure_RespectsHardByteCapByLeavingStdoutEmpty_Issue45 "1.0.0-test")); Assert.Equal(CommandExitCodes.InvalidArgument, exitCode); - Assert.Equal(string.Empty, stdout); - Assert.Contains("raw JSON node count exceeded", stderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, stderr); + using var document = JsonDocument.Parse(stdout); + var error = document.RootElement; + Assert.Equal("E028_RESPONSE_BUDGET_TOO_SMALL", error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("find", error.GetProperty("command").GetString()); + Assert.Equal(128, error.GetProperty("requested_bytes").GetInt64()); + Assert.Equal(128, error.GetProperty("effective_bytes").GetInt64()); + Assert.True(error.GetProperty("minimum_required_bytes_known").GetBoolean()); + Assert.True(error.GetProperty("minimum_required_bytes_uncertain").GetBoolean()); + Assert.True( + error.GetProperty("retry").GetProperty("recommended_bytes").GetInt64() + > error.GetProperty("minimum_required_bytes").GetInt64()); } finally { @@ -601,6 +612,209 @@ public void Diagnostics_AreMetadataControlRecordsInsteadOfProjectedRows_Issue458 } } + [Fact] + public void BoundedResponse_BudgetPreflightIsParseableForEmptyAndEscapedNonEmptyRows_Issue4909() + { + var projectRoot = TestProjectHelper.CreateTempProject("bounded_response_budget_4909"); + try + { + var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); + TestProjectHelper.InsertIndexedFile( + dbPath, + "src/日本語/Quote\"Target.cs", + "csharp", + """ + namespace Demo; + public sealed class Target + { + public void Run() { } + } + """); + + var zeroArgs = new[] + { + "definition", "Target", "--db", dbPath, "--format", "compact", + "--max-json-bytes", "0", + }; + var (zeroExitCode, zeroStdout, zeroStderr) = CaptureConsole(() => ProgramRunner.Run( + zeroArgs, + _jsonOptions, + "1.0.0-test")); + + Assert.Equal(CommandExitCodes.UsageError, zeroExitCode); + Assert.Equal(string.Empty, zeroStderr); + using var zeroDocument = JsonDocument.Parse(zeroStdout); + var zeroError = zeroDocument.RootElement; + Assert.Equal("E028_RESPONSE_BUDGET_TOO_SMALL", zeroError.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", zeroError.GetProperty("category").GetString()); + Assert.Equal("definition", zeroError.GetProperty("command").GetString()); + Assert.Equal(0, zeroError.GetProperty("requested_bytes").GetInt64()); + Assert.Equal(JsonValueKind.Null, zeroError.GetProperty("effective_bytes").ValueKind); + Assert.False(zeroError.GetProperty("minimum_required_bytes_known").GetBoolean()); + Assert.Equal( + "normal_payload_not_materialized", + zeroError.GetProperty("minimum_required_bytes_unavailable_reason").GetString()); + + var (duplicateExitCode, duplicateStdout, duplicateStderr) = CaptureConsole(() => ProgramRunner.Run( + [ + "definition", "Target", "--db", dbPath, "--format", "compact", + "--max-json-bytes", "10", "--max-json-bytes", "0", + ], + _jsonOptions, + "1.0.0-test")); + + Assert.Equal(CommandExitCodes.UsageError, duplicateExitCode); + Assert.Equal(string.Empty, duplicateStderr); + using var duplicateDocument = JsonDocument.Parse(duplicateStdout); + var duplicateError = duplicateDocument.RootElement; + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + duplicateError.GetProperty("error_code").GetString()); + Assert.Equal(0, duplicateError.GetProperty("requested_bytes").GetInt64()); + + foreach (var orderedArgs in new[] + { + new[] + { + "definition", "Target", "--db", dbPath, "--format", "compact", + "--limit", "0", "--max-json-bytes", "0", + }, + new[] + { + "definition", "Target", "--db", dbPath, "--format", "compact", + "--max-json-bytes", "0", "--limit", "0", + }, + }) + { + var (orderedExitCode, orderedStdout, orderedStderr) = CaptureConsole(() => + ProgramRunner.Run( + orderedArgs, + _jsonOptions, + "1.0.0-test")); + + Assert.Equal(CommandExitCodes.UsageError, orderedExitCode); + Assert.Equal(string.Empty, orderedStderr); + using var orderedDocument = JsonDocument.Parse(orderedStdout); + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + orderedDocument.RootElement.GetProperty("error_code").GetString()); + Assert.Equal(0, orderedDocument.RootElement.GetProperty("requested_bytes").GetInt64()); + } + + var tinyArgs = new[] + { + "definition", "Target", "--db", dbPath, "--format", "compact", + "--max-json-bytes", "1", + }; + var (tinyExitCode, tinyStdout, tinyStderr) = CaptureConsole(() => ProgramRunner.Run( + tinyArgs, + _jsonOptions, + "1.0.0-test")); + + Assert.Equal(CommandExitCodes.UsageError, tinyExitCode); + Assert.Equal(string.Empty, tinyStderr); + using var tinyDocument = JsonDocument.Parse(tinyStdout); + var tinyError = tinyDocument.RootElement; + Assert.Equal("E028_RESPONSE_BUDGET_TOO_SMALL", tinyError.GetProperty("error_code").GetString()); + Assert.Equal("definition", tinyError.GetProperty("command").GetString()); + Assert.StartsWith( + "cdidx definition ", + tinyError.GetProperty("usage").GetString(), + StringComparison.Ordinal); + Assert.Equal(1, tinyError.GetProperty("requested_bytes").GetInt64()); + Assert.Equal(1, tinyError.GetProperty("effective_bytes").GetInt64()); + Assert.True(tinyError.GetProperty("minimum_required_bytes_known").GetBoolean()); + Assert.True(tinyError.GetProperty("minimum_required_bytes_uncertain").GetBoolean()); + Assert.Equal( + "runtime_metadata_or_embedded_budget_varies_between_invocations", + tinyError.GetProperty("minimum_required_bytes_uncertainty_reason").GetString()); + var minimumRequiredBytes = tinyError.GetProperty("minimum_required_bytes").GetInt64(); + var recommendedBytes = tinyError.GetProperty("retry").GetProperty("recommended_bytes").GetInt64(); + Assert.True(recommendedBytes > minimumRequiredBytes); + + var retryArgs = tinyArgs.ToArray(); + retryArgs[^1] = recommendedBytes.ToString(System.Globalization.CultureInfo.InvariantCulture); + var (retryExitCode, retryStdout, retryStderr) = CaptureConsole(() => ProgramRunner.Run( + retryArgs, + _jsonOptions, + "1.0.0-test")); + + Assert.Equal(CommandExitCodes.Success, retryExitCode); + Assert.Equal(string.Empty, retryStderr); + Assert.True(Encoding.UTF8.GetByteCount(retryStdout) <= recommendedBytes); + using var retryDocument = JsonDocument.Parse(retryStdout); + var retryResult = Assert.Single(retryDocument.RootElement.GetProperty("results").EnumerateArray()); + Assert.Equal("src/日本語/Quote\"Target.cs", retryResult.GetProperty("file").GetString()); + + var (emptyExitCode, emptyStdout, emptyStderr) = CaptureConsole(() => ProgramRunner.Run( + [ + "definition", "MissingSymbol", "--db", dbPath, "--format", "compact", + "--max-json-bytes", "1", + ], + _jsonOptions, + "1.0.0-test")); + + Assert.Equal(CommandExitCodes.UsageError, emptyExitCode); + Assert.Equal(string.Empty, emptyStderr); + using var emptyDocument = JsonDocument.Parse(emptyStdout); + var emptyError = emptyDocument.RootElement; + Assert.Equal("E028_RESPONSE_BUDGET_TOO_SMALL", emptyError.GetProperty("error_code").GetString()); + Assert.Contains( + "complete empty bounded response envelope", + emptyError.GetProperty("message").GetString(), + StringComparison.Ordinal); + Assert.True(emptyError.GetProperty("minimum_required_bytes_known").GetBoolean()); + + var findArgs = new[] + { + "find", "Target", "--db", dbPath, "--json", "--max-json-bytes", "1", + }; + var (findExitCode, findStdout, findStderr) = CaptureConsole(() => ProgramRunner.Run( + findArgs, + _jsonOptions, + "1.0.0-test")); + + Assert.Equal(CommandExitCodes.UsageError, findExitCode); + Assert.Equal(string.Empty, findStderr); + using var findDocument = JsonDocument.Parse(findStdout); + var findError = findDocument.RootElement; + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + findError.GetProperty("error_code").GetString()); + Assert.Contains( + "complete bounded error envelope", + findError.GetProperty("message").GetString(), + StringComparison.Ordinal); + + var findRetryArgs = findArgs.ToArray(); + findRetryArgs[^1] = findError + .GetProperty("retry") + .GetProperty("recommended_bytes") + .GetInt64() + .ToString(System.Globalization.CultureInfo.InvariantCulture); + var (findRetryExitCode, findRetryStdout, findRetryStderr) = CaptureConsole(() => + ProgramRunner.Run( + findRetryArgs, + _jsonOptions, + "1.0.0-test")); + + Assert.Equal(CommandExitCodes.UsageError, findRetryExitCode); + Assert.Equal(string.Empty, findRetryStderr); + using var findRetryDocument = JsonDocument.Parse(findRetryStdout); + Assert.Equal( + CommandErrorCodes.UsageError, + findRetryDocument.RootElement + .GetProperty("metadata") + .GetProperty("error") + .GetProperty("error_code") + .GetString()); + } + finally + { + TestProjectHelper.DeleteDirectory(projectRoot); + } + } + private static (int ExitCode, string Stdout, string Stderr) CaptureConsole(Func action) => ConsoleCapture.Capture(action); diff --git a/tests/CodeIndex.Tests/ProgramRunnerTests.cs b/tests/CodeIndex.Tests/ProgramRunnerTests.cs index 5e9691bd1..d00bd194f 100644 --- a/tests/CodeIndex.Tests/ProgramRunnerTests.cs +++ b/tests/CodeIndex.Tests/ProgramRunnerTests.cs @@ -451,10 +451,72 @@ public void RunRecipesAlias_InvalidArgumentsKeepRecipesDiagnostic_Issue4574(stri appVersion: "1.10.0")); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.Empty(stdout); - Assert.Contains(expectedMessage, stderr, StringComparison.Ordinal); - Assert.Contains("Usage: cdidx recipes", stderr, StringComparison.Ordinal); - Assert.DoesNotContain("Usage: cdidx search", stderr, StringComparison.Ordinal); + if (expectedMessage.StartsWith("exceeds --max-json-bytes", StringComparison.Ordinal)) + { + Assert.Empty(stderr); + using var document = JsonDocument.Parse(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("recipes", error.GetProperty("command").GetString()); + Assert.Contains("cdidx recipes ", error.GetProperty("usage").GetString(), StringComparison.Ordinal); + } + else + { + Assert.Empty(stdout); + Assert.Contains(expectedMessage, stderr, StringComparison.Ordinal); + Assert.Contains("Usage: cdidx recipes", stderr, StringComparison.Ordinal); + Assert.DoesNotContain("Usage: cdidx search", stderr, StringComparison.Ordinal); + } + } + + [Theory] + [InlineData("hotspots")] + [InlineData("deps")] + [InlineData("inspect")] + [InlineData("excerpt")] + [InlineData("symbols")] + public void RunQueryAliases_ZeroJsonBudgetUsesStructuredResponseBudgetError_Issue4909(string command) + { + var args = command switch + { + "hotspots" => new[] { command, "--json", "--summary-only", "--max-json-bytes", "0" }, + "deps" => [command, "--json", "--summary-only", "--max-json-bytes", "0"], + "inspect" => [command, "Target", "--json", "--max-json-bytes", "0"], + "excerpt" => [command, "missing.cs", "--start", "1", "--json", "--max-json-bytes", "0"], + _ => [command, "Target", "--count", "--json", "--max-json-bytes", "0"], + }; + var (exitCode, stdout, stderr) = CaptureConsole(() => ProgramRunner.Run( + args, + appVersion: "1.10.0")); + + Assert.Equal(CommandExitCodes.UsageError, exitCode); + Assert.Empty(stderr); + using var document = JsonDocument.Parse(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal(command, error.GetProperty("command").GetString()); + Assert.Equal(0, error.GetProperty("requested_bytes").GetInt64()); + Assert.Equal(JsonValueKind.Null, error.GetProperty("effective_bytes").ValueKind); + } + + [Fact] + public void RunQuery_MultipleParseErrorsKeepBudgetMessageAndOtherValidationErrors_Issue4909() + { + var (exitCode, stdout, stderr) = CaptureConsole(() => ProgramRunner.Run( + ["map", "--json", "--limit", "0", "--max-json-bytes", "0"], + appVersion: "1.10.0")); + + Assert.Equal(CommandExitCodes.UsageError, exitCode); + Assert.Empty(stderr); + using var document = JsonDocument.Parse(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Contains("--max-json-bytes", error.GetProperty("message").GetString(), StringComparison.Ordinal); + Assert.DoesNotContain("--limit", error.GetProperty("message").GetString(), StringComparison.Ordinal); + var otherError = Assert.Single(error.GetProperty("other_validation_errors").EnumerateArray()); + Assert.Contains("--limit", otherError.GetString(), StringComparison.Ordinal); } [Fact] diff --git a/tests/CodeIndex.Tests/ProjectionFieldRegistryIssue4836Tests.cs b/tests/CodeIndex.Tests/ProjectionFieldRegistryIssue4836Tests.cs index aa7822eee..837ec4c1c 100644 --- a/tests/CodeIndex.Tests/ProjectionFieldRegistryIssue4836Tests.cs +++ b/tests/CodeIndex.Tests/ProjectionFieldRegistryIssue4836Tests.cs @@ -355,12 +355,15 @@ public void EarlyProjectionRegistryResponses_HonorMaxJsonBytes_Issue4836(string "1.0.0-test")); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.True(Encoding.UTF8.GetByteCount(stdout) <= maxJsonBytes); - Assert.Equal(string.Empty, stdout); - Assert.Contains( - $"--max-json-bytes {maxJsonBytes} is too small", - stderr, - StringComparison.Ordinal); + Assert.Equal(string.Empty, stderr); + using var document = JsonDocument.Parse(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("status", error.GetProperty("command").GetString()); + Assert.Equal(maxJsonBytes, error.GetProperty("requested_bytes").GetInt64()); + Assert.Equal(maxJsonBytes, error.GetProperty("effective_bytes").GetInt64()); + Assert.True(error.GetProperty("minimum_required_bytes_known").GetBoolean()); } [Theory] diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerGraphTests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerGraphTests.cs index 0a7861b67..b52185351 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerGraphTests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerGraphTests.cs @@ -227,20 +227,26 @@ public void RunDeps_JsonFormatsShareMaxBytesFixture_Issue4112() _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.Equal(string.Empty, stdout); - Assert.Contains("deps JSON output is", stderr); - Assert.Contains("exceeds --max-json-bytes 1", stderr); - Assert.Contains("Usage: cdidx deps", stderr); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("deps", error.GetProperty("command").GetString()); + Assert.Equal(1, error.GetProperty("requested_bytes").GetInt64()); + Assert.True(error.GetProperty("minimum_required_bytes_known").GetBoolean()); var (graphExitCode, graphStdout, graphStderr) = CaptureConsole(() => QueryCommandRunner.RunDeps( ["--db", dbPath, "--format", "json-graph", "--max-json-bytes", "1", "--lang", "sql"], _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, graphExitCode); - Assert.Equal(string.Empty, graphStdout); - Assert.Contains("deps JSON output is", graphStderr); - Assert.Contains("exceeds --max-json-bytes 1", graphStderr); - Assert.Contains("Usage: cdidx deps", graphStderr); + Assert.Equal(string.Empty, graphStderr); + using var graphDocument = ParseJsonOutput(graphStdout); + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + graphDocument.RootElement.GetProperty("error_code").GetString()); + Assert.Equal("deps", graphDocument.RootElement.GetProperty("command").GetString()); } finally { @@ -299,10 +305,13 @@ public void RunDeps_MissingGraphJsonModesShareZeroPayloadFixture_Issues4112And46 _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.Equal(string.Empty, stdout); - Assert.Contains("deps JSON output is", stderr); - Assert.Contains("exceeds --max-json-bytes 1", stderr); - Assert.Contains("Usage: cdidx deps", stderr); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("deps", error.GetProperty("command").GetString()); + Assert.Equal(1, error.GetProperty("requested_bytes").GetInt64()); var (summaryExitCode, summaryStdout, _) = CaptureConsole(() => QueryCommandRunner.RunDeps( ["--db", readOnlyUri, "--json", "--summary-only"], @@ -459,10 +468,13 @@ private void Shared() _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.Equal(string.Empty, stdout); - Assert.Contains("hotspots JSON output is", stderr); - Assert.Contains("exceeds --max-json-bytes 1", stderr); - Assert.Contains("Usage: cdidx hotspots", stderr); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("hotspots", error.GetProperty("command").GetString()); + Assert.Equal(1, error.GetProperty("requested_bytes").GetInt64()); } finally { @@ -481,10 +493,13 @@ public void RunHotspots_JsonMaxBytes_AppliesToMissingGraphZeroPayload_Issue4112( _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.Equal(string.Empty, stdout); - Assert.Contains("hotspots JSON output is", stderr); - Assert.Contains("exceeds --max-json-bytes 1", stderr); - Assert.Contains("Usage: cdidx hotspots", stderr); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("hotspots", error.GetProperty("command").GetString()); + Assert.Equal(1, error.GetProperty("requested_bytes").GetInt64()); } finally { diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerInspectTests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerInspectTests.cs index a46e8c886..957e4ad88 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerInspectTests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerInspectTests.cs @@ -1522,10 +1522,13 @@ public int Compute() _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.Equal(string.Empty, stdout); - Assert.Contains("inspect JSON output", stderr); - Assert.Contains("--max-json-bytes 64", stderr); - Assert.Contains("--compact", stderr); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("inspect", error.GetProperty("command").GetString()); + Assert.Equal(64, error.GetProperty("requested_bytes").GetInt64()); } finally { diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerMapTests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerMapTests.cs index cb7123749..d6c402aa2 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerMapTests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerMapTests.cs @@ -1,4 +1,5 @@ using System.Globalization; +using System.Text; using System.Text.Json; using CodeIndex.Cli; using CodeIndex.Database; @@ -495,9 +496,53 @@ public void RunMap_CompactJsonHonorsMaxJsonBytesAndReportsNextCommands_Issue4183 _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, capExitCode); - Assert.Equal(string.Empty, capStdout); - Assert.Contains("map JSON output", capStderr, StringComparison.Ordinal); - Assert.Contains("Usage: cdidx map", capStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, capStderr); + using var capDocument = ParseJsonOutput(capStdout); + var capError = capDocument.RootElement; + Assert.Equal("E028_RESPONSE_BUDGET_TOO_SMALL", capError.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", capError.GetProperty("category").GetString()); + Assert.Equal("map", capError.GetProperty("command").GetString()); + Assert.StartsWith("cdidx map ", capError.GetProperty("usage").GetString(), StringComparison.Ordinal); + Assert.Equal(1, capError.GetProperty("requested_bytes").GetInt64()); + Assert.Equal(1, capError.GetProperty("effective_bytes").GetInt64()); + Assert.True(capError.GetProperty("minimum_required_bytes_known").GetBoolean()); + Assert.False(capError.GetProperty("minimum_required_bytes_uncertain").GetBoolean()); + var minimumRequiredBytes = capError.GetProperty("minimum_required_bytes").GetInt64(); + Assert.True(minimumRequiredBytes > 1); + Assert.Equal( + minimumRequiredBytes, + capError.GetProperty("retry").GetProperty("recommended_bytes").GetInt64()); + + var (exactExitCode, exactStdout, exactStderr) = CaptureConsole(() => QueryCommandRunner.RunMap( + [ + "--db", dbPath, "--compact", "--max-json-bytes", + minimumRequiredBytes.ToString(CultureInfo.InvariantCulture), + ], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, exactExitCode); + Assert.Equal(string.Empty, exactStderr); + Assert.Equal(minimumRequiredBytes, Encoding.UTF8.GetByteCount(exactStdout)); + using var exactDocument = ParseJsonOutput(exactStdout); + Assert.Equal( + minimumRequiredBytes, + exactDocument.RootElement.GetProperty("output_byte_limit").GetInt64()); + + var (zeroExitCode, zeroStdout, zeroStderr) = CaptureConsole(() => QueryCommandRunner.RunMap( + ["--db", dbPath, "--json", "--max-json-bytes", "0"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.UsageError, zeroExitCode); + Assert.Equal(string.Empty, zeroStderr); + using var zeroDocument = ParseJsonOutput(zeroStdout); + var zeroError = zeroDocument.RootElement; + Assert.Equal("E028_RESPONSE_BUDGET_TOO_SMALL", zeroError.GetProperty("error_code").GetString()); + Assert.Equal(0, zeroError.GetProperty("requested_bytes").GetInt64()); + Assert.Equal(JsonValueKind.Null, zeroError.GetProperty("effective_bytes").ValueKind); + Assert.False(zeroError.GetProperty("minimum_required_bytes_known").GetBoolean()); + Assert.Equal( + "normal_payload_not_materialized", + zeroError.GetProperty("minimum_required_bytes_unavailable_reason").GetString()); } finally { diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerOutlineIssue4880Tests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerOutlineIssue4880Tests.cs index 733533154..a75409423 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerOutlineIssue4880Tests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerOutlineIssue4880Tests.cs @@ -123,10 +123,17 @@ public sealed class LevelTwo "1.0.0-test")); Assert.Equal(CommandExitCodes.UsageError, smallExitCode); - Assert.Equal(string.Empty, smallStdout); - Assert.Contains($"Error [{CommandErrorCodes.UsageError}]", smallStderr, StringComparison.Ordinal); - Assert.Contains("bounded response metadata and one projected row", smallStderr, StringComparison.Ordinal); - Assert.Contains("--outline-fields", smallStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, smallStderr); + using var smallDocument = JsonDocument.Parse(smallStdout); + var smallError = smallDocument.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, smallError.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", smallError.GetProperty("category").GetString()); + Assert.Equal("outline", smallError.GetProperty("command").GetString()); + Assert.Contains( + "bounded response metadata and one projected row", + smallError.GetProperty("message").GetString(), + StringComparison.Ordinal); + Assert.True(smallError.GetProperty("minimum_required_bytes_known").GetBoolean()); } finally { @@ -264,9 +271,15 @@ public void Outline_MaxJsonBytesPreservesUnsupportedOutputSelectorValidation_Iss "1.0.0-test")); Assert.Equal(CommandExitCodes.UsageError, cappedExitCode); - Assert.Equal(string.Empty, cappedStdout); - Assert.Contains($"Error [{CommandErrorCodes.UsageError}]", cappedStderr, StringComparison.Ordinal); - Assert.Contains("is not supported", cappedStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, cappedStderr); + using var cappedDocument = JsonDocument.Parse(cappedStdout); + var cappedError = cappedDocument.RootElement; + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + cappedError.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", cappedError.GetProperty("category").GetString()); + Assert.Equal("outline", cappedError.GetProperty("command").GetString()); + Assert.True(cappedError.GetProperty("minimum_required_bytes_known").GetBoolean()); } } finally diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerSearchTests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerSearchTests.cs index 755e3426b..d10950458 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerSearchTests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerSearchTests.cs @@ -1,3 +1,4 @@ +using System.Globalization; using System.Reflection; using System.Text; using System.Text.RegularExpressions; @@ -1348,8 +1349,12 @@ public void RunSearch_NamedQueriesHonorCompactProjectionAndPreserveRichJson_Issu _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, capExitCode); - Assert.Equal(string.Empty, capStdout); - Assert.Contains("named-query search JSON output", capStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, capStderr); + using var capDocument = ParseJsonOutput(capStdout); + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + capDocument.RootElement.GetProperty("error_code").GetString()); + Assert.Equal("search", capDocument.RootElement.GetProperty("command").GetString()); } finally { @@ -3442,8 +3447,12 @@ public void RunSearch_ListRecipesNamesJsonEmitsDeterministicSmallPayload_Issue40 _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, capExitCode); - Assert.Equal(string.Empty, capStdout); - Assert.Contains("recipe-name list JSON output", capStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, capStderr); + using var capDocument = ParseJsonOutput(capStdout); + Assert.Equal( + "E028_RESPONSE_BUDGET_TOO_SMALL", + capDocument.RootElement.GetProperty("error_code").GetString()); + Assert.Equal("search", capDocument.RootElement.GetProperty("command").GetString()); var (textCapExitCode, textCapStdout, textCapStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( ["--list-recipes", "--names", "--max-json-bytes", "1"], @@ -3458,8 +3467,66 @@ public void RunSearch_ListRecipesNamesJsonEmitsDeterministicSmallPayload_Issue40 _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, compactNamesCapExitCode); - Assert.Equal(string.Empty, compactNamesCapStdout); - Assert.Contains("recipe-name list JSON output", compactNamesCapStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, compactNamesCapStderr); + using var compactCapDocument = ParseJsonOutput(compactNamesCapStdout); + Assert.Equal( + "E028_RESPONSE_BUDGET_TOO_SMALL", + compactCapDocument.RootElement.GetProperty("error_code").GetString()); + } + + [Fact] + public void RunRecipeList_ResponseBudgetFailuresAreJsonAndExactMinimumIsRetryable_Issue4909() + { + var (capExitCode, capStdout, capStderr) = CaptureConsole(() => QueryCommandRunner.RunRecipeList( + ["--names", "--json", "--max-json-bytes", "1"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.UsageError, capExitCode); + Assert.Equal(string.Empty, capStderr); + using var capDocument = ParseJsonOutput(capStdout); + var capError = capDocument.RootElement; + Assert.Equal("E028_RESPONSE_BUDGET_TOO_SMALL", capError.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", capError.GetProperty("category").GetString()); + Assert.Equal("recipes", capError.GetProperty("command").GetString()); + Assert.StartsWith("cdidx recipes ", capError.GetProperty("usage").GetString(), StringComparison.Ordinal); + Assert.Equal(1, capError.GetProperty("requested_bytes").GetInt64()); + Assert.Equal(1, capError.GetProperty("effective_bytes").GetInt64()); + Assert.True(capError.GetProperty("minimum_required_bytes_known").GetBoolean()); + Assert.False(capError.GetProperty("minimum_required_bytes_uncertain").GetBoolean()); + var minimumRequiredBytes = capError.GetProperty("minimum_required_bytes").GetInt64(); + Assert.True(minimumRequiredBytes > 1); + + var (exactExitCode, exactStdout, exactStderr) = CaptureConsole(() => QueryCommandRunner.RunRecipeList( + [ + "--names", "--json", "--max-json-bytes", + minimumRequiredBytes.ToString(CultureInfo.InvariantCulture), + ], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, exactExitCode); + Assert.Equal(string.Empty, exactStderr); + Assert.Equal(minimumRequiredBytes, Encoding.UTF8.GetByteCount(exactStdout)); + using var exactDocument = ParseJsonOutput(exactStdout); + Assert.Contains( + exactDocument.RootElement.GetProperty("names").EnumerateArray(), + name => name.GetString() == "risky-code"); + + var (zeroExitCode, zeroStdout, zeroStderr) = CaptureConsole(() => QueryCommandRunner.RunRecipeList( + ["--names", "--json", "--max-json-bytes", "0"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.UsageError, zeroExitCode); + Assert.Equal(string.Empty, zeroStderr); + using var zeroDocument = ParseJsonOutput(zeroStdout); + var zeroError = zeroDocument.RootElement; + Assert.Equal("E028_RESPONSE_BUDGET_TOO_SMALL", zeroError.GetProperty("error_code").GetString()); + Assert.Equal("recipes", zeroError.GetProperty("command").GetString()); + Assert.Equal(0, zeroError.GetProperty("requested_bytes").GetInt64()); + Assert.Equal(JsonValueKind.Null, zeroError.GetProperty("effective_bytes").ValueKind); + Assert.False(zeroError.GetProperty("minimum_required_bytes_known").GetBoolean()); + Assert.Equal( + "normal_payload_not_materialized", + zeroError.GetProperty("minimum_required_bytes_unavailable_reason").GetString()); } [Fact] @@ -6299,16 +6366,24 @@ public void Run(Exception ex) _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, capExitCode); - Assert.Equal(string.Empty, capStdout); - Assert.Contains("recipe count summary JSON output", capStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, capStderr); + using var capDocument = ParseJsonOutput(capStdout); + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + capDocument.RootElement.GetProperty("error_code").GetString()); + Assert.Equal("search", capDocument.RootElement.GetProperty("command").GetString()); var (aggregationCapExitCode, aggregationCapStdout, aggregationCapStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( ["--recipe", "risky-code/raw-diagnostic-echo", "--db", dbPath, "--format", "count", "--summary-only", "--group-by", "file", "--origin", "code", "--max-json-bytes", "1"], _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, aggregationCapExitCode); - Assert.Equal(string.Empty, aggregationCapStdout); - Assert.Contains("recipe aggregation JSON output", aggregationCapStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, aggregationCapStderr); + using var aggregationCapDocument = ParseJsonOutput(aggregationCapStdout); + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + aggregationCapDocument.RootElement.GetProperty("error_code").GetString()); + Assert.Equal("search", aggregationCapDocument.RootElement.GetProperty("command").GetString()); } finally { @@ -6368,8 +6443,12 @@ public void Run(Exception ex) _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, capExitCode); - Assert.Equal(string.Empty, capStdout); - Assert.Contains("recipe compact JSON output", capStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, capStderr); + using var capDocument = ParseJsonOutput(capStdout); + Assert.Equal( + "E028_RESPONSE_BUDGET_TOO_SMALL", + capDocument.RootElement.GetProperty("error_code").GetString()); + Assert.Equal("search", capDocument.RootElement.GetProperty("command").GetString()); } finally { @@ -6577,16 +6656,24 @@ public void RunSearch_MaxJsonBytesBoundsCompactOutput_Issue4119() _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, zeroExitCode); - Assert.Equal(string.Empty, zeroStdout); - Assert.Contains("compact search results JSON output", zeroStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, zeroStderr); + using var zeroDocument = ParseJsonOutput(zeroStdout); + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + zeroDocument.RootElement.GetProperty("error_code").GetString()); + Assert.Equal("search", zeroDocument.RootElement.GetProperty("command").GetString()); var (groupedZeroExitCode, groupedZeroStdout, groupedZeroStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( ["MissingToken", "--db", dbPath, "--format", "grouped", "--max-json-bytes", "1"], _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, groupedZeroExitCode); - Assert.Equal(string.Empty, groupedZeroStdout); - Assert.Contains("grouped search results JSON output", groupedZeroStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, groupedZeroStderr); + using var groupedZeroDocument = ParseJsonOutput(groupedZeroStdout); + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + groupedZeroDocument.RootElement.GetProperty("error_code").GetString()); + Assert.Equal("search", groupedZeroDocument.RootElement.GetProperty("command").GetString()); } finally { @@ -6625,8 +6712,134 @@ public void RunSearch_NdjsonByteCapReportsPartialOrRejectsUndersizedTerminal_Iss _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, zeroExitCode); - Assert.Equal(string.Empty, zeroStdout); - Assert.Contains("terminal record", zeroStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, zeroStderr); + using var zeroDocument = ParseJsonOutput(zeroStdout); + var zeroError = zeroDocument.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, zeroError.GetProperty("error_code").GetString()); + Assert.Equal("search", zeroError.GetProperty("command").GetString()); + Assert.Equal(1, zeroError.GetProperty("requested_bytes").GetInt64()); + Assert.True(zeroError.GetProperty("minimum_required_bytes_known").GetBoolean()); + Assert.True(zeroError.GetProperty("minimum_required_bytes_uncertain").GetBoolean()); + + var (resultsOnlyExitCode, resultsOnlyStdout, resultsOnlyStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( + ["token", "--db", dbPath, "--json=ndjson", "--results-only", "--max-json-bytes", "1"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.UsageError, resultsOnlyExitCode); + Assert.Equal(string.Empty, resultsOnlyStderr); + using var resultsOnlyDocument = ParseJsonOutput(resultsOnlyStdout); + var resultsOnlyError = resultsOnlyDocument.RootElement; + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + resultsOnlyError.GetProperty("error_code").GetString()); + Assert.Equal("search", resultsOnlyError.GetProperty("command").GetString()); + Assert.True(resultsOnlyError.GetProperty("minimum_required_bytes_known").GetBoolean()); + Assert.False(resultsOnlyError.GetProperty("minimum_required_bytes_uncertain").GetBoolean()); + } + finally + { + TestProjectHelper.DeleteDirectory(projectRoot); + } + } + + [Theory] + [InlineData(1, 1)] + [InlineData(20_000_000, 16 * 1024 * 1024)] + public void WriteJsonObject_ResponseAboveEffectiveMaximumRequiresSizeReduction_Issue4909( + int requestedMaxJsonBytes, + int expectedEffectiveBytes) + { + var options = QueryCommandRunner.ParseArgs( + [ + "needle", + "--json=array", + "--max-json-bytes", + requestedMaxJsonBytes.ToString(System.Globalization.CultureInfo.InvariantCulture), + ], + jsonDefault: false); + Assert.Equal(requestedMaxJsonBytes, options.RequestedMaxJsonBytes); + Assert.Equal(expectedEffectiveBytes, options.MaxJsonBytes); + var oversizedJson = JsonSerializer.Serialize(new + { + payload = new string('x', QueryCommandRunner.MaxSearchJsonByteLimit + 1), + }); + + var (exitCode, stdout, stderr) = CaptureConsole(() => + QueryCommandRunner.WriteJsonObjectWithOptionalByteLimit( + oversizedJson, + options, + "oversized test payload", + "Reduce the test payload.", + _jsonOptions, + "search")); + + Assert.Equal(CommandExitCodes.UsageError, exitCode); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var error = document.RootElement; + Assert.Equal(requestedMaxJsonBytes, error.GetProperty("requested_bytes").GetInt64()); + Assert.Equal(expectedEffectiveBytes, error.GetProperty("effective_bytes").GetInt64()); + Assert.True( + error.GetProperty("minimum_required_bytes").GetInt64() + > QueryCommandRunner.MaxSearchJsonByteLimit); + var retry = error.GetProperty("retry"); + Assert.Equal("reduce_response_size", retry.GetProperty("action").GetString()); + Assert.Equal(JsonValueKind.Null, retry.GetProperty("option").ValueKind); + Assert.Equal(JsonValueKind.Null, retry.GetProperty("recommended_bytes").ValueKind); + Assert.Equal( + QueryCommandRunner.MaxSearchJsonByteLimit, + retry.GetProperty("maximum_effective_bytes").GetInt64()); + } + + [Fact] + public void RunSearch_PrettyIssueDraftBudgetRecommendationIsRetryable_Issue4909() + { + var projectRoot = TestProjectHelper.CreateTempProject("cdidx_pretty_issue_draft_budget_4909"); + try + { + var dbPath = TestProjectHelper.CreateProjectDb(projectRoot); + TestProjectHelper.InsertIndexedFile( + dbPath, + "src/pretty.cs", + "csharp", + "public sealed class PrettyIssue4909Needle { }\n"); + var prettyJsonOptions = new JsonSerializerOptions(_jsonOptions) + { + WriteIndented = true, + }; + var args = new[] + { + "PrettyIssue4909Needle", + "--db", dbPath, + "--format", "issue-drafts", + "--exact-substring", + "--limit", "1", + "--snippet-lines", "0", + "--max-json-bytes", "1", + }; + + var (exitCode, stdout, stderr) = CaptureConsole(() => + QueryCommandRunner.RunSearch(args, prettyJsonOptions)); + + Assert.Equal(CommandExitCodes.UsageError, exitCode); + Assert.Equal(string.Empty, stderr); + using var errorDocument = JsonDocument.Parse(stdout); + var recommendedBytes = errorDocument.RootElement + .GetProperty("retry") + .GetProperty("recommended_bytes") + .GetInt64(); + + var retryArgs = args.ToArray(); + retryArgs[^1] = recommendedBytes.ToString( + System.Globalization.CultureInfo.InvariantCulture); + var (retryExitCode, retryStdout, retryStderr) = CaptureConsole(() => + QueryCommandRunner.RunSearch(retryArgs, prettyJsonOptions)); + + Assert.Equal(CommandExitCodes.Success, retryExitCode); + Assert.Equal(string.Empty, retryStderr); + Assert.True(Encoding.UTF8.GetByteCount(retryStdout) <= recommendedBytes); + using var retryDocument = JsonDocument.Parse(retryStdout); + Assert.Single(retryDocument.RootElement.GetProperty("drafts").EnumerateArray()); } finally { @@ -10298,8 +10511,12 @@ public void Run(System.Exception ex) _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, capExitCode); - Assert.Equal(string.Empty, capStdout); - Assert.Contains("issue-draft JSON output", capStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, capStderr); + using var capDocument = ParseJsonOutput(capStdout); + Assert.Equal( + "E028_RESPONSE_BUDGET_TOO_SMALL", + capDocument.RootElement.GetProperty("error_code").GetString()); + Assert.Equal("search", capDocument.RootElement.GetProperty("command").GetString()); } finally { @@ -12507,10 +12724,13 @@ public void RunExcerpt_JsonMaxBytesRejectsOversizedPayload_Issue4311() _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.Equal(string.Empty, stdout); - Assert.Contains("excerpt JSON output", stderr); - Assert.Contains("--max-json-bytes 64", stderr); - Assert.Contains("--no-semantic-tokens", stderr); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("excerpt", error.GetProperty("command").GetString()); + Assert.Equal(64, error.GetProperty("requested_bytes").GetInt64()); } finally { @@ -15696,8 +15916,13 @@ public void RunSearch_SearchFieldsJsonArrayHonorsMaxJsonBytes_Issue4119() _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.Equal(string.Empty, stdout); - Assert.Contains("projected search result array JSON output", stderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("search", error.GetProperty("command").GetString()); + Assert.Equal(1, error.GetProperty("requested_bytes").GetInt64()); } finally { diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerStreamingIssue4561Tests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerStreamingIssue4561Tests.cs index 4aa8114df..d0db457e9 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerStreamingIssue4561Tests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerStreamingIssue4561Tests.cs @@ -24,8 +24,18 @@ public void RunSearch_NdjsonHardCapIncludesTerminalAndRequiresExplicitPartialOpt _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, tinyExitCode); - Assert.Equal(string.Empty, tinyStdout); - Assert.Contains("terminal record", tinyStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, tinyStderr); + using (var tinyDocument = ParseJsonOutput(tinyStdout)) + { + var error = tinyDocument.RootElement; + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + error.GetProperty("error_code").GetString()); + Assert.Equal("search", error.GetProperty("command").GetString()); + Assert.Equal(1, error.GetProperty("requested_bytes").GetInt64()); + Assert.True(error.GetProperty("minimum_required_bytes_known").GetBoolean()); + Assert.True(error.GetProperty("minimum_required_bytes_uncertain").GetBoolean()); + } var (partialExitCode, partialStdout, partialStderr) = CaptureConsole(() => QueryCommandRunner.RunSearch( ["Issue4561Needle", "--db", dbPath, "--json=ndjson", "--max-json-bytes", "600"], diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerSymbolTests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerSymbolTests.cs index f0bbf4499..875b3c87f 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerSymbolTests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerSymbolTests.cs @@ -387,9 +387,13 @@ public void RunSymbols_CountJsonMaxJsonBytesRejectsBareVerbatimZero_Issue4165() _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.Equal(string.Empty, stdout); - Assert.Contains("symbols count JSON output", stderr); - Assert.Contains("exceeds --max-json-bytes 1", stderr); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("symbols", error.GetProperty("command").GetString()); + Assert.Equal(1, error.GetProperty("requested_bytes").GetInt64()); } [Fact] @@ -599,9 +603,13 @@ public void RunDefinition_JsonMaxBytesRejectsOversizedRows_Issue4311() _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, exitCode); - Assert.Equal(string.Empty, stdout); - Assert.Contains("definition JSON output", stderr); - Assert.Contains("--max-json-bytes 20", stderr); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var error = document.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, error.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", error.GetProperty("category").GetString()); + Assert.Equal("definition", error.GetProperty("command").GetString()); + Assert.Equal(20, error.GetProperty("requested_bytes").GetInt64()); } finally { @@ -7131,8 +7139,15 @@ public void HandleRequest() { } _jsonOptions)); Assert.Equal(CommandExitCodes.UsageError, boundedExitCode); - Assert.Equal(string.Empty, boundedStdout); - Assert.Contains("exceeds --max-json-bytes 20", boundedStderr); + Assert.Equal(string.Empty, boundedStderr); + using var boundedDocument = ParseJsonOutput(boundedStdout); + var boundedError = boundedDocument.RootElement; + Assert.Equal( + CommandErrorCodes.ResponseBudgetTooSmall, + boundedError.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", boundedError.GetProperty("category").GetString()); + Assert.Equal("definition", boundedError.GetProperty("command").GetString()); + Assert.Equal(20, boundedError.GetProperty("requested_bytes").GetInt64()); } finally { diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerUnusedIssue4905Tests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerUnusedIssue4905Tests.cs index 2ceebcc42..fa96ffc29 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerUnusedIssue4905Tests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerUnusedIssue4905Tests.cs @@ -153,9 +153,17 @@ public void RunUnused_MaxJsonBytesHandlesMinimumEmptyExactBoundaryAndHelp_Issue4 "1.0.0-test")); Assert.Equal(CommandExitCodes.UsageError, smallExitCode); - Assert.Equal(string.Empty, smallStdout); - Assert.Contains($"Error [{CommandErrorCodes.UsageError}]", smallStderr, StringComparison.Ordinal); - Assert.Contains("bounded response metadata and one projected row", smallStderr, StringComparison.Ordinal); + Assert.Equal(string.Empty, smallStderr); + using var smallDocument = JsonDocument.Parse(smallStdout); + var smallError = smallDocument.RootElement; + Assert.Equal(CommandErrorCodes.ResponseBudgetTooSmall, smallError.GetProperty("error_code").GetString()); + Assert.Equal("response_budget", smallError.GetProperty("category").GetString()); + Assert.Equal("unused", smallError.GetProperty("command").GetString()); + Assert.Contains( + "bounded response metadata and one projected row", + smallError.GetProperty("message").GetString(), + StringComparison.Ordinal); + Assert.Equal(64, smallError.GetProperty("requested_bytes").GetInt64()); var (emptyExitCode, emptyStdout, emptyStderr) = CaptureConsole(() => ProgramRunner.Run(