Avoid duplicate unused symbols in by-bucket JSON

README.md

@@ -78,6 +78,7 @@ cdidx references UserService --fields path,line,reference_kind --limit 20 --max-
 cdidx inspect QueryCommandRunner --outline-only
 cdidx outline src/CodeIndex/Cli/QueryCommandRunner.cs --json --kind function --limit 50 --max-json-bytes 16384
 cdidx unused --compact --by-bucket --max-json-bytes 16384
+cdidx unused --json --by-bucket --max-json-bytes 16384
 cdidx map --compact --max-json-bytes 65536
 cdidx map --format issue-drafts --limit 10
 cdidx search --recipe risky-code --format compact --max-json-bytes 65536
@@ -93,6 +94,16 @@ cdidx license --json
 cdidx validate
 ```
 
+For full `unused --json --by-bucket` output, `symbols` is the canonical row
+collection and each `by_bucket` entry is a lightweight compatibility projection
+with a zero-based `symbol_index` into that collection plus common locator
+fields. `by_bucket_format` is `canonical_symbol_index_v1`; compact output uses
+`summary_v1`. `--max-json-bytes` bounds the complete UTF-8 response and, when
+needed, returns a canonical-row prefix with `truncated`, `omitted_count`, and
+`next_cursor` so following pages neither duplicate nor skip symbols.
+The MCP `unused_symbols` equivalent uses `symbols_by_bucket` and
+`symbols_by_bucket_format` for the same indexed projection.
+
 For C# explicit-interface implementations, symbol output and outlines keep the
 short display name (`Run`, `Value`, `Changed`, or `Item`). Qualified exact-name
 queries use the interface spelling, such as `IFoo.Run<T>`, `IFoo.Value`,
@@ -575,6 +586,7 @@ cdidx references UserService --fields path,line,reference_kind --limit 20 --max-
 cdidx inspect QueryCommandRunner --outline-only
 cdidx outline src/CodeIndex/Cli/QueryCommandRunner.cs --json --kind function --limit 50 --max-json-bytes 16384
 cdidx unused --compact --by-bucket --max-json-bytes 16384
+cdidx unused --json --by-bucket --max-json-bytes 16384
 cdidx map --compact --max-json-bytes 65536
 cdidx map --format issue-drafts --limit 10
 cdidx search --recipe risky-code --format compact --max-json-bytes 65536
@@ -590,6 +602,15 @@ cdidx license --json
 cdidx validate
 ```
 
+完全形式の `unused --json --by-bucket` 出力では、`symbols` が正規の行集合で、
+各 `by_bucket` 要素はその集合への 0 始まりの `symbol_index` と共通の位置情報を持つ
+軽量な互換投影です。`by_bucket_format` は `canonical_symbol_index_v1`、compact
+出力では `summary_v1` です。`--max-json-bytes` は UTF-8 応答全体を上限内に収め、
+必要な場合は正規行の prefix と `truncated`、`omitted_count`、`next_cursor` を返すため、
+後続ページで symbol の重複や欠落が発生しません。
+MCP の `unused_symbols` では、同じ index 付き投影を `symbols_by_bucket` と
+`symbols_by_bucket_format` で公開します。
+
 C# の明示的 interface 実装では、symbol 出力と outline は短い表示名
 （`Run`、`Value`、`Changed`、`Item`）を維持します。修飾した exact-name query には
 `IFoo.Run<T>`、`IFoo.Value`、`IFoo.Changed`、`IFoo.Item` のような interface 表記を使い、

TESTING_GUIDE.md

@@ -28,6 +28,7 @@ Use the full suite by default. Use targeted filters only while iterating locally
 - Test parallelism: enabled by default across independent test classes. Tests that touch process-global state such as SQLite pool resets, environment variables, or current-directory overrides must use an explicit non-parallel collection. Console-sensitive classes share one non-parallel xUnit collection, so they remain serial with each other and do not run beside independent classes that may write request-id or global diagnostics to the process console. Use `ConsoleCapture` for ordinary capture and keep every direct `Console.Out` / `Console.Error` swap under `TestConsoleLock.Gate`. Snapshot and assert global console writers under the same gate so another test cannot replace a writer between capture completion and the assertion. That gate aliases the production `ConsoleStreamOwnership` gate so console synchronization and scoped production redirects cannot retain a test writer after its capture ends.
 - MCP structured-output schema coverage is split between the catalog assertions in `McpServerToolsListTests` and actual-result validation in `McpServerOutputSchemaTests`. It requires every full catalog entry to advertise an `outputSchema`, validates actual success and typed-error results for all 24 tools plus search-recipe, empty-excerpt, and truncated variants, and rejects incomplete, versionless-error, excessive-depth, and every cross-tool success pair with the deliberately small local evaluator. Keep the per-tool `tool` discriminator, required fields, bounds, and runtime field names/nesting in sync. Extend that evaluator only for JSON Schema keywords emitted by `McpToolOutputSchemas`; compact catalog entries intentionally omit complete schemas.
 - Markdown unused-audit coverage indexes one real Markdown fixture containing common backtick and tilde fence-language markers. Keep default suppression, `documentation_surface` totals, reason tags, and `--all` recovery in that shared fixture.
+- Unused by-bucket JSON coverage keeps `symbols` as the only full-row collection and checks every lightweight bucket membership against its canonical `symbol_index`. Preserve empty, single-row, large multi-bucket-set, CLI full/compact, MCP, and bounded UTF-8 cursor fixtures; byte-limited pages must stay within the requested budget and reassemble without duplicate or skipped canonical rows.
 - FTS optimization recommendation coverage keeps the shared evaluator exact at one write below, at, and one write above the 25-write threshold. Status, explain, optimize dry-run, optimize execution, and vacuum maintenance guidance must expose the same `recommended`, `action`, `reason`, `threshold_writes`, `observed_writes`, and `state`; stale batches, known WAL-stale snapshots, forward-incompatible schema stamps, and unavailable legacy counters/page snapshots suppress the recommendation, query-only status performs no source writes, and execution uses the focused counter/page/forward-contract/freshness snapshot instead of full status scans. A hot-WAL fixture opened through an explicit `immutable=1` URI must prove that status, standalone optimize dry-run, and the `index --optimize` dry-run alias preserve the same stale recommendation. A WAL or freelist state of `unknown` cannot select the optimize command, and a successful optimize reports the reset counter afterward.
 - Full-scan CLI and MCP no-op coverage treats one repository-wide reusable-stat snapshot read and one folded-readiness verification as performance contracts. Keep assertions for one snapshot read, one stat lookup per candidate, one folded verification, and no content load for unchanged files when changing incremental indexing.
 - Reference-graph refresh coverage treats graph-neutral indexing as a performance contract across incremental full scan, scoped `--files` update, and MCP indexing. Keep zero-refresh assertions for new and modified source files without symbols/references, plus a single batched refresh assertion when existing or new graph identity rows change. A healthy incremental generation must restrict identity/candidate/recursion work to transaction-committed dirty files, old and new `(language, folded name)` dependencies, and their old/new reciprocal edges; retain C#/Python language-transition and unchanged-target parity with a subsequent full refresh, rolled-back file batches, cancellation/retry, orphan-candidate cleanup, and the controlled 4,100-of-4,100 broad-scope fallback. Fresh/rebuild runs, missing identity contracts, and dirty sets of at least 4,096 references covering at least 50% of the graph must keep the full-refresh path. Query-plan coverage must keep all four scoped update phases and all ten candidate inserts on dirty-table-driven reference primary-key seeks, keep C# instantiate grouping on lookup names plus `idx_symbols_name_folded`, and prove that a sub-4,096 dirty set does not count the whole reference table without an explicit diagnostic hook.
@@ -976,6 +977,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests"
 - テスト並列実行: 独立したテストクラス間ではデフォルトで有効です。SQLite pool の解放、環境変数の変更、カレントディレクトリの上書きのような process-global 状態を触るテストは、明示的な non-parallel collection に入れてください。console-sensitive class は同じ non-parallel な xUnit collection を共有するため、互いに直列実行され、request-id や global diagnostics を process console へ書く可能性がある独立 class とも並列実行されません。通常の capture には `ConsoleCapture` を使い、`Console.Out` / `Console.Error` を直接差し替える場合は `TestConsoleLock.Gate` で保護してください。global console writer の snapshot 取得と assertion も同じ gate 内で行い、capture 完了から assertion までの間に別のテストが writer を差し替えないようにします。この gate は本番の `ConsoleStreamOwnership` gate と同一なので、console 同期処理や scoped redirect が capture 終了後も test writer を保持することを防ぎます。
 - MCP structured-output schema の coverage は `McpServerToolsListTests` の catalog assertion と `McpServerOutputSchemaTests` の actual-result validation に分けます。full catalog の全 entry が `outputSchema` を公開することを要求し、全24 tool の実際の success / typed-error result に加えて search recipe、empty excerpt、truncated variant を検証し、不完全な payload、version のない error、過剰な nesting、全 tool 間の success 組み合わせを意図的に小さく保った local evaluator で拒否します。tool ごとの `tool` discriminator、required field、bound、runtime の field 名 / nesting を同期してください。この evaluator は `McpToolOutputSchemas` が出力する JSON Schema keyword に必要な場合だけ拡張し、compact catalog entry は完全な schema を意図的に省略します。
 - Markdown の unused audit coverage は、一般的な backtick / tilde fence の language marker を含む実 Markdown fixture を1回 index します。同じ fixture で既定抑制、`documentation_surface` totals、reason tag、`--all` による復元を維持してください。
+- unused の bucket 別 JSON coverage では、完全な行集合を `symbols` だけに保ち、各 lightweight bucket membership を正規の `symbol_index` と照合します。空集合、1行、大規模な複数 bucket 集合、CLI の full / compact、MCP、UTF-8 byte 上限付き cursor の fixture を維持してください。byte 制限された各 page は要求 budget 内に収まり、連結時に正規行の重複や欠落が発生してはいけません。
 - FTS optimization recommendation coverage は、25 write threshold の1つ下、ちょうど、1つ上で shared evaluator の境界を固定します。status、explain、optimize dry-run、optimize execution、vacuum maintenance guidance は同じ `recommended`、`action`、`reason`、`threshold_writes`、`observed_writes`、`state` を公開し、stale batch、既知の WAL-stale snapshot、forward-incompatible な schema stamp、利用できない legacy counter / page snapshot は recommendation を抑止します。query-only status は source に書き込まず、execution は full status scan ではなく counter / page / forward-contract / freshness に限定した snapshot を使います。hot WAL fixture を明示的な `immutable=1` URI で開き、status、standalone optimize dry-run、`index --optimize` dry-run alias が同じ stale recommendation を保持することも証明します。WAL または freelist の state が `unknown` の場合は optimize command を選択せず、成功した optimize は reset 後の counter を返す必要があります。
 - full-scan CLI と MCP の no-op coverage は、リポジトリ全体の reusable-stat snapshot read と folded-readiness verification がそれぞれ 1 回であることを performance contract とします。incremental indexing を変更するときは、snapshot read が 1 回、候補ごとの stat lookup が 1 回、folded verification が 1 回、unchanged file の content load が 0 回という assertion を維持してください。
 - reference-graph refresh coverage は、incremental full scan、scoped `--files` update、MCP indexing を横断する graph-neutral indexing を performance contract とします。symbol/reference を持たない新規・変更 source file では refresh 0 回を維持し、既存または新規の graph identity 行が変化する場合は batch 全体で refresh 1 回を assertion してください。健全な incremental generation では identity / candidate / recursion 処理を transaction commit 済みの dirty file、旧・新の `(language, folded name)` 依存、旧・新の逆辺に限定します。C# / Python の言語遷移、未変更targetを参照する新規callerと後続full refreshのparity、rollback file batch、cancel後retry、孤立candidate cleanup、4,100件中4,100件をdirtyにする制御broad-scope fallbackを維持してください。fresh/rebuild、identity契約欠落、または4,096件以上かつgraphの50%以上を占めるdirty集合ではfull-refresh経路を維持します。query-plan coverageでは、scoped updateの4 phaseとcandidate INSERT 10本をdirty table起点のreference主キーseekに保ち、C# instantiate groupingをlookup nameと`idx_symbols_name_folded`起点にし、明示的なdiagnostic hookがない4,096件未満のdirty集合ではreference table全件COUNTを行わないことを検証してください。

changelog.d/unreleased/4904.fixed.md

@@ -0,0 +1,24 @@
+---
+category: fixed
+issues:
+  - 4904
+affected:
+  - README.md
+  - TESTING_GUIDE.md
+  - src/CodeIndex/Cli/CliFlagSchema.cs
+  - src/CodeIndex/Cli/ConsoleUi.cs
+  - src/CodeIndex/Cli/QueryCommandRunner.Unused.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.SymbolAnalysis.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.cs
+  - src/CodeIndex/Mcp/McpToolOutputSchemas.cs
+  - tests/CodeIndex.Tests/McpServerToolsCallTests.cs
+  - tests/CodeIndex.Tests/QueryCommandRunnerSymbolTests.cs
+---
+
+## English
+
+- **By-bucket unused JSON now serializes each full symbol row once (#4904)** — CLI and MCP responses keep full rows in the canonical `symbols` collection and expose bucket membership through lightweight indexed projections; CLI output also supports exact UTF-8 `--max-json-bytes` paging without duplicate or skipped canonical rows.
+
+## 日本語
+
+- **bucket 別の unused JSON が各 symbol の完全な行を一度だけシリアライズするようになりました (#4904)** — CLI と MCP の応答では完全な行を正規の `symbols` 集合に保持し、bucket membership は軽量な index 付き投影で公開します。また CLI 出力は、正規行の重複や欠落を発生させずに UTF-8 byte 数を厳密に制限する `--max-json-bytes` paging に対応しました。

src/CodeIndex/Cli/CodeIndexExceptionFormatter.cs

@@ -13,7 +13,11 @@ namespace CodeIndex.Cli;
 /// </summary>
 internal static class CodeIndexExceptionFormatter
 {
-    public static void Write(CodeIndexException ex, string[] args, JsonSerializerOptions jsonOptions)
+    public static void Write(
+        CodeIndexException ex,
+        string[] args,
+        JsonSerializerOptions jsonOptions,
+        int? maxJsonBytes = null)
     {
         var message = CommandErrorWriter.FormatSanitizedExceptionMessage(ex);
         if (HasJsonFlag(args))
@@ -25,10 +29,15 @@ public static void Write(CodeIndexException ex, string[] args, JsonSerializerOpt
                 ErrorCode: ex.Code,
                 Path: ex.Path,
                 Category: ex.Category);
-            Console.WriteLine(JsonSerializer.Serialize(
+            var json = JsonSerializer.Serialize(
                 payload,
-                CliJsonSerializerContextFactory.Create(jsonOptions).CommandErrorJsonResult));
-            return;
+                CliJsonSerializerContextFactory.Create(jsonOptions).CommandErrorJsonResult);
+            if (!maxJsonBytes.HasValue
+                || JsonEnvelopeWrapper.JsonFitsResponseBudget(json, maxJsonBytes.Value))
+            {
+                Console.WriteLine(json);
+                return;
+            }
         }
 
         // Keep human output close to the existing `Error [Exxx]: ...` shape that