Fix excerpt semantic token coordinates

DEVELOPER_GUIDE.md

@@ -1073,7 +1073,7 @@ When the match line falls inside an indexed symbol range, `search --json` and MC
 
 `find --json` remains line-delimited for repeated matches and adds bounded match-span/truncation metadata to each row: `length` reports the 1-based `column` span length, `original_line_length` reports the source line length before any line-width clamp, and `snippet_truncation_context.line_count` / `char_counts` / `total_chars` / optional `reason` describe snippet clamping. `reason` is `line_width` when `--max-line-width` elides one or more snippet lines.
 
-`excerpt --json` includes `semantic_tokens`, a lightweight range list with 1-based start/end positions, token `type`, and `modifiers`, so IDE and LLM clients can render or post-process excerpt spans without reparsing the raw `content` string. Excerpt rows also expose `requested_start_line`, `requested_end_line`, `effective_start_line`, `effective_end_line`, `content_truncation_reasons`, and optional `content_recovery` so clients can tell when `--max-line-width` caused `line_width_cap` and replay `cdidx excerpt ... --max-line-width 0 --json` for the omitted text. Body-bearing JSON rows use matching `body_requested_*`, `body_effective_*`, `body_content_truncation_reasons`, and `body_content_recovery` fields; body reasons include `body_line_cap` for snippet/body line caps and `body_byte_cap` for definition body byte caps.
+`excerpt --json` includes `semantic_tokens`, a lightweight range list with 1-based source start/end positions, token `type`, and `modifiers`, so IDE and LLM clients can render or post-process excerpt spans without reparsing the raw `content` string. `semantic_token_coordinate_space` is `source`; when `--max-line-width` clamps returned content, `content_line_spans` maps each returned content line and visible content-column span back to the matching source line and source-column span, while clamp markers remain unmapped and are not emitted as semantic tokens. Excerpt rows also expose `requested_start_line`, `requested_end_line`, `effective_start_line`, `effective_end_line`, `content_truncation_reasons`, and optional `content_recovery` so clients can tell when `--max-line-width` caused `line_width_cap` and replay `cdidx excerpt ... --max-line-width 0 --json` for the omitted text. Body-bearing JSON rows use matching `body_requested_*`, `body_effective_*`, `body_content_truncation_reasons`, and `body_content_recovery` fields; body reasons include `body_line_cap` for snippet/body line caps and `body_byte_cap` for definition body byte caps.
 
 `inspect` and MCP `analyze_symbol` bundle the primary definition, nearby symbols from the same file, references, callers, callees, file metadata, workspace freshness/git metadata, and graph-support metadata into one response. When those bundled graph sections actually depend on SQL-backed reads, the payload also mirrors `sql_graph_contract_ready` / `sql_graph_contract_degraded_reason` (plus the existing camelCase aliases on MCP responses); mixed-language bundles that only return C# / JS / etc. graph rows omit the SQL trust signal entirely. This is intended for symbol-oriented AI workflows that would otherwise need several back-to-back calls. Call graph sections remain language-aware: for unsupported languages, clients can now distinguish "unsupported" from "no hits" via `graphSupported` / `graphSupportReason`, and should prefer `search` instead of assuming graph data will exist.
 
@@ -3288,7 +3288,7 @@ literal-safe な `search` query は reader 層で FTS5 sanitization 前に 1000
 
 `find --json` は繰り返し一致でも line-delimited のまま維持し、各 row に bounded な match span / truncation metadata を追加します。`length` は 1-based の `column` から始まる一致長、`original_line_length` は行幅クランプ前のソース行長、`snippet_truncation_context.line_count` / `char_counts` / `total_chars` / 任意の `reason` は snippet クランプを表します。`--max-line-width` によって snippet 行が省略された場合、`reason` は `line_width` になります。
 
-`excerpt --json` は 1-based の開始/終了位置、token `type`、`modifiers` を持つ軽量 range list の `semantic_tokens` を返すため、IDE や LLM クライアントは生の `content` 文字列を再パースせずに抜粋範囲を描画・後処理できます。excerpt row は `requested_start_line`、`requested_end_line`、`effective_start_line`、`effective_end_line`、`content_truncation_reasons`、任意の `content_recovery` も返すため、`--max-line-width` による `line_width_cap` を検出し、省略部分を `cdidx excerpt ... --max-line-width 0 --json` で再取得できます。body を持つ JSON row も対応する `body_requested_*`、`body_effective_*`、`body_content_truncation_reasons`、`body_content_recovery` を返します。body reason には snippet/body 行数上限の `body_line_cap` と definition body byte 上限の `body_byte_cap` があります。
+`excerpt --json` は 1-based の source 開始/終了位置、token `type`、`modifiers` を持つ軽量 range list の `semantic_tokens` を返すため、IDE や LLM クライアントは生の `content` 文字列を再パースせずに抜粋範囲を描画・後処理できます。`semantic_token_coordinate_space` は `source` です。`--max-line-width` で返却内容がクランプされた場合、`content_line_spans` は返却 content 行と可視 content column span を、対応する source 行と source column span に対応付けます。clamp marker は未対応領域として扱い、semantic token には含めません。excerpt row は `requested_start_line`、`requested_end_line`、`effective_start_line`、`effective_end_line`、`content_truncation_reasons`、任意の `content_recovery` も返すため、`--max-line-width` による `line_width_cap` を検出し、省略部分を `cdidx excerpt ... --max-line-width 0 --json` で再取得できます。body を持つ JSON row も対応する `body_requested_*`、`body_effective_*`、`body_content_truncation_reasons`、`body_content_recovery` を返します。body reason には snippet/body 行数上限の `body_line_cap` と definition body byte 上限の `body_byte_cap` があります。
 
 `inspect` と MCP の `analyze_symbol` は、主定義、同一ファイル内の近傍シンボル、参照、caller、callee、ファイルメタデータ、さらにワークスペース鮮度/git メタデータと graph 対応メタデータを1レスポンスにまとめます。bundle 内の graph 節が実際に SQL ベースの read に依存する場合だけ、`sql_graph_contract_ready` / `sql_graph_contract_degraded_reason`（MCP では既存の camelCase alias も）も返します。mixed-language bundle で C# / JS などの graph row しか返っていない場合は SQL trust signal を出さないため、無関係なクエリが stale SQL state に引きずられません。複数の連続クエリを避けたい AI ワークフロー向けです。call graph 系の節は言語差分を考慮しており、未対応言語では `graphSupported` / `graphSupportReason` によって「未対応」と「ヒットなし」を区別できます。その場合は `search` を優先して使う前提です。
 

changelog.d/unreleased/3563.fixed.md

@@ -0,0 +1,18 @@
+---
+category: fixed
+issues:
+  - 3563
+affected:
+  - src/CodeIndex/Cli/QueryCommandRunner.cs
+  - src/CodeIndex/Database/DbReader.FilesStatus.cs
+  - src/CodeIndex/Models/QueryResults.cs
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Excerpt semantic tokens now use source coordinates for clamped content (#3563)** — `excerpt --json` reports `semantic_token_coordinate_space: "source"` and includes `content_line_spans` so clients can map visible clamped content back to original source columns without treating clamp markers as tokens.
+
+## 日本語
+
+- **clamp された excerpt content の semantic token が source 座標を使うようになりました (#3563)** — `excerpt --json` は `semantic_token_coordinate_space: "source"` と `content_line_spans` を返し、クライアントが clamp marker を token として扱わずに可視 content を元の source column へ対応付けられるようになりました。

src/CodeIndex/Cli/JsonOutputContracts.cs

@@ -472,6 +472,7 @@ internal sealed record VersionInfoJsonResult(
 [JsonSerializable(typeof(ExportImportCommandRunner.ExportManifest))]
 [JsonSerializable(typeof(ExportImportCommandRunner.ImportDryRunResult))]
 [JsonSerializable(typeof(ExportImportCommandRunner.ImportValidationPhaseResult))]
+[JsonSerializable(typeof(ExcerptContentLineSpan))]
 [JsonSerializable(typeof(ExcerptRecoveryHint))]
 [JsonSerializable(typeof(ExcerptSemanticToken))]
 [JsonSerializable(typeof(FileDependencyResult))]

src/CodeIndex/Cli/QueryCommandRunner.cs

@@ -3858,11 +3858,19 @@ private static List<ExcerptSemanticToken> BuildExcerptSemanticTokens(FileExcerpt
     {
         var tokens = new List<ExcerptSemanticToken>();
         var lines = excerpt.Content.Replace("\r\n", "\n").Split('\n');
-        for (var lineIndex = 0; lineIndex < lines.Length; lineIndex++)
+        var spans = excerpt.ContentLineSpans.Count == 0
+            ? BuildIdentityExcerptContentLineSpans(excerpt, lines)
+            : excerpt.ContentLineSpans;
+        foreach (var span in spans)
         {
-            var line = lines[lineIndex];
-            var column = 0;
-            while (column < line.Length)
+            if (span.ContentLine <= 0 || span.ContentLine > lines.Length)
+                continue;
+
+            var line = lines[span.ContentLine - 1];
+            var startColumn = Math.Clamp(span.ContentStartColumn - 1, 0, line.Length);
+            var endColumn = Math.Clamp(span.ContentEndColumn - 1, startColumn, line.Length);
+            var column = startColumn;
+            while (column < endColumn)
             {
                 if (!IsSemanticTokenStart(line[column]))
                 {
@@ -3872,16 +3880,18 @@ private static List<ExcerptSemanticToken> BuildExcerptSemanticTokens(FileExcerpt
 
                 var start = column;
                 column++;
-                while (column < line.Length && IsSemanticTokenPart(line[column]))
+                while (column < endColumn && IsSemanticTokenPart(line[column]))
                     column++;
 
                 var tokenText = line[start..column];
+                var sourceStartColumn = span.SourceStartColumn + ((start + 1) - span.ContentStartColumn);
+                var sourceEndColumn = span.SourceStartColumn + ((column + 1) - span.ContentStartColumn);
                 tokens.Add(new ExcerptSemanticToken
                 {
-                    StartLine = excerpt.StartLine + lineIndex,
-                    StartColumn = start + 1,
-                    EndLine = excerpt.StartLine + lineIndex,
-                    EndColumn = column + 1,
+                    StartLine = span.SourceLine,
+                    StartColumn = sourceStartColumn,
+                    EndLine = span.SourceLine,
+                    EndColumn = sourceEndColumn,
                     Type = ClassifySemanticToken(tokenText),
                 });
             }
@@ -3890,6 +3900,25 @@ private static List<ExcerptSemanticToken> BuildExcerptSemanticTokens(FileExcerpt
         return tokens;
     }
 
+    private static List<ExcerptContentLineSpan> BuildIdentityExcerptContentLineSpans(FileExcerptResult excerpt, string[] lines)
+    {
+        var spans = new List<ExcerptContentLineSpan>(lines.Length);
+        for (var i = 0; i < lines.Length; i++)
+        {
+            spans.Add(new ExcerptContentLineSpan
+            {
+                ContentLine = i + 1,
+                SourceLine = excerpt.StartLine + i,
+                ContentStartColumn = 1,
+                ContentEndColumn = lines[i].Length + 1,
+                SourceStartColumn = 1,
+                SourceEndColumn = lines[i].Length + 1,
+            });
+        }
+
+        return spans;
+    }
+
     private static bool IsSemanticTokenStart(char value) =>
         char.IsLetter(value) || value == '_' || char.IsDigit(value);
 

src/CodeIndex/Database/DbReader.FilesStatus.cs

@@ -561,14 +561,32 @@ FROM chunks c
         var focusLineIndex = focusLine.HasValue ? selectedLines.IndexOf(focusLine.Value) : -1;
         if (focusLineIndex >= 0 && focusColumn.HasValue && focusColumn.Value > contentLines[focusLineIndex].Length)
             return null;
-        var clampedContent = maxLineWidth.HasValue
-            ? LineWidthFormatter.ClampLines(
-                contentLines,
-                maxLineWidth.Value,
-                focusLineIndex >= 0 ? focusLineIndex : null,
-                focusLineIndex >= 0 ? focusColumn : null,
-                focusLength)
-            : new ClampedTextResult(string.Join("\n", contentLines), false);
+        var excerptLines = new string[contentLines.Count];
+        var contentLineSpans = new List<ExcerptContentLineSpan>(contentLines.Count);
+        var contentTruncated = false;
+        for (var i = 0; i < contentLines.Count; i++)
+        {
+            var clampedLine = maxLineWidth.HasValue
+                ? LineWidthFormatter.ClampLine(
+                    contentLines[i],
+                    maxLineWidth.Value,
+                    i == focusLineIndex ? focusColumn : null,
+                    focusLength)
+                : ClampedTextResult.Unclamped(contentLines[i]);
+
+            excerptLines[i] = clampedLine.Text;
+            contentTruncated |= clampedLine.Truncated;
+            var visibleLength = Math.Max(0, clampedLine.OriginalVisibleEndColumn - clampedLine.OriginalVisibleStartColumn + 1);
+            contentLineSpans.Add(new ExcerptContentLineSpan
+            {
+                ContentLine = i + 1,
+                SourceLine = selectedLines[i],
+                ContentStartColumn = clampedLine.TextVisibleStartColumn,
+                ContentEndColumn = clampedLine.TextVisibleStartColumn + visibleLength,
+                SourceStartColumn = clampedLine.OriginalVisibleStartColumn,
+                SourceEndColumn = clampedLine.OriginalVisibleStartColumn + visibleLength,
+            });
+        }
 
         return new FileExcerptResult
         {
@@ -580,12 +598,13 @@ FROM chunks c
             RequestedEndLine = requestedEndCeiling,
             EffectiveStartLine = selectedLines[0],
             EffectiveEndLine = selectedLines[^1],
-            Content = clampedContent.Text,
-            ContentTruncated = clampedContent.Truncated,
-            ContentTruncationReasons = clampedContent.Truncated ? ["line_width_cap"] : [],
-            ContentRecovery = clampedContent.Truncated
+            Content = string.Join("\n", excerptLines),
+            ContentTruncated = contentTruncated,
+            ContentTruncationReasons = contentTruncated ? ["line_width_cap"] : [],
+            ContentRecovery = contentTruncated
                 ? FileExcerptResult.CreateRecoveryHint(path, selectedLines[0], selectedLines[^1])
                 : null,
+            ContentLineSpans = contentLineSpans,
         };
     }
 

src/CodeIndex/Mcp/McpToolHandlers.cs

@@ -3564,20 +3564,106 @@ internal static void ApplyExcerptOutputBudget(JsonObject payload, int maxOutputB
             return;
 
         var builder = new StringBuilder();
+        var retainedLineCount = 0;
+        var firstRetainedLine = true;
         foreach (var line in content.Replace("\r\n", "\n").Split('\n'))
         {
-            var candidate = builder.Length == 0 ? line : builder.ToString() + "\n" + line;
+            var candidate = firstRetainedLine ? line : builder.ToString() + "\n" + line;
             if (Encoding.UTF8.GetByteCount(candidate) > maxOutputBytes)
                 break;
             builder.Clear();
             builder.Append(candidate);
+            retainedLineCount++;
+            firstRetainedLine = false;
         }
         payload[contentKey] = builder.ToString();
+        TrimExcerptCoordinatePayload(payload, retainedLineCount);
         payload["contentTruncated"] = true;
         payload["truncated"] = true;
         payload["truncation_reason"] = "output_size_cap";
     }
 
+    private static void TrimExcerptCoordinatePayload(JsonObject payload, int retainedLineCount)
+    {
+        var spansKey = FirstPayloadKey(payload, "contentLineSpans", "content_line_spans", "ContentLineSpans");
+        var retainedSpans = new List<ExcerptPayloadSpan>();
+        var hasSpanMapping = false;
+        if (spansKey is not null && payload[spansKey] is JsonArray spans)
+        {
+            hasSpanMapping = true;
+            var trimmedSpans = new JsonArray();
+            foreach (var spanNode in spans)
+            {
+                if (spanNode is not JsonObject span)
+                    continue;
+                var contentLine = GetPayloadInt(span, "contentLine", "content_line", "ContentLine");
+                if (!contentLine.HasValue || contentLine.Value > retainedLineCount)
+                    continue;
+
+                trimmedSpans.Add(span.DeepClone());
+                var sourceLine = GetPayloadInt(span, "sourceLine", "source_line", "SourceLine");
+                var sourceStartColumn = GetPayloadInt(span, "sourceStartColumn", "source_start_column", "SourceStartColumn");
+                var sourceEndColumn = GetPayloadInt(span, "sourceEndColumn", "source_end_column", "SourceEndColumn");
+                if (sourceLine.HasValue && sourceStartColumn.HasValue && sourceEndColumn.HasValue)
+                    retainedSpans.Add(new ExcerptPayloadSpan(sourceLine.Value, sourceStartColumn.Value, sourceEndColumn.Value));
+            }
+
+            payload[spansKey] = trimmedSpans;
+        }
+
+        var tokensKey = FirstPayloadKey(payload, "semanticTokens", "semantic_tokens", "SemanticTokens");
+        if (tokensKey is null || payload[tokensKey] is not JsonArray tokens)
+            return;
+        if (!hasSpanMapping)
+        {
+            if (retainedLineCount == 0)
+                payload[tokensKey] = new JsonArray();
+            return;
+        }
+
+        var trimmedTokens = new JsonArray();
+        if (retainedLineCount > 0 && retainedSpans.Count > 0)
+        {
+            foreach (var tokenNode in tokens)
+            {
+                if (tokenNode is not JsonObject token)
+                    continue;
+                var startLine = GetPayloadInt(token, "startLine", "start_line", "StartLine");
+                var endLine = GetPayloadInt(token, "endLine", "end_line", "EndLine");
+                var startColumn = GetPayloadInt(token, "startColumn", "start_column", "StartColumn");
+                var endColumn = GetPayloadInt(token, "endColumn", "end_column", "EndColumn");
+                if (!startLine.HasValue || !endLine.HasValue || !startColumn.HasValue || !endColumn.HasValue)
+                    continue;
+                if (retainedSpans.Any(span =>
+                    startLine.Value == span.SourceLine &&
+                    endLine.Value == span.SourceLine &&
+                    startColumn.Value >= span.SourceStartColumn &&
+                    endColumn.Value <= span.SourceEndColumn))
+                {
+                    trimmedTokens.Add(token.DeepClone());
+                }
+            }
+        }
+
+        payload[tokensKey] = trimmedTokens;
+    }
+
+    private static string? FirstPayloadKey(JsonObject payload, params string[] keys)
+        => keys.FirstOrDefault(payload.ContainsKey);
+
+    private static int? GetPayloadInt(JsonObject obj, params string[] keys)
+    {
+        foreach (var key in keys)
+        {
+            if (obj[key] is JsonNode node)
+                return node.GetValue<int>();
+        }
+
+        return null;
+    }
+
+    private readonly record struct ExcerptPayloadSpan(int SourceLine, int SourceStartColumn, int SourceEndColumn);
+
     private JsonNode ExecuteFindInFile(JsonNode? id, JsonNode? args)
     {
         if (!TryReadRequiredStringParameter(args, "query", out var query, out var requiredError))