Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions DEVELOPER_GUIDE.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1586,7 +1586,7 @@ Use `--json` for machine-readable output (AI agents):
{"path":"src/Auth/TokenService.cs","lang":"csharp","chunk_start_line":1,"chunk_end_line":80,"snippet_start_line":40,"snippet_end_line":47,"snippet":"if (claims.Count == 0)\\n throw new InvalidOperationException();\\nreturn GenerateToken(claims);","match_lines":[42,47],"highlights":[{"line":47,"text":"return GenerateToken(claims);","terms":["GenerateToken"]}],"context_before":2,"context_after":3,"score":9.8}
```

Add `--json-envelope` to wrap the per-line stream into a single document with a `metadata` block (command, `cdidx_version`, `elapsed_ms`, `db_path`, `result_count`, `exit_code`, optional `query_normalized` / `indexed_at_head_sha`) and a `results` array. The flag implies `--json` and works on every query command (`search`, `definition`, `references`, `callers`, `callees`, `symbols`, `files`, `find`, `excerpt`, `map`, `inspect`, `outline`, `status`, `validate`, `languages`, `impact`, `deps`, `unused`, `hotspots`). Wrapped commands can capture up to 10,485,760 output characters; if the budget is exceeded, cdidx returns a JSON envelope with empty `results`, non-zero `metadata.exit_code`, and `metadata.error`, and suggests using `--limit` / `--top` or streaming `--json`. The flat NDJSON / array output stays the default for one release; the envelope will become the default in the next major release, at which point the flat form will be opt-in via `--json-flat`.
Add `--json-envelope` to wrap the per-line stream into a single document with a `metadata` block (command, `cdidx_version`, `elapsed_ms`, `db_path`, `result_count`, `exit_code`, optional `query_normalized` / `indexed_at_head_sha`) and a `results` array. `indexed_at_head_sha` has the same meaning as status `indexed_head_sha`: it identifies the checkout captured by the latest successful full scan, `--files`, `--commits`, or `--changed-between` refresh. It does not advance after a failed or rolled-back refresh; databases created before `indexed_head_sha` fall back to the legacy full-scan-only `indexed_head_commit`. If the latest-head key exists but its value is unavailable because Git HEAD could not be resolved, the envelope omits `indexed_at_head_sha` instead of reporting the legacy baseline. Every envelope binds this stamp to the validated index generation used for its rows and rejects the response with restart guidance if that generation changes during execution; bounded responses also bind their cursor to that snapshot. The flag implies `--json` and works on every query command (`search`, `definition`, `references`, `callers`, `callees`, `symbols`, `files`, `find`, `excerpt`, `map`, `inspect`, `outline`, `status`, `validate`, `languages`, `impact`, `deps`, `unused`, `hotspots`). Wrapped commands can capture up to 10,485,760 output characters; if the budget is exceeded, cdidx returns a JSON envelope with empty `results`, non-zero `metadata.exit_code`, and `metadata.error`, and suggests using `--limit` / `--top` or streaming `--json`. The flat NDJSON / array output stays the default for one release; the envelope will become the default in the next major release, at which point the flat form will be opt-in via `--json-flat`.

Add `--profile` to any read command when debugging slow queries. It appends one JSON object after the normal result with `profile.phases` (`name`, `elapsed_ms`, `rows_scanned`), `profile.query_plan` (`EXPLAIN QUERY PLAN` rows), and `profile.queries` (the SQL text). Add `--slow-query-ms <n>` to log profiled SQL statements that meet the threshold to the persistent tool log.

Expand Down Expand Up @@ -4745,7 +4745,7 @@ src/Auth/TokenService.cs:42-58
{"path":"src/Auth/TokenService.cs","start_line":42,"end_line":58,"content":"public string GenerateToken(...)...","lang":"csharp","score":9.8}
```

`--json-envelope` を追加すると、1 行ごとの stream を `metadata`(command、`cdidx_version`、`elapsed_ms`、`db_path`、`result_count`、`exit_code`、任意の `query_normalized` / `indexed_at_head_sha`)と `results` 配列を持つ 1 つの JSON document に包みます。この flag は `--json` を暗黙に有効化し、各 query command で使えます。wrapped command の捕捉出力は最大 10,485,760 文字です。超過した場合は、空の `results`、非 0 の `metadata.exit_code`、`metadata.error` を持つ JSON envelope を返し、`--limit` / `--top` または streaming `--json` の利用を促します。
`--json-envelope` を追加すると、1 行ごとの stream を `metadata`(command、`cdidx_version`、`elapsed_ms`、`db_path`、`result_count`、`exit_code`、任意の `query_normalized` / `indexed_at_head_sha`)と `results` 配列を持つ 1 つの JSON document に包みます。`indexed_at_head_sha` の意味は status の `indexed_head_sha` と同じで、最後に成功した full scan、`--files`、`--commits`、`--changed-between` refresh が取り込んだ checkout を示します。失敗または rollback された refresh では進まず、`indexed_head_sha` より前の database では legacy の full-scan 限定 `indexed_head_commit` に fallback します。最新 HEAD key が存在していても Git HEAD を解決できず値が未設定の場合は、legacy baseline を返さず `indexed_at_head_sha` を省略します。すべての envelope はこの stamp を row と同じ検証済み index generation に固定し、実行中に generation が変わった場合は response を拒否して再実行案内を返します。bounded response では cursor も同じ snapshot に固定します。この flag は `--json` を暗黙に有効化し、各 query command で使えます。wrapped command の捕捉出力は最大 10,485,760 文字です。超過した場合は、空の `results`、非 0 の `metadata.exit_code`、`metadata.error` を持つ JSON envelope を返し、`--limit` / `--top` または streaming `--json` の利用を促します。

### シンボル検索(関数、クラスなど)

Expand Down
23 changes: 23 additions & 0 deletions changelog.d/unreleased/4854.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
category: fixed
issues:
- 4854
affected:
- src/CodeIndex/Cli/DbPathResolver.cs
- src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs
- src/CodeIndex/Cli/JsonEnvelopeWrapper.cs
- src/CodeIndex/Database/DbReader.cs
- tests/CodeIndex.Tests/JsonEnvelopeWrapperTests.cs
- tests/CodeIndex.Tests/IndexCommandRunnerTests.cs
- tests/CodeIndex.Tests/McpServerToolsCallTests.cs
- USER_GUIDE.md
- DEVELOPER_GUIDE.md
---

## English

- **Query envelopes now agree with status on the latest indexed HEAD (#4854)** — `metadata.indexed_at_head_sha` now uses the latest successful full or partial refresh stamp established by #1509, #1527, and #4313, while retaining the legacy full-scan HEAD fallback only for older databases without the latest key, omitting unresolved current stamps, binding regular and bounded responses to a validated generation, rejecting concurrent refresh races, and preserving the prior stamp after failed or rolled-back refreshes.

## 日本語

- **query envelope と status が最新の indexed HEAD で一致するようになりました (#4854)** — `metadata.indexed_at_head_sha` は #1509、#1527、#4313 で確立された最新の成功 full / partial refresh stamp を使うようになり、最新 key のない古い database だけを legacy full-scan HEAD へ fallback し、解決不能な current stamp を省略し、通常および bounded response を検証済み generation に固定して同時 refresh の競合を拒否しつつ、失敗または rollback された refresh 後の直前 stamp を維持します。
14 changes: 14 additions & 0 deletions src/CodeIndex/Cli/DbPathResolver.cs
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,20 @@ private static void SeedPathCasingFromWorkspaceStamp(string? indexedProjectRoot,
public static string? TryReadIndexedHeadSha(string dbPath)
=> TryReadMetaString(dbPath, CodeIndex.Database.DbContext.IndexedHeadShaMetaKey);

public static string? TryReadIndexedHeadForResponse(string dbPath)
{
try
{
using var connection = OpenMetadataConnection(dbPath);
connection.Open();
return CodeIndex.Database.DbReader.TryGetIndexedHeadForResponse(connection);
}
catch (Exception ex) when (IsMetadataProbeException(ex))
{
return null;
}
}

public static DateTimeOffset? TryReadIndexedHeadTimestamp(string dbPath)
{
var raw = TryReadMetaString(dbPath, CodeIndex.Database.DbContext.IndexedHeadTimestampMetaKey);
Expand Down
13 changes: 9 additions & 4 deletions src/CodeIndex/Cli/JsonEnvelopeWrapper.Bounded.cs
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ internal static partial class JsonEnvelopeWrapper
private const string LegacyResponseCursorPrefix = "response:v1:";
private const string ResponseCursorPrefix = "response:v2:";
private static readonly AsyncLocal<BoundedExecutionContext?> BoundedExecution = new();
internal static Action? ResponseSnapshotValidatedForTesting { get; set; }

private static readonly HashSet<string> BoundedResponseCommands =
ProjectionFieldRegistry.SupportedCommands.ToHashSet(StringComparer.Ordinal);
Expand Down Expand Up @@ -274,6 +275,7 @@ private static int RunBoundedResponse(
"The index generation changed while this page was being read.",
"Restart pagination without --cursor after the active index refresh completes.");
}
ResponseSnapshotValidatedForTesting?.Invoke();
var envelope = BuildBoundedEnvelopeWithinBudget(
command,
queryNormalized,
Expand Down Expand Up @@ -351,7 +353,8 @@ JsonObject BuildCandidate(int count)
exitCode,
error: commandError is null ? null : (JsonObject)commandError.DeepClone(),
streamTerminal: adjustedStreamTerminal,
streamControlRecords: streamControlRecords);
streamControlRecords: streamControlRecords,
responseSnapshot: snapshot);
var metadata = (JsonObject)envelope["metadata"]!;
metadata["result_stable_at"] = snapshot.ResultStableAt;
if (commandError is not null)
Expand Down Expand Up @@ -1454,7 +1457,8 @@ private readonly record struct ResponseCount(

private readonly record struct ResponseSnapshot(
string GenerationFingerprint,
string? ResultStableAt);
string? ResultStableAt,
string? IndexedHead);

private static ResponseSnapshot SafeReadResponseSnapshot(
string dbPath,
Expand Down Expand Up @@ -1486,11 +1490,12 @@ private static ResponseSnapshot BuildResponseSnapshot(DbReader reader)
var generation = reader.GetPaginationGeneration();
return new(
BuildResponseValueFingerprint(generation.Identity),
generation.StableAt);
generation.StableAt,
reader.GetIndexedHeadForResponse());
}

private static ResponseSnapshot BuildFallbackResponseSnapshot(string appVersion)
=> new(BuildResponseValueFingerprint("catalog\0" + appVersion), null);
=> new(BuildResponseValueFingerprint("catalog\0" + appVersion), null, null);

private static string BuildResponseValueFingerprint(string value)
{
Expand Down
70 changes: 65 additions & 5 deletions src/CodeIndex/Cli/JsonEnvelopeWrapper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ internal static int RunWrapped(
var innerArgs = PrepareInnerArgs(args);
var queryNormalized = ExtractQueryArg(args);
var (resolvedDbPath, dbPathExplicit) = ResolveQueryDbPath(args);
var responseSnapshot = SafeReadResponseSnapshot(resolvedDbPath, dbPathExplicit, appVersion);

using var captured = new BoundedStringWriter(MaxCapturedOutputChars);
var stopwatch = Stopwatch.StartNew();
Expand Down Expand Up @@ -143,7 +144,8 @@ internal static int RunWrapped(
stopwatch.Elapsed.TotalMilliseconds,
new JsonArray(),
exitCode,
envelopeError);
envelopeError,
responseSnapshot: responseSnapshot);

Console.WriteLine(overflowEnvelope.ToJsonString(jsonOptions));
return exitCode;
Expand Down Expand Up @@ -190,6 +192,24 @@ internal static int RunWrapped(
};
results = [];
}

var completedSnapshot = SafeReadResponseSnapshot(resolvedDbPath, dbPathExplicit, appVersion);
if (!string.Equals(
responseSnapshot.GenerationFingerprint,
completedSnapshot.GenerationFingerprint,
StringComparison.Ordinal))
{
return WriteResponseSnapshotChangedError(
command,
queryNormalized,
resolvedDbPath,
dbPathExplicit,
appVersion,
stopwatch.Elapsed.TotalMilliseconds,
jsonOptions,
responseSnapshot);
}
ResponseSnapshotValidatedForTesting?.Invoke();
var envelope = BuildEnvelope(
command,
queryNormalized,
Expand All @@ -201,12 +221,48 @@ internal static int RunWrapped(
exitCode,
parseError,
streamTerminal,
streamControlRecords);
streamControlRecords,
responseSnapshot);

Console.WriteLine(envelope.ToJsonString(jsonOptions));
return exitCode;
}

private static int WriteResponseSnapshotChangedError(
string command,
string? queryNormalized,
string dbPath,
bool dbPathExplicit,
string appVersion,
double elapsedMs,
JsonSerializerOptions jsonOptions,
ResponseSnapshot responseSnapshot)
{
const string message = "The index generation changed while this response was being read.";
const string hint = "Restart the command after the active index refresh completes.";
CommandErrorWriter.WriteStderr($"Error [{CommandErrorCodes.UsageError}]: {message}");
CommandErrorWriter.WriteStderr($"Hint: {hint}");
var error = new JsonObject
{
["message"] = message,
["hint"] = hint,
["error_code"] = CommandErrorCodes.UsageError,
};
var envelope = BuildEnvelope(
command,
queryNormalized,
dbPath,
dbPathExplicit,
appVersion,
elapsedMs,
[],
CommandExitCodes.UsageError,
error,
responseSnapshot: responseSnapshot);
Console.WriteLine(envelope.ToJsonString(jsonOptions));
return CommandExitCodes.UsageError;
}

private static bool HasArgument(string[] args, string option)
=> args.Any(arg => string.Equals(arg, option, StringComparison.Ordinal)
|| arg.StartsWith(option + "=", StringComparison.Ordinal));
Expand All @@ -222,7 +278,8 @@ private static JsonObject BuildEnvelope(
int exitCode,
JsonObject? error = null,
JsonObject? streamTerminal = null,
JsonArray? streamControlRecords = null)
JsonArray? streamControlRecords = null,
ResponseSnapshot? responseSnapshot = null)
{
var metadata = new JsonObject
{
Expand All @@ -244,7 +301,9 @@ private static JsonObject BuildEnvelope(
if (streamControlRecords is { Count: > 0 })
metadata["stream_control_records"] = streamControlRecords.DeepClone();

var indexedHead = SafeReadIndexedHead(dbPath, dbPathExplicit);
var indexedHead = responseSnapshot.HasValue
? responseSnapshot.Value.IndexedHead
: SafeReadIndexedHead(dbPath, dbPathExplicit);
if (!string.IsNullOrEmpty(indexedHead))
metadata["indexed_at_head_sha"] = indexedHead;

Expand All @@ -262,7 +321,8 @@ private static JsonObject BuildEnvelope(
var resolvedPath = dbPath;
if (!dbPathExplicit && !File.Exists(LongPath.EnsureWindowsPrefix(resolvedPath)))
return null;
return DbPathResolver.TryReadIndexedHeadCommit(DbPathResolver.NormalizeDbPath(resolvedPath));
var normalizedPath = DbPathResolver.NormalizeDbPath(resolvedPath);
return DbPathResolver.TryReadIndexedHeadForResponse(normalizedPath);
}
catch
{
Expand Down
39 changes: 39 additions & 0 deletions src/CodeIndex/Database/DbReader.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1091,6 +1091,45 @@ AND COUNT(s.family_key) < COUNT(*)
/// </summary>
internal string? GetMetaString(string key) => TryGetMetaString(_conn, key);

/// <summary>
/// Read the latest indexed HEAD used by response metadata. A present
/// <c>indexed_head_sha</c> row is authoritative even when its value is NULL;
/// only databases without that key fall back to the legacy full-scan stamp.
/// response metadata 用の最新 indexed HEAD を読む。<c>indexed_head_sha</c> row が
/// 存在する場合は NULL 値でも優先し、その key がない legacy DB だけ full-scan stamp に fallback する。
/// </summary>
internal string? GetIndexedHeadForResponse()
=> TryGetIndexedHeadForResponse(_conn);

internal static string? TryGetIndexedHeadForResponse(SqliteConnection conn)
{
try
{
using var cmd = conn.CreateCommand();
cmd.CommandText = """
SELECT value
FROM codeindex_meta
WHERE key = CASE
WHEN EXISTS (
SELECT 1
FROM codeindex_meta
WHERE key = @latestHead
)
THEN @latestHead
ELSE @legacyHead
END
LIMIT 1
""";
SqliteCommandPolicy.Add(cmd, "@latestHead", DbContext.IndexedHeadShaMetaKey);
SqliteCommandPolicy.Add(cmd, "@legacyHead", DbContext.IndexedHeadCommitMetaKey);
return cmd.ExecuteScalar() as string;
}
catch (SqliteException)
{
return null;
}
}

private static string? TryGetMetaString(SqliteConnection conn, string key)
{
// Inline the codeindex_meta lookup to avoid creating a DbContext here.
Expand Down
Loading
Loading