Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1523,7 +1523,7 @@ Extractor strategy by language surface:
| Scala | Uses a separate block-call pass for `name { ... }` / `name { x => ... }` forms so idiomatic calls such as `foreach {}`, `Try {}`, and `synchronized {}` stay visible. |
| Common Lisp / Racket | Use a lightweight S-expression scanner that masks strings, line comments, and `#| ... |#` block comments before extracting definitions and function ranges. |
| HTML | Uses a dedicated character-level state machine instead of the regex pattern loop. It walks tag openers, quoted/unquoted attribute values including multi-line values, and masks `<script>` / `<style>` / `<textarea>` / `<title>` bodies plus `<!-- ... -->` comments so attribute-lookalike strings inside those regions do not leak phantom symbols. |
| JSON / JSON Lines | JSON emits `object`, `array`, `property`, and bounded primitive-array `value` symbols with indexed paths. `.jsonl` and `.ndjson` parse each non-empty physical line independently, prefix symbols with a stable zero-based record path such as `[0].result.path`, and emit repository-local path references from each valid record without flattening malformed neighbors. |
| JSON / JSON Lines | JSON emits `object`, `array`, `property`, and bounded primitive-array `value` symbols with indexed paths. Array indexes attach directly to the parent path (`command_cases[0]`, not `command_cases.command_cases[0]`), and object/array parent kinds preserve the hierarchy used by outline depth. Root arrays start at paths such as `[0]`. `.jsonl` and `.ndjson` parse each non-empty physical line independently, prefix symbols with a stable zero-based record path such as `[0].result.path`, and emit repository-local path references from each valid record without flattening malformed neighbors. |
| TOML / repository metadata | TOML tables and keys, EditorConfig sections and keys, Git/Docker ignore rules, Git attribute rules/attributes, and `.rules` blocks/keys are emitted as bounded structural symbols. References are limited to repository-local paths or globs; remote URLs, absolute filesystem paths, and parent traversal are suppressed. |
| Windows application manifests | Manifest element paths, assembly identities, execution levels, and supported-OS values remain structural symbols. Dependent assembly identities emit `dependency` references, while local `file`, `codeBase`, and probing paths emit `project_reference` edges. |
| XML / NuGet.config | Generic XML emits bounded element and attribute paths. NuGet.config additionally promotes package sources, source mappings, signature validation mode, trusted signer names, certificate fingerprints, and `allowUntrustedRoot` values to semantic `property` symbols with `nuget.*` subkinds. |
Expand Down Expand Up @@ -4875,7 +4875,7 @@ LIMIT 20;
| Scala | `name { ... }` / `name { x => ... }` 形式を拾う専用の block-call path があり、`foreach {}`、`Try {}`、`synchronized {}` のような慣用的な呼び出しも graph から消えないようにしています。 |
| Common Lisp / Racket | 文字列、行 comment、`#| ... |#` block comment を mask してから definition と function range を抽出する軽量な S-expression scanner を使います。 |
| HTML | 汎用の正規表現 loop を使わず、専用の文字単位 state machine で tag opener、引用符付き/なし attribute value(複数行値を含む)、`<script>` / `<style>` / `<textarea>` / `<title>` body、`<!-- ... -->` comment を扱い、attribute 名に似た body 内文字列から phantom symbol が漏れないようにします。 |
| JSON / JSON Lines | JSON は `object`、`array`、`property` と、上限付きの primitive-array `value` symbol を index 付き path で出力します。`.jsonl` と `.ndjson` は空でない物理行を個別に parse し、`[0].result.path` のような安定した 0 始まり record path を付けます。各有効 record から repository-local path reference を出し、不正な隣接 record の内容は平坦化しません。 |
| JSON / JSON Lines | JSON は `object`、`array`、`property` と、上限付きの primitive-array `value` symbol を index 付き path で出力します。配列 index は親 path に直接連結し(`command_cases.command_cases[0]` ではなく `command_cases[0]`)、object / array の親 kind を保持して outline の depth に階層を反映します。root array は `[0]` のような path から始まります。`.jsonl` と `.ndjson` は空でない物理行を個別に parse し、`[0].result.path` のような安定した 0 始まり record path を付けます。各有効 record から repository-local path reference を出し、不正な隣接 record の内容は平坦化しません。 |
| TOML / repository metadata | TOML の table / key、EditorConfig の section / key、Git / Docker ignore rule、Git attribute の rule / attribute、`.rules` の block / key を上限付き structural symbol として出力します。reference は repository-local な path / glob に限定し、remote URL、絶対 filesystem path、親 directory traversal は抑止します。 |
| Windows application manifest | manifest element path、assembly identity、execution level、supported OS value を structural symbol として維持します。依存 assembly identity は `dependency` reference、local な `file` / `codeBase` / probing path は `project_reference` edge を出力します。 |
| XML / NuGet.config | 汎用 XML は上限付きの element / attribute path を出力します。NuGet.config ではさらに package source、source mapping、署名検証モード、trusted signer 名、証明書 fingerprint、`allowUntrustedRoot` の値を `nuget.*` subkind 付きの semantic `property` symbol にします。 |
Expand Down
23 changes: 23 additions & 0 deletions changelog.d/unreleased/4874.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
category: fixed
issues:
- 4874
affected:
- src/CodeIndex/Indexer/Symbols/SymbolExtractor.StructuredData.cs
- src/CodeIndex/Indexer/Symbols/SymbolExtractor.Contracts.cs
- src/CodeIndex/Database/DbSymbolReader.Outline.cs
- tests/CodeIndex.Tests/DatabaseTests.cs
- tests/CodeIndex.Tests/IndexCommandRunnerTests.cs
- tests/CodeIndex.Tests/SymbolExtractorRepositoryMetadataTests.cs
- tests/CodeIndex.Tests/SymbolExtractorTests.cs
- tests/CodeIndex.Tests/QueryCommandRunnerInspectTests.cs
- DEVELOPER_GUIDE.md
---

## English

- **JSON arrays now preserve their paths and hierarchy (#4874)** — array indexes attach directly to their parent path, root arrays are extracted, and object/array parent metadata keeps `symbols`, `outline`, and `definition` results at the correct depth without duplicated names such as `command_cases.command_cases[0]`.

## 日本語

- **JSON 配列の path と階層を正しく保持するようになりました (#4874)** — 配列 index を親 path に直接連結し、root array を抽出するとともに、object / array の親 metadata を保持することで、`symbols`、`outline`、`definition` が `command_cases.command_cases[0]` のような重複名を出さず正しい depth を返すようになりました。
107 changes: 87 additions & 20 deletions src/CodeIndex/Database/DbSymbolReader.Outline.cs
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ public partial class DbReader
}

var startColumnOrderSql = GetSymbolColumnSql("start_column", "CAST(2147483647 AS INTEGER)");
var structuredHierarchyOrderSql = lang is "json" or "jsonl"
? "LENGTH(CAST(s.name AS BLOB)) ASC,"
: string.Empty;
var includeReferenceCountSql = includeReferenceCounts && _hasReferencesTable;
var referenceCountSql = includeReferenceCountSql
? "CASE WHEN COALESCE(symbol_defs.definition_sites, 0) = 1 THEN COALESCE(symbol_rank.reference_count, 0) ELSE 0 END"
Expand Down Expand Up @@ -96,6 +99,7 @@ FROM symbols s
WHERE s.file_id = @fileId
ORDER BY s.line ASC,
{startColumnOrderSql} ASC,
{structuredHierarchyOrderSql}
s.kind COLLATE BINARY ASC,
s.name COLLATE BINARY ASC,
s.id ASC";
Expand All @@ -104,6 +108,7 @@ FROM symbols s
SqliteCommandPolicy.AddNullableText(symCmd, "@lang", lang);

var symbols = new List<OutlineSymbol>();
var isJsonStructuredData = lang is "json" or "jsonl";
using (var reader = symCmd.ExecuteTrackedReader())
{
while (reader.TrackedRead())
Expand All @@ -123,15 +128,18 @@ FROM symbols s
Signature = GetNullableString(reader, 7),
ContainerKind = GetNullableString(reader, 8),
ContainerName = containerName,
Path = BuildOutlineSymbolPath(containerQualifiedName ?? containerName, name),
Path = BuildOutlineSymbolPath(
containerQualifiedName ?? containerName,
name,
isJsonStructuredData),
Visibility = GetNullableString(reader, 11),
ReturnType = GetNullableString(reader, 12),
ReferenceCount = GetNullableInt32(reader, 13),
});
}
}

PopulateOutlineDepths(symbols);
PopulateOutlineDepths(symbols, isJsonStructuredData);
ApplyQueryOutputSignatureLimits(symbols);
PopulateOutlineDisplayNames(symbols, lang);

Expand Down Expand Up @@ -172,16 +180,34 @@ private static string BuildOutlineDisplayName(OutlineSymbol symbol, string? lang
: symbol.Name;
}

private static string BuildOutlineSymbolPath(string? containerQualifiedName, string name)
private static string BuildOutlineSymbolPath(
string? containerQualifiedName,
string name,
bool allowArrayIndexBoundary)
{
if (string.IsNullOrWhiteSpace(containerQualifiedName))
return name;

return name.StartsWith(containerQualifiedName + ".", StringComparison.Ordinal)
return IsQualifiedOutlineChildPath(containerQualifiedName, name, allowArrayIndexBoundary)
? name
: $"{containerQualifiedName}.{name}";
}

private static bool IsQualifiedOutlineChildPath(
string containerPath,
string childPath,
bool allowArrayIndexBoundary)
{
if (!childPath.StartsWith(containerPath, StringComparison.Ordinal)
|| childPath.Length <= containerPath.Length)
{
return false;
}

var boundary = childPath[containerPath.Length];
return boundary == '.' || (allowArrayIndexBoundary && boundary == '[');
}

private static bool IsCallableOutlineSymbol(string kind)
{
return kind is "function" or "operator" or "method" or "constructor";
Expand Down Expand Up @@ -327,15 +353,27 @@ private static string SimplifyParameterForOutline(string parameter, string? lang
return string.Join(" ", parts.Take(parts.Length - 1));
}

private static void PopulateOutlineDepths(List<OutlineSymbol> symbols)
private static void PopulateOutlineDepths(
List<OutlineSymbol> symbols,
bool isJsonStructuredData)
{
var depthCache = new Dictionary<int, int>();
var activeStack = new HashSet<int>();
for (var i = 0; i < symbols.Count; i++)
symbols[i].Depth = GetOutlineDepth(symbols, i, depthCache, activeStack);
symbols[i].Depth = GetOutlineDepth(
symbols,
i,
depthCache,
activeStack,
isJsonStructuredData);
}

private static int GetOutlineDepth(List<OutlineSymbol> symbols, int index, Dictionary<int, int> depthCache, HashSet<int> activeStack)
private static int GetOutlineDepth(
List<OutlineSymbol> symbols,
int index,
Dictionary<int, int> depthCache,
HashSet<int> activeStack,
bool isJsonStructuredData)
{
if (depthCache.TryGetValue(index, out var cachedDepth))
return cachedDepth;
Expand All @@ -347,20 +385,35 @@ private static int GetOutlineDepth(List<OutlineSymbol> symbols, int index, Dicti
var depth = 0;
if (!string.IsNullOrEmpty(symbol.ContainerName))
{
var parentIndex = FindOutlineContainerIndex(symbols, index, symbol.ContainerName, symbol.ContainerKind);
var parentIndex = FindOutlineContainerIndex(
symbols,
index,
symbol.ContainerName,
symbol.ContainerKind,
isJsonStructuredData);
if (parentIndex >= 0)
depth = GetOutlineDepth(symbols, parentIndex, depthCache, activeStack) + 1;
depth = GetOutlineDepth(
symbols,
parentIndex,
depthCache,
activeStack,
isJsonStructuredData) + 1;
}

activeStack.Remove(index);
depthCache[index] = depth;
return depth;
}

private static int FindOutlineContainerIndex(List<OutlineSymbol> symbols, int childIndex, string containerName, string? containerKind)
private static int FindOutlineContainerIndex(
List<OutlineSymbol> symbols,
int childIndex,
string containerName,
string? containerKind,
bool isJsonStructuredData)
{
var child = symbols[childIndex];
var expectedContainerPath = GetOutlineContainerPath(child);
var expectedContainerPath = GetOutlineContainerPath(child, isJsonStructuredData);
for (var i = childIndex - 1; i >= 0; i--)
{
var candidate = symbols[i];
Expand All @@ -375,32 +428,46 @@ private static int FindOutlineContainerIndex(List<OutlineSymbol> symbols, int ch
}
if (candidate.Line > child.Line)
continue;
if (IsOutlineContainerMatch(candidate, child.Line))
if (IsOutlineContainerMatch(candidate, child.Line, isJsonStructuredData))
return i;
}

return -1;
}

private static string? GetOutlineContainerPath(OutlineSymbol symbol)
private static string? GetOutlineContainerPath(
OutlineSymbol symbol,
bool allowArrayIndexBoundary)
{
if (string.IsNullOrWhiteSpace(symbol.Path) || string.IsNullOrWhiteSpace(symbol.Name))
return null;

var suffix = "." + symbol.Name;
return symbol.Path.EndsWith(suffix, StringComparison.Ordinal)
? symbol.Path[..^suffix.Length]
: null;
if (symbol.Path.EndsWith(suffix, StringComparison.Ordinal))
return symbol.Path[..^suffix.Length];

return !string.IsNullOrWhiteSpace(symbol.ContainerName)
&& string.Equals(symbol.Path, symbol.Name, StringComparison.Ordinal)
&& IsQualifiedOutlineChildPath(
symbol.ContainerName,
symbol.Name,
allowArrayIndexBoundary)
? symbol.ContainerName
: null;
}

private static bool IsOutlineContainerMatch(OutlineSymbol candidate, int childLine)
private static bool IsOutlineContainerMatch(
OutlineSymbol candidate,
int childLine,
bool isJsonStructuredData)
{
if (candidate.StartLine <= childLine && candidate.EndLine >= childLine)
return true;

// file-scoped namespaces have no body range, so they do not enclose children by lines
// even though they are the correct logical container.
return candidate.Kind == "namespace"
// File-scoped namespaces and structured-data containers have no body range, so they
// do not enclose children by lines even though their exact paths identify the parent.
return (candidate.Kind == "namespace"
|| (isJsonStructuredData && candidate.Kind is "object" or "array" or "record"))
&& candidate.BodyStartLine == null
&& candidate.BodyEndLine == null
&& candidate.Line <= childLine;
Expand Down
8 changes: 6 additions & 2 deletions src/CodeIndex/Indexer/Symbols/SymbolExtractor.Contracts.cs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ public static partial class SymbolExtractor
public const int ApplicationManifestContractVersion = 3;
public const int MarkdownContractVersion = 3;
public const int DependencyLockContractVersion = 3;
public const int JsonContractVersion = 3;
public const int JsonLinesContractVersion = 3;

private static readonly string[] ExplicitReferenceGraphContractLanguages =
["crystal", "groovy", "tcl", "prolog", "ambiguous_pl"];
Expand Down Expand Up @@ -68,11 +70,13 @@ public static int GetContractVersion(string? lang)
"crystal" or "groovy" or "tcl" => DynamicReferenceGraphContractVersion,
"prolog" or "ambiguous_pl" => PrologReferenceGraphContractVersion,
"ada" or "ambiguous_m" or "cython" or "d" or "julia" or "matlab" or "nim" or "objc" => ScientificNativeGraphContractVersion,
"config" or "dockerignore" or "editorconfig" or "gitattributes" or "gitignore" or "jsonl" or "toml" => RepositoryMetadataContractVersion,
"config" or "dockerignore" or "editorconfig" or "gitattributes" or "gitignore" or "toml" => RepositoryMetadataContractVersion,
"app_manifest" => ApplicationManifestContractVersion,
"markdown" => MarkdownContractVersion,
"dependency_lock" => DependencyLockContractVersion,
"cmake" or "dependency_manifest" or "graphql" or "html" or "json" or "justfile" or "msbuild" or "solution" or "yaml" => ExpandedLanguageContractVersion,
"json" => JsonContractVersion,
"jsonl" => JsonLinesContractVersion,
"cmake" or "dependency_manifest" or "graphql" or "html" or "justfile" or "msbuild" or "solution" or "yaml" => ExpandedLanguageContractVersion,
_ => DefaultContractVersion,
};
}
Expand Down
Loading
Loading