fix: preserve encoded Git paths and native ranges

[lzehrung]

Summary

• Convert native query capture byte offsets to UTF-16 source ranges.
• Preserve NUL-delimited Git filenames and absolute blob-hash inputs.
• Parse quoted and whitespace-bearing diff headers without losing paths.
• Broaden import/alias identifier extraction to each source language's real identifier grammar (JS/TS ID_Start/ID_Continue, Python PEP 3131, Rust/Go/Java/Kotlin/C# Unicode letters/XID grammars, PHP any byte >= 0x80) across native-query, text-fallback, and specifier extraction.
• Keep runtime and type-only import edges to the same target distinct during graph edge dedup, preferring the edge with better provenance instead of silently dropping one.
• Disambiguate diff --git a/X b/Y header paths using the unambiguous ---/+++ lines (and equal-halves preference when neither is available), instead of always taking the earliest b/ split.

Verification

• Focused native-query, Git-path, cache-invalidation, and streaming-parser regressions passed.
• Final review approved the UTF-16 and path-encoding behavior.

Deferred follow-up (out of scope for this PR)

• docs/plans/2026-08-17-unicode-identifier-normalization.md: NFC/NFKC normalization for Rust/Python name resolution so decomposed and composed forms of the same identifier resolve as one symbol. Net-new, multi-file work spanning import extraction, symbol indexing, and navigation matching.
• docs/plans/2026-08-17-unicode-import-e2e-fixtures.md: end-to-end fixture coverage proving Unicode-named imports resolve through goto/references for the newly-broadened languages (Java, Kotlin, C#, Go, PHP, Rust).

[Copilot]

Pull request overview

This PR tightens correctness around Unicode and Git diff/path handling: native Tree-sitter capture byte offsets are converted to UTF-16 string indexes, Git filename handling preserves raw encoding (including NUL-delimited output), and the unified-diff parser now correctly decodes quoted/octal-escaped headers.

Changes:

• Convert native query capture byte offsets/points into UTF-16 Range indexes/columns via a per-source byte→string index map.
• Make Git diff/path plumbing robust to non-ASCII, whitespace-bearing, and rename paths using -z, explicit rename detection, and safer blob-hash collection.
• Add regressions covering quoted diff headers, Unicode identifiers (goto/references), and Unicode import/alias extraction across languages.

Reviewed changes

Copilot reviewed 45 out of 45 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/streaming-parser.test.ts	Adds C12 regression coverage for quoted diff --git headers (UTF-8 octal escapes, embedded quotes, trailing whitespace).
tests/samples/python/.regressions/unicode_def.py	Adds Python Unicode-identifier fixture for native semantic parity.
tests/samples/python/.regressions/unicode_consumer.py	Adds consumer fixture referencing the Unicode identifier.
tests/references.test.ts	Adds C11 regression ensuring cross-file reference counts match for Unicode vs ASCII control.
tests/native-semantic-parity.test.ts	Extends native parity suite with Python Unicode fixtures and expectations.
tests/native-query-results.test.ts	Adds direct unit test for UTF-8 byte offsets → UTF-16 range conversion.
tests/languages/zig.test.ts	Adds Zig range regression around multibyte text preceding an ASCII identifier.
tests/languages/unicodeSymbolRange.ts	Adds shared helper asserting published ranges match source.indexOf() for native-driven symbols.
tests/languages/swift.test.ts	Adds Swift Unicode symbol-range regression using the shared helper.
tests/languages/rust.test.ts	Adds Rust Unicode symbol-range regression using the shared helper.
tests/languages/php.test.ts	Adds PHP Unicode symbol-range regression using the shared helper.
tests/languages/kotlin.test.ts	Adds Kotlin Unicode symbol-range regression using the shared helper.
tests/languages/java.test.ts	Adds Java Unicode symbol-range regression using the shared helper.
tests/languages/go.test.ts	Adds Go Unicode symbol-range regression using the shared helper.
tests/languages/csharp.test.ts	Adds C# Unicode symbol-range regression using the shared helper.
tests/languages/cpp.test.ts	Adds C++ Unicode symbol-range regression using the shared helper.
tests/index.test.ts	Updates Python sample module count and asserts regression fixtures are indexed.
tests/import-extraction-unicode-identifiers.test.ts	Adds targeted tests ensuring import/alias extractors accept Unicode identifiers across languages.
tests/impact-git-provider.test.ts	Adds C12 regression ensuring review/impact diff path resolves Unicode filenames and renames.
tests/goto.test.ts	Adds C11 regression ensuring goto-definition lands on correct UTF-16 index for Unicode identifiers.
tests/git-revision-safety.test.ts	Updates expectations for explicit --find-renames insertion in git diff args.
tests/git-diff-semantics.test.ts	Adds C12 regressions for non-ASCII/space/leading-space filenames and renames through diff parsing.
tests/fast-graph-edgecases.test.ts	Adds regression ensuring runtime + type-only edges to same target survive dedup.
tests/fallback-import-extraction-messages.test.ts	Adds regression ensuring fallback import-extraction logs are human-readable.
tests/duplicates.test.ts	Adjusts case-variant path selection in duplicate handling test.
tests/cache-invalidation.test.ts	Adds regressions for git signature hashing in repo-subdir roots and whitespace-leading paths.
tests/build-index-import-options.test.ts	Adds regression ensuring shared import options propagate to embedded SFC blocks.
tests/snapshots/native-semantic-parity.test.ts.snap	Updates snapshot for the added Python Unicode native-parity fixture.
src/util/specifiers.ts	Updates JS/TS + Python specifier extraction regexes to accept Unicode identifiers.
src/util/git.ts	Preserves NUL-delimited paths, forces rename detection, and makes blob-hash collection robust to special paths/subdir roots.
src/native/queryResults.ts	Converts native capture ranges from UTF-8 bytes to UTF-16 indexes via a provided byte→string map.
src/native/projectedTree.ts	Replaces ad-hoc byte mapping with shared byte→string index map utilities.
src/native/byteIndex.ts	Adds reusable UTF-8 byte offset → UTF-16 string index/column conversion utilities.
src/languages/importStatementParsers.ts	Broadens multiple language import statement parsers to accept Unicode identifiers.
src/indexer/locals-and-exports.ts	Ensures all native capture → Range conversions use the byte→string index map.
src/indexer/imports/python.ts	Broadens Python import/alias regex handling for Unicode identifiers.
src/indexer/imports/nativeCaptures.ts	Broadens object-pattern binding extraction to accept Unicode identifiers.
src/indexer/imports/languageSpecific.ts	Broadens Go/Java/Kotlin text import fallbacks to accept Unicode identifiers.
src/indexer/imports/jsTextImports.ts	Broadens JS/TS text import extraction to accept Unicode identifiers.
src/indexer/build-index.ts	Centralizes shared import options so embedded blocks receive the same settings/events.
src/indexer/build-cache/reports.ts	Improves fallback import extraction log messages for clarity and stability.
src/impact/parse.ts	Decodes git-quoted/octal-escaped paths and parses quoted diff headers without losing paths/whitespace.
src/graphs/specifiers.ts	Broadens Python module specifier parsing to accept Unicode identifiers.
src/graph-edge-collector.ts	Exposes provenance comparison helper for use during edge merges.
src/graph-builder.ts	Fixes edge merge identity to keep runtime vs type-only edges distinct and prefer better provenance.

Suppressed comments (1)

src/indexer/imports/python.ts:130

• Similarly, the import ... as ... fallback regex uses \p{L}/\p{N} which is narrower than Python’s Unicode identifier rules; this can still miss valid module names/aliases containing combining marks or other ID_Continue characters. Aligning this with ID_Start/ID_Continue keeps behavior consistent with other identifier handling in the codebase.

  const importPattern = /^(?:\s*)import\s+([\p{L}_][\p{L}\p{N}_.]*)\s*(?:as\s+([\p{L}_][\p{L}\p{N}_]*))?/gmu;
  for (const match of pySrc.matchAll(importPattern)) {

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 45 out of 45 changed files in this pull request and generated no new comments.

Suppressed comments (10)

src/indexer/imports/jsTextImports.ts:65

• Namespace import aliases in JS/TS are also Unicode identifiers (ID_Start/ID_Continue plus ZWNJ/ZWJ), but this regex only matches \p{L}/\p{N}/$_ and can drop valid aliases.

    const namespaceMatch = clause.match(/^\*\s+as\s+([\p{L}_$][\p{L}\p{N}_$]*)$/u);

src/indexer/imports/jsTextImports.ts:115

• CommonJS require() bindings should use JS identifier semantics (ID_Start/ID_Continue plus ZWNJ/ZWJ). Using only \p{L}/\p{N}/$_ can still miss valid local names.

  const defaultRequirePattern =
    /(?:^|[;{}])\s*(?:export\s+)?(?:const|let|var)\s+([\p{L}_$][\p{L}\p{N}_$]*)\s*=\s*require\s*\(\s*(["'])(?<module>[^"']+)\2\s*\)/gmu;
  for (const match of source.matchAll(defaultRequirePattern)) {

src/indexer/imports/jsTextImports.ts:142

• Destructured CommonJS bindings should use JS identifier semantics (ID_Start/ID_Continue plus ZWNJ/ZWJ); this pattern still excludes valid identifiers such as those with combining marks.

      const namedMatch = spec.match(/^([\p{L}_$][\p{L}\p{N}_$]*)(?::\s*([\p{L}_$][\p{L}\p{N}_$]*))?$/u);

src/indexer/imports/jsTextImports.ts:165

• import = require(...) local bindings should use JS identifier semantics (ID_Start/ID_Continue plus ZWNJ/ZWJ); this regex currently only matches \p{L}/\p{N}/$_.

  const importEqualsPattern =
    /(?:^|[;{}])\s*import\s+([\p{L}_$][\p{L}\p{N}_$]*)\s*=\s*require\s*\(\s*(["'])(?<module>[^"']+)\2\s*\)/gmu;
  for (const match of source.matchAll(importEqualsPattern)) {

src/languages/importStatementParsers.ts:54

• Same issue for the Rust use ... as alias parser: the alias regex is still \p{L}/\p{N}-only even though Rust identifiers are XID-based.

  const aliasMatch = useBody.match(/^(.*?)\s+as\s+([\p{L}_][\p{L}\p{N}_]*)$/u);
  const rawPath = aliasMatch?.[1]?.trim() ?? useBody;

src/indexer/build-cache/reports.ts:128

• The message selection checks supportsReducedModeRegexRecovery(language) before checking event.reason === "fast", so JS/TS/TSX fast-mode fallback events will be logged as “native import recovery degraded” instead of accurately reporting fast mode.

    if (event.reason === "reduced-mode") {
      message = `Native parser unavailable for ${event.language}; using reduced import extraction.`;
    } else if (supportsReducedModeRegexRecovery(event.language)) {
      message = `Native import recovery degraded for ${event.language}; using native-owned fallback extraction.`;
    } else if (event.reason === "fast") {
      message = `Fast mode active for ${event.language}; using regex-based import extraction instead of the native parser.`;

src/indexer/imports/nativeCaptures.ts:33

• This regex still only accepts letters/digits/$_, but JS/TS identifiers are defined in terms of Unicode ID_Start/ID_Continue (plus ZWNJ/ZWJ). Valid bindings like "℘" (U+2118) or names with combining marks will still be silently dropped.

    // JS/TS identifiers permit Unicode ID_Start/ID_Continue plus $/_, not just ASCII.
    const match = withoutDefault.match(/^([\p{L}_$][\p{L}\p{N}_$]*)(?::\s*([\p{L}_$][\p{L}\p{N}_$]*))?$/u);

src/indexer/imports/jsTextImports.ts:34

• These identifier regexes use \p{L}/\p{N}, which is narrower than JS/TS’s actual Unicode identifier rules (ID_Start/ID_Continue plus ZWNJ/ZWJ). This can still miss valid import bindings (e.g. Other_ID_Start chars or combining-mark continuations).

This issue also appears in the following locations of the same file:

• line 65
• line 113
• line 142
• line 163

  // JS/TS identifiers permit Unicode ID_Start/ID_Continue plus $/_, not just ASCII.
  const typeOnlyMatch = spec.match(/^type\s+([\p{L}_$][\p{L}\p{N}_$]*)(?:\s+as\s+([\p{L}_$][\p{L}\p{N}_$]*))?$/u);
  if (typeOnlyMatch) {
    const imported = typeOnlyMatch[1]!;
    return { imported, local: typeOnlyMatch[2] ?? imported, typeOnly: true };
  }

  const namedMatch = spec.match(/^([\p{L}_$][\p{L}\p{N}_$]*)(?:\s+as\s+([\p{L}_$][\p{L}\p{N}_$]*))?$/u);
  if (!namedMatch) return null;

src/languages/importStatementParsers.ts:38

• The comment says Rust identifiers use Unicode XID_Start/XID_Continue, but the patterns here only accept \p{L}/\p{N} (letters/digits). That will still reject valid Rust identifiers covered by XID (e.g. certain "Other_ID_Start" code points).

This issue also appears on line 53 of the same file.

  // Rust identifiers permit Unicode XID_Start/XID_Continue, not just ASCII.
  const modMatch = trimmed.match(/^mod\s+([\p{L}_][\p{L}\p{N}_]*)\s*;?$/u);
  if (modMatch?.[1]) {
    return {
      kind: "module",
      from: modMatch[1],
      local: modMatch[1],
      isExternCrate: false,
    };
  }

  const externMatch = trimmed.match(
    /^extern\s+crate\s+([\p{L}_][\p{L}\p{N}_]*)(?:\s+as\s+([\p{L}_][\p{L}\p{N}_]*))?\s*;?$/u,
  );

src/native/queryResults.ts:21

• This docstring is Rust-specific, but rangeFromNativeCapture is used for native Tree-sitter captures across multiple languages. That can mislead future callers about which backends require byte->UTF-16 conversion.

/**
 * Rust's Tree-sitter captures expose UTF-8 byte offsets. `Range` and every downstream
 * consumer (source slicing, portable handles, rename edits) expect UTF-16 string indexes,
 * so every capture must convert through the caller's per-file `byteIndexMap` here.
 */

[Copilot]

Pull request overview

Copilot reviewed 46 out of 47 changed files in this pull request and generated no new comments.

Suppressed comments (10)

src/graphs/specifiers.ts:284

• The from native-import parser also excludes combining-mark continuations. When another statement already contributed a specifier, fallback extraction does not run and this valid module is omitted; use XID_Start/XID_Continue per dotted segment here as well.

          const mFrom = /^\s*from\s+(\.*)([\p{L}_][\p{L}\p{N}_.]*)?\s+import\b/u.exec(stmtText);

src/languages/importStatementParsers.ts:171

• The same restricted alias pattern breaks valid non-ASCII PHP aliases in ordinary use clauses, causing the as suffix to become part of the imported namespace instead of the local alias.

  const aliasMatch = clause.match(/^(.*?)\s+as\s+([\p{L}_][\p{L}\p{N}_]*)$/iu);

src/util/git.ts:47

• Git's C-style pathname quoting also emits \a, \b, \v, and \f for legal control bytes. Leaving those escapes literal produces a different filename, so diffs containing such paths still cannot be resolved correctly.

    if (next === "n") {

src/util/git.ts:452

• NUL delimiting does not fully preserve UTF-8 here because runGit converts each stdout Buffer chunk independently with chunk.toString(). Stream chunks may split a multibyte filename, replacing both halves with U+FFFD before this split; use a StringDecoder (or concatenate buffers before decoding) in runGit so changed-file and unified-diff paths remain intact for large outputs too.

    const relFiles = stdout.split("\0").filter(Boolean);

src/impact/parse.ts:201

• The earliest- b/ split still corrupts a legal unquoted path containing that substring, such as dir b/file.ts; Git does not C-quote ordinary spaces. A modified file then keeps the malformed header path because finalizeFile does not use the later unambiguous ---/+++ paths. Defer/override path selection from those file headers (and rename/copy headers) rather than accepting this ambiguity.

const DIFF_GIT_HEADER_PLAIN = /^a\/(.+?) b\/(.+)$/;

src/languages/importStatementParsers.ts:154

• PHP permits every non-ASCII byte in identifiers, but \p{L}/\p{N} excludes valid aliases containing marks or symbols (for example an emoji). In this group-use path the failed alias match incorrectly folds as <alias> into fullPath; accept any non-ASCII code point after decoding UTF-8.

This issue also appears on line 171 of the same file.

      // PHP identifiers permit any byte >= 0x80, which in practice means non-ASCII UTF-8.
      const aliasMatch = body.match(/^(.*?)\s+as\s+([\p{L}_][\p{L}\p{N}_]*)$/iu);

src/util/identifiers.ts:5

• These new identifier patterns expand language-support behavior across several languages, but this change does not update docs/language-parity.md and docs/scenario-catalog.md. Repository guidance requires both capability and fixture-coverage docs to stay aligned whenever language support changes.

export const ECMASCRIPT_IDENTIFIER_SOURCE = String.raw`[$_\p{ID_Start}](?:[$_\p{ID_Continue}]|\u200c|\u200d)*`;

/** Unicode XID identifiers, with underscores permitted at every position. */
export const XID_IDENTIFIER_SOURCE = String.raw`[_\p{XID_Start}][_\p{XID_Continue}]*`;

src/util/specifiers.ts:258

• These PEP 3131 patterns omit XID_Continue characters such as combining marks. For import cafe\u0301, the unanchored first regex returns the incorrect module cafe; dotted names also currently allow a digit immediately after .. Build each segment from XID_Start/XID_Continue so fallback graph edges retain the real module name.

    const reImport = /^\s*import\s+([\p{L}_][\p{L}\p{N}_.]*)/gmu;
    for (const match of cleaned.matchAll(reImport)) out.push(match[1]!);
    const reFrom = /^\s*from\s+(\.+(?:[\p{L}_][\p{L}\p{N}_.]*)?|[\p{L}_][\p{L}\p{N}_.]*)\s+import/gmu;

src/graphs/specifiers.ts:279

• This native-import branch still rejects valid PEP 3131 names whose continuation contains a combining mark, despite the adjacent extraction path now supporting them. In a file that also has any ordinary import, out is nonempty and the fallback is skipped, so the Unicode module silently disappears from the graph.

This issue also appears on line 284 of the same file.

              const parsed = spec.match(/^([\p{L}_][\p{L}\p{N}_.]*)(?:\s+as\s+[\p{L}_][\p{L}\p{N}_]*)?$/u);

src/graph-builder.ts:119

• This changes graph semantics by preserving separate runtime/type-only edges and selecting provenance, but that behavior is not mentioned in the PR title or description, which only covers native ranges and Git paths. Document and verify this additional scope explicitly (or split it into a focused PR) so reviewers and release notes do not miss the graph-output contract change.

        // typeOnly is part of identity: a runtime import and a type-only import to the same
        // target are distinct edges (e.g. `import { X }` plus `import type { X }`), and
        // collapsing them on from/raw/target alone silently drops the weaker of the two.
        const kind = edge.typeOnly ? "type-only" : "runtime";
        const key = `${edge.from}::${edge.raw}::${target}::${kind}`;
        const previous = byKey.get(key);
        if (!previous || hasBetterProvenance(edge, previous)) byKey.set(key, edge);

[Copilot]

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated no new comments.

Suppressed comments (6)

src/languages/importStatementParsers.ts:415

• This character class still drops valid Java imports. Java identifier starts include $ and other currency/connector characters, while identifier parts also include combining and format characters, so inputs such as import com.$Widget; or a decomposed Unicode class name return null. Use shared Java identifier-start/part segment patterns and apply the same grammar to the text fallback.

  const match = stmtText.trim().match(/^\s*import\s+(static\s+)?([\p{L}_][\p{L}\p{N}_.]*(?:\.\*)?)\s*;?\s*$/u);

src/indexer/imports/languageSpecific.ts:90

• The text fallback remains narrower than Java's identifier grammar: it rejects valid $/currency/connector starts and combining or format continuation characters. This causes fallback extraction to omit valid Java imports even though the native statement parser is being broadened; share the corrected per-segment Java pattern with parseJavaImportStatement.

  const importPattern = /^\s*import\s+(static\s+)?([\p{L}_][\p{L}\p{N}_.]*(?:\.\*)?)\s*;/gmu;

src/languages/importStatementParsers.ts:443

• These replacement classes are still not the C# identifier grammar. Valid verbatim identifiers such as @class and continuations in the combining, connecting, or formatting categories are rejected, and the same reduced class is repeated for alias, static, and plain using forms. Define shared C# start/part patterns, support the optional @ prefix per identifier segment, and cover all three forms.

  const aliasMatch = trimmed.match(
    /^(?:global\s+)?using\s+([\p{L}_][\p{L}\p{N}_]*)\s*=\s*([\p{L}_][\p{L}\p{N}_.]*)\s*;?$/u,

src/indexer/locals-and-exports.ts:386

• tree is normally a ProjectedSyntaxTree created in build-index.ts:231, whose constructor already builds this byte-to-string table. For every non-ASCII native-query file, this call scans the source and allocates a second byte-sized Uint32Array, doubling conversion CPU and peak map memory. Reuse the projected tree's map or build one map and inject it into both consumers.

    if (!byteIndexMap) byteIndexMap = buildByteToStringIndexMap(source);

docs/language-parity.md:95

• Correct the malformed ECMAScript identifier names and missing spaces in this public documentation.

- Import and alias binding extraction accepts each source language's real identifier grammar rather than an ASCII-narrowed approximation: JS/TS/TSX use ID*Start/ID_Continue plus `$`/`*`and ZWNJ/ZWJ continuations, Python uses PEP 3131 XID_Start/XID_Continue, Rust/Go/Java/Kotlin use their Unicode letter/XID identifier rules, and PHP accepts any byte`>= 0x80` at any identifier position. This covers native-query statement parsing, text/regex fallback recovery, and specifier extraction for those languages.

docs/scenario-catalog.md:188

• Correct the malformed ID_Start and underscore notation in this scenario description.

| Unicode import identifiers           | `tests/import-extraction-unicode-identifiers.test.ts`                                      | CommonJS destructuring, `import =`/`require()` equals bindings, and text-fallback import/alias extraction accept full ID*Start/ID_Continue identifiers (including `$`/`*` and ZWNJ/ZWJ continuations), not just ASCII/letter-digit subsets.                                                                                                             | Internal regression test                              | 2026-08-16 |

[Copilot]

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/util/identifiers.ts:30

• This does not match the C# identifier grammar it documents. Only _ (not every Pc character) is allowed at the start, Nl is a valid start category, and continuation accepts Nd rather than all of N; the current pattern therefore drops valid aliases such as one starting with a Roman numeral while accepting invalid starts such as U+203F. Use the ECMA-334 category sets and add boundary cases to the parser test.

export const CSHARP_IDENTIFIER_SOURCE = String.raw`@?[\p{L}\p{Pc}_][\p{L}\p{N}\p{Pc}\p{Mn}\p{Mc}\p{Cf}_]*`;

docs/language-parity.md:95

• The PR also broadens C# using aliases and adds C# Unicode tests, but this parity claim omits C#, and the C# scenario table has no corresponding entry. Document the new C# identifier behavior in both parity and scenario-catalog docs so the language-support change stays aligned with the repository's documentation contract.

- Import and alias binding extraction accepts each source language's real identifier grammar rather than an ASCII-narrowed approximation: JS/TS/TSX use `ID_Start`/`ID_Continue` plus `$` and `_` (and ZWNJ/ZWJ continuations), Python uses PEP 3131 `XID_Start`/`XID_Continue`, Rust/Go/Java/Kotlin use their Unicode letter/XID identifier rules, and PHP accepts any byte `>= 0x80` at any identifier position. This covers native-query statement parsing, text/regex fallback recovery, and specifier extraction for those languages.

src/util/identifiers.ts:22

• This is narrower than Character.isJavaIdentifierStart/Part in some valid cases and wider in invalid ones. Java permits Nl at the start and identifier-ignorable Cf characters in continuations, while \p{N} also admits rejected No characters; valid imports using a letter-number or ZWNJ can be dropped and malformed fraction characters accepted. Encode the Java categories explicitly and add boundary cases to the parser test.

export const JAVA_IDENTIFIER_SOURCE = String.raw`[\p{L}\p{Sc}\p{Pc}$][\p{L}\p{N}\p{Sc}\p{Pc}\p{Mn}\p{Mc}$]*`;

[Copilot]

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/util/identifiers.ts:25

• Java's Character.isJavaIdentifierPart accepts combining marks (Mn/Mc), so this pattern still rejects valid decomposed identifiers such as cafe\u0301. Those imports are consequently dropped by both the statement parser and Java text fallback; include the combining-mark categories and add a decomposed-identifier regression.

 * formatting characters (Cf, e.g. ZWNJ/ZWJ). Combining marks and non-decimal number
 * categories (No) are not part of the Java grammar.
 */
export const JAVA_IDENTIFIER_SOURCE = String.raw`[\p{L}\p{Nl}\p{Sc}\p{Pc}][\p{L}\p{Nl}\p{Sc}\p{Pc}\p{Nd}\p{Cf}]*`;

src/impact/parse.ts:231

• Splitting at the first b/ still corrupts a legal unquoted path containing that sequence. For example, diff --git a/dir b/file.ts b/dir b/file.ts produces file.ts b/dir b/file.ts instead of dir b/file.ts, and modified files never replace it from the +++ header. Select the separator whose a/ and b/ suffixes match before falling back for rename/copy headers.

  const plain = remainder.match(DIFF_GIT_HEADER_PLAIN);
  if (!plain) return null;
  return buildInitiatedFile(`a/${plain[1]}`, `b/${plain[2]}`);

[Copilot]

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/impact/parse.ts:201

• This fallback still misparses valid unquoted Git paths containing the separator text b/. For example, Git emits diff --git a/foo b/bar b/foo b/bar for a modified file named foo b/bar; this regex splits at the first internal b/, and finalizeFile does not replace the result with the later ---/+++ paths, so review/impact reports bar b/foo b/bar instead of the real path. Please retain split candidates and disambiguate them from the decoded file/rename headers (including added/deleted cases) rather than accepting the earliest split.

// fallback keeps the original earliest-" b/"-split heuristic, which is the best a regex can
// do when neither side is delimited (a literal " b/" inside an unquoted path is a
// pre-existing, accepted limitation, unchanged by this fix).
const DIFF_GIT_HEADER_BOTH_QUOTED = new RegExp(`^(${QUOTED_PATH_SEGMENT}) (${QUOTED_PATH_SEGMENT})$`);
const DIFF_GIT_HEADER_A_QUOTED = new RegExp(`^(${QUOTED_PATH_SEGMENT}) b\\/(.+)$`);
const DIFF_GIT_HEADER_B_QUOTED = new RegExp(`^a\\/(.+?) (${QUOTED_PATH_SEGMENT})$`);
const DIFF_GIT_HEADER_PLAIN = /^a\/(.+?) b\/(.+)$/;

[Copilot]

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/indexer/imports/languageSpecific.ts:130

• The text-fallback regex repeats the same over-broad Kotlin grammar: it accepts non-Nd numbers and lets a dotted segment begin with a digit or be empty. Since this path constructs imports when native bindings are unavailable, malformed source can produce false dependency edges; match a sequence of valid identifier segments instead.

  // Kotlin identifiers permit Unicode letters, not just ASCII.
  const importPattern = /^\s*import\s+([\p{L}_][\p{L}\p{N}_.]*(?:\.\*)?)(?:\s+as\s+([\p{L}_][\p{L}\p{N}_]*))?\s*$/gmu;

src/languages/importStatementParsers.ts:384

• This pattern does not enforce Kotlin's grammar per dotted segment: \p{N} accepts non-decimal number categories, and allowing dots inside the continuation class accepts invalid imports such as pkg.2mod or pkg..Name. Match each segment separately and use only Nd digits so invalid statements do not create graph bindings.

  const match = stmtText
    .trim()
    .match(/^\s*import\s+([\p{L}_][\p{L}\p{N}_.]*(?:\.\*)?)(?:\s+as\s+([\p{L}_][\p{L}\p{N}_]*))?\s*$/mu);

src/indexer/imports/languageSpecific.ts:42

• This is broader than Go's import-name grammar in two ways: \p{N} also includes letter/other numbers even though Go digits are Nd, and . is a standalone import token rather than an identifier prefix. As written, invalid aliases such as a½ and .alias can replace the native binding's local name; use an alternation for dot and an Nd-only identifier.

This issue also appears on line 129 of the same file.

  const importPattern = /^\s*(?:import\s+)?(?:(?<alias>[._\p{L}][\p{L}\p{N}_]*)\s+)?["'`](?<from>[^"'`]+)["'`]/gmu;

src/impact/parse.ts:243

• The earliest-separator fallback still misparses an unchanged ambiguous path when Git emits no ---/+++ lines, as happens for binary or mode-only changes. For foo b/bar, it records foo versus bar b/foo b/bar, and finalizeFile then reports a rename; examine all separators first and prefer the candidate whose a/ and b/ payloads are equal.

  const plain = remainder.match(DIFF_GIT_HEADER_PLAIN);
  if (!plain) return null;
  return buildInitiatedFile(`a/${plain[1]}`, `b/${plain[2]}`);

[Copilot]

Pull request overview

Copilot reviewed 50 out of 51 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/languages/importStatementParsers.ts:34

• Rust normalizes identifiers to NFC before name resolution, but these new XID matches return the original spelling. A decomposed import path or alias can therefore fail to match a canonically equivalent composed declaration that rustc resolves successfully. Normalize parsed Rust identifiers and apply the same normalization when matching indexed symbols.

const RUST_MODULE_PATTERN = new RegExp(String.raw`^mod\s+(${XID_IDENTIFIER_SOURCE})\s*;?$`, "u");
const RUST_EXTERN_CRATE_PATTERN = new RegExp(
  String.raw`^extern\s+crate\s+(${XID_IDENTIFIER_SOURCE})(?:\s+as\s+(${XID_IDENTIFIER_SOURCE}))?\s*;?$`,
  "u",
);
const RUST_USE_ALIAS_PATTERN = new RegExp(String.raw`^(.*?)\s+as\s+(${XID_IDENTIFIER_SOURCE})$`, "u");

src/util/identifiers.ts:25

• Character.isJavaIdentifierPart also accepts identifier-ignorable ISO control ranges U+0000-U+0008, U+000E-U+001B, and U+007F-U+009F. Omitting them leaves this parser narrower than the Java API named in the comment and rejects otherwise valid import identifiers.

export const JAVA_IDENTIFIER_SOURCE = String.raw`[\p{L}\p{Nl}\p{Sc}\p{Pc}][\p{L}\p{Nl}\p{Sc}\p{Pc}\p{Nd}\p{Mn}\p{Mc}\p{Cf}]*`;

tests/import-extraction-unicode-identifiers.test.ts:28

• These assertions cover parser and binding seams, but most newly claimed languages still lack an end-to-end cross-file case proving that a Unicode import reaches graph resolution, go-to-definition, and references. Add representative semantic fixtures in the nearest language suites and shared navigation/parity suites; otherwise resolver or normalization mismatches can pass all of these tests.
  src/indexer/imports/python.ts:134
• PEP 3131 also requires identifier text to be normalized to NFKC before comparison. Keeping the raw capture means a decomposed cafe\u0301 import will not resolve a composed caf\u00e9 module or symbol even though Python treats them as the same identifier. Normalize module segments, imported names, and aliases consistently across binding extraction, specifier extraction, and Python symbol matching.

      const aliasMatch = item.match(PYTHON_NAMED_IMPORT_PATTERN);
      if (!aliasMatch) continue;
      const imported = aliasMatch[1]!;
      const local = aliasMatch[2] ?? imported;

[Copilot]

Pull request overview

Copilot reviewed 52 out of 53 changed files in this pull request and generated no new comments.

Suppressed comments (5)

docs/plans/2026-08-17-unicode-import-e2e-fixtures.md:21

• This defers coverage that AGENTS.md:21 requires in the same change. The PR changes cross-file import handling for Java, Kotlin, C#, Go, PHP, and Rust, but only Python exercises the full binding -> graph edge -> goto/references/native-parity pipeline; parser-level tests cannot catch a binding that is accepted but fails to resolve. Add the planned per-language fixtures and shared semantic tests before merging.

cross-file language scenario, add or update the nearest language test in
`tests/languages/*.test.ts` and the shared semantic coverage in
`tests/goto.test.ts`, `tests/references.test.ts`, and
`tests/native-semantic-parity.test.ts` when the language uses the native
runtime" — this PR's identifier-breadth change qualifies and that coverage

src/util/git.ts:406

• The argv implementation is introduced specifically to support filenames containing newlines, but the added hash regression only covers leading whitespace. Add a tracked filename containing \n (skipped on platforms that cannot create it) and assert getGitBlobHashes returns its hash; otherwise the key behavior replacing --stdin-paths remains unverified.

    // `hash-object --stdin-paths` accepts newline-delimited input, so it cannot represent a
    // pathname containing a newline. Passing an absolute pathname as an argv value keeps every
    // legal Git pathname atomic and also works when projectRoot is below the repository root.

src/languages/importStatementParsers.ts:416

• This accepts @ and formatting characters as C# syntax but then exposes their raw spelling to consumers. C# treats @ as a lexical escape and removes formatting characters for identifier comparison, so using @Widget = Some.Type; must match a use spelled Widget, and aliases differing only by an allowed Cf character are also identical; raw-string matching will miss both. Preserve source text for ranges, but canonicalize C# comparison/resolution keys and add end-to-end coverage for equivalent spellings.

const CSHARP_DOTTED_NAME_SOURCE = String.raw`${CSHARP_IDENTIFIER_SOURCE}(?:\.${CSHARP_IDENTIFIER_SOURCE})*`;
const CSHARP_USING_ALIAS_PATTERN = new RegExp(
  String.raw`^(?:global\s+)?using\s+(${CSHARP_IDENTIFIER_SOURCE})\s*=\s*(${CSHARP_DOTTED_NAME_SOURCE})\s*;?$`,

src/util/identifiers.ts:26

• Java's identifier-ignorable characters are allowed lexically but are ignored when identifiers are compared. After this regex accepts Cf/control characters, the parsers retain them in from/imported, so an import such as Foo\u200C will not match a declaration named Foo even though Java defines them as the same identifier. Keep raw text for display/ranges, but remove identifier-ignorable characters from Java comparison/resolution keys and add an end-to-end regression.

 * and `Character.isIdentifierIgnorable` characters: formatting characters (Cf, e.g.
 * ZWNJ/ZWJ) plus the ISO control ranges U+0000-U+0008, U+000E-U+001B, and U+007F-U+009F.
 * Non-decimal number categories (No) are not part of the Java grammar.
 */
export const JAVA_IDENTIFIER_SOURCE = String.raw`[\p{L}\p{Nl}\p{Sc}\p{Pc}][\p{L}\p{Nl}\p{Sc}\p{Pc}\p{Nd}\p{Mn}\p{Mc}\p{Cf}\u0000-\u0008\u000E-\u001B\u007F-\u009F]*`;

docs/plans/2026-08-17-unicode-identifier-normalization.md:27

• This is inaccurate: Java ignores Character.isIdentifierIgnorable characters when comparing identifiers, and C# excludes formatting characters (and the verbatim @ prefix) from identifier identity. They do not apply NFC/NFKC, but they still require language-specific canonical comparison; excluding them from the plan leaves the newly accepted forms unresolved.

Java, C#, Kotlin, and Go do **not** normalize identifiers (each raw code
point sequence is a distinct identifier per their specs), so this plan is
scoped to Python and Rust only.