Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
aa01b18
fix: convert native capture byte offsets to UTF-16 string indexes (C11)
lzehrung Aug 15, 2026
b5416be
fix: correct git path encoding and subdirectory blob-hash resolution …
lzehrung Aug 15, 2026
31b08aa
test: cover subdirectory-root git blob hash resolution (C1)
lzehrung Aug 15, 2026
85ec36d
fix: include typeOnly in graph edge dedup key (C3)
lzehrung Aug 15, 2026
6759995
refactor: extract shared import-option builder for primary/embedded s…
lzehrung Aug 15, 2026
227df05
fix: give every fallback-extraction reason a human message (D11)
lzehrung Aug 15, 2026
5cd6c02
fix: accept Unicode identifiers in import/alias extractors (C11 sibli…
lzehrung Aug 15, 2026
e5e7228
test: cover non-ASCII/space filenames and quoted renames through the …
lzehrung Aug 15, 2026
2a65895
test: cover non-ASCII filenames and renames through the review/impact…
lzehrung Aug 15, 2026
7152b12
test: cover quoted diff --git headers directly in the unified-diff pa…
lzehrung Aug 15, 2026
f15343b
test: cover rename detection with diff.renames=false configured local…
lzehrung Aug 15, 2026
cb867b4
test: cover C11 range identity for every usesQueryDrivenLocals language
lzehrung Aug 15, 2026
ea70899
test: extend shared goto/references/native-semantic-parity coverage f…
lzehrung Aug 15, 2026
9df51f6
test: update gitDiffArgs exact-array assertions for --find-renames (C4)
lzehrung Aug 15, 2026
a94f778
style: apply prettier to changed files
lzehrung Aug 15, 2026
c1fa347
fix: preserve encoded git paths in repairs
lzehrung Aug 16, 2026
4148f53
test: stabilize fixture and identity regressions
lzehrung Aug 16, 2026
687deaf
fix: parse Unicode identifier continuations
lzehrung Aug 16, 2026
7dc31cb
fix: support Unicode import bindings
lzehrung Aug 16, 2026
7d7947b
refactor: extract shared decodeGitPath utility to util/git
lzehrung Aug 16, 2026
3f51c09
fix: address review feedback on Git path decoding and identifier breadth
lzehrung Aug 16, 2026
13e0190
fix: address second review batch (Java/C# identifiers, byte-index reu…
lzehrung Aug 16, 2026
c65f9e5
fix: correct Java/C# identifier Unicode categories, document C# in pa…
lzehrung Aug 16, 2026
e1fce21
fix: include combining marks in Java identifier continuation
lzehrung Aug 16, 2026
5699b7e
fix: disambiguate diff --git header paths using --- / +++ lines
lzehrung Aug 17, 2026
9f5389a
fix: address 4th review batch (Kotlin/Go identifiers, diff header dis…
lzehrung Aug 17, 2026
2f9f9b6
fix: add Java identifier-ignorable ISO control ranges; defer normaliz…
lzehrung Aug 17, 2026
3494884
test: cover getGitBlobHashes for a newline-containing pathname; corre…
lzehrung Aug 17, 2026
e4c37fd
fix: convert AST-grep capture columns from UTF-8 bytes to UTF-16
lzehrung Aug 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/coverage/js.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ Source: `coverage/js/lcov.info`

| Metric | Hit | Found | Coverage |
| --------- | ----: | ----: | -------: |
| Lines | 27353 | 30141 | 90.75% |
| Functions | 4559 | 4839 | 94.21% |
| Branches | 20655 | 26070 | 79.23% |
| Lines | 27491 | 30264 | 90.84% |
| Functions | 4565 | 4844 | 94.24% |
| Branches | 20716 | 26123 | 79.30% |

## Least-covered Files

Expand Down Expand Up @@ -42,6 +42,7 @@ These files have line records but no function or branch records, so they are tra
| File | Lines | Functions | Branches |
| --------------------------------------- | ------: | --------: | -------: |
| `src/cliBootstrap.ts` | 0.00% | n/a | n/a |
| `src/util/identifiers.ts` | 100.00% | n/a | n/a |
| `src/languages/definitions/jsFamily.ts` | 100.00% | n/a | n/a |
| `src/duplicate-keywords.ts` | 100.00% | n/a | n/a |
| `src/impact/types.ts` | 100.00% | n/a | n/a |
Expand All @@ -50,6 +51,5 @@ These files have line records but no function or branch records, so they are tra
| `src/languages/definitions/hbs.ts` | 100.00% | n/a | n/a |
| `src/languages/definitions/markdown.ts` | 100.00% | n/a | n/a |
| `src/languages/definitions/mdx.ts` | 100.00% | n/a | n/a |
| `src/languages/definitions/rst.ts` | 100.00% | n/a | n/a |

Generated from LCOV by `node ./scripts/coverage-markdown.mjs`.
1 change: 1 addition & 0 deletions docs/language-parity.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ Notes:
- JavaScript expands `module.exports = { ...source }` for statically resolvable CommonJS imports and local object literals. Dynamic spread sources remain explicit namespace-reexport markers rather than silently disappearing.
- Node `package.json#exports` condition matching follows author key order with mutually exclusive `import`/`require` modes threaded from ESM `import`/`import()` versus CommonJS `require()` (and TypeScript `import x = require(...)`). Nested conditions, array fallbacks, and `default` termination match Node for those cases. Custom `--conditions`, `browser`/`types`/`development`/`production`, import attributes, and `#imports` maps are not modeled.
- Ruby treats `Constant = Struct.new(...)` as a class-kind symbol and a synthetic detailed class declaration. Runtime-computed class factories remain outside this recognition.
- Import and alias binding extraction accepts each source language's real identifier grammar rather than an ASCII-narrowed approximation: JS/TS/TSX use `ID_Start`/`ID_Continue` plus `$` and `_` (and ZWNJ/ZWJ continuations), Python uses PEP 3131 `XID_Start`/`XID_Continue`, Rust/Go/Kotlin use their Unicode letter/XID identifier rules, Java follows `Character.isJavaIdentifierStart`/`isJavaIdentifierPart` (Unicode letters, letter-numbers, currency symbols, connecting punctuation, decimal digits, combining marks, and identifier-ignorable formatting characters), C# follows the ECMA-334 `identifier-start-character`/`identifier-part-character` grammar including the `@` verbatim-identifier prefix, and PHP accepts any byte `>= 0x80` at any identifier position. This covers native-query statement parsing, text/regex fallback recovery, and specifier extraction for those languages.

## Project file discovery coverage

Expand Down
141 changes: 141 additions & 0 deletions docs/plans/2026-08-17-unicode-identifier-normalization.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# Unicode identifier canonicalization for name resolution (2026-08-17)

Status: Planned. Not started; no code in this plan has landed.

## Problem

PR #262 broadened import/alias extraction regexes (`src/util/identifiers.ts`,
`src/languages/importStatementParsers.ts`, `src/indexer/imports/*.ts`,
`src/graphs/specifiers.ts`, `src/util/specifiers.ts`) to accept each source
language's real identifier grammar, including combining-mark continuations
(Mn/Mc) for Java, C#, and PHP, and identifier-ignorable formatting/control
characters (Cf, plus a handful of ISO control ranges) for Java.

Accepting a wider identifier at the regex layer is necessary but not
sufficient for correct resolution: four languages define two spellings of
"the same" identifier as equal for name-resolution purposes, and
`codegraph` currently compares raw captured text everywhere, so it will
treat those equal spellings as different symbols.

- **Python (PEP 3131)**: identifiers are compared after NFKC normalization.
`café` (NFC, U+00E9) and `cafe\u0301` (NFD, "e" + combining acute) are the
_same_ identifier to CPython.
- **Rust**: `rustc` normalizes identifiers to NFC before name resolution
(tracked via `rustc_lexer`/`rustc_parse` identifier normalization since
the RFC on non-ASCII idents). Same NFC/NFD pair collapses to one name.
- **Java (JLS §3.8)**: two identifiers are the same "if, after ignoring
characters for which `Character.isIdentifierIgnorable` returns true, they
have the same sequence of characters." This is not Unicode normalization;
it is deletion of `Cf` formatting characters (ZWNJ/ZWJ/bidi/etc.) and a
handful of ISO control ranges — the exact character set
`JAVA_IDENTIFIER_SOURCE` (added in #262) now accepts as legal continuation
characters. `Foo` and `Foo\u200C` are the same field to `javac` but would
currently resolve as two different symbols here.
- **C# (ECMA-334)**: two identifiers match if identical after (1) removing
a leading `@` verbatim-identifier prefix, (2) resolving
unicode-escape-sequences, and (3) removing `Cf` formatting characters.
`@Widget` and `Widget` name the same symbol; `Widget` and `Widget\u200C`
do too. `CSHARP_IDENTIFIER_SOURCE` (added in #262) accepts both the `@`
prefix and `Cf` continuation but nothing downstream removes them for
comparison.

Kotlin and Go have no such rule (raw code point sequences are compared
directly per their specs, and neither grammar admits `Cf`/ignorable
characters at all), so this plan does not touch them. PHP compares raw
bytes with no normalization step either. JS/TS (ECMAScript) also performs
no identifier normalization for name resolution — two different Unicode
spellings are genuinely different bindings.

Today, `codegraph` captures whatever byte sequence appears at each site
(import statement, declaration, reference) and compares those sequences
verbatim. This is a real, silent navigation/reference gap, not a parsing
gap — it cannot be fixed by adjusting a regex character class.

## Why this is a separate PR

Fixing this only where PR #262 touched code (import binding extraction)
would be incomplete and misleading: it would make imports parse but not
resolve, or resolve inconsistently depending on which side of a match was
canonicalized. Correct behavior requires canonicalizing at every point a
Python, Rust, Java, or C# identifier is captured or compared:

1. **Import/alias extraction** (already regex-broadened in #262):
- `src/indexer/imports/python.ts` (`collectPythonImportsFromSource`)
- `src/graphs/specifiers.ts` (native Python `import`/`from` parsing)
- `src/util/specifiers.ts` (`extractPythonSpecifiers` fallback)
- `src/languages/importStatementParsers.ts` (`parseRustImportStatement`,
`parseJavaImportStatement`, `parseCsharpUsingDirective`)
- `src/indexer/imports/languageSpecific.ts` (Java text fallback)
2. **Symbol declaration indexing** — not touched by #262, and the actual
source of the "declaration name" side of every match:
- `src/indexer/locals-and-exports.ts` (native capture → `SymbolDef.localName`)
- Wherever Rust/Python/Java/C# detailed symbol extraction reads a node's
text as a declaration name (`src/graphs/symbol-graph-detailed/*`,
native query capture text for `name`/`tname` captures).
3. **Navigation/resolution matching**:
- `src/indexer/navigation.ts` (`findReferences`)
- `src/indexer/navigation-resolve.ts` (`resolveExport`, import → declaration matching)
- `src/indexer/navigation-references.ts` (scope-based reference matching)
- `src/agent/renamePreview.ts`, `src/agent/refactorPlan.ts` (candidate
matching reuses the navigation layer, so should inherit this for free
once navigation canonicalizes)
4. **Symbol/reference hashing and IDs** — `defNodeId` in
`src/graphs/symbol-graph.ts` includes `localName` verbatim in the node
ID; canonicalizing only for comparison (not for the stored ID/display
name) avoids changing portable handles or displayed source text.

## Proposed approach

- Add `canonicalizeIdentifierForComparison(name: string, languageId: string): string`
to `src/util/identifiers.ts` with one explicit branch per language that
needs it, and an explicit passthrough default for every other language
(never a default `.normalize()`/strip call, so adding a new language
never silently opts in):
- `"python"`: `name.normalize("NFKC")`.
- `"rust"`: `name.normalize("NFC")`.
- `"java"`: strip every code point in the `JAVA_IDENTIFIER_SOURCE`
continuation class's `Cf`/ISO-control set (reuse the same ranges
documented on `JAVA_IDENTIFIER_SOURCE` so the two never drift apart).
- `"csharp"`/`"cs"`: strip a single leading `@`, then strip `Cf`
characters (reuse the `Cf` portion of `CSHARP_IDENTIFIER_SOURCE`).
- everything else: return `name` unchanged.
- Canonicalize **only at comparison sites**, never at storage sites: keep
`SymbolDef.localName`, import binding `imported`/`local`, and displayed
text exactly as they appear in source (required for accurate ranges,
rename edits, and portable handles). Build a canonicalized comparison key
alongside the raw name wherever lookups currently do `a === b` or
`map.get(name)` on a Python/Rust/Java/C# identifier, and use that key for
the lookup while keeping the raw name for everything else.
- Concretely: extend whatever lookup structure `resolveExport`/`findReferences`
use (name → declaration map) to key by
`canonicalizeIdentifierForComparison` instead of the raw string, for
Python, Rust, Java, and C# only.

## Verification plan

- Unit tests in `tests/import-extraction-unicode-identifiers.test.ts`
proving each canonicalization branch collapses the documented equal
pairs (`café`/`cafe\u0301` for Python, `Foo`/`Foo\u200C` for Java,
`@Widget`/`Widget` for C#, NFC/NFD pairs for Rust) to the same key,
covering both the extraction and declaration side.
- New cross-file fixtures (see the companion E2E fixture plan
`2026-08-17-unicode-import-e2e-fixtures.md`) per canonicalizing language:
a declaration file using one spelling and a consumer importing the
equal-but-differently-spelled form, asserting `goto`/`references`
resolve across the pair.
- Explicit regression proving Kotlin/Go/PHP/JS/TS do **not** canonicalize
(two differently-spelled-but-"equal" forms remain distinct symbols for
those languages), so this change cannot silently over-canonicalize them.
- Update `docs/language-parity.md`: state which languages canonicalize
identifiers for resolution (Python NFKC, Rust NFC, Java
identifier-ignorable stripping, C# `@`-prefix + formatting-character
stripping) and which do not.

## Non-goals

- No change to displayed/stored identifier text, portable search handles,
or rename-edit content — canonicalization is comparison-only.
- No canonicalization for languages without a documented spec rule
(Kotlin, Go, PHP, JS/TS) even though PR #262 broadened their extraction
grammars; those characters remain part of the identifier's identity for
those languages, matching their real compilers.
82 changes: 82 additions & 0 deletions docs/plans/2026-08-17-unicode-import-e2e-fixtures.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# End-to-end fixture coverage for Unicode import identifiers (2026-08-17)

Status: Planned. Not started; no code in this plan has landed.

## Problem

PR #262 broadened import/alias extraction across JS/TS, Python, PHP, Rust,
Go, Java, Kotlin, and C# (`src/util/identifiers.ts` and the parsers/fallback
extractors that consume it) and added parser-level unit coverage in
`tests/import-extraction-unicode-identifiers.test.ts`. Those tests prove the
regexes and binding-construction functions accept/reject the right inputs
in isolation, matching each language's real identifier grammar.

They do not prove a Unicode-named import survives the full pipeline: native
parse → import binding → graph edge → symbol declaration → `goto`/
`references` resolution. Per `AGENTS.md`, "when adding or changing a
cross-file language scenario, add or update the nearest language test in
`tests/languages/*.test.ts` and the shared semantic coverage in
`tests/goto.test.ts`, `tests/references.test.ts`, and
`tests/native-semantic-parity.test.ts` when the language uses the native
runtime" — this PR's identifier-breadth change qualifies and that coverage
is currently missing.

## Scope

One cross-file scenario per already-native language touched by the
identifier-breadth work, following the existing fixture pattern in
`tests/samples/<language>/` (see `tests/samples/python/.regressions/
unicode_def.py` / `unicode_consumer.py`, already added by this PR, as the
template):

| Language | Sample directory | Unicode case to cover |
| -------- | ------------------------------------ | ------------------------------------------------------------------------------------------ |
| Java | `tests/samples/java/.regressions/` | `$`-prefixed and combining-mark class/import name |
| Kotlin | `tests/samples/kotlin/.regressions/` | Unicode `import ... as alias` |
| C# | `tests/samples/csharp/.regressions/` | `using alias = Namespace;` with a combining-mark alias, plus `@class`-style verbatim alias |
| Go | `tests/samples/go/.regressions/` | Unicode-letter import alias |
| PHP | `tests/samples/php/.regressions/` | non-`\p{L}` `use ... as alias` (e.g. emoji) |
| Rust | `tests/samples/rust/.regressions/` | `use ... as alias` with XID continuation beyond `\p{L}`/`\p{N}` |

JS/TS/TSX and Python already have adjacent native-semantic-parity coverage
from this PR (`tests/samples/python/.regressions/unicode_*.py`,
`tests/native-semantic-parity.test.ts` Python fixtures); this plan extends
the same pattern to the remaining six languages.

## Per-language work (repeat for each row above)

1. Add a two-file fixture: a declaration file with a Unicode-named
exported symbol, and a consumer file that imports it using the
Unicode form the corresponding parser fix now accepts.
2. Extend `tests/languages/<language>.test.ts` with a case asserting the
dependency graph includes the edge between consumer and declaration
file (mirrors existing `LanguageTestDefinition` fixtures in that file).
3. Extend `tests/goto.test.ts` with a case asserting go-to-definition from
the consumer's Unicode-named reference resolves to the declaration.
4. Extend `tests/references.test.ts` with a case asserting the declaration
appears in `findReferences` results from the consumer's usage site.
5. If the language uses the native runtime (all six do), extend
`tests/native-semantic-parity.test.ts` with the same fixture pair so
native-mode regression coverage catches drift.
6. Add a `docs/scenario-catalog.md` row per language (companion to the
parser-level rows already added by PR #262) pointing at the new
`tests/languages/*.test.ts` case as the "Sample".

## Verification plan

- `npx vitest run tests/languages/<language>.test.ts tests/goto.test.ts
tests/references.test.ts tests/native-semantic-parity.test.ts` per
language as each is added.
- Full `npm run check` once all six languages are covered.
- Confirm each new case fails against the pre-PR-#262 regex (sanity check
that the fixture actually exercises the fixed code path, not an
already-passing ASCII-only case).

## Non-goals

- No new fixtures for languages whose identifier grammar was not changed
by PR #262 (Ruby, Swift, Zig, C, C++, SQL, etc.).
- No fixture coverage for the NFC/NFKC normalization work — that is
tracked separately in `2026-08-17-unicode-identifier-normalization.md`
and should reuse this plan's fixture pattern for Python/Rust once it
lands, rather than duplicating fixture setup here.
Loading