Preserve live LSP document-symbol hierarchy

[Widthdom]

Summary

• Re-extract document symbols from the latest accepted live full-text buffer while preserving the indexed language and the existing hierarchy builder.
• Reject stale or duplicate document versions and retain bounded version tombstones across live-buffer eviction and oversized updates.
• Bound live symbol extraction to the LSP response budget and fall back to indexed symbols when the indexed extractor is unavailable.
• Add regressions for hierarchy/range parity, successive CRLF/LF and Unicode changes, stale versions, extension-sensitive language selection, dense inputs, extractor fallback, and eviction behavior.

Root cause

Full-document changes were read from the live document store, but textDocument/documentSymbol continued to serve the flattened indexed symbol rows. Re-extracting without the indexed language, version safeguards, or a materialization bound also introduced correctness and resource risks, so the fix keeps those contracts explicit.

Validation

• dotnet build CodeIndex.sln -c Release --no-restore — succeeded with 0 warnings and 0 errors.
• dotnet test tests/CodeIndex.Tests/CodeIndex.Tests.csproj -c Debug --no-restore --filter 'FullyQualifiedName~LspServer' — 120/120 passed on both net8.0 and net9.0 after the latest-main lifecycle integration.
• dotnet test CodeIndex.sln -c Release --no-build --no-restore — net8.0: 10,638 passed, 7 skipped, 1 existing load-sensitive failure; net9.0: 10,165 passed, 419 skipped, 1 existing load-sensitive failure. The two failures were the existing MCP batch-initialize regression test and the CI performance smoke while both full suites ran concurrently.
• Isolated Release rerun of those two existing tests — net8.0: 2/2 passed; net9.0: 1 passed and the performance test skipped by design.
• dotnet run --project tools/CodeIndex.Changelog --configuration Release --no-build -- check — validated 26 fragments.
• Post-commit dotnet build CodeIndex.sln -c Debug --no-restore — succeeded with 0 warnings and 0 errors.
• Post-commit local index freshness check — workspace matched HEAD with no dirty files.

Adversarial review

Two required review rounds were completed and all actionable findings were addressed:

• Preserve the indexed language for extension-sensitive live buffers such as C++ headers.
• Bound live extraction before materializing dense symbol sets.
• Preserve version tombstones across live-buffer eviction and oversized updates.
• Fall back to indexed symbols when the indexed extractor is unavailable.

Documentation and changelog

• Updated the English and Japanese sections of README.md, USER_GUIDE.md, and DEVELOPER_GUIDE.md.
• Added changelog.d/unreleased/4851.fixed.md.

Follow-up candidates

None.

Fixes #4851