Fix indexing diagnostics, caps, and scan edge cases

[Widthdom]

Summary

• Add bounded indexing diagnostics for null bytes, regex timeouts, reference caps, generated-code suppression, project marker budgets, dangling-entry scans, and scan checkpoint schema checks.
• Reduce line/chunk materialization and reuse unchanged reference-preparation lines where safe.
• Add focused CLI/MCP regression coverage and changelog fragments for the handled issues.

Issues

Fixes #3835
Fixes #3785
Fixes #3763
Fixes #3762
Fixes #3720
Fixes #3719
Fixes #3800
Fixes #3712

Validation

• dotnet build
• dotnet test tests/CodeIndex.Tests/CodeIndex.Tests.csproj --framework net8.0 --no-build
• dotnet run --project tools/CodeIndex.Changelog -- check --no-build
• dotnet ./src/CodeIndex/bin/Debug/net8.0/cdidx.dll index . --json
• dotnet ./src/CodeIndex/bin/Debug/net8.0/cdidx.dll status --check --json

Review Notes

• No requested issue was skipped; issue threads only had this branch's work-start comments.
• No additional unrelated issue was opened.
• Codex adversarial review was run; its Add traversal budgets to solution and project-marker discovery #3762 warning-forwarding finding was fixed, and follow-up MCP coverage was added for Improve file scanning diagnostics for LFS, encoding, symlink, marker, and shebang edge cases #3835/Add a per-file reference extraction cap #3719.
• AGENTS.md, CLAUDE.md, and AGENT_GUIDE.md were not modified.