Detect UTF-16 shebang BOMs

[Widthdom]

Summary

• Detect BOMs before decoding extensionless shebang probes.
• Support UTF-8, UTF-8 BOM, UTF-16 LE, and UTF-16 BE shebang scripts without treating UTF-16 NUL bytes as binary content.
• Add regression coverage for all supported shebang encodings.

Validation

• dotnet build CodeIndex.sln -c Release -p:UseSharedCompilation=false
• dotnet test tests/CodeIndex.Tests/CodeIndex.Tests.csproj -c Release --filter FileIndexerTests -p:UseSharedCompilation=false
• dotnet test CodeIndex.sln -c Release -p:UseSharedCompilation=false
• dotnet ./src/CodeIndex/bin/Debug/net8.0/cdidx.dll status --check --json
• Codex adversarial review: No blocking/actionable issues found.

Documentation and Changelog

• Added changelog fragment: changelog.d/unreleased/1830.fixed.md
• No README or guide update needed; this preserves existing shebang behavior and fixes an encoding-specific detection gap.

Fixes #1830