perf(vindex): size native batches by active indexes

[jerry-024]

Purpose

global-index.thread-num controls index job fan-out and OSS range-read permits. It also reduced the native batch chunk size even when a query had only one active Vindex entry, making the chunk size depend on the configured maximum instead of actual index parallelism.

This PR decouples native batch sizing from range-read concurrency and enforces the 64 MiB native working-set budget process-wide through an admission-controlled memory pool, so larger per-query chunks cannot multiply memory across concurrent queries.

Brief change log

• Derive Vindex batch parallelism as min(planned index entries, global-index.thread-num), with a minimum of one, and use it to size native batch chunks in both the global-index batch and primary-key vector search paths.
• Add a process-wide 64 MiB native batch memory pool: each batch chunk reserves its actual working set (filter bytes + queries x per-query bytes) before running and releases it afterwards, so concurrent batch queries interleave per chunk instead of oversubscribing memory. Single-query chunks bypass the pool.
• Keep global-index.thread-num as the index job and range-read permit limit, including the existing 32-range cap per read batch.
• Uncontended chunk size and latency are unchanged; under concurrency, chunks from different queries interleave, which may delay individual requests while improving fairness and overall memory behavior.
• Add coverage for single-entry and multi-entry sizing, the 32-range read boundary, chunk reservations tracking the actual chunk size, and pool admission (a fitting reservation passes immediately; an oversized one waits until bytes are released).

Tests

• cargo +1.97.0 test -p paimon
• cargo +1.97.0 clippy -p paimon --lib --tests -- -D warnings
• cargo +1.97.0 fmt --all -- --check

API and Format

No public API or storage format changes.

Documentation

No documentation changes are required because this does not add or change a user-facing option.

[JingsongLi]

When a request to NativeBatchMemoryPool::acquire exceeds the fixed 64 MiB capacity, it waits indefinitely for permits that can never be obtained. This issue—which can be triggered by large filters or a single native batch—poses a risk of production deadlocks. It is recommended that requests exceeding the capacity either exclusively occupy the entire pool or fail immediately, rather than waiting indefinitely.

[JingsongLi]

+1