issue #587: cap input segments per vector-index compaction cycle

[eolivelli]

Fixes #587.

A vector-index compaction cycle that selected 53 input segments stalled the Indexing Service for 2+ hours. The downstream PQ-retraining step (jvector PQRetrainer.extractVectorsSequential) samples training vectors per input segment with random remote-storage reads, so its I/O cost scales with the number of input segments. VectorIndexCompactor.chooseSegmentsToMerge bounded the picked set only by a byte cap, which never bites when many segments are individually small — leaving the input count effectively unbounded.

This PR is the HerdDB-side mitigation. The complementary jvector-side fix (parallelizing the per-source extraction so the per-read latency is hidden) is in eolivelli/jvector#12. The two changes are independent — HerdDB CI builds jvector from eolivelli/jvector main and is unaffected by the jvector PR's merge state — but full latency relief needs both.

Changes

• VectorIndexCompactor — new 7-arg chooseSegmentsToMerge overload with a maxInputs parameter (6-arg overload delegates with the cap disabled). After the fire/no-fire trigger decision, the normal byte-capped selection is truncated smallest-first to at most maxInputs segments, with an INFO log when truncation happens. The micro-segment fast path ([IS compaction] Prioritize merging micro-segments to relieve back-pressure faster #570) is deliberately exempt — those cycles must stay fast slot-reclaiming merges and the PQ-retraining-I/O concern does not apply to them. Added clampMaxInputs (<=0 disables, 1→2) and computeTieredMaxInputs.
• PersistentVectorStore — DEFAULT_VECTOR_INDEX_COMPACTION_MAX_INPUTS = 16, a vectorIndexCompactionMaxInputs field, setCompactionMaxInputs/getCompactionMaxInputs. The base cap is tier-scaled (2×/4×/8× at 100/300/500 segments) per cycle alongside the byte/count caps, so the per-cycle drain rate rises with the backlog and the cap cannot starve the tailer toward the back-pressure threshold. The cycle still fires on the same triggers and merges leftover segments in subsequent cycles.
• IndexingServerConfiguration / IndexingServiceEngine — new vector.index.compaction.maxInputs config key (default 16), wired into the store and the startup config log.

Tests

• VectorIndexCompactorChooseTest — new cases: a 53-segment pick is truncated to the 16 smallest in order; maxInputs=0 disables the cap; the cap never changes the fire/no-fire trigger decision; a picked set within the cap is returned untruncated; the micro-segment fast-path result is not capped; clampMaxInputs normalisation.
• Issue587CompactionInputCapTest — new end-to-end test: builds a 50-segment backlog with the cap enabled at its default, drives multiple compaction cycles, and asserts every cycle merges at most the cap and the segment count strictly converges (no starvation).
• Issue354TieredCompactionTest — new computeTieredMaxInputs unit tests (scaling, overflow, disabled-cap); the two end-to-end tiered tests disable the orthogonal input cap so their "drain the whole backlog in one cycle" premise still holds.
• Pre-PR validation green: spotless:check apache-rat:check install -DskipTests spotbugs:check -Pci (the exact CI gate).
• Hammer suite green (twice): DirectMultipleConcurrentUpdatesSuite{NoIndexes,WithNonUniqueIndexes,WithUniqueIndexes}Test, BLinkConcurrentSearchInsertTest.

🤖 Generated with Claude Code