Motivation
Vector reallocation during the calibration streaming loop is a latency spike that grows with dataset size. The current hard-coded reserve(8) assumes at most 8 batches — correct for toy datasets, wrong for any real evaluation run. On a 100k-sample dataset with batch size 32 that is ~3125 batches and ~11 reallocations, each doubling the allocation and copying all accumulated logit chunks.
Current State
src/core.hpp:2258–2261:
std::vector<torch::Tensor> logits_chunks;
std::vector<torch::Tensor> target_chunks;
logits_chunks.reserve(8); // TODO: Modify
target_chunks.reserve(8);
Both evaluation_inputs.size(0) (total samples) and streaming_options.batch_size are available at this point in the function, so the correct capacity is computable for free.
Proposed Change
const auto n_samples = evaluation_inputs.size(0);
const auto batch_sz = static_cast<int64_t>(streaming_options.batch_size);
const auto estimated_batches = (n_samples + batch_sz - 1) / batch_sz;
std::vector<torch::Tensor> logits_chunks;
std::vector<torch::Tensor> target_chunks;
logits_chunks.reserve(static_cast<std::size_t>(estimated_batches));
target_chunks.reserve(static_cast<std::size_t>(estimated_batches));
No behavioural change — purely an allocation optimisation.
Acceptance Criteria
Motivation
Vector reallocation during the calibration streaming loop is a latency spike that grows with dataset size. The current hard-coded
reserve(8)assumes at most 8 batches — correct for toy datasets, wrong for any real evaluation run. On a 100k-sample dataset with batch size 32 that is ~3125 batches and ~11 reallocations, each doubling the allocation and copying all accumulated logit chunks.Current State
src/core.hpp:2258–2261:Both
evaluation_inputs.size(0)(total samples) andstreaming_options.batch_sizeare available at this point in the function, so the correct capacity is computable for free.Proposed Change
No behavioural change — purely an allocation optimisation.
Acceptance Criteria
reserve()size derived from actual input size and batch size// TODO: Modifycomment removed