Skip to content

Replace hardcoded reserve(8) with dataset-size-aware reservation in calibration streaming #8

Description

@MoonFlowww

Motivation

Vector reallocation during the calibration streaming loop is a latency spike that grows with dataset size. The current hard-coded reserve(8) assumes at most 8 batches — correct for toy datasets, wrong for any real evaluation run. On a 100k-sample dataset with batch size 32 that is ~3125 batches and ~11 reallocations, each doubling the allocation and copying all accumulated logit chunks.

Current State

src/core.hpp:2258–2261:

std::vector<torch::Tensor> logits_chunks;
std::vector<torch::Tensor> target_chunks;
logits_chunks.reserve(8); // TODO: Modify
target_chunks.reserve(8);

Both evaluation_inputs.size(0) (total samples) and streaming_options.batch_size are available at this point in the function, so the correct capacity is computable for free.

Proposed Change

const auto n_samples       = evaluation_inputs.size(0);
const auto batch_sz        = static_cast<int64_t>(streaming_options.batch_size);
const auto estimated_batches = (n_samples + batch_sz - 1) / batch_sz;

std::vector<torch::Tensor> logits_chunks;
std::vector<torch::Tensor> target_chunks;
logits_chunks.reserve(static_cast<std::size_t>(estimated_batches));
target_chunks.reserve(static_cast<std::size_t>(estimated_batches));

No behavioural change — purely an allocation optimisation.

Acceptance Criteria

  • reserve() size derived from actual input size and batch size
  • Zero reallocations during a standard streaming evaluation loop
  • // TODO: Modify comment removed
  • No change to outputs, metrics, or evaluation semantics

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions