Skip to content

perf(vindex): split build timing logs by phase - #723

Draft
jerry-024 wants to merge 1 commit into
apache:mainfrom
jerry-024:perf/vindex-build-timing
Draft

perf(vindex): split build timing logs by phase#723
jerry-024 wants to merge 1 commit into
apache:mainfrom
jerry-024:perf/vindex-build-timing

Conversation

@jerry-024

@jerry-024 jerry-024 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Purpose

Vector index build diagnostics currently do not distinguish storage I/O, Parquet decoding, index training, and index population costs. This change adds opt-in phase-level timing logs so build bottlenecks can be identified without adding log noise when diagnostics are not enabled.

Enable the logs with:

PAIMON_LOG_VECTOR_INDEX_BUILD_TIMING=1

Brief change log

  • Split source batch wait, OSS/file reads, and Parquet decoding into independent timings.
  • Split raw temporary-file writes/rereads, index training, and index add into independent timings.
  • Report serialization/upload, snapshot commit, total, unattributed time, and build counters in one structured log event.
  • Keep timing collection and logging disabled by default.

Tests

  • cargo test -p paimon
  • Run vector index build tests with timing disabled and enabled; verify logs are emitted only when enabled and phase fields are populated.
  • cargo fmt --all -- --check
  • git diff --check

API and Format

No public API or storage format changes.

Documentation

No documentation changes. The environment variable is intended for opt-in build diagnostics.

@jerry-024
jerry-024 marked this pull request as draft August 17, 2026 10:42
@jerry-024 jerry-024 changed the title perf(vindex): add detailed build timing logs perf(vindex): split build timing logs by phase Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant