feat: add ActivePartitionBatchRing.GetKeysByPartition()#943
Merged
Conversation
Signed-off-by: Marco Pracucci <marco@pracucci.com>
pracucci
changed the title
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
4 tasks
pracucci
added a commit
to grafana/mimir
that referenced
this pull request
Apr 2, 2026
…l to produce Kafka records (#14898) #### What this PR does In this PR I'm introducing an optimization to reduce distributor CPU utilization when writing to ingest storage with a large number of partitions (order of thousands) by batching all partitions into a single `ProduceSync()` call instead of one per partition. Previously, `DoBatchWithOptions` spawned a goroutine per partition, each making its own `ProduceSync()` call. This was the dominant CPU cost (~35% cumulative in profiling with 3.5K partitions). Now, `sendWriteRequestToPartitions()` uses `ActivePartitionBatchRing.GetKeysByPartition()` to group keys by partition, builds a single `[]PartitionWriteRequest`, and calls `MultiWriteSync()` which sends all records in one `ProduceSync()` call. To do this change, I've updated the vendored dskit to get `ActivePartitionBatchRing.GetKeysByPartition()` ([dskit PR](grafana/dskit#943)). There are a couple of small behavioural changes: - `cortex_ingest_storage_writer_latency_seconds` now tracks the latency to write to all partitions in a single call, instead of per-partition latency. - Success metrics (`sent_bytes_total`, `input_bytes_total`, latency) are only tracked when all records succeed, to keep them consistent on partial failures. When testing this optimization in a Mimir test cluster with 3.5K Kafka partitions, the distributors CPU was reduced by -33%: <img width="720" height="471" alt="screenshot_2026-03-27_at_16 16 59_720" src="https://github.com/user-attachments/assets/041afbdb-08a0-4355-b6c3-40f93535aecc" /> #### Which issue(s) this PR fixes or relates to N/A #### Checklist - [x] Tests updated. - [ ] Documentation added. - [x] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features. --------- Signed-off-by: Marco Pracucci <marco@pracucci.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
In this PR I'm adding
ActivePartitionBatchRing.GetKeysByPartition(). This will be used to optimise the Mimir distributors when writing to Kafka partitions.ActivePartitionBatchRing.GetKeysByPartition()is a simplified alternative to `DoBatchWithOptions() for partition rings where replication factor and quorum is always 1.ActivePartitionBatchRing.GetKeysByPartition()Uses a two-pass approach with a single backing array for all index slices, achieving only 5 allocations regardless of partition count:Which issue(s) this PR fixes:
N/A
Checklist