Skip an allocation in our hot path#1250
Open
sgrif wants to merge 1 commit into
Open
Conversation
Profiling showed this allocation was as much as .5% of our wall time. This function unfortunately does need to allocate, as one of the strategies involves randomizing the elements and another involves sorting them. Neither of those are possible without allocation. We also can't do any funky shit like re-using a thread local allocation, as `candidates` does need to be held past an await point. That leaves us with stack allocation as our only remaining option. This will always take up 256 bytes on the stack, and only heap allocate when we have more elements than that. The largest deployment that we know of will have 14 targets here, so this is potentially twice as large as it needs to be. But we can start by giving ourselves some wiggle room and we can always shrink this down later. The cost here is that the size of the future being returned by this function will be 222 bytes larger, so copying the future itself becomes a more expensive operation. In practice this should happen exactly once, as it gets moved to the heap when spawned, so it should be a net improvement over a random heap allocation. But profiling should be done to confirm.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
levkk
reviewed
Jul 21, 2026
| RoundRobin => { | ||
| let first = self.round_robin.fetch_add(1, Ordering::Relaxed) % candidates.len(); | ||
| let mut reshuffled = vec![]; | ||
| let mut reshuffled = SmallVec::with_capacity(candidates.len()); |
Collaborator
There was a problem hiding this comment.
Literally had to look up the code to understand that it infers the size of the new vec from the type of the original. This is legit a cool lib and a really cool use of the type system.
Contributor
Author
There was a problem hiding this comment.
Yeah, it's wild that they managed to make this work before const generics (2.0 uses const generics but it's still alpha)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Profiling showed this allocation was as much as .5% of our wall time. This function unfortunately does need to allocate, as one of the strategies involves randomizing the elements and another involves sorting them. Neither of those are possible without allocation.
We also can't do any funky shit like re-using a thread local allocation, as
candidatesdoes need to be held past an await point.That leaves us with stack allocation as our only remaining option. This will always take up 256 bytes on the stack, and only heap allocate when we have more elements than that. The largest deployment that we know of will have 14 targets here, so this is potentially twice as large as it needs to be. But we can start by giving ourselves some wiggle room and we can always shrink this down later.
The cost here is that the size of the future being returned by this function will be 222 bytes larger, so copying the future itself becomes a more expensive operation. In practice this should happen exactly once, as it gets moved to the heap when spawned, so it should be a net improvement over a random heap allocation. But profiling should be done to confirm.