[TASK-431] Byte-sized cap instead of hard one for ArrowWriter#443
Merged
leekeiabstraction merged 5 commits intoapache:mainfrom Apr 13, 2026
Merged
[TASK-431] Byte-sized cap instead of hard one for ArrowWriter#443leekeiabstraction merged 5 commits intoapache:mainfrom
leekeiabstraction merged 5 commits intoapache:mainfrom
Conversation
Contributor
Author
Contributor
Author
Contributor
charlesdong1991
left a comment
charlesdong1991
left a comment
There was a problem hiding this comment.
Very nice PR! leave some minor comments/questions
Contributor
Author
|
@charlesdong1991 Ty for the review, addressed comments |
Contributor
Author
|
@leekeiabstraction TY for the review, answered your comment |
Contributor
Author
|
@luoyuxia @leekeiabstraction @charlesdong1991 PTAL 🙏 |
leekeiabstraction
approved these changes
Apr 13, 2026
Contributor
leekeiabstraction
left a comment
leekeiabstraction
left a comment
There was a problem hiding this comment.
LGTM, approved. Note there's rebase conflict that needs to be addressed before merge.
fresh-borzoni
force-pushed
the
byte-sized-cap
branch
from
dc7f07b to
71d95d3
Compare
Contributor
Author
|
@leekeiabstraction Ty, rebased |
leekeiabstraction
approved these changes
Apr 13, 2026
Contributor
leekeiabstraction
left a comment
leekeiabstraction
left a comment
There was a problem hiding this comment.
LGTM, made some small changes to remove unused functions to allow build to pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
closes #431
Arrow log batches were previously capped at a hard 256-record limit regardless of actual data size, meaning a batch of 256 tiny ints (~1KB) was treated the same as 256 large rows (~10MB). This replaces that fixed cap with byte-size-based fullness matching Java's ArrowWriter, so batches now fill to the configured writer_batch_size (default 2MB).
The fullness check uses a threshold-based optimization to avoid computing sizes on every append, it estimates how many records should fit, skips checks until that count is reached, then recalculates. Size estimation reads buffer lengths directly from Arrow builders (O(num_columns), zero allocation), with a pre-computed IPC overhead constant that captures metadata and body framing for the schema.
Compression is accounted for via an adaptive ArrowCompressionRatioEstimator shared across batches for the same table. It starts at 1.0 (assume no compression) and adjusts asymmetrically after each batch is serialized, quick to increase when compression worsens, slow to decrease when it improves. This matches Java's conservative approach to avoid underestimating batch sizes.
Also aligns VARIABLE_WIDTH_AVG_BYTES (64 -> 8) and INITIAL_ROW_CAPACITY (256 -> 1024) with Java Arrow defaults.
Writer pooling (ArrowWriterPool) and DynamicWriteBatchSizeEstimator are out of scope -> TODOs added.