GH-50216: [C++][Parquet] Add RleBitPackedToBitmapDecoder by AntoinePrv · Pull Request #50217 · apache/arrow

AntoinePrv · 2026-06-18T09:25:44Z

Rationale for this change

Add a RleBitPackedToBitmapDecoder capable of decoding a Parquet mixed Rle / BitPacked byte stream directly into a bitmap. This is an optimization to be used when target and source bit_width = 1 that will improve decoding nullable columns in a follow-up PR.

What changes are included in this PR?

New classes, that reuse previous Runs and parser.
Their API is slighly different from the existing one as it fits a partilcular case.

RleRunToBitmapDecoder
BitPackedRunToBitmapDecoder
RleRunToBitmapDecoder

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

GitHub Issue: [C++][Parquet] Rle-Bitpacked to Bitmap decoder #50216

github-actions · 2026-06-18T09:30:29Z

⚠️ GitHub issue #50216 has been automatically assigned in GitHub to PR creator.

Copilot

Pull request overview

Adds a specialized path for decoding Parquet mixed RLE / BitPacked streams with bit_width=1 directly into Arrow bitmaps, aiming to speed up validity/definition-level decoding.

Changes:

Added bitmap-oriented decoders for RleRun and BitPackedRun, plus RleBitPackedToBitmapDecoder for mixed streams.
Refactored RleBitPackedParser to provide ParseWithCallable() and adjusted RleBitPackedDecoder to use it.
Added new bit utilities (Get32Bits, CopyBits) and comprehensive unit tests for the new bitmap decoders.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
cpp/src/arrow/util/rle_encoding_internal.h	Adds `RleRun::value()`, moves callable-based parsing into the parser, and updates `RleBitPackedDecoder` accordingly.
cpp/src/arrow/util/rle_bitmap_internal.h	Introduces bitmap span + run-to-bitmap decoders and the new `RleBitPackedToBitmapDecoder`.
cpp/src/arrow/util/rle_bitmap_test.cc	Adds tests covering alignment, chunking, reset, and truncated input scenarios for bitmap decoding.
cpp/src/arrow/util/CMakeLists.txt	Registers the new test file in the util test target.
cpp/src/arrow/util/bit_util.h	Adds `Get32Bits` and `CopyBits` helpers used by the new bitmap decoder implementation.
cpp/src/arrow/util/bit_util_test.cc	Adds unit tests for `Get32Bits` and `CopyBits`.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

+  void Reset(const RunType& run) noexcept {
+    values_left_ = run.values_count();
+    if (run.value_little_endian() == 0) {
+      value_pattern_ = uint8_t{0};
+    } else {
+      value_pattern_ = uint8_t{0xFF};
+    }
+  }


AntoinePrv · 2026-06-18T12:26:10Z

@pitrou this is ready

pitrou · 2026-06-18T13:28:09Z

                                      rle_size_t batch_size) -> rle_size_t {
  using ControlFlow = RleBitPackedParser::ControlFlow;

+  if (ARROW_PREDICT_FALSE(batch_size == 0 || exhausted())) {


Is there a particular reason this became necessary?

Came from the Copilot review, which I found to be relevant. I may not be a possible path in practice in all of Arrow, but better safe than sorry.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

pitrou

Thanks for the update @AntoinePrv !

pitrou · 2026-06-22T14:32:21Z

+/// Make a vector of `size` pseudo-random bytes, deterministic for a given `seed`.
+std::vector<uint8_t> MakeRandomBytes(size_t size, uint32_t seed = 56) {
+  std::vector<uint8_t> bytes(size);
+  std::minstd_rand gen(seed);


Hmm, the default_random_engine might give better randomness, though not sure it's useful here.

I picked this one because it is small and cheap and I assume sufficient here.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

+  void Reset(const RunType& run) noexcept {
+    values_left_ = run.values_count();
+    if (run.value_little_endian() == 0) {
+      value_pattern_ = uint8_t{0};
+    } else {
+      value_pattern_ = uint8_t{0xFF};
+    }
+  }


AntoinePrv · 2026-06-23T12:18:19Z

@pitrou this is ready

pitrou · 2026-06-23T13:32:17Z

@AntoinePrv Note there is a compilation error on MSVC.

pitrou · 2026-06-23T13:41:44Z

@github-actions crossbow submit -g cpp

pitrou

+1, but need to fix the MSVC-based CI build.

github-actions · 2026-06-23T13:44:59Z

Revision: 6fc9785

Submitted crossbow builds: ursacomputing/crossbow @ actions-fb93408670

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

pitrou · 2026-06-23T14:37:51Z

Other related failures:

this compilation error on Ubuntu 22.04
these Valgrind errors: I think you just have to implement a PrintTo for the new test parameter type(s) such that GTest doesn't read uninitialized memory

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

AntoinePrv requested a review from pitrou as a code owner June 18, 2026 09:25

Copilot AI review requested due to automatic review settings June 18, 2026 09:25

Copilot started reviewing on behalf of AntoinePrv June 18, 2026 09:28 View session

github-actions Bot added the awaiting review Awaiting review label Jun 18, 2026

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/rle_encoding_internal.h Outdated

Comment thread cpp/src/arrow/util/rle_bitmap_internal.h

Comment thread cpp/src/arrow/util/bit_util.h Outdated

Comment thread cpp/src/arrow/util/rle_encoding_internal.h

github-actions Bot added awaiting committer review Awaiting committer review Component: C++ and removed awaiting review Awaiting review labels Jun 18, 2026

Copilot AI review requested due to automatic review settings June 18, 2026 12:03

Copilot started reviewing on behalf of AntoinePrv June 18, 2026 12:04 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/rle_bitmap_internal.h Outdated

Comment on lines +169 to +176

void Reset(const RunType& run) noexcept {

values_left_ = run.values_count();

if (run.value_little_endian() == 0) {

value_pattern_ = uint8_t{0};

} else {

value_pattern_ = uint8_t{0xFF};

}

}

pitrou reviewed Jun 18, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 22, 2026 09:22

AntoinePrv force-pushed the rle-bitmap branch from 07f2371 to 086af88 Compare June 22, 2026 09:22

Copilot started reviewing on behalf of AntoinePrv June 22, 2026 09:23 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/rle_bitmap_internal.h

pitrou reviewed Jun 22, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/bit_util.h Outdated

pitrou reviewed Jun 22, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 23, 2026 11:57

Copilot started reviewing on behalf of AntoinePrv June 23, 2026 11:58 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/rle_bitmap_internal.h

Comment on lines +83 to +90

void Reset(const RunType& run) noexcept {

values_left_ = run.values_count();

if (run.value_little_endian() == 0) {

value_pattern_ = uint8_t{0};

} else {

value_pattern_ = uint8_t{0xFF};

}

}

AntoinePrv added 7 commits June 23, 2026 14:17

Add CopyBits

ef5dbbc

Add Get32Bits

109723b

Add RleRun::value

c4ee2f9

Add Rle bitmap

aa97150

Simplify parser decoder maping

00ef6d6

Add BitPackedRunToBitMapDecoder

cf5ac4e

Extra check

c0692c1

AntoinePrv added 13 commits June 23, 2026 14:17

Extra doc

fb0d18b

Comment style

347c2ec

Test layout improvements

0008ac8

Uniform BitMap > Bitmap

163aa10

Fix class specialization

d26dba0

Explicit little endian name

363c709

Safe decoder on zero batch_size

116fc5b

Aply some review comment

b5d742d

Use CopyBitmap

ab16793

Improve tests

ed641b6

Strong clobber test

88ba25e

Fix assertion

41fc6c7

Adjust for review

6fc9785

AntoinePrv force-pushed the rle-bitmap branch from c80e478 to 6fc9785 Compare June 23, 2026 12:18

pitrou approved these changes Jun 23, 2026

View reviewed changes

Fix uniform int distribution type

441408d

Copilot AI review requested due to automatic review settings June 23, 2026 14:34

Copilot started reviewing on behalf of AntoinePrv June 23, 2026 14:35 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/rle_bitmap_internal.h

Comment thread cpp/src/arrow/util/rle_bitmap_internal.h

Comment thread cpp/src/arrow/util/CMakeLists.txt

Conversation

AntoinePrv commented Jun 18, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

AntoinePrv commented Jun 18, 2026

Uh oh!

pitrou Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

AntoinePrv Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

AntoinePrv Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

AntoinePrv commented Jun 23, 2026

Uh oh!

pitrou commented Jun 23, 2026

Uh oh!

pitrou commented Jun 23, 2026

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

pitrou commented Jun 23, 2026

Uh oh!

AntoinePrv commented Jun 18, 2026 •

edited by github-actions Bot

Loading