[ISSUE #10936] Avoid temporary buffers during batch message encoding

[ai-yang]

Which Issue(s) This PR Fixes

• Fixes [Enhancement] Avoid temporary full-message buffers during batch encoding #10936
• Related to fix and optimize methods for sending and storing batch message #2931, [ISSUE #2931 ]fix and optimize methods for sending and storing batch message #2932, [Enhancement] Optimize message properties encode/decode to reduce allocation #10442, [ISSUE #10442] Optimize message properties encode/decode to reduce allocation #10444, and [ISSUE #9195] Propagate MessageBatch user properties to inner messages #10588

Brief Description

MessageDecoder.encodeMessages currently encodes every message into its own complete byte[], retains those arrays, then allocates a final result and copies every message again.

This change removes those temporary full-message buffers:

• the first pass caches each body reference and the serialized property bytes while computing the exact aggregate size;
• one final ByteBuffer is allocated;
• the second pass writes every message directly into that buffer;
• single-message and batch encoding share internal size and field-writing helpers;
• a byte-for-byte compatibility test covers an empty body, Unicode properties, a 4 KiB body, edge flag values and an empty list;
• an independent fixed golden vector verifies the wire field layout without relying on the shared helper as its oracle.

The public API and encoded wire format are unchanged.

How Did You Test This Change?

Correctness

The new tests compare encodeMessages(messages) with the concatenation of encodeMessage(message) for every input and verify an independent fixed wire vector. The complete common module suite passes:

Tests run: 245, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

Command:

mvn -o \
  -Dspotbugs-plugin.version=4.2.2 -Dspotbugs.skip=true \
  -Dcheckstyle.skip=true -Drat.skip=true \
  -pl common test

Relevant downstream tests also pass:

AppendCallbackTest + BatchPutMessageTest + AppendPropCRCTest:
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0

ProduceAccumulatorTest:
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

JMH benchmark

JMH 1.36 configuration:

• throughput mode with GC profiler;
• one thread pinned to CPU 5;
• 3 forks;
• 5 warm-up and 7 measurement iterations per fork, 500 ms each;
• fixed -Xms512m -Xmx512m heap;
• OpenJDK 11.0.31, Ubuntu 22.04.4, Intel Xeon Gold 6133.

The benchmark's baseline() method reconstructs the previous aggregation algorithm in the same binary (per-message full encodings followed by a final copy), so the table labels it as a reconstructed baseline.

Scenario	Reconstructed baseline ops/s	This PR ops/s	Change	Baseline B/op	This PR B/op	Change
10 x 128 B	340,743 +/- 6,348	381,842 +/- 9,236	+12.06%	6,760	4,552	-32.66%
100 x 128 B	32,247 +/- 1,099	36,246 +/- 242	+12.40%	67,152	45,192	-32.70%
10 x 1 KiB	179,223 +/- 3,518	240,718 +/- 10,326	+34.31%	24,680	13,512	-45.25%
100 x 1 KiB	16,954 +/- 594	22,411 +/- 594	+32.19%	246,352	134,792	-45.28%

Values are means plus or minus JMH's 99.9% confidence-interval half-width. The baseline and optimized throughput confidence intervals do not overlap in any scenario.

The 100 x 1 KiB case was repeated on OpenJDK 8u492:

Throughput: 14,472.346 +/- 329.526 -> 18,954.609 +/- 113.257 ops/s (+30.97%)
Allocation: 288,568 -> 173,832 B/op (-39.76%)

Its throughput confidence intervals are also disjoint.

I also benchmarked the actual production method from separate baseline and optimized checkouts in A-B-A order for 100 x 1 KiB:

Production checkout	Throughput ops/s	99.9% CI	Allocation B/op
Baseline A1	17,352.526 +/- 400.418	[16,952.108, 17,752.943]	246,352.044
This PR	22,600.219 +/- 880.650	[21,719.569, 23,480.870]	134,792.035
Baseline A2	16,876.707 +/- 671.661	[16,205.046, 17,548.368]	246,352.045

This is +30.24%/+33.91% throughput versus the two baseline runs and -45.28% allocation. The confidence intervals are disjoint.

Benchmark source, methodology, checksums, and all raw per-fork/per-iteration JSON results: https://gist.github.com/ai-yang/c3e3f65351054263dec6f7b8d67ad3cd

git diff --check also passes.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.49%. Comparing base (293f588) to head (d66b4f8).

Additional details and impacted files

@@              Coverage Diff              @@
##             develop   #10938      +/-   ##
=============================================
- Coverage      48.62%   48.49%   -0.13%     
+ Complexity     13692    13652      -40     
=============================================
  Files           1381     1381              
  Lines         101464   101472       +8     
  Branches       13187    13187              
=============================================
- Hits           49337    49210     -127     
- Misses         46142    46250     +108     
- Partials        5985     6012      +27     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[RockteMQ-AI]

Summary

Eliminates intermediate byte[] allocations in MessageDecoder.encodeMessages by computing the aggregate size first, then writing all messages directly into a single ByteBuffer. The public API and wire format are unchanged. JMH benchmarks show 12–34% throughput gain and 33–45% allocation reduction.

Findings

• [Info] common/src/main/java/org/apache/rocketmq/common/message/MessageDecoder.java:738–762 — The two-pass approach (size computation + direct write) is clean. One minor note: propertiesLength is computed in both encodedMessageSize and the internal encodeMessage helper. Consider caching it in the propertiesBytes[] array alongside bodies to avoid the redundant cast, though the overhead is negligible.
• [Info] common/src/main/java/org/apache/rocketmq/common/message/MessageDecoder.java:738 — The first pass calls message.getBody() and caches the reference. If a caller mutates the body array between the two passes, the size computed in pass 1 could mismatch the data written in pass 2. This is the same implicit assumption the old code made, so it's not a regression — just worth noting for future maintainers.

Verdict

Solid optimization with good test coverage (golden vector + equivalence test). The refactored internal helpers (encodedMessageSize, internal encodeMessage) are well-factored and the public API remains untouched.

---

Automated review by github-manager-bot