Replication streaming compression (per-replica) - review vs #3531 base by roshkhatri · Pull Request #21 · roshkhatri/valkey

roshkhatri · 2026-07-24T19:06:05Z

Review-only PR: isolates the per-replica replication-stream compression work, squashed to one commit on sarthak's latest valkey-io#3531 base (dfcc37a, final).

Base: streaming-compression-rio-pr @ dfcc37a (Streaming Compression support for RDB valkey-io/valkey#3531 final)
Head: replication-streaming-compression-v2 (single commit: Streaming Compression support for replication stream)

Supersedes #19 (old base lineage). Does not update the upstream PR.

roshkhatri · 2026-07-30T03:03:09Z

+    # Re-runs the replication suite with repl-compression=lz4-stream applied
+    # globally via --config. Test files opt in by carrying the top-level "repl-compression"
+    # tag (see tests/support/server.tcl), so this job selects them with
+    # --tags repl-compression instead of a hardcoded filename list that would
+    # silently drift if a file is renamed. Exercises the streaming-compression
+    # transport across the replication surface (full sync, dual-channel, buffer
+    # management, AOF-sync, etc.) to catch regressions that only surface under
+    # compressed replication.
+    # Note: a few tests are intentionally NOT tagged for this job:
+    #  - integration/replication-psync: its start_server lives inside a reusable
+    #    proc, incompatible with a top-level-only tag.
+    #  - integration/replication-buffer.tcl and the buffer-memory tests in
+    #    integration/dual-channel-replication.tcl: they assert exact replication
+    #    buffer/backlog memory and byte volumes, which compression legitimately
+    #    changes (per-replica codec buffers, fewer wire bytes). They run in the
+    #    regular (uncompressed) job.


This can be shortned

roshkhatri · 2026-07-30T03:14:18Z

    return 0;
 }

+/* Sink path: compress directly into the caller's sds tail, no scratch/emit. */


remove , no scratch/emit.

roshkhatri · 2026-07-30T03:48:16Z


+/* Set by the repl-compression apply callback when the setting changes. Switching
+ * a live link between plaintext and compressed requires a reconnect, which is
+ * irreversible, so the reconcile is deferred until after the whole CONFIG SET


shorten this

roshkhatri · 2026-07-30T03:49:55Z

+    /* Switching existing replicas between plaintext and compressed requires a
+     * reconnect, which is irreversible. Only record intent here; configSetCommand
+     * runs the reconcile after the command commits, so a rolled-back CONFIG SET
+     * disconnects nothing. */


shorten this

roshkhatri · 2026-07-30T03:50:20Z

+    /* Primary client with a live replication decoder (probing or compressed)
+     * must be decoded on the main thread; the IO-thread read path does not
+     * invoke replDecompressQueryBuf. Once the probe resolves to plaintext the
+     * decoder is destroyed and reads return here. */


shorten this

roshkhatri · 2026-07-30T03:50:28Z

+        /* Check for unsent compressed data in the compressed output buffer.
+         * Skip while CLIENT_PENDING_IO: the IO thread owns out_buf and may
+         * realloc it, so the main thread must not read it here
+         * (postWriteToReplica re-checks once the write job completes). */


shorten this

roshkhatri · 2026-07-30T03:52:43Z

+ * malicious or buggy primary; disconnect rather than allocate unbounded. */
+#define REPL_STREAM_DECODER_OUTPUT_MAX (256 * 1024 * 1024)
+
+int replDecompressQueryBuf(client *c, size_t new_data_start) {


Does this belong to some replication file?

roshkhatri · 2026-07-30T03:53:39Z

+
+/* Reconcile each replica's transport with the current repl-compression setting
+ * after a runtime change. A live link cannot switch between plaintext and a
+ * compressed VCS/LZ4 frame mid-stream, so a mismatched replica is dropped and
+ * reconnects to renegotiate:
+ *   - compression disabled: drop replicas still on a compressed stream;
+ *   - compression enabled:  drop online replicas that advertised the capability
+ *     but are still plaintext, so they come back compressed.
+ * Disconnect is async because we run inside the main event loop; the iteration
+ * is safe since freeClientAsync does not remove from server.replicas
+ * synchronously. Replicas still syncing are left alone: they pick up the current


shorten this

roshkhatri · 2026-07-30T03:53:58Z

+    /* If we are ourselves a replica, the compression capability was negotiated
+     * with our primary at handshake time. A config change must renegotiate, so
+     * drop the upstream link; the replication cron reconnects and re-advertises
+     * capa with the new setting. */


shorten this

Vendor lz4, lz4hc, lz4frame and xxhash from https://github.com/lz4/lz4 (v1.10.0) under deps/lz4 with build wiring for make and cmake. Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

Add Valkey Compressed Stream (VCS) support for RDB persistence with lz4 as the first whole-stream codec. The RDB is wrapped in a VCS envelope and compressed as a single stream at the rio layer. On load, the RDB path probes for the envelope and transparently decompresses when present, falling back to the plain RDB path otherwise. Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

Apply the outstanding review changes and retain the null-safe RDB stream-writer check required by encoded-length calculation. Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

Signed-off-by: Roshan Khatri <roshanvkhatri@gmail.com>

github-actions Bot assigned roshkhatri Jul 24, 2026

roshkhatri force-pushed the replication-streaming-compression-v2 branch 4 times, most recently from 2f851c3 to 3688b40 Compare July 29, 2026 23:07

roshkhatri commented Jul 30, 2026

View reviewed changes

roshkhatri force-pushed the replication-streaming-compression-v2 branch 3 times, most recently from 5d2009b to 5442bed Compare July 30, 2026 08:24

roshkhatri mentioned this pull request Jul 30, 2026

Full-sync streaming compression - review vs replication base #22

Open

roshkhatri force-pushed the streaming-compression-rio-pr branch from 3975e1a to dfcc37a Compare August 3, 2026 16:06

roshkhatri force-pushed the replication-streaming-compression-v2 branch 6 times, most recently from 88ad2f0 to caf1b3d Compare August 4, 2026 02:13

roshkhatri mentioned this pull request Aug 4, 2026

Full-sync streaming compression - standalone on #3531 base #23

Open

roshkhatri force-pushed the replication-streaming-compression-v2 branch 2 times, most recently from 70ab3ba to 8ac3117 Compare August 5, 2026 07:58

sarthakaggarwal97 added 8 commits August 11, 2026 08:44

Vendor lz4 1.10.0

5ad6c9b

Vendor lz4, lz4hc, lz4frame and xxhash from https://github.com/lz4/lz4 (v1.10.0) under deps/lz4 with build wiring for make and cmake. Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

Address streaming compression review feedback

3ff8965

Apply the outstanding review changes and retain the null-safe RDB stream-writer check required by encoded-length calculation. Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

Address follow-up streaming compression review

0cb4e31

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

Address streaming RDB review follow-ups

33ef3f2

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

Remove ineffective RDB checker assertion

6e32606

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

Address latest streaming compression review feedback

c36240a

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

Add explicit LZF RDB compression mode

62a5899

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

roshkhatri force-pushed the streaming-compression-rio-pr branch from dfcc37a to 62a5899 Compare August 11, 2026 23:12

roshkhatri force-pushed the replication-streaming-compression-v2 branch from 8ac3117 to 2886b6f Compare August 13, 2026 18:27

roshkhatri force-pushed the streaming-compression-rio-pr branch from 62a5899 to 26d48a4 Compare August 13, 2026 18:28

roshkhatri force-pushed the replication-streaming-compression-v2 branch from 2886b6f to d8c14e4 Compare August 13, 2026 19:21

Streaming Compression support for replication stream

a41c3e7

Signed-off-by: Roshan Khatri <roshanvkhatri@gmail.com>

roshkhatri force-pushed the replication-streaming-compression-v2 branch from d8c14e4 to a41c3e7 Compare August 13, 2026 20:21

roshkhatri force-pushed the streaming-compression-rio-pr branch from 26d48a4 to e45f2d6 Compare August 14, 2026 19:43

Navigation Menu

Replication streaming compression (per-replica) - review vs #3531 base - #21

Replication streaming compression (per-replica) - review vs #3531 base#21
roshkhatri wants to merge 9 commits into
streaming-compression-rio-prfrom
replication-streaming-compression-v2

roshkhatri commented Jul 24, 2026 •

edited

Loading

Uh oh!

roshkhatri Jul 30, 2026

Uh oh!

roshkhatri Jul 30, 2026

Uh oh!

roshkhatri Jul 30, 2026

Uh oh!

roshkhatri Jul 30, 2026

Uh oh!

roshkhatri Jul 30, 2026

Uh oh!

roshkhatri Jul 30, 2026

Uh oh!

roshkhatri Jul 30, 2026

Uh oh!

roshkhatri Jul 30, 2026

Uh oh!

roshkhatri Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

roshkhatri commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

roshkhatri commented Jul 24, 2026 •

edited

Loading