Replication compression configs and handshake negotiation - #16
Open
roshkhatri wants to merge 15 commits into
Open
Replication compression configs and handshake negotiation#16roshkhatri wants to merge 15 commits into
roshkhatri wants to merge 15 commits into
Conversation
Add streaming LZ4-backed RDB compression with rio decorators, stream envelope handling, integration changes, and the follow-up fixes and config cleanup needed on top of unstable. Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
- Remove dead code: rdbIsValidMagic() and unused #include <string.h> in rdb.h - Remove redundant first RIO_FLAG_SKIP_RDB_CHECKSUM set in rdbSaveInternal - Remove unrelated changes: config_parse_depth, USE_FAST_FLOAT, write-make-settings - Validate full 8-byte VKCS envelope in aof.c rdbFileUsesStreamingCompression - Add SAFETY comment for rdbRioHasCorruptCompressedInput cast invariant - Rename all snake_case identifiers to camelCase per Valkey conventions: types (compression_algo_t -> compressionAlgo, stream_compressor_t -> streamCompressor, compress_rio_t -> compressRio, etc.), functions (stream_writer_create -> streamWriterCreate, compress_rio_finish -> compressRioFinish, write_vkcs_envelope -> writeVkcsEnvelope, etc.), and static variables (compression_lz4_codec_impl -> compressionLz4CodecImpl) - Drop _t suffix from all types to match Valkey convention - Fix typo: streamWriterIsErrord -> streamWriterIsErrored - Replace silent dummy buffer allocation with assert(needed > 0) in streamWriterEnsureOutBuf Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
* Address streaming RDB compression review * Skip RDB CRC for streaming compression * Remove brittle 32-bit compression unit test --------- Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com> Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
--------- Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
- rdbSaveInternal: the old comment claimed per-string LZF was disabled 'when algo != LZF', but the actual gate is RIO_FLAG_STREAMING_COMPRESSION on the wrapper rio. Standalone rios (DUMP, AOF rewrite, diskless) keep using LZF regardless of algo. - rdbInputStreamPrepare: flag the synchronous probe IO so a future non-blocking caller (replication) doesn't accidentally block the loop. - rdbRioHasCorruptCompressedInput: the cast is sound today only because one producer sets RIO_FLAG_STREAMING_DECOMPRESSION. Replace the 'SAFETY' assertion with a note telling the next person to add a type discriminator before adding a second producer. - rdbSaveRawString: minor wording cleanup on the per-string LZF gate. Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Match the surrounding Valkey style: drop comments that restate the code, drop the Ownership/Threading/Returns boilerplate from headers, collapse repeated 'capacity-shortage is retriable' notes into one explanation per function. Behavior is unchanged. Net -285 lines; integration tests (21/21) and build are clean. Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
… config
Introduces REPLICA_CAPA_COMPRESSION handshake negotiation and a
replcompression boolean config. A replica with replcompression=yes
advertises capa compression during the PSYNC handshake when using
diskless load. The primary records REPLICA_CAPA_COMPRESSION on the
replica's capa bitmask but takes no action on it yet — compressed
replication transport lands in a follow-up.
Compression parameters (algorithm, level) are fixed internally via
REPL_COMPRESSION_ALGO (ALGO_LZ4) and REPL_COMPRESSION_LEVEL (5, HC
mode) and will become configurable in a later release.
- src/server.h: new REPLICA_CAPA_COMPRESSION (1<<4) flag, matching
_STR, repl_compression field, and REPL_COMPRESSION_{ALGO,LEVEL}
constants
- src/config.c: new replcompression bool config
- src/replication.c: primary records capa in replconfCommand; replica
advertises capa compression when server.repl_compression and
useDisklessLoad(); argv/lens arrays grown from 9 to 11 entries;
useDisklessLoad() extracted into a local reused by skip-rdb-checksum
and compression checks
- valkey.conf: new replcompression config documentation block
- src/unit/test_repl_compression.cpp: 3 GTest cases covering capa bit
uniqueness, capa string, and ALGO_LZ4 non-zero
- tests/integration/repl-compression.tcl: 9 TCL integration tests
covering config CRUD, CONFIG REWRITE persistence, and handshake
behavior
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
roshkhatri
force-pushed
the
repl-compression-handshake-v2
branch
from
May 22, 2026 19:38
c5ecfeb to
92db9be
Compare
roshkhatri
force-pushed
the
streaming-compression-rio-pr
branch
from
May 22, 2026 19:41
750a667 to
68b9d75
Compare
roshkhatri
force-pushed
the
streaming-compression-rio-pr
branch
from
August 3, 2026 16:06
3975e1a to
dfcc37a
Compare
roshkhatri
force-pushed
the
streaming-compression-rio-pr
branch
3 times, most recently
from
August 14, 2026 19:43
26d48a4 to
e45f2d6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
… config
Introduces REPLICA_CAPA_COMPRESSION handshake negotiation and a replcompression boolean config. A replica with replcompression=yes advertises capa compression during the PSYNC handshake when using diskless load. The primary records REPLICA_CAPA_COMPRESSION on the replica's capa bitmask but takes no action on it yet — compressed replication transport lands in a follow-up.
Compression parameters (algorithm, level) are fixed internally via REPL_COMPRESSION_ALGO (ALGO_LZ4) and REPL_COMPRESSION_LEVEL (5, HC mode) and will become configurable in a later release.