Full-sync streaming compression - standalone on #3531 base - #23
Open
roshkhatri wants to merge 1 commit into
Open
Full-sync streaming compression - standalone on #3531 base#23roshkhatri wants to merge 1 commit into
roshkhatri wants to merge 1 commit into
Conversation
roshkhatri
force-pushed
the
repl-streaming-compression-fullsync-v3
branch
15 times, most recently
from
August 11, 2026 22:22
aaafee1 to
20d0431
Compare
roshkhatri
force-pushed
the
streaming-compression-rio-pr
branch
from
August 11, 2026 23:12
dfcc37a to
62a5899
Compare
roshkhatri
force-pushed
the
repl-streaming-compression-fullsync-v3
branch
5 times, most recently
from
August 13, 2026 16:54
8a5389c to
c274d2d
Compare
roshkhatri
commented
Aug 13, 2026
| retval = rdbSaveToReplicasSockets(req, rdbver, rsiptr); | ||
| else { | ||
| int save_flags = RDBFLAGS_REPLICATION | RDBFLAGS_KEEP_CACHE; | ||
| /* The round is compressed when mincapa carries the compress-sync bit |
Owner
Author
There was a problem hiding this comment.
simplify this and reduce the compress the comment in a gist
roshkhatri
commented
Aug 13, 2026
| * Disk-based sync snapshots can also become AOF bases, which currently do | ||
| * not record whether the reused RDB has whole-stream compression. */ | ||
| if (rdbflags & RDBFLAGS_REPLICATION) use_streaming_compression = false; | ||
| /* Replication full sync compresses only when the cohort negotiated it |
Owner
Author
There was a problem hiding this comment.
reduce and compress the comment to a gist
roshkhatri
force-pushed
the
streaming-compression-rio-pr
branch
from
August 13, 2026 18:28
62a5899 to
26d48a4
Compare
roshkhatri
force-pushed
the
repl-streaming-compression-fullsync-v3
branch
from
August 13, 2026 18:28
c274d2d to
fb3c3cc
Compare
roshkhatri
force-pushed
the
streaming-compression-rio-pr
branch
from
August 14, 2026 19:43
26d48a4 to
e45f2d6
Compare
roshkhatri
force-pushed
the
repl-streaming-compression-fullsync-v3
branch
3 times, most recently
from
August 14, 2026 21:39
ce8f008 to
8212f1d
Compare
roshkhatri
force-pushed
the
repl-streaming-compression-fullsync-v3
branch
2 times, most recently
from
August 14, 2026 22:36
40c6617 to
30f5934
Compare
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
roshkhatri
force-pushed
the
repl-streaming-compression-fullsync-v3
branch
from
August 14, 2026 23:51
30f5934 to
67ed2b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full-sync RDB payload compression (disk-based, diskless, dual-channel) as one commit directly on sarthak's streaming-compression-rio-pr (valkey-io#3531) tip dfcc37a, independent of the incremental replication PR.
Adds the pieces fullsync needs: REPLCONF capa compression negotiation, the repl-compression config (no | lz4, modifiable, applies to subsequent syncs), the per-handshake provisional latch, and the rio conn partial-read machinery (rioConnFillBuffer/rioConnReadSome) deferred from valkey-io#3531 review.
Compression rule: the primary compresses a full sync only when every replica in the attaching cohort advertised the capability; if any attaching replica is not capable, the payload is sent plaintext to all of them. $EOF framing stays plaintext; the frame checksum replaces the RDB CRC64; a link dropped mid-frame is recoverable truncation (resync), not corruption.
Verified: build zero warnings, unit 329/329, fullsync suite 16/16, repl-compression matrix green, replication 68/68, aof-sync 7/7, clang-format clean.
Siblings: PR #22 (same feature stacked on the incremental base), PR #21 (incremental), upstream valkey-io#4075.