Validate stream listpack live and deleted counts on load - #27
Draft
roshkhatri wants to merge 9 commits into
Draft
Conversation
roshkhatri
force-pushed
the
restore-count-validation
branch
2 times, most recently
from
August 11, 2026 00:24
ce522d9 to
c9a822f
Compare
- Subset of valkey-io#2183 --------- Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech> Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Include wrappers.h directly so OBJ_STRING and the object helper declarations are available when the custom matchers are parsed. Signed-off-by: harrylin98 <harrylin980107@gmail.com>
Fix various typos related to setup and set up. Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Fix various typos around otherwise with a comma Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
…lkey-io#4380) Fixes paths which used a client's cached slot id for keys not directly associated with the client. --------- Signed-off-by: Nitai Caro <caronita@amazon.com> Co-authored-by: Nitai Caro <caronita@amazon.com>
…io#4323) In an earlier commit we didn't properly reset the redaction bitmap during exec, and used the wrong one for lua scripts. This fixes that to properly redact commands. Added new regression tests and all existing tests still pass. --------- Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
…d Tcl tests (valkey-io#2243) Fix spelling and grammar issues across 17 files This is a subset of valkey-io#2183. --------- Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com> Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>
…-io#3922) ## Problem Two crafted-`RESTORE` crashes in stream loading. In both, the payload passes the existing structural validation but violates an invariant downstream code relies on. **Any client with `RESTORE` access can remotely crash the server.** ### 1. Length vs. tombstones A stream can claim a positive `length` while every listpack entry is a tombstone (`STREAM_ITEM_FLAG_DELETED`). The length is loaded directly from the payload and only checked against the rax being non-empty. `streamLastValidID()` then finds no non-tombstone entry while `s->length` is non-zero and aborts: ``` serverPanic("Corrupt stream, length is %llu, but no max id", ...) // t_stream.c ``` Triggered by `XSETID` / `XADD` / `XREADGROUP`. Confirmed: a 2-entry stream with both entries flagged `DELETED` and `length=1` loads OK, then `XSETID` panics. ### 2. Negative field counts A master entry (or a per-entry field count for non-`SAMEFIELDS` entries) can declare a **negative** number of fields. The validator only checked `lpGetIntegerIfValid()`'s success flag, not the sign. The negative count drives listpack traversal in `streamIteratorGetID()`, walking past the listpack and asserting (`lpAssertValidEntry`) on `XRANGE` and similar reads. Confirmed: crafted payload loads OK, then `XRANGE` aborts at `listpack.c`. ## Fix 1. `streamValidateListpackIntegrity()` already parses each listpack's master entry count (live entries). Sum it across listpacks via a new out-parameter and reject the payload if it does not match the loaded length. This reuses the assertion-safe parsing rather than iterating the stream with `streamIteratorGetID()`, which can itself hit entry-level assertions on *other* malformed payloads (an earlier iterate-based version regressed three existing corrupt-dump tests). 2. Reject negative `primary_fields` and per-entry `fields` counts during validation. ## Testing - Two `RESTORE`-path integration tests in `tests/integration/corrupt-dump.tcl`. - Both verified to **fail pre-fix** (panic / assert) and **pass post-fix**. - Confirmed legitimate streams — including ones with real tombstones (5 entries, 2 deleted) and multi-field entries — still load and read correctly. - Full `integration/corrupt-dump` suite: 75 passed, 0 failed (including the three stream consumer-group tests an earlier iterate-based approach broke). > [!NOTE] > Found via structure-aware fuzzing + code review of the RESTORE path. This issue was generated by AI but verified, with love, by a human. --------- Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Roshan Khatri <roshanvkhatri@gmail.com>
roshkhatri
force-pushed
the
restore-count-validation
branch
from
August 12, 2026 22:18
5e36707 to
c671ead
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
streamValidateListpackIntegrity()checked only thatlive + deletedmatched the number of records it walked, never the split between them. A payload declaringlive=1, deleted=1for two records that are both live validates cleanly, because the traversal consumes two records either way and lands exactly on the terminator.XDELthen trusts the cached live count to decide when a node is exhausted:Deleting one entry frees the node holding the other, and
s->lengthcomes from a separate RDB field that is only decremented once, leaving the key permanently self-contradictory:That state propagates to replicas, the AOF and every subsequent RDB save.
streamTrim()has the same exposure from the other direction:lpReplaceInteger(lp, &p, entries - deleted_from_lp)writes a negative live count when the header understates. Reachable by anything that supplies a payload:RESTORE, RDB load, replication.This PR
Counts the
STREAM_ITEM_FLAG_DELETEDflag while validating and rejects the payload if either counter disagrees with the records actually present. Also tracks entries consumed so the declared field and record counts are bounded against what the listpack holds.Validation stays entirely at ingestion, per valkey-io#3721. No format or protocol change, and valid data behaves exactly as before.
Testing
Five gtest cases against the validator. Three fail on the unpatched validator:
TestRejectsUnderstatedLiveCount,TestRejectsOverstatedLiveCount,TestRejectsAllRecordsDeletedWithLiveHeader. The twoAccepts*cases are controls against over-rejection.One integration test in
corrupt-dump.tclbuilt from a valid two-entry control by changing the primary entry count from 2 to 1 and its deleted count from 0 to 1, then recomputing the CRC64. Verified both ways: base accepts it and reproduces the data loss, this rejects it.corrupt-dump,stream,stream-cgroups: 230 passed.Scope
The second commit reduces this PR per review feedback. The access-time detection layer is reverted: the tri-state stream iterator, the listpack random-helper return codes and the per-entry re-validation. valkey-io#3721 made dump payload validation unconditional and already rejects a listpack whose header count disagrees with its contents, so those shapes never reach the read path, and re-validating each entry during iteration tripled the cost of stream reads.
Net effect is 14 files / +597−124 down to 3 files / +147−6. Gone with it: the
XRANGEandXREVRANGEpartial-return behavior, theEIOcontract change onVM_StreamIteratorNextID(), and every new corruption panic. The tests for the reverted behavior are dropped too; the stream field count case was already covered upstream, and the listpack header count case belongs with the check it guards.