Skip to content

Validate stream listpack live and deleted counts on load - #27

Draft
roshkhatri wants to merge 9 commits into
unstablefrom
restore-count-validation
Draft

Validate stream listpack live and deleted counts on load#27
roshkhatri wants to merge 9 commits into
unstablefrom
restore-count-validation

Conversation

@roshkhatri

@roshkhatri roshkhatri commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Problem

streamValidateListpackIntegrity() checked only that live + deleted matched the number of records it walked, never the split between them. A payload declaring live=1, deleted=1 for two records that are both live validates cleanly, because the traversal consumes two records either way and lands exactly on the terminator.

XDEL then trusts the cached live count to decide when a node is exhausted:

aux = lpGetInteger(p);          /* the DECLARED live count */
if (aux == 1) {
    lpFree(lp);                                  /* frees the whole node */
    raxRemove(si->stream->rax, si->ri.key, si->ri.key_len, NULL);
}
...
si->stream->length--;           /* decremented once, for one entry */

Deleting one entry frees the node holding the other, and s->length comes from a separate RDB field that is only decremented once, leaving the key permanently self-contradictory:

RESTORE      +OK
XLEN         2
XDEL 1-0     1
XLEN         1          <- claims one entry
XRANGE - +   (empty)    <- both are gone

That state propagates to replicas, the AOF and every subsequent RDB save. streamTrim() has the same exposure from the other direction: lpReplaceInteger(lp, &p, entries - deleted_from_lp) writes a negative live count when the header understates. Reachable by anything that supplies a payload: RESTORE, RDB load, replication.

This PR

Counts the STREAM_ITEM_FLAG_DELETED flag while validating and rejects the payload if either counter disagrees with the records actually present. Also tracks entries consumed so the declared field and record counts are bounded against what the listpack holds.

Validation stays entirely at ingestion, per valkey-io#3721. No format or protocol change, and valid data behaves exactly as before.

Testing

Five gtest cases against the validator. Three fail on the unpatched validator: TestRejectsUnderstatedLiveCount, TestRejectsOverstatedLiveCount, TestRejectsAllRecordsDeletedWithLiveHeader. The two Accepts* cases are controls against over-rejection.

One integration test in corrupt-dump.tcl built from a valid two-entry control by changing the primary entry count from 2 to 1 and its deleted count from 0 to 1, then recomputing the CRC64. Verified both ways: base accepts it and reproduces the data loss, this rejects it. corrupt-dump, stream, stream-cgroups: 230 passed.

Scope

The second commit reduces this PR per review feedback. The access-time detection layer is reverted: the tri-state stream iterator, the listpack random-helper return codes and the per-entry re-validation. valkey-io#3721 made dump payload validation unconditional and already rejects a listpack whose header count disagrees with its contents, so those shapes never reach the read path, and re-validating each entry during iteration tripled the cost of stream reads.

Net effect is 14 files / +597−124 down to 3 files / +147−6. Gone with it: the XRANGE and XREVRANGE partial-return behavior, the EIO contract change on VM_StreamIteratorNextID(), and every new corruption panic. The tests for the reverted behavior are dropped too; the stream field count case was already covered upstream, and the listpack header count case belongs with the check it guards.

@roshkhatri
roshkhatri force-pushed the restore-count-validation branch 2 times, most recently from ce522d9 to c9a822f Compare August 11, 2026 00:24
jsoref and others added 4 commits August 11, 2026 16:03
- Subset of valkey-io#2183

---------

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Include wrappers.h directly so OBJ_STRING and the object helper declarations are available when the custom matchers are parsed.

Signed-off-by: harrylin98 <harrylin980107@gmail.com>
Fix various typos related to setup and set up.

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Fix various typos around otherwise with a comma

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
@roshkhatri roshkhatri changed the title Harden RESTORE count validation Validate stream listpack live and deleted counts on load Aug 12, 2026
nitaicaro and others added 5 commits August 12, 2026 10:18
…lkey-io#4380)

Fixes paths which used a client's cached slot id for keys not directly associated with the client.
---------

Signed-off-by: Nitai Caro <caronita@amazon.com>
Co-authored-by: Nitai Caro <caronita@amazon.com>
…io#4323)

In an earlier commit we didn't properly reset the redaction bitmap
during exec, and used the wrong one for lua scripts. This fixes that to
properly redact commands. Added new regression tests and all existing
tests still pass.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
…d Tcl tests (valkey-io#2243)

Fix spelling and grammar issues across 17 files

This is a subset of valkey-io#2183.

---------

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>
…-io#3922)

## Problem

Two crafted-`RESTORE` crashes in stream loading. In both, the payload
passes the existing structural validation but violates an invariant
downstream code relies on. **Any client with `RESTORE` access can
remotely crash the server.**

### 1. Length vs. tombstones
A stream can claim a positive `length` while every listpack entry is a
tombstone (`STREAM_ITEM_FLAG_DELETED`). The length is loaded directly
from the payload and only checked against the rax being non-empty.
`streamLastValidID()` then finds no non-tombstone entry while
`s->length` is non-zero and aborts:
```
serverPanic("Corrupt stream, length is %llu, but no max id", ...)   // t_stream.c
```
Triggered by `XSETID` / `XADD` / `XREADGROUP`. Confirmed: a 2-entry
stream with both entries flagged `DELETED` and `length=1` loads OK, then
`XSETID` panics.

### 2. Negative field counts
A master entry (or a per-entry field count for non-`SAMEFIELDS` entries)
can declare a **negative** number of fields. The validator only checked
`lpGetIntegerIfValid()`'s success flag, not the sign. The negative count
drives listpack traversal in `streamIteratorGetID()`, walking past the
listpack and asserting (`lpAssertValidEntry`) on `XRANGE` and similar
reads. Confirmed: crafted payload loads OK, then `XRANGE` aborts at
`listpack.c`.

## Fix

1. `streamValidateListpackIntegrity()` already parses each listpack's
master entry count (live entries). Sum it across listpacks via a new
out-parameter and reject the payload if it does not match the loaded
length. This reuses the assertion-safe parsing rather than iterating the
stream with `streamIteratorGetID()`, which can itself hit entry-level
assertions on *other* malformed payloads (an earlier iterate-based
version regressed three existing corrupt-dump tests).
2. Reject negative `primary_fields` and per-entry `fields` counts during
validation.

## Testing

- Two `RESTORE`-path integration tests in
`tests/integration/corrupt-dump.tcl`.
- Both verified to **fail pre-fix** (panic / assert) and **pass
post-fix**.
- Confirmed legitimate streams — including ones with real tombstones (5
entries, 2 deleted) and multi-field entries — still load and read
correctly.
- Full `integration/corrupt-dump` suite: 75 passed, 0 failed (including
the three stream consumer-group tests an earlier iterate-based approach
broke).

> [!NOTE]
> Found via structure-aware fuzzing + code review of the RESTORE path.
This issue was generated by AI but verified, with love, by a human.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Roshan Khatri <roshanvkhatri@gmail.com>
@roshkhatri
roshkhatri force-pushed the restore-count-validation branch from 5e36707 to c671ead Compare August 12, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants