Skip to content

wiki: re-ground the data tiering wiki on the current tree - #19

Merged
xdk-amz merged 1 commit into
unstablefrom
wiki/reground-citations
Jul 30, 2026
Merged

wiki: re-ground the data tiering wiki on the current tree#19
xdk-amz merged 1 commit into
unstablefrom
wiki/reground-citations

Conversation

@xdk-amz

@xdk-amz xdk-amz commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Brings the wiki back in line with the code it documents.

before after
verify_citations errors 42 0
verify_citations warnings 121 49
keg_lint 0 err / 0 warn / 0 orphans unchanged
typed-edge graph 203 edges 215 edges

Citations

Four pages were grounded on a layout that no longer exists. Every citation is re-pointed and
every line number re-read from the current file:

Cited Actual
modules/non-key-spilling/ modules/flash-tiering/
modules/storage_example/ src/storage/storage_mock.c
modules/storage_flashcache_module/ src/storage/storage_flashcache_real.c
tests/unit/ext-storage*.tcl tests/unit/data-tiering/

The interface reference no longer lists a getter (storageGetRocksDBType) that exists nowhere in
the tree — there is no engine-native RocksDB backend, only an alias returning the in-memory mock.
The test page now reflects 19 files / 225 test blocks, every count derived from the tcl
sources, and records that ext-storage.tcl never runs in practice for want of a module path.

Two headers named storage.h exist — the compiled src/storage/storage.h and an older
uncompiled src/storage.h that nothing includes — so this interface is cited by full path
throughout and the hazard is documented.

Persistence

Tiered values are persisted. rdbSaveKeyValuePair materializes them into a standard RDB
entry, so RDB files, the AOF-preamble base (the default) and disk-target replica full sync all
carry tiered data; the fork-snapshot protocol (settle to no COPYING_* key → park the backend IO
thread → pause GC → fork → resume → done) is what makes that consistent. Surviving gaps: the
non-preamble AOF rewrite, cluster slot migration, and tiering placement never being replicated.
With a backend that cannot snapshot, the save paths fail closed rather than write a lossy
snapshot (L8).

Flagged, not demonstrated. rdbSaveToReplicasSockets is the default replication path
(repl-diskless-sync defaults to yes) and forks and materializes tiered values in the child,
yet it is the only fork/save entry point that never calls extStorageSnapshotPrepare() and it
carries no fail-closed guard. No test covers a replica full sync with tiered values. Needs a
maintainer decision on whether the omission is deliberate.

Feature reconciliation

Six tiering states, not five. TIERING_STATE_PENDING_DELETION = 5 is documented in the state
table, transitions, blocking matrix and the rendered diagram, and distinguished from
PENDING_EVICT.

SWAPDB logical/physical db-id indirection, with the complete translating-site inventory —
including the one deliberate non-translating call (the READ_RETRY resubmit, where the id is
already physical). Getting the direction wrong does not fault; it silently addresses another
database's keyspace.

Mid-execution synchronous fetch as a second fetch mode alongside block-and-retry: its two
triggers (LOOKUP_SYNCFETCH, and implicitly at execution_nesting > 1), its drain-ordering rule,
and that it has no timeout — a wedged backend blocks the main thread inside command execution.

Runtime maxmemory-policy guard restricting the policy to allkeys-lru, allkeys-lfu or
noeviction while tiering is active.

Contradictions and limitations

known-limitations is restructured: 6 contradictions, 8 functional limits, 4 resolved. Nothing
was deleted — entries that no longer hold moved to Resolved with the citations proving it.

Both spill controllers are present and selectable via ext-storage-spilling-strategy, so
items_spillover_batch_size and the SPILL_CONCURRENT_BASE cap are strategy-gated rather
than dead or deleted.

Three source-comment defects are recorded, because the wiki had believed them:

  • the two aof.c tiered-skip warnings name each other's path (C8);
  • two comments assert extStorageUpdateSpillConcurrency was removed while it is defined and
    called (C9) — this is what made the wiki record C7 and R2 as deletions that never happened;
  • ext_storage.h still says "one of 5 states" and still refers to the removed
    keys_tiering_state hashtable.

extStorageEvictFlashKey has no caller in the tree and is noted as dead code where described.

Tooling

verify_citations computes staleness from each cited file's last git commit date rather than
filesystem mtime, which in a fresh clone or after a checkout is the checkout time for every file.

The diagrams Makefile pins FONTCONFIG_PATH. Without it graphviz can silently fall back to a
symbol font, producing a diagram with correct structure whose every label is dingbats.

Remaining

49 warnings: 13 of the soft "symbol not in cited sources" class, and 36 STALE on pages this branch
does not touch (concentrated in adr-index, memory-accounting, 01-architecture). No page's
updated: date was advanced except where its content was re-verified against the tree.

Note that verify_citations was green while several pages asserted the opposite of what the code
does — it proves citations point at real lines, not that the prose matches them.


Engine findings filed separately — these are code defects the wiki can only describe:

Both are cross-referenced from the pages that describe them.

@xdk-amz
xdk-amz force-pushed the wiki/reground-citations branch from 6f4f57b to 8134bfe Compare July 30, 2026 22:05
@xdk-amz xdk-amz changed the title wiki: re-ground citations on the current tree (42 errors → 0) wiki: re-ground the data tiering wiki on the current tree Jul 30, 2026
Brings the wiki back in line with the code it documents. `verify_citations` goes
from 42 errors to 0; warnings from 121 to 49. `keg_lint` stays at 0 errors, 0
warnings, 0 orphans; the typed-edge graph grows 203 -> 215 edges.

## Citations

Four pages were grounded on a layout that no longer exists. Every citation is
re-pointed and every line number re-read from the current file:

  modules/non-key-spilling/          -> modules/flash-tiering/
  modules/storage_example/           -> src/storage/storage_mock.c
  modules/storage_flashcache_module/ -> src/storage/storage_flashcache_real.c
  tests/unit/ext-storage*.tcl        -> tests/unit/data-tiering/

The interface reference no longer lists a getter (storageGetRocksDBType) that
exists nowhere in the tree: there is no engine-native RocksDB backend, only an
alias returning the in-memory mock. The test page now reflects 19 files / 225
test blocks, each count derived from the tcl sources, and records that
ext-storage.tcl never runs in practice for want of a module path.

Two headers named storage.h exist -- the compiled src/storage/storage.h and an
older uncompiled src/storage.h that nothing includes -- so this interface is
cited by full path throughout, and the hazard is documented.

## Persistence

Tiered values are persisted. rdbSaveKeyValuePair materializes them into a
standard RDB entry, so RDB files, the AOF-preamble base (the default) and
disk-target replica full sync all carry tiered data; the fork-snapshot protocol
(settle to no COPYING_* key, park the backend IO thread, pause GC, fork, resume,
done) is what makes that consistent. The surviving gaps are the non-preamble AOF
rewrite, cluster slot migration, and tiering placement never being replicated.
With a backend that cannot snapshot, the save paths fail closed rather than write
a lossy snapshot (L8).

Flagged, not demonstrated: rdbSaveToReplicasSockets is the default replication
path (repl-diskless-sync defaults to yes) and forks and materializes tiered
values in the child, yet it is the only fork/save entry point that never calls
extStorageSnapshotPrepare() and carries no fail-closed guard. No test covers a
replica full sync with tiered values. Needs a maintainer decision.

## Feature reconciliation

Six tiering states, not five: TIERING_STATE_PENDING_DELETION = 5 is documented in
the state table, transitions, blocking matrix and the rendered diagram, and
distinguished from PENDING_EVICT.

The SWAPDB logical/physical db-id indirection is documented with its complete
translating-site inventory, including the one deliberate non-translating call
(the READ_RETRY resubmit, already physical). Getting the direction wrong does not
fault -- it silently addresses another database's keyspace.

Mid-execution synchronous fetch is documented as a second fetch mode alongside
block-and-retry: its two triggers (LOOKUP_SYNCFETCH, and implicitly at
execution_nesting > 1), its drain-ordering rule, and that it has no timeout -- a
wedged backend blocks the main thread inside command execution.

The runtime maxmemory-policy guard restricts the policy to allkeys-lru,
allkeys-lfu or noeviction while tiering is active.

## Contradictions and limitations

known-limitations is restructured: 6 contradictions, 8 functional limits, 4
resolved. Nothing was deleted; entries that no longer hold moved to Resolved with
the citations proving it.

Both spill controllers are present and selectable via
ext-storage-spilling-strategy, so items_spillover_batch_size and the
SPILL_CONCURRENT_BASE cap are strategy-gated rather than dead or deleted.

Three source-comment defects are recorded because the wiki had believed them:
the two aof.c tiered-skip warnings name each other's path (C8); two comments
assert extStorageUpdateSpillConcurrency was removed while it is defined and
called (C9); and ext_storage.h still says "one of 5 states" and still refers to
the removed keys_tiering_state hashtable. extStorageEvictFlashKey has no caller in
the tree and is noted as dead code where it is described.

## Tooling

verify_citations computes staleness from each cited file's last git commit date
rather than filesystem mtime, which in a fresh clone or after a checkout is the
checkout time for every file.

The diagrams Makefile pins FONTCONFIG_PATH. Without it graphviz can silently fall
back to a symbol font, producing a diagram with correct structure whose every
label is dingbats.

Remaining 49 warnings: 13 of the soft "symbol not in cited sources" class, and 36
STALE on pages this branch does not touch. No page's updated: date was advanced
except where its content was re-verified against the tree.

## Tracking

The two engine-side findings are filed so they are not carried only by the wiki:

  #20  diskless replication forks and materializes tiered values without the
       snapshot protocol (flagged, not reproduced)
  #21  extStorageEvictFlashKey is dead code -- no caller in the tree

Both are referenced from the pages that describe them.
@xdk-amz
xdk-amz force-pushed the wiki/reground-citations branch from 8134bfe to 4c293a6 Compare July 30, 2026 22:38
@xdk-amz
xdk-amz merged commit 44f9bd4 into unstable Jul 30, 2026
40 of 82 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant