wiki: re-ground the data tiering wiki on the current tree - #19
Merged
Conversation
xdk-amz
force-pushed
the
wiki/reground-citations
branch
from
July 30, 2026 22:05
6f4f57b to
8134bfe
Compare
Brings the wiki back in line with the code it documents. `verify_citations` goes from 42 errors to 0; warnings from 121 to 49. `keg_lint` stays at 0 errors, 0 warnings, 0 orphans; the typed-edge graph grows 203 -> 215 edges. ## Citations Four pages were grounded on a layout that no longer exists. Every citation is re-pointed and every line number re-read from the current file: modules/non-key-spilling/ -> modules/flash-tiering/ modules/storage_example/ -> src/storage/storage_mock.c modules/storage_flashcache_module/ -> src/storage/storage_flashcache_real.c tests/unit/ext-storage*.tcl -> tests/unit/data-tiering/ The interface reference no longer lists a getter (storageGetRocksDBType) that exists nowhere in the tree: there is no engine-native RocksDB backend, only an alias returning the in-memory mock. The test page now reflects 19 files / 225 test blocks, each count derived from the tcl sources, and records that ext-storage.tcl never runs in practice for want of a module path. Two headers named storage.h exist -- the compiled src/storage/storage.h and an older uncompiled src/storage.h that nothing includes -- so this interface is cited by full path throughout, and the hazard is documented. ## Persistence Tiered values are persisted. rdbSaveKeyValuePair materializes them into a standard RDB entry, so RDB files, the AOF-preamble base (the default) and disk-target replica full sync all carry tiered data; the fork-snapshot protocol (settle to no COPYING_* key, park the backend IO thread, pause GC, fork, resume, done) is what makes that consistent. The surviving gaps are the non-preamble AOF rewrite, cluster slot migration, and tiering placement never being replicated. With a backend that cannot snapshot, the save paths fail closed rather than write a lossy snapshot (L8). Flagged, not demonstrated: rdbSaveToReplicasSockets is the default replication path (repl-diskless-sync defaults to yes) and forks and materializes tiered values in the child, yet it is the only fork/save entry point that never calls extStorageSnapshotPrepare() and carries no fail-closed guard. No test covers a replica full sync with tiered values. Needs a maintainer decision. ## Feature reconciliation Six tiering states, not five: TIERING_STATE_PENDING_DELETION = 5 is documented in the state table, transitions, blocking matrix and the rendered diagram, and distinguished from PENDING_EVICT. The SWAPDB logical/physical db-id indirection is documented with its complete translating-site inventory, including the one deliberate non-translating call (the READ_RETRY resubmit, already physical). Getting the direction wrong does not fault -- it silently addresses another database's keyspace. Mid-execution synchronous fetch is documented as a second fetch mode alongside block-and-retry: its two triggers (LOOKUP_SYNCFETCH, and implicitly at execution_nesting > 1), its drain-ordering rule, and that it has no timeout -- a wedged backend blocks the main thread inside command execution. The runtime maxmemory-policy guard restricts the policy to allkeys-lru, allkeys-lfu or noeviction while tiering is active. ## Contradictions and limitations known-limitations is restructured: 6 contradictions, 8 functional limits, 4 resolved. Nothing was deleted; entries that no longer hold moved to Resolved with the citations proving it. Both spill controllers are present and selectable via ext-storage-spilling-strategy, so items_spillover_batch_size and the SPILL_CONCURRENT_BASE cap are strategy-gated rather than dead or deleted. Three source-comment defects are recorded because the wiki had believed them: the two aof.c tiered-skip warnings name each other's path (C8); two comments assert extStorageUpdateSpillConcurrency was removed while it is defined and called (C9); and ext_storage.h still says "one of 5 states" and still refers to the removed keys_tiering_state hashtable. extStorageEvictFlashKey has no caller in the tree and is noted as dead code where it is described. ## Tooling verify_citations computes staleness from each cited file's last git commit date rather than filesystem mtime, which in a fresh clone or after a checkout is the checkout time for every file. The diagrams Makefile pins FONTCONFIG_PATH. Without it graphviz can silently fall back to a symbol font, producing a diagram with correct structure whose every label is dingbats. Remaining 49 warnings: 13 of the soft "symbol not in cited sources" class, and 36 STALE on pages this branch does not touch. No page's updated: date was advanced except where its content was re-verified against the tree. ## Tracking The two engine-side findings are filed so they are not carried only by the wiki: #20 diskless replication forks and materializes tiered values without the snapshot protocol (flagged, not reproduced) #21 extStorageEvictFlashKey is dead code -- no caller in the tree Both are referenced from the pages that describe them.
xdk-amz
force-pushed
the
wiki/reground-citations
branch
from
July 30, 2026 22:38
8134bfe to
4c293a6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the wiki back in line with the code it documents.
verify_citationserrorsverify_citationswarningskeg_lintCitations
Four pages were grounded on a layout that no longer exists. Every citation is re-pointed and
every line number re-read from the current file:
modules/non-key-spilling/modules/flash-tiering/modules/storage_example/src/storage/storage_mock.cmodules/storage_flashcache_module/src/storage/storage_flashcache_real.ctests/unit/ext-storage*.tcltests/unit/data-tiering/The interface reference no longer lists a getter (
storageGetRocksDBType) that exists nowhere inthe tree — there is no engine-native RocksDB backend, only an alias returning the in-memory mock.
The test page now reflects 19 files / 225 test blocks, every count derived from the tcl
sources, and records that
ext-storage.tclnever runs in practice for want of a module path.Two headers named
storage.hexist — the compiledsrc/storage/storage.hand an olderuncompiled
src/storage.hthat nothing includes — so this interface is cited by full paththroughout and the hazard is documented.
Persistence
Tiered values are persisted.
rdbSaveKeyValuePairmaterializes them into a standard RDBentry, so RDB files, the AOF-preamble base (the default) and disk-target replica full sync all
carry tiered data; the fork-snapshot protocol (settle to no
COPYING_*key → park the backend IOthread → pause GC → fork → resume → done) is what makes that consistent. Surviving gaps: the
non-preamble AOF rewrite, cluster slot migration, and tiering placement never being replicated.
With a backend that cannot snapshot, the save paths fail closed rather than write a lossy
snapshot (L8).
Feature reconciliation
Six tiering states, not five.
TIERING_STATE_PENDING_DELETION = 5is documented in the statetable, transitions, blocking matrix and the rendered diagram, and distinguished from
PENDING_EVICT.SWAPDB logical/physical db-id indirection, with the complete translating-site inventory —
including the one deliberate non-translating call (the
READ_RETRYresubmit, where the id isalready physical). Getting the direction wrong does not fault; it silently addresses another
database's keyspace.
Mid-execution synchronous fetch as a second fetch mode alongside block-and-retry: its two
triggers (
LOOKUP_SYNCFETCH, and implicitly atexecution_nesting > 1), its drain-ordering rule,and that it has no timeout — a wedged backend blocks the main thread inside command execution.
Runtime
maxmemory-policyguard restricting the policy toallkeys-lru,allkeys-lfuornoevictionwhile tiering is active.Contradictions and limitations
known-limitationsis restructured: 6 contradictions, 8 functional limits, 4 resolved. Nothingwas deleted — entries that no longer hold moved to Resolved with the citations proving it.
Both spill controllers are present and selectable via
ext-storage-spilling-strategy, soitems_spillover_batch_sizeand theSPILL_CONCURRENT_BASEcap are strategy-gated ratherthan dead or deleted.
Three source-comment defects are recorded, because the wiki had believed them:
aof.ctiered-skip warnings name each other's path (C8);extStorageUpdateSpillConcurrencywas removed while it is defined andcalled (C9) — this is what made the wiki record C7 and R2 as deletions that never happened;
ext_storage.hstill says "one of 5 states" and still refers to the removedkeys_tiering_statehashtable.extStorageEvictFlashKeyhas no caller in the tree and is noted as dead code where described.Tooling
verify_citationscomputes staleness from each cited file's last git commit date rather thanfilesystem mtime, which in a fresh clone or after a checkout is the checkout time for every file.
The diagrams Makefile pins
FONTCONFIG_PATH. Without it graphviz can silently fall back to asymbol font, producing a diagram with correct structure whose every label is dingbats.
Remaining
49 warnings: 13 of the soft "symbol not in cited sources" class, and 36 STALE on pages this branch
does not touch (concentrated in
adr-index,memory-accounting,01-architecture). No page'supdated:date was advanced except where its content was re-verified against the tree.Note that
verify_citationswas green while several pages asserted the opposite of what the codedoes — it proves citations point at real lines, not that the prose matches them.
Engine findings filed separately — these are code defects the wiki can only describe:
extStorageEvictFlashKeyis dead code, no caller in the treeBoth are cross-referenced from the pages that describe them.