docs(storage): document storage.rocks.* memory config options#496
Draft
kriszyp wants to merge 2 commits into
Draft
docs(storage): document storage.rocks.* memory config options#496kriszyp wants to merge 2 commits into
kriszyp wants to merge 2 commits into
Conversation
Adds documentation for four new storage configuration parameters introduced in v5.1.0: - storage.rocks.blockCacheSize — explicit override for the shared RocksDB block cache size - storage.rocks.writeBufferManagerSize — process-wide cap on memtable memory across all databases - storage.rocks.writeBufferManagerCostToCache — share memtable accounting with the block cache for unified observability - storage.rocks.writeBufferManagerAllowStall — hard cap vs soft cap behavior when the WriteBufferManager limit is reached - Adds bullet-list entries to reference/configuration/options.md under the existing `storage` section. - Adds a new "RocksDB Memory" detail section to reference/database/storage-tuning.md between "Read & Write Behavior" and "Storage Reclamation", with workload-recipe-style guidance for when to lower the cache or enable the manager. Documents the multi-tenant shared-host scenario as the primary use case for tuning these knobs. Source PR: HarperFast/harper#780 Co-Authored-By: Claude Sonnet 4.7 <noreply@anthropic.com>
kriszyp
requested review from
Ethan-Arrowood,
cb1kenobi and
dawsontoth
and removed request for
dawsontoth
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-496 This preview will update automatically when you push new commits. |
Reframes the RocksDB Memory section to focus on general memory management rather than Fabric-specific multi-tenant tuning (that context now lives in host-manager's DESIGN.md, where the actual tuning decision is made). Adds a new "How RocksDB reads are cached" subsection explaining the three-tier hierarchy: block cache -> OS page cache -> disk with concrete latency expectations and the implication for sizing: shrinking the block cache shifts hits from the decompressed in-process cache to the kernel's dynamically-sized compressed page cache, not directly to disk. This helps operators reason about why a smaller block cache isn't necessarily worse on memory-constrained hosts. Drops the "shared-host / multi-tenant" framing from the section intro and from the blockCacheSize discussion. Softens the WriteBufferManager rationale to talk about "many tables" generically rather than naming a 14-table customer profile. Co-Authored-By: Claude Sonnet 4.7 <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-496 This preview will update automatically when you push new commits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents four new storage configuration parameters being added in v5.1.0 (source: HarperFast/harper#780).
What's added
reference/configuration/options.md— four new bullets under the existingstoragesection:rocks.blockCacheSizerocks.writeBufferManagerSizerocks.writeBufferManagerCostToCacherocks.writeBufferManagerAllowStallreference/database/storage-tuning.md— a new "RocksDB Memory" detail section between "Read & Write Behavior" and "Storage Reclamation":Scope decision
This PR is intentionally general guidance for any RocksDB deployment. The Fabric-specific multi-tenant tuning rationale (why we picked 15% block cache + 5% WBM in host-manager, the canopy memory data, the dependency chain across the three implementation PRs) is captured in host-manager's DESIGN.md rather than the public docs, since it's our infrastructure concern rather than something operators of self-hosted Harper need to reason about.
Where to focus review
What's NOT changed
Build / format
Generated by Claude. — Claude Sonnet 4.7