fix(cnight-observation): chunk the bulk cache catch-up pull#1889
Draft
skylar-simoncelli wants to merge 1 commit into
Draft
fix(cnight-observation): chunk the bulk cache catch-up pull#1889skylar-simoncelli wants to merge 1 commit into
skylar-simoncelli wants to merge 1 commit into
Conversation
The bulk cNIGHT observation cache warms/slides its window in a background refresh that pulled the whole [from_block, target_end] range in one bulk_pull with LARGE_LIMIT. When a node is far behind tip (cold start or after a lag) that span can be hundreds of thousands of Cardano blocks, and a single query over it is a multi-minute db-sync scan that stalls block import. Pull the range in bounded block-span chunks (REFRESH_CHUNK_BLOCKS = 5000), committing snapshot_end_block after each chunk so an interrupted catch-up resumes from the last committed block. Cache-warming only: the union of chunks covers the same range, so the consensus inherent payload is unchanged (the per-block get_utxos_up_to_capacity path still queries through tip). Assisted-by: Claude:claude-opus-4-8[1m]
4153dd5 to
06442ec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Investigating the "slow db-sync query" spikes on devnet/stagenet (node 2.0.x) traced the load to the cNIGHT observation path, not to db-sync resourcing. Two facts pin it to the node code, not the DB:
EXPLAIN ANALYZEof the cNIGHT observation join shape ran ~53 s on an otherwise idle DB. The plan is sensible (index scans, accurate estimates) — the cost is the range width scanned, not a missing index or contention.This PR fixes one concrete driver of that width: the bulk cache's catch-up pull.
Change
RefreshContext::refreshwarms/slides the in-memory cNIGHT window in the background. It pulled the whole[from_block, target_end]range in a singlebulk_pullwithLARGE_LIMIT. In steady state that range is small ([old_end + 1, target_end]), but when a node is far behind tip (cold start, or after a lag) the span can be hundreds of thousands of Cardano blocks, and one query over it is a multi-minute db-sync scan that stalls block import.The refresh now pulls the range in bounded block-span chunks (
REFRESH_CHUNK_BLOCKS = 5000), committingsnapshot_end_blockafter each chunk so an interrupted catch-up resumes from the last committed block instead of restarting the giant query.Consensus-safe / scope: this is a cache-warming detail only. The union of chunks covers exactly the same range, so no inherent payload changes. The per-block consensus path (
MidnightCNightObservationDataSourceImpl::get_utxos_up_to_capacity) is untouched and still queries through tip — deliberately, per its existing replay-semantics comment.Relationship to other PRs
Testing Evidence
rustfmt --edition 2024clean.cargo check/clippy/tests.refresh(chunk_end clamping, resume via committedsnapshot_end_block). Happy to add once a reviewer confirms the approach.Submission Checklist