Skip to content

fix(cnight-observation): chunk the bulk cache catch-up pull#1889

Draft
skylar-simoncelli wants to merge 1 commit into
mainfrom
skylar/chunk-cnight-catchup-pull
Draft

fix(cnight-observation): chunk the bulk cache catch-up pull#1889
skylar-simoncelli wants to merge 1 commit into
mainfrom
skylar/chunk-cnight-catchup-pull

Conversation

@skylar-simoncelli

Copy link
Copy Markdown
Contributor

Overview

Investigating the "slow db-sync query" spikes on devnet/stagenet (node 2.0.x) traced the load to the cNIGHT observation path, not to db-sync resourcing. Two facts pin it to the node code, not the DB:

  • devnet (2.0.x) and qanet (1.0.0) read the same shared Aurora; only the 2.0.x one is slow. stagenet (2.0.x) has its own dedicated Aurora and is slow too. The variable that tracks the symptom is the node version, not the database.
  • An EXPLAIN ANALYZE of the cNIGHT observation join shape ran ~53 s on an otherwise idle DB. The plan is sensible (index scans, accurate estimates) — the cost is the range width scanned, not a missing index or contention.

This PR fixes one concrete driver of that width: the bulk cache's catch-up pull.

Change

RefreshContext::refresh warms/slides the in-memory cNIGHT window in the background. It pulled the whole [from_block, target_end] range in a single bulk_pull with LARGE_LIMIT. In steady state that range is small ([old_end + 1, target_end]), but when a node is far behind tip (cold start, or after a lag) the span can be hundreds of thousands of Cardano blocks, and one query over it is a multi-minute db-sync scan that stalls block import.

The refresh now pulls the range in bounded block-span chunks (REFRESH_CHUNK_BLOCKS = 5000), committing snapshot_end_block after each chunk so an interrupted catch-up resumes from the last committed block instead of restarting the giant query.

Consensus-safe / scope: this is a cache-warming detail only. The union of chunks covers exactly the same range, so no inherent payload changes. The per-block consensus path (MidnightCNightObservationDataSourceImpl::get_utxos_up_to_capacity) is untouched and still queries through tip — deliberately, per its existing replay-semantics comment.

Relationship to other PRs

Testing Evidence

  • rustfmt --edition 2024 clean.
  • Not compiled locally (dev-machine build policy); relying on CI for cargo check/clippy/tests.
  • Follow-up wanted: a unit test around the chunk boundary in refresh (chunk_end clamping, resume via committed snapshot_end_block). Happy to add once a reviewer confirms the approach.

Submission Checklist

  • Changes are backward-compatible
  • PR description explains why the change is needed
  • Self-reviewed the diff
  • Included a change file
  • Bump node minor version (n/a — behavior-preserving perf fix)
  • Additional tests (see follow-up above)

@skylar-simoncelli skylar-simoncelli added the bot:ai-assisted Authored or substantially edited by an AI agent label Jul 15, 2026
The bulk cNIGHT observation cache warms/slides its window in a background
refresh that pulled the whole [from_block, target_end] range in one
bulk_pull with LARGE_LIMIT. When a node is far behind tip (cold start or
after a lag) that span can be hundreds of thousands of Cardano blocks, and
a single query over it is a multi-minute db-sync scan that stalls block
import.

Pull the range in bounded block-span chunks (REFRESH_CHUNK_BLOCKS = 5000),
committing snapshot_end_block after each chunk so an interrupted catch-up
resumes from the last committed block. Cache-warming only: the union of
chunks covers the same range, so the consensus inherent payload is
unchanged (the per-block get_utxos_up_to_capacity path still queries
through tip).

Assisted-by: Claude:claude-opus-4-8[1m]
@skylar-simoncelli skylar-simoncelli force-pushed the skylar/chunk-cnight-catchup-pull branch from 4153dd5 to 06442ec Compare July 15, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:ai-assisted Authored or substantially edited by an AI agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant