Skip to content

Stop the permanent store from caching a block as done when it isn't - #60

Open
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/permanent-store-cache-poisoning
Open

Stop the permanent store from caching a block as done when it isn't#60
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/permanent-store-cache-poisoning

Conversation

@damilolaedwards

Copy link
Copy Markdown

Summary

processBlock logged a failure to record a block in the permanent_blocks table but still marked the block as done in its in-memory cache and returned success. Any later attempt to process that same block, such as retention re-checking before deleting the original, would hit the cache and skip straight to success without ever noticing the database never actually got the record. Retention would then delete the original block and its blob, leaving an orphaned copy at the permanent location that nothing in the system tracks anymore.

The cache is now only updated once recording actually succeeds, and the failure is reported back to the caller instead of swallowed. ProcessedChan changes from a plain close signal to a channel carrying the actual result, so retention can tell the difference between "safe to delete, this was archived" and "archiving failed, leave it alone and retry later." The three previously silent queuing outcomes (store disabled, stopped, queue full) now report through the same channel: disabled reports success, since a deployment that never turned this feature on never had a durability promise to keep; stopped and queue-full report an error, since the block genuinely wasn't archived.

Changes

  • pkg/server/service/indexer/permanent_store.go: processBlock uses a named return so every exit path reports its real outcome, cache is only updated on confirmed success
  • pkg/server/service/indexer/retention.go: purgeOldBeaconBlocks checks the result before deleting the original, retries next cycle on failure instead
  • pkg/server/service/indexer/permanent_store_test.go: mechanical update to the existing tests' channel type
  • pkg/server/service/indexer/permanent_store_cache_test.go (new): two tests using a real, targeted database failure (drops the permanent_blocks table so recording genuinely fails, rather than mocking the error) to prove the cache isn't poisoned and that a retry after the underlying problem clears succeeds normally

Note on overlap with #59

This PR touches the same QueueBlock and purgeOldBeaconBlocks logic that #59 changed, since correctly reporting success or failure required redoing that channel handling with a richer type (chan error instead of chan struct{}). This PR's version includes everything #59 fixed (every path still reports back, nothing blocks forever) plus the cache correctness fix. Whichever of the two merges first, the other will need a rebase.

Test plan

  • go build ./... and go vet ./... pass
  • New tests pass, and their log output shows the full mechanism working: copy succeeds, record fails and is logged, a second attempt tries again instead of trusting a stale cache, a third attempt after the underlying table exists again succeeds and is recorded
  • Updated the existing test suite's channel type so it still compiles
  • Pre-existing Docker-dependent tests elsewhere in this package could not be run in this environment (no Docker available), unrelated to this change

processBlock logged a failure to record a block in the permanent_blocks
table but still marked the block as done in its in-memory cache and
returned success. Any later attempt to process that same block, such
as retention re-checking before deleting the original, would hit the
cache and skip straight to success without ever noticing the database
never actually got the record. Retention would then delete the
original block and its blob, leaving an orphaned copy at the permanent
location that nothing in the system tracks anymore.

The cache is now only updated once recording actually succeeds, and
the failure is reported back to the caller instead of swallowed.
ProcessedChan changes from a plain close signal to a channel carrying
the actual result, so retention can tell the difference between "this
block is safe to delete because it was archived" and "archiving
failed, leave it alone and retry later." Queuing outcomes that were
previously silent (store disabled, stopped, queue full) now report
through the same channel so callers get one consistent way to find
out what happened.
@damilolaedwards
damilolaedwards requested a review from Savid as a code owner July 26, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant