Stop the permanent store from caching a block as done when it isn't - #60
Open
damilolaedwards wants to merge 1 commit into
Open
Conversation
processBlock logged a failure to record a block in the permanent_blocks table but still marked the block as done in its in-memory cache and returned success. Any later attempt to process that same block, such as retention re-checking before deleting the original, would hit the cache and skip straight to success without ever noticing the database never actually got the record. Retention would then delete the original block and its blob, leaving an orphaned copy at the permanent location that nothing in the system tracks anymore. The cache is now only updated once recording actually succeeds, and the failure is reported back to the caller instead of swallowed. ProcessedChan changes from a plain close signal to a channel carrying the actual result, so retention can tell the difference between "this block is safe to delete because it was archived" and "archiving failed, leave it alone and retry later." Queuing outcomes that were previously silent (store disabled, stopped, queue full) now report through the same channel so callers get one consistent way to find out what happened.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
processBlocklogged a failure to record a block in thepermanent_blockstable but still marked the block as done in its in-memory cache and returned success. Any later attempt to process that same block, such as retention re-checking before deleting the original, would hit the cache and skip straight to success without ever noticing the database never actually got the record. Retention would then delete the original block and its blob, leaving an orphaned copy at the permanent location that nothing in the system tracks anymore.The cache is now only updated once recording actually succeeds, and the failure is reported back to the caller instead of swallowed.
ProcessedChanchanges from a plain close signal to a channel carrying the actual result, so retention can tell the difference between "safe to delete, this was archived" and "archiving failed, leave it alone and retry later." The three previously silent queuing outcomes (store disabled, stopped, queue full) now report through the same channel: disabled reports success, since a deployment that never turned this feature on never had a durability promise to keep; stopped and queue-full report an error, since the block genuinely wasn't archived.Changes
pkg/server/service/indexer/permanent_store.go:processBlockuses a named return so every exit path reports its real outcome, cache is only updated on confirmed successpkg/server/service/indexer/retention.go:purgeOldBeaconBlockschecks the result before deleting the original, retries next cycle on failure insteadpkg/server/service/indexer/permanent_store_test.go: mechanical update to the existing tests' channel typepkg/server/service/indexer/permanent_store_cache_test.go(new): two tests using a real, targeted database failure (drops thepermanent_blockstable so recording genuinely fails, rather than mocking the error) to prove the cache isn't poisoned and that a retry after the underlying problem clears succeeds normallyNote on overlap with #59
This PR touches the same
QueueBlockandpurgeOldBeaconBlockslogic that #59 changed, since correctly reporting success or failure required redoing that channel handling with a richer type (chan errorinstead ofchan struct{}). This PR's version includes everything #59 fixed (every path still reports back, nothing blocks forever) plus the cache correctness fix. Whichever of the two merges first, the other will need a rebase.Test plan
go build ./...andgo vet ./...pass