Skip to content

fix(core): identify Kopia snapshots by a stable identity in verify and delete - #74

Open
gabriele-wolfox wants to merge 5 commits into
mainfrom
dev/cnp-9006
Open

fix(core): identify Kopia snapshots by a stable identity in verify and delete#74
gabriele-wolfox wants to merge 5 commits into
mainfrom
dev/cnp-9006

Conversation

@gabriele-wolfox

@gabriele-wolfox gabriele-wolfox commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Problem

WALRetentionQueueAwareness fails intermittently with backup verification detected corruption: exit status 65 on a backup that is intact.

Post-relay tier2 maintenance unpins the snapshots of the backup just taken, and kopia snapshot pin --remove rewrites each manifest under a new ID. Verification resolves the manifest IDs and uses them a moment later, so a rewrite in between makes Kopia report found 0 of the N requested snapshot IDs to verify, which klio classified as corruption. The observed gap between resolution and rewrite was around 150 ms, hence the flake. The same misreport is reachable outside tests: any backup whose tier1 verification overlaps the unpin of its own snapshots can be failed as corrupt, and exit 65 is non-retryable.

Changes

  • Verify resolves each snapshot to its root object ID, which the pin rewrite leaves untouched and which the server side already uses for the unpin. Directory and file roots take different Kopia flags, since a backup mixes both
    (the control data file is snapshotted on its own), so the root entry type picks the flag.
  • Classification: a verify result reporting only unresolved snapshot IDs is no longer corruption. Nothing was read, so it carries no evidence about repository integrity, and it is now retryable. Errors naming a missing object
    or blob still fail as corruption.
  • Delete had the same resolve-then-use shape: deleting a rewritten ID matches nothing, so the command failed and the backup survived. It now re-resolves and retries. It deliberately does not switch to root object IDs: unchanged content dedupes to the same root across backups and Kopia deletes every snapshot matching the ID it is given, so that would delete another backup's snapshots.
  • e2e: the WAL retention feature now verifies its newest backup on tier1, after that backup has been through a maintenance pass.
  • AGENTS.md: records when to use each identity, and corrects the existing note that said a stale manifest ID makes delete remove the wrong snapshot.

Testing

  • Reproduced the failure deterministically by widening the window, then confirmed the fix with the window still forced open, and again with it removed.
  • Full core suite and golangci-lint clean on both modules; 15 new unit subtests.
  • WALRetentionQueueAwareness passes locally on kind.

Assisted-by: Claude

Closes #73

@gabriele-wolfox
gabriele-wolfox force-pushed the dev/cnp-9006 branch 2 times, most recently from de5fdfb to e9b63b2 Compare August 11, 2026 13:06
@gabriele-wolfox
fix(core): verify backups by root object ID
0b6202c 
Tier1 backup verification resolved a backup to its Kopia snapshot manifest
IDs and passed them to "kopia snapshot verify". Post-relay tier2 maintenance
unpins those same snapshots, and "kopia snapshot pin --remove" rewrites a
snapshot manifest under a new ID and deletes the old one, so an ID resolved a
moment earlier could already be gone by the time verification ran. Kopia then
reported "found 0 of the N requested snapshot IDs to verify", and klio turned
that into a corruption failure with exit code 65, failing an intact backup.
This is the intermittent WALRetentionQueueAwareness e2e failure, and the same
misreport is reachable outside tests.

Resolve backups to the root object ID of each snapshot instead. Root object
IDs are a stable identity that the pin rewrite leaves untouched, and the
server side already identifies the snapshots it unpins the same way.
Directory and file roots need different flags, and a backup mixes both: pgdata
and metadata are directory snapshots, while the control data file is
snapshotted on its own. Passing a file root to --directory-id makes Kopia
parse file content as a directory listing and report healthy data as corrupt,
so the root entry type now decides the flag and an unknown type is an error
rather than a guess.

Also stop classifying an unresolved snapshot set as corruption. Such a result
means nothing was verified, so it carries no evidence about repository
integrity, and it is now returned as a retryable error. Errors about a missing
object or blob keep failing as corruption.

Assisted-by: Claude
Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
@gabriele-wolfox
fix(core): re-resolve snapshot IDs when deleting a backup
d554c90 
Backup deletion listed a backup's snapshots and then deleted them by manifest
ID, which a concurrent "kopia snapshot pin" can rewrite in between. Deleting a
replaced ID matches nothing and fails with "no snapshots matched", so the
deletion reports an error and the backup survives.

Resolve the snapshots again and retry, up to three attempts. Each attempt
lists the current IDs, and snapshots removed by an earlier attempt are no
longer listed, so the loop converges without deleting anything twice.

Deleting by root object ID, the identity used for verification, would be wrong
here: unchanged content dedupes to the same root across backups, and Kopia
deletes every snapshot matching the ID it is given. Two backups of an idle
tablespace share a root object, so deleting one backup by root ID would take
the other backup's snapshot with it.

The delete loop moved behind a narrow snapshotStore interface so the retry
behaviour is covered by unit tests without reworking how Connection holds its
Kopia client.

Assisted-by: Claude
Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
@gabriele-wolfox
test(e2e): verify the newest backup on tier1
175848d 
The WAL retention feature already drives three backups through a full
maintenance pass, which unpins their snapshots and so rewrites every snapshot
manifest under a new ID. Verifying the newest backup at the end of the feature
asserts that verification still resolves it afterwards, and exercises that
resolution against a real backup: each backup mixes a file root (the control
data file, snapshotted on its own) with directory roots (pgdata and metadata),
which need different Kopia flags.

Only the newest backup is verified, because "klio backup list" spans both tiers
and so also reports the backup the feature deletes, which no longer has tier1
snapshots.

Also drop the tracker references from the test comments added with the fix, and
describe the failure mode instead.

Assisted-by: Claude
Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
@gabriele-wolfox
docs(agents): document Kopia snapshot identity choice
6bfad69 
Add a section on when to identify a snapshot by manifest ID and when by root
object ID. A manifest ID is not stable, since "kopia snapshot pin" rewrites it,
so reads that must survive concurrent maintenance use the root object ID, while
deletions must not: unchanged content dedupes to the same root across backups
and Kopia deletes every snapshot matching the ID it is given.

Also record that directory and file roots take different verify flags, that a
verify result reporting only unresolved snapshot IDs is not corruption, and
correct the existing note about a stale manifest ID: a delete of a rewritten ID
matches nothing and fails rather than deleting the wrong snapshot.

Assisted-by: Claude
Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
@gabriele-wolfox
fix(core): tighten verify/delete after the root-object-ID switch
2410b28 
The unresolved-snapshot verify classification can never trigger: once
buildVerifyArgs only emits --directory-id/--file-id flags, Kopia never
produces the "requested snapshot IDs to verify" error it matched. Removed
it; classifyVerifyError again treats any errorCount > 0 as corruption.

Also:
- verifySpecificBackups no longer aborts every backup when one fails to
  resolve; it joins the error and still verifies the rest.
- DeleteBackup now waits between retry attempts instead of retrying
  immediately, matching the backoff already used for the same pin-rewrite
  race elsewhere.
- AGENTS.md no longer calls the tier1 unpin a safe root-ID read: unpinning
  by root ID mutates every snapshot sharing that root, same as delete. It
  stays only because the step is best-effort.
- Reworded the e2e verification step to say it exercises root-object-ID
  resolution, not the rewrite race itself.

Assisted-by: Claude
Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

WALRetentionQueueAwareness e2e intermittently fails with "backup verification detected corruption: exit status 65"

1 participant

@gabriele-wolfox