The WALRetentionQueueAwareness e2e feature fails intermittently.
A backup inside the test is marked failed with rpc error: code = Unknown desc = backup verification detected corruption: exit status 65.
There is no corruption: the tier1 verification the plugin runs after a backup resolves the backup's Kopia snapshot IDs, and those IDs are rewritten underneath it before verification starts, so Kopia reports that it found none of the snapshots it was asked to check.
The e2e fails at the "backup not completed" assertion because the CNPG Backup never reaches completed.
Which backup of the feature is hit varies per run.
Failure signature:
unable to parse snapshot manifest <id>: snapshot not found, then found 0 of the 3 requested snapshot IDs to verify, errorCount:1, then Backup verification detected corruption.
A few-hundred-millisecond window between the plugin resolving the snapshot IDs and invoking verification.
The server-side consumer unpins those same snapshots in that interval during post-relay tier2 maintenance, and kopia snapshot pin --remove rewrites a snapshot's manifest under a new ID and deletes the old one.
Some failed e2e:
The
WALRetentionQueueAwarenesse2e feature fails intermittently.A backup inside the test is marked failed with
rpc error: code = Unknown desc = backup verification detected corruption: exit status 65.There is no corruption: the tier1 verification the plugin runs after a backup resolves the backup's Kopia snapshot IDs, and those IDs are rewritten underneath it before verification starts, so Kopia reports that it found none of the snapshots it was asked to check.
The e2e fails at the "backup not completed" assertion because the CNPG
Backupnever reachescompleted.Which backup of the feature is hit varies per run.
Failure signature:
unable to parse snapshot manifest <id>: snapshot not found, then found 0 of the 3 requested snapshot IDs to verify, errorCount:1, thenBackup verification detected corruption.A few-hundred-millisecond window between the plugin resolving the snapshot IDs and invoking verification.
The server-side consumer unpins those same snapshots in that interval during post-relay tier2 maintenance, and
kopia snapshot pin --removerewrites a snapshot's manifest under a new ID and deletes the old one.Some failed e2e: