Skip to content

Retention pruners do unbatched single-transaction DELETE on the fastest-growing table #280

Description

@richardmhope

Summary

The audit and feed-item pruners do SELECT count(*) then a single unbatched DELETE ... WHERE cutoff in one transaction (src/iceberg/services/audit.py:225-232, src/iceberg/services/feeds.py:522-530). Config comments call AuditEvent "the fastest-growing table on a scanned public instance" (config.py:276-278).

Failure scenario

The first cron run on an existing instance (365-day window) can delete millions of rows in one statement: long row-lock hold, WAL/replication spike, potential job timeout, and on failure zero progress (no batched commits). The returned value is also the pre-delete count, not result.rowcount — a second full-table scan and a value that can drift from rows actually deleted.

Notes

FK integrity is fine — nothing references AuditEvent or FeedItem (their FKs point outward), and ingested feed items are correctly spared.

Suggested fix

Use a LIMIT-batched delete loop with per-batch commits (the standard shape for large retention deletes); return result.rowcount instead of a pre-count.

Found in Fable review of PR #232 → HEAD.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingfable-reviewCorrectness findings from Claude Fable code reviewpriority: mediumImportant, not blocking

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions