Skip to content

fix(tracking): reclaim history.db pages after retention cleanup - #3479

Open
imkp1 wants to merge 1 commit into
rtk-ai:developfrom
imkp1:fix/tracking-reclaim-history-db-space
Open

fix(tracking): reclaim history.db pages after retention cleanup#3479
imkp1 wants to merge 1 commit into
rtk-ai:developfrom
imkp1:fix/tracking-reclaim-history-db-space

Conversation

@imkp1

@imkp1 imkp1 commented Aug 8, 2026

Copy link
Copy Markdown

Summary

  • cleanup_old() deleted rows past the 90-day window but nothing returned the freed pages, so history.db stayed at its high-water mark for good. Retention bounded the row count, not the file size.
  • New databases are now created with auto_vacuum=INCREMENTAL and hand the pages back after every cleanup that actually removed rows. The pragma has to be set before journal_mode=WAL, because switching to WAL writes the header and a database can no longer change its vacuum mode after that.
  • A database created before this change is converted once, by the first cleanup that frees pages, and reclaims its accumulated slack in the same step. needs_full_vacuum is what keeps that a one-time event.
  • reset_all() reclaims too. Resetting stats to zero should return the space, not just empty the tables.
  • Reclamation is best-effort throughout. A database that cannot be shrunk right now must never fail the command being recorded.

There is no "vacuum once the freelist passes X%" rule here. Reclaiming is driven by whether a delete removed rows, and the work incremental_vacuum does is proportional to the pages that delete freed.

The two secondary query items at the end of the issue (SQL-side SUM/COUNT in get_summary_filtered, and the parameter-dependent OR that stops the project-scope filter using its index) are left out on purpose. They are a separate change and this repo asks for one focused fix per PR. Happy to follow up with them.

Test plan

  • cargo fmt --all --check && cargo clippy --all-targets && cargo test

  • Four new unit tests in src/core/tracking.rs: a new database reports auto_vacuum=2; cleanup leaves an empty freelist and a smaller page count; a database built without auto_vacuum is converted and shrunk on its first cleanup; reset_all reclaims. They assert on auto_vacuum, freelist_count and before/after page_count rather than on bytes or milliseconds, since in WAL mode the on-disk size only follows a checkpoint.

  • Manual, against a database with 200k rows all past the retention window:

    before   rows 200000   auto_vacuum 0   page_count 3853   15.8 MB
    rtk proxy echo hi
    after    rows      1   auto_vacuum 2   page_count    7   28 KB
    
  • Manual worst case for the one-time conversion, 100k rows still inside the window so the vacuum has to rewrite all of them: the run that converts takes 145 ms, the runs after it 32 ms. It happens once per database.

  • Steady state, same 100k-row database, 60 interleaved runs of each binary: 8.31 ms median before this change, 8.16 ms after. When nothing expires, cleanup_old deletes 0 rows and does no extra work at all.

  • Steady state for a user past the window, where rows expire on almost every insert: 100k rows spread across the last 91 days, 50 interleaved runs each, 8.07 ms median before and 7.88 ms after. incremental_vacuum only touches the pages that delete just freed, so this stays a handful of pages per command.

Closes #3393


🤖 Opened with Superhuman, an open-source contribution agent.

cleanup_old() deleted rows past the 90-day window but nothing returned the
freed pages, so history.db stayed at its high-water mark for good. Retention
bounded the row count, not the file size.

New databases are now created with auto_vacuum=INCREMENTAL and give the pages
back after every cleanup that removed rows. A database created before this
change is converted once, by the first cleanup that frees pages, and reclaims
its accumulated slack at the same time.

Reclamation is best-effort: a database that cannot be shrunk right now must
never fail the command being recorded.
@imkp1
imkp1 marked this pull request as ready for review August 8, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

history.db never shrinks — a database with one row still occupies 87.8 MB after 90-day cleanup

1 participant