Skip to content

fix(tiering): honor tiered_retention policies in the flag phase#48

Merged
taran-dbx merged 1 commit into
mainfrom
fix/tiered-retention-flag
Jun 8, 2026
Merged

fix(tiering): honor tiered_retention policies in the flag phase#48
taran-dbx merged 1 commit into
mainfrom
fix/tiered-retention-flag

Conversation

@taran-dbx

Copy link
Copy Markdown
Collaborator

Problem

add_tiered_retention_policy('stock_ticks', tier_after => '10 minutes', drop_after => '60 minutes') registered the policy, but the tiering job logged:

Found 0 table(s) with tiering policies
Tiering complete: 0 flagged, 0 deferred

add_tiered_retention_policy writes policy_type = 'tiered_retention' (config tier_after/drop_after), but the flag/tier path only knew about 'tiering' (config after). The drop path already handled IN ('retention','tiered_retention'), so the drop horizon worked — only the flagging was dead. Three gaps:

  1. tiering_job.py filtered policy_type = 'tiering' → "Found 0 tables".
  2. _get_chunks_to_tier filtered 'tiering' and read config->>'after' (not tier_after).
  3. add_tiered_retention_policy never installed write-tracking, never backfilled last_write_lsn, and never set tiering_enabled — so even reaching tier_chunk, every chunk deferred (NULL last_write_lsn = "cannot prove durable").

Fix

  • tiering_job.pyWHERE policy_type IN ('tiering','tiered_retention') (with DISTINCT in case both exist).
  • _get_chunks_to_tier — same IN (...) filter; horizon = COALESCE(config->>'after', config->>'tier_after').
  • add_tiered_retention_policy — install _install_tiering_write_tracking, backfill last_write_lsn = pg_current_wal_lsn() on active chunks, and set tiering_enabled = TRUE (mirrors add_tiering_policy).

tiering_enabled itself gates nothing (no code reads it) — it's a status flag, fixed for honesty.

Test plan

  • make build clean (14 modules, version 0.1.2)
  • add_tiered_retention_policy then run the tiering job → chunks older than tier_after flagged tiered, then dropped at drop_after
  • _chronotable_registry.tiering_enabled = true after the call

Ships as v0.1.2. Requires reinstalling dist/lakets.sql and redeploying the maintenance jobs.

add_tiered_retention_policy writes policy_type='tiered_retention', but the
flag/tier phase only matched 'tiering' -- so the tiering job reported
'Found 0 table(s)' and chunks were never flagged 'tiered' (only the drop
horizon worked).

- tiering_job.py + _get_chunks_to_tier: match policy_type IN
  ('tiering','tiered_retention') and read the horizon from 'after' or
  'tier_after'.
- add_tiered_retention_policy: install the write-tracking trigger, backfill
  last_write_lsn on existing active chunks, and set tiering_enabled = TRUE --
  without these tier_chunk treats last_write_lsn as NULL and defers forever.

Release 0.1.2.
@taran-dbx taran-dbx merged commit 250e7bd into main Jun 8, 2026
8 checks passed
@taran-dbx taran-dbx deleted the fix/tiered-retention-flag branch June 8, 2026 20:54
@github-actions github-actions Bot added documentation Improvements or additions to documentation sql databricks-workflows labels Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

databricks-workflows documentation Improvements or additions to documentation sql

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant