fix(tiering): honor tiered_retention policies in the flag phase#48
Merged
Conversation
add_tiered_retention_policy writes policy_type='tiered_retention', but the
flag/tier phase only matched 'tiering' -- so the tiering job reported
'Found 0 table(s)' and chunks were never flagged 'tiered' (only the drop
horizon worked).
- tiering_job.py + _get_chunks_to_tier: match policy_type IN
('tiering','tiered_retention') and read the horizon from 'after' or
'tier_after'.
- add_tiered_retention_policy: install the write-tracking trigger, backfill
last_write_lsn on existing active chunks, and set tiering_enabled = TRUE --
without these tier_chunk treats last_write_lsn as NULL and defers forever.
Release 0.1.2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
add_tiered_retention_policy('stock_ticks', tier_after => '10 minutes', drop_after => '60 minutes')registered the policy, but the tiering job logged:add_tiered_retention_policywritespolicy_type = 'tiered_retention'(configtier_after/drop_after), but the flag/tier path only knew about'tiering'(configafter). The drop path already handledIN ('retention','tiered_retention'), so the drop horizon worked — only the flagging was dead. Three gaps:tiering_job.pyfilteredpolicy_type = 'tiering'→ "Found 0 tables"._get_chunks_to_tierfiltered'tiering'and readconfig->>'after'(nottier_after).add_tiered_retention_policynever installed write-tracking, never backfilledlast_write_lsn, and never settiering_enabled— so even reachingtier_chunk, every chunk deferred (NULLlast_write_lsn= "cannot prove durable").Fix
tiering_job.py—WHERE policy_type IN ('tiering','tiered_retention')(withDISTINCTin case both exist)._get_chunks_to_tier— sameIN (...)filter; horizon =COALESCE(config->>'after', config->>'tier_after').add_tiered_retention_policy— install_install_tiering_write_tracking, backfilllast_write_lsn = pg_current_wal_lsn()on active chunks, and settiering_enabled = TRUE(mirrorsadd_tiering_policy).tiering_enableditself gates nothing (no code reads it) — it's a status flag, fixed for honesty.Test plan
make buildclean (14 modules, version 0.1.2)add_tiered_retention_policythen run the tiering job → chunks older thantier_afterflaggedtiered, then dropped atdrop_after_chronotable_registry.tiering_enabled = trueafter the callShips as v0.1.2. Requires reinstalling
dist/lakets.sqland redeploying the maintenance jobs.