fix(recorder): path-idempotent storage seeding + name->path lookups - #585
Merged
Conversation
Boot seeding (recorder main + the seed binary) upserted the two configured
storages by NAME only. After an operator renames a storage in the console
(PUT /config/storages/{id}), no row carries the configured *_STORAGE_NAME, so
the next boot INSERTs a second, empty row for the same directory: the duplicate
Live/Archive rows that showed up next to the real 2TB NVMe / 16TB Spinner rows.
PR #557 added a warn-only detector (duplicate_path_under_other_name) and
deliberately did nothing else; its revisit trigger (the duplicates actually
materialize and confuse the operator) has now fired. The ghost rows are also not
inert: the recorder resolves runtime defaults by config NAME (archive.rs
free-space-floor fallback, reconcile.rs live/archive stage labelling), so once
the real rows are renamed those lookups miss and silently degrade.
- Promote the detector from warn to a GATE. Both seed paths now share one
decision function, db::seed_storage_path_idempotent: retarget a same-named
row's path; else if a row already covers the path under another name, SKIP the
insert (never create a duplicate); else insert (fresh install). It never
renames or deletes the operator's row, and folds the resolved row into the
in-pass set so a shared live==archive directory resolves to one row.
- Add db::get_storage_by_path (deterministic: oldest created_at wins on a shared
path) + db::get_storage_by_name_or_path (name first, then path), and use the
name-or-path variant at the two runtime sites so a renamed install keeps full
behavior and, after the empty rows are cleaned up, the paths still resolve to
the real rows.
- Path comparison normalizes trailing slashes (normalize_storage_path).
- No migration, no UNIQUE(path): existing DBs already carry duplicate paths and
a shared live==archive layout is legal.
Tests (DB-backed, gated on TEST_DATABASE_URL): the exact #557 scenario run twice
(still 2 rows, operator names untouched, no growth), fresh-install two rows,
same-name retarget, shared-layout one row, get_storage_by_path oldest-wins +
trailing-slash, and the name->path free-floor/labelling fallback; plus pure
tests for normalize_storage_path and the trailing-slash-aware detector.
Cannot orphan footage: existing segments keep their storage_id, the default
policy keeps live_storage_id, ensure_default_policy wires by PATH, and the fix
only ever skips creating a NEW empty row. DECISIONS.md entry added (supersedes
#557's detect-never-fix). The one-time cleanup of the two already-present empty
rows remains a maintainer DB action, sequenced after this deploys.
Signed-off-by: badbread <badbread@users.noreply.github.com>
Signed-off-by: badbread <badbread@users.noreply.github.com>
# Conflicts: # docs/DECISIONS.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #584.
Problem
db::upsert_storageis idempotent by storage name only(
ON CONFLICT (name) DO UPDATE SET path). The recorder seeds the two configuredstorages on every boot (
main.rs::seed_storages), and theseedbinary does the sameat container start, keyed on
LIVE_STORAGE_NAME/ARCHIVE_STORAGE_NAME. After anoperator renames a storage in the console (
PUT /config/storages/{id}), no row carriesthe configured name, so the next boot INSERTs a second, empty row for the same directory:
the duplicate
Live/Archiverows sitting next to the real renamed rows.PR #557 added a warn-only detector (
duplicate_path_under_other_name) but deliberatelydid not prevent the duplicate. Its revisit trigger (the duplicates actually materialize
and confuse the operator) has now fired. The ghost rows are also not inert: the recorder
resolves runtime defaults by config name (the free-space-floor fallback in
archive.rs, live/archive stage labelling inreconcile.rs), so once the real rows arerenamed those lookups miss and silently degrade.
Change
function,
db::seed_storage_path_idempotent: retarget a same-named row's path; else ifa row already covers the path under another name, skip the insert (never create a
duplicate) and return the covering row; else insert (fresh install). It never renames
or deletes the operator's row, and folds the resolved row into the in-pass set so a
shared
live == archivedirectory resolves to one row.db::get_storage_by_path(deterministic: oldestcreated_atwins on a shared path)and
db::get_storage_by_name_or_path(name first, then path), and use the name-or-pathvariant at the two runtime sites so a renamed install keeps full behavior and, after the
empty rows are cleaned up, the paths still resolve to the real rows.
normalize_storage_path).UNIQUE(path): existing DBs already carry duplicate paths and a sharedlive == archivelayout is legal.docs/DECISIONS.mdentry added, superseding fix: correct release-prep copy defects and code/compose config drift #557's detect-never-fix stance.Why this cannot orphan footage
Existing segments keep their
storage_id, the default policy keeps itslive_storage_id,ensure_default_policywires the default policy by path, and the fix only ever skipscreating a new empty row — it never mutates or removes any storage row. Referenced
storage rows are further protected by the
ON DELETE RESTRICTFK onsegments.storage_id.Tests
DB-backed tests (gated on
TEST_DATABASE_URL; confirmed they actually run on a throwawayPostgres, not skipped):
seed_is_path_idempotent_against_renamed_rows(the exact fix: correct release-prep copy defects and code/compose config drift #557 scenario run twice: stilltwo rows, operator names untouched, no growth);
seed_fresh_db_creates_both_defaults;seed_retargets_existing_name_in_place;seed_shared_layout_yields_one_row;get_storage_by_path_prefers_oldest(determinism + trailing-slash);get_storage_by_name_or_path_falls_back_to_path(the free-floor / labelling fallback).Plus pure-function tests for
normalize_storage_pathand the trailing-slash-awaredetector.
Gate on the build box:
cargo fmt --all -- --check,cargo clippy --all-targets -D warnings, andcargo test --workspaceagainst a throwaway Postgres all green.Follow-up (not in this PR)
The one-time cleanup of the two already-present empty duplicate rows on an affected
deployment is a maintainer DB action, sequenced after this deploys. A referenced row
cannot be deleted (FK
RESTRICT), and the empty ghosts are unreferenced, so the cleanupis a small, verified transactional
DELETE.