Polecat-owned rolling range partitions for time-series document tables (#386)

[jeremydmiller]

Closes #386. Depends on JasperFx/weasel#401, shipped in Weasel 9.21.0 (bumped here from 9.18.0). SQL Server counterpart to JasperFx/marten#5094.

Problem

PartitionOn(x => x.BucketEnd).ByExternallyManagedRange(boundaries) was the only workable option for a time-series document table: Polecat seeds the partition function/scheme once and the application owns everything after that. For a time-partitioned table "everything after that" is the whole lifecycle — rolling a period forward means ALTER PARTITION SCHEME … NEXT USED followed by ALTER PARTITION FUNCTION … SPLIT RANGE, and retiring an aged one means MERGE RANGE, so applications end up hand-writing partition DDL on a schedule forever.

Opting out of Weasel means opting out of its ordering and dependency management, not just its DDL generation. Field case in JasperFx/CritterWatch#886 (the Marten flavor of the same table): a hand-rolled rebuild re-created a partitioned table in a schema missing a helper function the generated index DDL depended on, and an ordinary computed index failed with a bare 42883. The index was correct; bypassing Weasel was the defect.

What this adds

opts.Schema.For<MetricsSample>()
    .PartitionOn(x => x.BucketEnd)
    // Keep 12 months of history, provision 3 months ahead.
    .ByRollingRange(PartitionPeriod.Month, periodsAhead: 3, periodsBehind: 12);

PartitionPeriod covers Hour, Day, Week, Month, Year. Overloads also take a RollingWindowPolicy directly, or a pre-built ManagedRangePartitions — pass the same manager to several document types to roll all of their tables forward in one pass (and to set Filegroup). A TimeProvider is injectable so the window is testable without waiting on the calendar.

How the two halves are driven

	Driven by	Why
Provision the leading edge	ordinary schema migration	with a manager attached CreateDelta is purely additive, so a rolled-forward window is a SPLIT, never a rebuild of the partition function and every table on it
Retire the trailing edge	PolecatActivator startup pass + Advanced.*	migration never removes data, so the retention half has to be driven separately

The startup pass is gated on the same opt-in as the migration itself (ApplyAllDatabaseChangesOnStartup) — that is how a host says "Polecat owns this schema", and retiring a partition is emphatically a schema change.

For hosts whose processes outlive periodsAhead — an hourly window especially — AdvancedOperations gains ApplyRollingPartitionsAsync, plus RollPartitionsForwardAsync (additive only) and DropAgedRollingPartitionsAsync (retention only) for callers who want the halves separately. Each returns Weasel TablePartitionStatus[], so a partial failure surfaces per table.

SQL Server specifics the implementation respects

• NEXT USED before every SPLIT, or the split fails.
• TRUNCATE before MERGE RANGE, in that order. MERGE RANGE on a partition holding rows moves them into the neighbour rather than reclaiming anything — the opposite of the point. Truncating first deallocates the pages in O(1) and the merge is then metadata-only. A failed truncate deliberately leaves the boundary in place.
• RANGE RIGHT throughout, so a boundary is the inclusive lower bound of its period, matching the half-open [From, To) intervals the policy produces.
• Partition functions and schemes are database-scoped objects named from the table + column, which is why each test owns its own document type.
• A SQL Server RANGE function always spans (-inf, +inf), so unlike PostgreSQL there is no overflow rejection to guard against and no DEFAULT partition to declare.

Design notes

• Managers are discovered from each database's own schema objects rather than tracked on StoreOptions. That stays correct for a document type that grows a rolling window later, and makes re-running the pass unconditionally idempotent. Reference identity is the key, exactly as ManagedRangePartitions.ResolveManagedTables matches tables back to their manager.
• ByRollingRange asserts at configuration time that the partition member is a DateTime/DateTimeOffset, and that a shared manager's column and SQL data type match what the member resolves to. Both turn what would be an opaque error during the first migration into a message that names the member.
• Polecat promotes the partition member into a real column and adds it to the primary key already, so unlike Marten there is no Duplicate(...) call to remember.
• The store-usage descriptor reports RollingRange with the window's current boundaries, since there is no declared list to report.

Acceptance criteria from the issue

• A rolling monthly document table needs no application-authored DDL — host_startup_rolls_forward_and_retires_under_ApplyAllDatabaseChangesOnStartup
• Roll-forward and retirement are policy outcomes, applied idempotently at startup — rolling_the_window_forward_is_additive_and_never_a_rebuild, the_apply_pass_is_idempotent
• Safe under concurrent node starts — concurrent_apply_passes_do_not_throw_and_converge_on_the_window
• Retention reclaim stays a partition operation rather than a mass DELETE — the_additive_and_retention_halves_can_be_run_separately

Docs

docs/documents/partitioning.md gains a strategy-selection table up front, a Rolling time windows section (policy, shared managers, how the two halves are driven, why TRUNCATE precedes MERGE, the retention warning, the Advanced.* cadence escape hatch), and an Externally-managed range partitions section documenting that door for what it is actually for. The stale "dropping aged partitions is not yet wired into the document API" limitation is gone.

Tests

11 new tests in src/Polecat.Tests/Storage/rolling_range_partitioning_tests.cs, driving the window with a mutable TimeProvider rather than the calendar: window shape on first migration, documents landing per period, overflow rows, additive roll-forward with data survival, a shared manager rolling two tables, the two halves run separately, idempotency, concurrent passes, the full host-startup pass, and the two configuration-time rejections.

Green on net10.0: Polecat.Tests 1623/0 (3 pre-existing skips), Polecat.EntityFrameworkCore.Tests 37/0 — the full suite also serving as the regression gate on the Weasel 9.18.0 → 9.21.0 bump, which picks up weasel#391 (managed tenant bucketing actually sharing a partition, #335's ordinal sharing) and weasel#399 along the way.

Known consumer: JasperFx/CritterWatch hand-writes exactly this for pc_doc_metricssample in its SQL Server flavor and can delete that code.

🤖 Generated with Claude Code