From 29db33269547d19dcc7e87c5db84a7fe3f72fc0d Mon Sep 17 00:00:00 2001 From: bravonatalie Date: Tue, 13 Jan 2026 21:39:40 +0700 Subject: [PATCH 1/4] rfc: initial proposal for refactoring the space-diff table --- rfc/refactor-space-diff-table.md | 172 +++++++++++++++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 rfc/refactor-space-diff-table.md diff --git a/rfc/refactor-space-diff-table.md b/rfc/refactor-space-diff-table.md new file mode 100644 index 0000000..f5d4609 --- /dev/null +++ b/rfc/refactor-space-diff-table.md @@ -0,0 +1,172 @@ +# RFC: Space Diff Refactoring + +## Authors + +- [Natalie Bravo](https://github.com/bravonatalie), [Storacha Network](https://storacha.network/) + +## Language + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119](https://datatracker.ietf.org/doc/html/rfc2119). + +## Introduction + +The current `space-diff` table has accumulated some structural and operational issues that impact billing, usage calculation, and system reliability. This RFC proposes structural changes to make usage calculation efficient, prevent duplicate diffs, and simplify long-term maintenance. + +### Problem Statement + +### 1. Duplicate space diffs + +Past bugs caused multiple diffs to be written for the same cause (e.g. failed uploads). This resulted in duplicated diffs that inflate usage, slow down queries and create “ghost” usage for spaces that should be empty after deletion. + +This behavior should be **structurally impossible** going forward. + +### 2. Usage calculation timeouts + +A single space can generate a very large number of diff entries within the current month. When this happens, usage record calculation often times out because the system needs to aggregate too many records. + +**Current mitigation (temporary):** + +* A *space diff compaction* script that: + * Aggregates many diffs into a single “summary” diff. + * Archives the original diffs into a separate table. + +This is an ad-hoc workaround and not a long-term solution. + +## Current `space-diff` usage model + +The `space-diff` table is the single **source of truth** for billing. It is written to by different sources depending on the protocol. + +### Source A: Modern Blob Protocol + +``` +blob/accept OR blob/remove → blob-registry.register() + 1. allocation table entry (legacy compatibility) + → TransactWrite { + 2. blob-registry table entry (primary storage) + 3. space-diff table entry (billing) + } + +``` + +- **Location**: `upload-api/stores/blob-registry.js` + +### Source B: Legacy Store Protocol + +Deprecated, but still operational for existing clients. + +``` +store/add OR store/remove receipt → UCAN stream → ucan-stream-handler → space-diff table + +``` + +- **Location**: `billing/functions/ucan-stream.js` + +### How usage is calculated today + +This flow is used during billing runs for each space: + +**Initial state** + +* Load the space snapshot from `space-snapshot` for the `from` date +* If no snapshot exists, assume the space was empty (`size = 0`) + +**Usage calculation** + +* Base usage = `initialSize × periodDurationMs` +* Fetch all space diffs for the billing period +* Iterate diffs in chronological order: + * `size += diff.delta` + * `usage += size × timeSinceLastChange` + * where `timeSinceLastChange = diff.receiptAt - lastReceiptAt` + +**Storage** + +* Store final space size in `space-snapshot` with `recordedAt = to` +* Store total usage in `usage` (byte-milliseconds) + +## Proposal + +### Fix for problem 1: Duplicate diffs + +To guarantee uniqueness and prevent future duplication: + +* Use **`cause` as the sort key (SK)** of the `space-diff` table +* This makes it impossible to insert two diffs for the same `(space, cause)` pair + +#### Open design concern + +Using `cause` as the SK removes natural chronological ordering. + +**Proposed solution** + +* Add a **GSI with a timestamp-based sort key** + +This enables: + +* Efficient chronological queries +* Time-based pagination +* Retention policies (e.g. deleting data older than 1 year) + +The additional cost is acceptable, especially since older diffs can be safely deleted after the retention window. + +#### Migration plan (high level) + +1. Create a **new `space-diff` table** with: + * Correct PK design + * `cause` as SK + * GSI for timestamp-based queries +2. Export data from the existing table +3. Deduplicate and transform records +4. Import data into the new table +5. Update application code to use the new schema +6. Decommission the old table after validation is complete + +### Fix for problem 2: Usage calculation timeouts + +Introduce a new table (e.g. `space-usage-month`) keyed by `provider#space#YYYY-MM` that is updated atomically on each diff write, making billing reads **O(1)**. + +#### Core idea + +Maintain a running usage accumulator instead of scanning historical diffs. + +**Algorithm** + +1. Track `lastSize` and `lastChangeAt` per `(provider, space, month)` +2. On each incoming diff: + * `usage += lastSize × (receiptAt - lastChangeAt)` + * `lastSize += delta` + * `lastChangeAt = receiptAt` +3. At end-of-month billing: + * `usage += lastSize × (periodEnd - lastChangeAt)` + * Finalize and snapshot + +**Additional fields** + +* `sizeStart` +* `sizeEnd` +* `lastReceiptAt` +* `subscription` + +**Behavior** + +* `space-diff` remains for audit and idempotency +* Billing reads exclusively from `space-usage-month` +* `calculatePeriodUsage`: + * First tries the aggregator + * Falls back to a GSI scan if missing +* Aggregator becomes the canonical source for the billing month + +**Retention** + +* Keep `space-diff` entries for N months using TTL +* Archive older diffs to S3 (TBD) + +**Considerations** + +- The accumulator MUST process diffs for a space in **ascending** `receiptAt` order. If the write path can deliver out-of-order events and strict ordering cannot be guaranteed, this solution SHOULD be revisited. Pragmatic mitigations include: + - Buffer within a small window and sort incoming diffs. + - Recompute a localized suffix by reading recent diffs via the time GSI and re-applying from the last stable checkpoint. + +- Alternative when strict ordering is infeasible: + - Use time-bucketed diffs (hour/day): persist per-bucket, order-independent aggregates (e.g., Σdelta and Σ(delta × (bucketEnd − receiptAt))). At billing time, iterate buckets in chronological order to compute exact monthly usage, where no event sorting required. + - Maintain a size-only monthly state (track `lastSize` and `lastChangeAt`) to accelerate space usage report. Note: this does NOT remove the need to iterate diffs for the billing run. From d988b3c5da5780ac9e98adff10da0a96e8239769 Mon Sep 17 00:00:00 2001 From: bravonatalie Date: Wed, 14 Jan 2026 23:23:01 +0700 Subject: [PATCH 2/4] fix: add adjustments based on the comments --- rfc/refactor-space-diff-table.md | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/rfc/refactor-space-diff-table.md b/rfc/refactor-space-diff-table.md index f5d4609..8cc9c57 100644 --- a/rfc/refactor-space-diff-table.md +++ b/rfc/refactor-space-diff-table.md @@ -115,11 +115,8 @@ The additional cost is acceptable, especially since older diffs can be safely de * Correct PK design * `cause` as SK * GSI for timestamp-based queries -2. Export data from the existing table -3. Deduplicate and transform records -4. Import data into the new table -5. Update application code to use the new schema -6. Decommission the old table after validation is complete +2. Enable dual-writes: on each diff event, write to both the existing table and the new table. Keep all readers (usage, reporting, billing) pointed at the existing table during January. +3. Cut over in February: switch usage reporting and billing reads to the new table; keep the existing table as read-only historical storage. ### Fix for problem 2: Usage calculation timeouts @@ -163,10 +160,4 @@ Maintain a running usage accumulator instead of scanning historical diffs. **Considerations** -- The accumulator MUST process diffs for a space in **ascending** `receiptAt` order. If the write path can deliver out-of-order events and strict ordering cannot be guaranteed, this solution SHOULD be revisited. Pragmatic mitigations include: - - Buffer within a small window and sort incoming diffs. - - Recompute a localized suffix by reading recent diffs via the time GSI and re-applying from the last stable checkpoint. - -- Alternative when strict ordering is infeasible: - - Use time-bucketed diffs (hour/day): persist per-bucket, order-independent aggregates (e.g., Σdelta and Σ(delta × (bucketEnd − receiptAt))). At billing time, iterate buckets in chronological order to compute exact monthly usage, where no event sorting required. - - Maintain a size-only monthly state (track `lastSize` and `lastChangeAt`) to accelerate space usage report. Note: this does NOT remove the need to iterate diffs for the billing run. +To avoid potential race conditions when two diffs for the same space read the current total at the same time, one option is to process diffs through a queue. This would also help preserve the correct ordering of diffs. From dae4e9902dcb97444133e98bbc4293d8f24e315e Mon Sep 17 00:00:00 2001 From: bravonatalie Date: Thu, 22 Jan 2026 22:59:23 +0700 Subject: [PATCH 3/4] fix: adjust usage calculation timeout solution --- rfc/refactor-space-diff-table.md | 42 ++++++-------------------------- 1 file changed, 8 insertions(+), 34 deletions(-) diff --git a/rfc/refactor-space-diff-table.md b/rfc/refactor-space-diff-table.md index 8cc9c57..b4394a4 100644 --- a/rfc/refactor-space-diff-table.md +++ b/rfc/refactor-space-diff-table.md @@ -120,44 +120,18 @@ The additional cost is acceptable, especially since older diffs can be safely de ### Fix for problem 2: Usage calculation timeouts -Introduce a new table (e.g. `space-usage-month`) keyed by `provider#space#YYYY-MM` that is updated atomically on each diff write, making billing reads **O(1)**. +Generate snapshots more frequently. -#### Core idea +One option is to move snapshot generation to a daily cadence. There are two possible approaches: -Maintain a running usage accumulator instead of scanning historical diffs. +1. **Decouple snapshot generation from the billing cron** -**Algorithm** + * Generate snapshots independently, without running the full billing pipeline -1. Track `lastSize` and `lastChangeAt` per `(provider, space, month)` -2. On each incoming diff: - * `usage += lastSize × (receiptAt - lastChangeAt)` - * `lastSize += delta` - * `lastChangeAt = receiptAt` -3. At end-of-month billing: - * `usage += lastSize × (periodEnd - lastChangeAt)` - * Finalize and snapshot +2. **Run the full billing process daily** -**Additional fields** + * This would naturally produce more snapshots and also push usage reports to Stripe more frequently -* `sizeStart` -* `sizeEnd` -* `lastReceiptAt` -* `subscription` +Since both approaches are pretty similar and would need to iterate over all customers and spaces to generate snapshots anyway, the main extra work with running the full billing flow is the usage calculation, writing usage records, and reporting to Stripe. -**Behavior** - -* `space-diff` remains for audit and idempotency -* Billing reads exclusively from `space-usage-month` -* `calculatePeriodUsage`: - * First tries the aggregator - * Falls back to a GSI scan if missing -* Aggregator becomes the canonical source for the billing month - -**Retention** - -* Keep `space-diff` entries for N months using TTL -* Archive older diffs to S3 (TBD) - -**Considerations** - -To avoid potential race conditions when two diffs for the same space read the current total at the same time, one option is to process diffs through a queue. This would also help preserve the correct ordering of diffs. +Given the upside of reporting to Stripe more frequently and the fact that this is simpler than setting up separate infra just for snapshot generation, we’ll move forward with option two. \ No newline at end of file From 36c2c21f02c245a72d6acce72d7ff42deba9cef4 Mon Sep 17 00:00:00 2001 From: bravonatalie Date: Tue, 10 Feb 2026 22:49:10 +0700 Subject: [PATCH 4/4] fix: remove usage timeout discussion and clarify different timeframe solutions for space-diff duplication --- rfc/refactor-space-diff-table.md | 75 +++++++++----------------------- 1 file changed, 21 insertions(+), 54 deletions(-) diff --git a/rfc/refactor-space-diff-table.md b/rfc/refactor-space-diff-table.md index b4394a4..6794384 100644 --- a/rfc/refactor-space-diff-table.md +++ b/rfc/refactor-space-diff-table.md @@ -1,4 +1,4 @@ -# RFC: Space Diff Refactoring +# RFC: Space Diff Deduplication ## Authors @@ -10,28 +10,14 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S ## Introduction -The current `space-diff` table has accumulated some structural and operational issues that impact billing, usage calculation, and system reliability. This RFC proposes structural changes to make usage calculation efficient, prevent duplicate diffs, and simplify long-term maintenance. +The current `space-diff` table has accumulated structural issues that impact billing and system reliability due to duplicate entries. This RFC proposes changes to prevent duplicate diffs and simplify long-term maintenance. ### Problem Statement -### 1. Duplicate space diffs - -Past bugs caused multiple diffs to be written for the same cause (e.g. failed uploads). This resulted in duplicated diffs that inflate usage, slow down queries and create “ghost” usage for spaces that should be empty after deletion. +Past bugs caused multiple diffs to be written for the same cause (e.g. failed uploads). This resulted in duplicated diffs that inflate usage, slow down queries and create "ghost" usage for spaces that should be empty after deletion. This behavior should be **structurally impossible** going forward. -### 2. Usage calculation timeouts - -A single space can generate a very large number of diff entries within the current month. When this happens, usage record calculation often times out because the system needs to aggregate too many records. - -**Current mitigation (temporary):** - -* A *space diff compaction* script that: - * Aggregates many diffs into a single “summary” diff. - * Archives the original diffs into a separate table. - -This is an ad-hoc workaround and not a long-term solution. - ## Current `space-diff` usage model The `space-diff` table is the single **source of truth** for billing. It is written to by different sources depending on the protocol. @@ -61,34 +47,33 @@ store/add OR store/remove receipt → UCAN stream → ucan-stream-handler → sp - **Location**: `billing/functions/ucan-stream.js` -### How usage is calculated today +## Proposal + +### Short-term solution: Deduplicate on write using a GSI -This flow is used during billing runs for each space: +Add a **GSI on `cause`** to the existing `space-diff` table. Before inserting a new diff, query the GSI to check whether a diff with the same `cause` already exists. If it does, skip the write. -**Initial state** +This approach: -* Load the space snapshot from `space-snapshot` for the `from` date -* If no snapshot exists, assume the space was empty (`size = 0`) +* Prevents new duplicates without changing the table schema +* Can be deployed quickly with minimal risk +* Does not require a migration or dual-write strategy -**Usage calculation** +**Limitation:** This is a best-effort guard — it adds a read-before-write cost and does not structurally prevent duplicates (a race condition is still theoretically possible). -* Base usage = `initialSize × periodDurationMs` -* Fetch all space diffs for the billing period -* Iterate diffs in chronological order: - * `size += diff.delta` - * `usage += size × timeSinceLastChange` - * where `timeSinceLastChange = diff.receiptAt - lastReceiptAt` +### Medium-term solution: TTL-based archival to Glacier -**Storage** +Add a **TTL attribute** to the `space-diff` table so that items older than 1 year are automatically expired by DynamoDB. Before expiration, use a **DynamoDB Streams + Lambda** pipeline to archive expired items to S3 Glacier. -* Store final space size in `space-snapshot` with `recordedAt = to` -* Store total usage in `usage` (byte-milliseconds) +This approach: -## Proposal +* Keeps the table lean over time, improving query performance +* Reduces storage costs for historical data +* Supports retention policies without manual cleanup -### Fix for problem 1: Duplicate diffs +### Long-term solution: New table with structural uniqueness -To guarantee uniqueness and prevent future duplication: +To guarantee uniqueness and prevent future duplication at the schema level: * Use **`cause` as the sort key (SK)** of the `space-diff` table * This makes it impossible to insert two diffs for the same `(space, cause)` pair @@ -116,22 +101,4 @@ The additional cost is acceptable, especially since older diffs can be safely de * `cause` as SK * GSI for timestamp-based queries 2. Enable dual-writes: on each diff event, write to both the existing table and the new table. Keep all readers (usage, reporting, billing) pointed at the existing table during January. -3. Cut over in February: switch usage reporting and billing reads to the new table; keep the existing table as read-only historical storage. - -### Fix for problem 2: Usage calculation timeouts - -Generate snapshots more frequently. - -One option is to move snapshot generation to a daily cadence. There are two possible approaches: - -1. **Decouple snapshot generation from the billing cron** - - * Generate snapshots independently, without running the full billing pipeline - -2. **Run the full billing process daily** - - * This would naturally produce more snapshots and also push usage reports to Stripe more frequently - -Since both approaches are pretty similar and would need to iterate over all customers and spaces to generate snapshots anyway, the main extra work with running the full billing flow is the usage calculation, writing usage records, and reporting to Stripe. - -Given the upside of reporting to Stripe more frequently and the fact that this is simpler than setting up separate infra just for snapshot generation, we’ll move forward with option two. \ No newline at end of file +3. Cut over later: switch usage reporting and billing reads to the new table; keep the existing table as read-only historical storage.