Storage deduplication#374
Merged
Merged
Conversation
- Add generation counter to ConnManager to prevent N goroutines from each reconnecting when they all fail on the same dead connection. Only the first caller reconnects; the rest piggyback on the fresh connection. - Serialize PublishMessage through pubMu when confirms are enabled to prevent orphan confirmations from causing false timeouts in publishAndWaitConfirm. - Fix Debugf format verb in GetDueScheduledBlogs log statement.
Adds SHA-256 dedup per design doc. - Schema 000005: storage_assets + storage_asset_refs (soft-delete, partial unique index). - Storage svc: new database pkg (CheckAsset, RegisterAsset race-safe, Create/Delete/ReplaceAssetRef, UpdateNSFW); gRPC handlers wired. - Consumer: BLOG_DELETE / USER_ACCOUNT_DELETE soft-delete CAS refs; legacy prefix delete kept as fallback for pre-CAS objects. - Gateway storage_v2: hash-first upload (memory <=32MB, spool above), CheckAsset reuse, RegisterAsset + Create/Replace/DeleteAssetRef on mutations; new GET /assets/sha256/:p1/:p2/:fileName. - Proto: new dedup messages + service methods. Follow-ups: reference-counted GC, profile-pic CAS, NSFW scanner.
|
🏷️ [bumpr] |
Signed-off-by: Dave Augustus <devpandey19924u@gmail.com>
Signed-off-by: Dave Augustus <devpandey19924u@gmail.com>
|
🏷️ [bumpr] |
daveaugustus
approved these changes
May 15, 2026
|
🚀 [bumpr] Bumped! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✅ Here's a clean, professional GitHub PR description:
feat: Implement Smart File Storage with Content-Addressable Deduplication
Summary
Introduced global Content-Addressable Storage (CAS) using SHA-256 hashing to eliminate duplicate file storage across the platform. The same image uploaded by multiple users/blogs will now be stored only once.
Key Changes:
storage_assetsandstorage_asset_refstables to separate physical assets from logical references.Benefits:
Important Notes:
Related Issues: [https://github.com//issues/371]