Global content-addressed file dedup across sites

[MudDev]

Spun out from #10 (point 11, raised by @mx5kevin).

Problem

Today every site stores its own copy of a file, keyed by (site_address, sha512). If the same large file (video, dataset, PDF) is referenced from multiple sites, each peer ends up storing N copies on disk and uploading it N times across the network.

This is especially wasteful for:

• Video sharing sites re-uploading the same clips
• Mirror sites and content aggregators
• Popular optional-file downloads

Proposal

Introduce a global content-addressed store where files are stored once per local peer, keyed by SHA512, and multiple sites reference them by hash.

Approach sketch

• New storage tier: data/_content/<sha512_prefix>/<sha512> with a refcount per site
• content.json file entries can reference a global hash instead of (or in addition to) a site-relative path
• When a site is removed, decrement refcounts; when refcount hits zero, file is a candidate for GC
• Push swarm (see Enhance error handling in PeerPortchecker and TorManager; update Open… #2 in Known issues in the predecessor system of EpixNet #10) serves files by hash regardless of which site requested them

Design questions

• Backwards compatibility: old sites keep working, new sites opt into global hashes
• Security: a malicious site can't "poison" the global store because hashes are verified
• Storage accounting: does a 1GB file count once or N times against the user's storage budget?
• Discovery: how do peers announce they have a hash that isn't tied to a specific site they've joined?

Non-goals

• Not a full CAS redesign of EpixNet storage, this is additive and opt-in
• Not required for small files where the overhead wouldn't pay off

Related

• Composes well with Known issues in the predecessor system of EpixNet #10 point 8 (magnet/IPFS bridging) since those systems are already content-addressed

Credit: @mx5kevin in #10.