Require `source` parameter for SDK initialisation for namespace isolation

[rvagg]

Pulling out of here so this doesn't get lost: #495 (comment)

Goal & background

We have a problem of multiple dapps using a single wallet not having clear separation of which app owns which data sets or data. We lack a single solution that gives us namespacing and any kind of on-chain ecosystem telemetry.

The linked comment comes from a discussion where this problem came up again and we decided that instead of coming up with messy contract-level solutions, that we should tackle it in the SDK and just require that the developer inject their namespace (unless they explicitly opt out).

Proposal

The solution we're using in Filecoin Pin is to always put source = filecoin-pin on the data set metadata and then those can be partitioned from anything else as needed. So the proposal is to extend that to the sdk (not core) and require it on initialisation:

source: string | null required on Synapse.create()

// Must provide — TypeScript enforces it
Synapse.create({ account, source: 'my-dapp' })     // identified
Synapse.create({ account, source: null })          // conscious opt-out
Synapse.create({ account })                        //  <== type error, must decide

It's mildly hostile, but this is a product-level problem that we need to solve, and this is the least messy version without going to an FWSSv2 design. null is easy and this is just a type constraint not a checked requirement (we could choose to harden this when we look at behaviour patterns and it's not up to scratch).

Partitioning happens at read and write:

• Any dapp or tool wanting to operate on a specific dapp's data can query by source and isolate its business to that; this includes dashboards that want to present a global view, they can partition by source.
• We do automatic data set selection by metadata matching, so a user using two dapps both using the SDK would get different data sets automatically for their dapps, with only the developers of those dapps having to provide a source.

We would follow the withCDN boolean down through the classes, this is how it works, we'd just do the same with source:

• Synapse.create({ withCDN })
  • stored as this._withCDN
    • passed to StorageManager constructor`
      • passed to combineMetadata(metadata, { withCDN })
        • adds { withCDN: '' } key to dataset metadata
          • metadataMatches() naturally scopes dataset reuse

Note also that there are ways that this can be overridden through that same chain, we could do the same. upload(data, { source: 'newsource' }) should be possible to override your original source. For now, let's exclude this override behaviour from an initial version of this and keep it simple.

We need to add SOURCE: 'source' to METADATA_KEYS, our list of common keys.

Caveats

• Empty string ('') would be treated the same as null: source isn't recorded at all on chain, no waste.
• When someone starts doing this and has existing data sets that don't have source on them, they're going to end up with new data sets because we match on metadata

[BigLep]

Thanks @rvagg . This makes sense. I assume we want to include this with M4.0 because we want builders to adopt this paradigm ASAP or at least explicitly opt out. Please adjust if that seems wrong. I also gave an initial estimate.

[a-simptoad]

Hey, Can i work on this issue?

[rvagg]

sorry @a-simptoad, not ready for this yet, it's parked as an issue until we land existing work so we can work on top of it.

[rvagg]

#647