[Feature]: dedupe CIDs in a run and against what the account already stores

[SgtPooki]

Storing the same bytes twice means paying twice. In a real 2.2M-CID inventory, 61,766 CIDs (~2.9%) appeared in two different spaces of the same account, so the overlap is routine rather than theoretical.

Two parts, with very different shapes.

1. Dedupe within a run

Mostly built. File intake canonicalizes each line through toCanonicalCidV1 and dedupes on that form (packages/core/src/cid-list.ts), and the CLI inserts with ON CONFLICT(cid) DO NOTHING (packages/cli/src/db.ts:236).

The gap is the console's union. cid-list.ts deliberately returns the input spelling, and the cids memo in app/src/app.tsx dedupes the paste plus file union with new Set over raw strings. Qm…X pasted and bafybei…X loaded from a file are the same content, so they survive as two rows, become two pieces, and get billed twice. Dedupe the union on the canonical form, keyed the same way rows and saved results are.

2. Skip CIDs the account already stores

Design is open, and the obvious approach does not scale.

Chain is the canonical record: every piece we add carries its ipfsRootCID as piece metadata (app/src/submit.ts:190). But getPieceMetadata(dataSetId, pieceId, key) is a forward lookup, and there is no reverse index from a root CID back to a piece. Answering "does this account already store X?" by walking the account's pieces costs one contract read per piece. At millions of pieces that is millions of reads and RPC exhaustion. Not viable.

IPNI (cid.contact) is not a substitute on its own, for two reasons:

• It reports announcement, not possession. --check-ipni is documented as exactly that in packages/cli/src/report.ts; possession is proven by PDP, not by a routing record.
• It is not account-scoped. A record can come from the operator's old pinning service or any unrelated party. Treating that as "already stored" would skip a CID the account does not hold and is not paying for, and the data would silently never land. That is the failure this tool exists to prevent.

Options to weigh:

1. The run manifest (shipped in browser and CLI). Exact and free for runs this tool performed; useless if the operator lost it or migrated by other means. Couples to the open manifest question.
2. Store the manifest as a piece in the data set, giving the account its own reverse index: one retrieval, reconciled against chain.
3. IPNI as a cheap prefilter only, then confirm the candidates against the account's own data sets on chain. Bounds the reads to candidates, but inherits the false-positive risk above.
4. An indexer or subgraph over piece metadata.

Done means

• A run never prepares the same content twice, whatever the CID spelling or which input carried it.
• Before preparing, an operator sees how many of their CIDs their account already stores, computed without a per-CID chain read.
• A skip is backed by evidence the account holds the piece, never by an announcement record.

[SgtPooki]

Priced the second part against the real inventory before building anything. It does not earn its cost.

What a duplicate is worth. In the 2.2M-CID inventory, the 61,766 CIDs present in two spaces come to 11.6 GB, which is 1.32% of the 877.6 GB corpus. At the mainnet FWSS rate of 2.5 USDFC/TiB/month:

	bytes	USDFC/month
the duplicates	11.6 GB	0.026
whole corpus	877.6 GB	2.00

Storing every duplicate costs about 3 cents a month, or 32 cents a year. The duplicates are also small: median 54 KiB, largest 19.5 MB.

What the check would cost. Piece metadata carries ipfsRootCID, but getPieceMetadata(dataSetId, pieceId, key) is a forward lookup with no reverse index, so a naive account scan is one contract read per piece: millions of reads and RPC exhaustion at this scale. Avoiding that needs a provider/routing prefilter, a possession probe, and an account-scoped chain confirmation, plus a guarantee that a false positive can never skip a CID the account does not hold. That is real design work to save cents.

Recommendation: split this issue.

• Keep and do now: in-run dedupe. It is cheap, nearly built, and it already catches the case observed here. Feeding both lists to one run dedupes them; the only gap is the console union in app/src/app.tsx, which dedupes paste plus file over raw strings while cid-list.ts returns input spellings, so Qm…X and bafybei…X survive as two pieces for identical content.
• Deprioritize: the already-stored check. Revisit if a cheap reverse index shows up (an indexer over piece metadata, or storing the manifest as a piece so an account carries its own index), or if someone reports a duplicate bill that actually hurts. Not worth blocking on.

A caveat on the framing: the money argument holds for storage. It does not cover wasted migration time or provider bandwidth on a re-run, which is the more plausible reason to want this later.

[SgtPooki]

Part 1 landed in 1fd26ec: the console union now dedupes on the canonical CIDv1 form, first occurrence keeps its spelling. Cross-dataset dedupe (part 2) remains open.