[Feature]: parallel p2p retrieval for console prepare (verified-fetch)

[SgtPooki]

Description

Console prepare pulls every root as one streaming ?format=car request from the single configured gateway, and the per-block gap-fill fallback hits the same host (app/src/commp.ts, packages/core/src/car-stream-source.ts). Retrieval concurrency is pinned to the hash pool: CONCURRENCY = HASH_POOL_SIZE = 4 in app/src/app.tsx. #57 made a 2,159,085-CID run manageable in the UI, but a live run against that inventory sat at ~400 ready after a few minutes — roughly 2 CIDs/s, which extrapolates to nearly two weeks for the full list, with every byte routed through one central endpoint.

Impact

Retrieval throughput is now the binding constraint on million-CID runs, and one gateway is both the bottleneck and a single point of rate limiting for the whole migration.

Next steps

• Fetch blocks in the browser with @helia/verified-fetch over js-libp2p so retrieval spreads across providers instead of one host; verification stays cryptographic per block. Adopt @helia/verified-fetch to replace custom trustless-gateway polling #19 evaluated the same move for the CLI.
• Decouple retrieval concurrency from the hash pool: hashing is CPU-bound (4 workers is right), but many more fetches can be in flight feeding those workers, especially with a 63 KiB median root.
• Invariant to hold: the sourceUrl handed to providers, and the canonical CAR bytes it serves, must not change — how the console verifies locally is independent of what the provider pulls (docs/onchain.md).

[SgtPooki]

The concurrency-decoupling step landed in bfab621. computePiece no longer claims a hash worker before fetching: a piece buffers its canonical CAR up to 4 MiB and hands off to a worker only when there are bytes to hash, so small pieces (the bulk of real inventories, 63 KiB median) never hold a CPU core through network time. The prepare pool went from 4 (one per hashing core) to 32.

Measured against the default gateway, disjoint 800–1,600-CID slices of a real inventory per configuration so caching could not bias the A/B:

in flight	sustained rate	failures
4 (before)	~2.0/s	0
16	~10/s	0
32 (now)	~18/s	0

Scaling is close to linear through 32, confirming the gateway round-trip — not bandwidth, CPU, or the hash pool — is the wait. A 2,159,085-CID run drops from ~13 days to ~33 hours. UI checks at 2M still pass (0.9s to first rows, 27ms click latency, reload-restore intact).

Leaving this open for the verified-fetch/js-libp2p work: 32 streams into one host is the polite ceiling for a central endpoint, and spreading retrieval across providers is what removes both the remaining wait and the single point of rate limiting.

[SgtPooki]

Provider-spread retrieval landed in c4da0f9: each root's CAR is requested from the providers delegated routing names for it (transport-ipfs-gateway-http, https-only, every block still hash-verified), with the configured gateway as the last candidate and gap-fill pinned to it. Discovery is best-effort — any lookup failure or timeout and the root goes straight to the gateway.

Validation, all on real inventory slices:

• Commitments: identical cid → pieceCid maps (100/100) across the pre-concurrency build, the concurrency build, and the provider-discovery build. Retrieval source cannot affect what gets committed.
• Load spread: on an 800-root run, 542 CARs came from the content's provider host, 259 from the configured gateway, 4 from a third host — roughly two-thirds of the bytes off the central funnel.
• Sustained behavior (10-minute soak, gateway-only, warm slice): the single gateway degrades under one client's sustained 32 streams — 19/s falling into stall/kill cycles at ~6/s, with the 120s watchdog reaping dozens of streams.
• Sustained behavior (55-minute soak, provider spread, fully cold 10k slice): 9,735 ready / 263 failed (97.4% first pass, failures are watchdog kills that requeue via Retry), steady ~3.2/s with no page errors and flat heap.

The warm/cold gap is the finding to carry forward: 13–19/s on cache-warmed slices vs ~3.2/s cold means the remaining wait is origin-side cold serving, not client parallelism. Next increments worth measuring: higher in-flight counts now that load is spread across hosts, and bitswap over browser-dialable transports — a 20-root probe of this corpus found WSS addrs on 19 and WebTransport on 11, so a js-libp2p path has real coverage when we want it.

[SgtPooki]

Bitswap-over-WebSockets landed in 840d536 as the last rescue tier, and it works in the browser.

How it's wired: the per-root routing lookup now also collects browser-dialable peer multiaddrs (/tls/ws AutoTLS listeners, /wss). Retrieval order per block is unchanged HTTP first — provider CAR, gateway CAR, gateway ?format=raw — and only when all of those fail does the console dial the root's own peers and issue a bitswap want. Peers are dialed explicitly from the discovery record rather than through helia's routed broker: on this corpus the routed path trips over a stale wss provider record and peers that publish empty Protocols, while a direct dial to the same peers answers in a few hundred milliseconds (verified in Node with browser-equivalent transports before wiring). The libp2p stack is dynamic-imported — its ~800 KB of chunks load the first time a run actually needs the rescue; the main bundle is unchanged.

Verified in a real browser: with every HTTP byte source blocked (gateway, provider gateways — only the routing endpoint reachable), 9 of 10 roots prepared end-to-end over WSS bitswap, and every piece matched the gateway path's commitment exactly (cid → pieceCid + rawSize, 0 mismatches). Offline error paths, bulk retry, and the 2M reload-restore all still pass; suite green.

Still open here: WebTransport as a second dialable transport (11/20 corpus coverage), and whether the rescue should be offered earlier than last-resort for roots whose HTTP candidates are known-dead.

[SgtPooki]

Retrieval is now a hedged race per root (8e2c849, app/src/source-race.ts). Every source can serve the same verified blocks, so the fastest one does — but candidates start staggered instead of simultaneously: the root's provider gateways lead (400ms apart), the community gateway joins at 3s, and bitswap joins last (3.4s, since a bitswap win walks the DAG per-block, the most expensive mode). A tier whose predecessors have all already failed starts immediately, so dead sources cost nothing. First block wins; losers abort. The per-block hedge is winner-aware: a bitswap-walked root leads with its peers and holds the gateway raw fetch back 1.5s as insurance, an HTTP gap does the reverse.

The offsets matter more than the race. Tuning them was measured, same-day cold slices throughout:

configuration	800 cold roots	community-gateway requests
sequential candidates (before)	799 in ~60s	226
race, community at 900ms	on par	439
race, bitswap as competitive tier	on par	4,881
race, tuned offsets (shipped)	790 in ~40s	247

So: fastest configuration measured, with shared-gateway traffic at the sequential baseline. The all-HTTP-dead rescue also dropped from 48s to 18s for 10 roots (failed tiers promote instantly instead of timing out in order). Commitments verified identical on the race path (100/100) and the bitswap-only path (9/9); offline/retry/reload probes and the full suite green.