The ipfs2filecoin campaign path is: download source content, pack ~1016 MiB aggregate CARs locally, upload each to the primary SP, and batch up to 40 parked pieces into one addPieces transaction. synapse-sdk already covers streaming upload and the second copy (upload(stream, {copies: 2}) uploads to the primary and has each secondary pull from it), so what is missing here is the scheduling.
Curio garbage-collects parked pieces that have not been added on chain. The window is roughly 2h, SP-configurable, and not discoverable. So the client has to flush before its own best guess expires.
Loop
Stamp parkedAt per piece on upload completion (client clock; nothing better is available). Flush when any of:
- batch reaches 40 pieces
now >= min(parkedAt) + assumed_window - margin- no work left
margin must cover build plus confirmation, because the transaction has to land before GC. Derive it from confirmations observed earlier in the same run rather than a constant.
Window estimation
Start well under 2h. On a detected GC, lower assumed_window and persist it per provider in the migration DB. Never raise it automatically: a successful run proves only that you stayed under the window, not that it is longer.
The default should be conservative because the costs are asymmetric. An early flush costs one extra transaction; a collected piece costs a full 1016 MiB re-upload, and that risk peaks exactly when the user's upload is slowest.
Uploads slower than the window
If one aggregate cannot be parked and confirmed inside the window, the flow cannot work at all:
floor = 1016 MiB / (assumed_window - margin) # ~1.3 Mbit/s at 2h with a 10 min margin
Below that, retain locally and upload in a burst instead, per rvagg's "don't dribble this in, retain locally if it's going to be slow". That is a different mode, not a parameter, so analyze should pick between them up front.
Open question
How does a collected piece manifest? Whether it surfaces as a failed addPieces, a missing piece on a pre-flight check, or something else determines how the GC signal is detected and how the lost aggregate is re-queued. Needs checking against Curio.
Not a concern
Gas is not the client's cost. SP.addPieces() posts to the provider's serviceURL with the client's EIP-712 authorization in extraData, so the provider submits the transaction and pays for it. A base-fee gate would therefore pause to save the provider gas while costing the user a full re-upload of every piece collected in the meantime. Do not gate this path on base fee.
Related: #69 (hosted CAR aggregator, which would remove the local-disk requirement entirely).
The ipfs2filecoin campaign path is: download source content, pack ~1016 MiB aggregate CARs locally, upload each to the primary SP, and batch up to 40 parked pieces into one
addPiecestransaction.synapse-sdkalready covers streaming upload and the second copy (upload(stream, {copies: 2})uploads to the primary and has each secondary pull from it), so what is missing here is the scheduling.Curio garbage-collects parked pieces that have not been added on chain. The window is roughly 2h, SP-configurable, and not discoverable. So the client has to flush before its own best guess expires.
Loop
Stamp
parkedAtper piece on upload completion (client clock; nothing better is available). Flush when any of:now >= min(parkedAt) + assumed_window - marginmarginmust cover build plus confirmation, because the transaction has to land before GC. Derive it from confirmations observed earlier in the same run rather than a constant.Window estimation
Start well under 2h. On a detected GC, lower
assumed_windowand persist it per provider in the migration DB. Never raise it automatically: a successful run proves only that you stayed under the window, not that it is longer.The default should be conservative because the costs are asymmetric. An early flush costs one extra transaction; a collected piece costs a full 1016 MiB re-upload, and that risk peaks exactly when the user's upload is slowest.
Uploads slower than the window
If one aggregate cannot be parked and confirmed inside the window, the flow cannot work at all:
Below that, retain locally and upload in a burst instead, per rvagg's "don't dribble this in, retain locally if it's going to be slow". That is a different mode, not a parameter, so
analyzeshould pick between them up front.Open question
How does a collected piece manifest? Whether it surfaces as a failed
addPieces, a missing piece on a pre-flight check, or something else determines how the GC signal is detected and how the lost aggregate is re-queued. Needs checking against Curio.Not a concern
Gas is not the client's cost.
SP.addPieces()posts to the provider'sserviceURLwith the client's EIP-712 authorization inextraData, so the provider submits the transaction and pays for it. A base-fee gate would therefore pause to save the provider gas while costing the user a full re-upload of every piece collected in the meantime. Do not gate this path on base fee.Related: #69 (hosted CAR aggregator, which would remove the local-disk requirement entirely).