Description
upload runs plan, then packs the entire CID list into CAR files, then starts uploading. Peak staging disk is therefore roughly the total size of the migration, even though CARs are evicted as copies commit during the upload phase.
Impact
The runbook has to tell users to budget free disk equal to their data size. Issue #70 set out to keep the disk footprint near the pack target ("upload each ASAP, to keep disk small"), and #69 (hosted aggregator) assumes the local flow is disk-light for most users.
Proposed next step
Interleave: build one bin, hand it to the upload loop, build the next while the upload streams. The commit batcher already tolerates pieces arriving over time, so only the pack scheduling moves.
Description
uploadruns plan, then packs the entire CID list into CAR files, then starts uploading. Peak staging disk is therefore roughly the total size of the migration, even though CARs are evicted as copies commit during the upload phase.Impact
The runbook has to tell users to budget free disk equal to their data size. Issue #70 set out to keep the disk footprint near the pack target ("upload each ASAP, to keep disk small"), and #69 (hosted aggregator) assumes the local flow is disk-light for most users.
Proposed next step
Interleave: build one bin, hand it to the upload loop, build the next while the upload streams. The commit batcher already tolerates pieces arriving over time, so only the pack scheduling moves.