Distributed SSD streaming

[wi8duy]

Looking at the SSD streaming feature, I was wondering whether you've experimented with streaming weights from multiple disks. Most Macs have many independent Thunderbolt 4/5 buses, so we could attach multiple external enclosures with NVME drives. If the weights were copied onto each drive, including the internal one, we could stream a different portion of the weights from each drive (a bit like RAID 1). This could a be "cheap" way to increase the SSD bandwidth in random reads, and maybe make these cache miss less expensive?

I am not really familiar with ds4's codebase, so this could be irrelevant, just some food for thoughts.

[lobanov]

It may be difficult to match M5 SSD read speeds with external storage, which are reported to be up to 14.5GB/s (I didn't benchmark personally). The fastest external Thunderbolt 5 SSD I saw a spec for had 6.5GB/s, so even RAID 1 storage would just about match the built-in storage speed assuming you believe official benchmarks.

[wi8duy]

This is not necessarily about replacing internal storage, but more supplementing it with copies of the same weights in different locations, so that the read workload is spread across drives (and SSD controllers). If I follow your numbers, and if we have 3 Thunderbolt 5 (like on the Mac Mini), and separate drives/enclosures, we could theoretically double the available bandwidth. This is of course theoretical, as the reads are mostly random from what I've gathered, and there might be bottlenecks elsewhere.

In a simplistic example, if we have 10 experts to load, we could load 6 from the internal drive, 2 from external drive 1, 2 from external drive 2. We could of course have a queueing system where we assign loading tasks based on the specifications of each drive...