A question about StreamingDataset

[miljenm]

Hi,

I've read your excellent blog https://www.databricks.com/blog/mosaicml-streamingdataset. However, there is one thing that is not clear to me. It says: "First, StreamingDataset lets different training machines download disjoint subsets of the dataset. This means that no matter how many machines you train on, you only download the data once." My question is - if the disjoint subsets of data are downloaded only once, does each training machine work only on its downloaded subset (with potential reshuffling on each epoch), or are they able somehow to exchange data?

Thank you in advance.