How to transmit massive small files parallelly?

[MaoJianwei]

How to transmit massive small files parallelly?

[0x2501]

May I suggest taking this to Discussions?

That said, you can use parallel:

find . -mindepth 1 -maxdepth 1 -type d | parallel -j 16 rsync -az {}/ otherhost:/destination/directory/{}/

or get a wrapper like fpsync

alternatively, use tar + mbuffer + ssh

tar -c -C /source . | zstd -T0 | mbuffer -s 128k -m 1G | ssh otherhost 'mbuffer -s 128k -m 1G | zstd -d -T0 | tar -x -C /destination'

[MaoJianwei]

May I suggest taking this to Discussions?

That said, you can use parallel:

find . -mindepth 1 -maxdepth 1 -type d | parallel -j 16 rsync -az {}/ otherhost:/destination/directory/{}/

or get a wrapper like fpsync

alternatively, use tar + mbuffer + ssh

tar -c -C /source . | zstd -T0 | mbuffer -s 128k -m 1G | ssh otherhost 'mbuffer -s 128k -m 1G | zstd -d -T0 | tar -x -C /destination'

Why does rsync not provide native method to parallel transmission?
Thanks :)

[0x2501]

Because rsync is decades old (1996!) and the world was very different then (single-core CPUs, spindle hard drives, etc), and now we have other utilities that help around that, so it really isn't a priority.

[MaoJianwei]

Because rsync is decades old (1996!) and the world was very different then (single-core CPUs, spindle hard drives, etc), and now we have other utilities that help around that, so it really isn't a priority.

I see, thanks :)