How to setup Thunderbolt networking?

[META-DREAMER]

Hey, really interested in your hardware setup with networking done over Thunderbolt natively (instead of using thunderbolt to 10gbe adapters)

How exactly did you set this up? I haven't been able to find any documentation or info on how to implement something like this or even a mention of networking over Thunderbolt in this way. Do you get the full 40gbps bandwidth?

[angelnu]

Basically you need thunderbolt-net: https://christian.kellner.me/2018/05/24/thunderbolt-networking-on-linux/

In my case I connect 3 Proxmox nodes where I run ceph (Kubernetes uses the traditional Ethernet network - 1 GB). Each of the 3 Proxmox ceph nodes has 3 Thunderbolt ports.

I have not used the Thunderbolt mesh recently as the performance without it was good enough and there was an annoying issue with the Thunderbolt mesh: when you restart the nodes the other nodes loose the connection and do not recover it until I manually go and do it. Most likely something that could be fixed but I moved my focus to something else before I figured it out.

There was a Proxmox tutorial I used - I tried to google it but I do not find it.

You might want to ask in the k8s-at-home discord - others there also tried the Thunderbolt network setup (and I am also member there).