switchdev mode for physical functions when using SRIOV

[dann1]

By default, the SRIOV capable physical functions operate on legacy mode. We are planning to extend the VF logic in the network drivers to handle VF passthrough while the PF is in switchdev mode.

We would like for one-deploy to set up the smartnic mode if chosen in the inventory. Something like

        - address: "0000:81:00.0"
          set_driver: omit
          set_numvfs: max
          set_mode: switchdev # default is legacy

Some examples of smartnic modes. Mellanox node

[root@sm15 ~]# dpdk-devbind.py --status-dev net

Network devices using kernel driver
===================================
0000:01:00.0 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller 16d8' numa_node=0 if=vmnic0 drv=bnxt_en unused=vfio-pci
0000:01:00.1 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller 16d8' numa_node=0 if=vmnic1 drv=bnxt_en unused=vfio-pci
0000:81:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' numa_node=0 if=pf0,eth7,eth5,eth3,eth1,eth6,eth4,eth2,eth0 drv=mlx5_core unused=vfio-pci
0000:81:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' numa_node=0 if=pf1 drv=mlx5_core unused=vfio-pci
0000:81:00.2 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf002 drv=mlx5_core unused=vfio-pci
0000:81:00.3 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf003 drv=mlx5_core unused=vfio-pci
0000:81:00.4 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf004 drv=mlx5_core unused=vfio-pci
0000:81:00.5 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf005 drv=mlx5_core unused=vfio-pci
0000:81:00.6 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf006 drv=mlx5_core unused=vfio-pci
0000:81:00.7 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf007 drv=mlx5_core unused=vfio-pci
0000:81:01.0 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf010 drv=mlx5_core unused=vfio-pci
0000:81:01.1 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf011 drv=mlx5_core unused=vfio-pci
0000:81:01.2 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf012 drv=mlx5_core unused=vfio-pci
0000:81:01.3 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf013 drv=mlx5_core unused=vfio-pci
0000:81:01.4 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf014 drv=mlx5_core unused=vfio-pci
0000:81:01.5 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf015 drv=mlx5_core unused=vfio-pci
0000:81:01.6 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf016 drv=mlx5_core unused=vfio-pci
0000:81:01.7 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf017 drv=mlx5_core unused=vfio-pci
0000:81:02.0 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf020 drv=mlx5_core unused=vfio-pci
0000:81:02.1 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 if=vf021 drv=mlx5_core unused=vfio-pci
[root@sm15 ~]# devlink dev eswitch show pci/0000:81:00.0
pci/0000:81:00.0: mode switchdev inline-mode link encap-mode basic
[root@sm15 ~]# devlink dev eswitch show pci/0000:81:00.1
pci/0000:81:00.1: mode legacy inline-mode none encap-mode basic
[root@sm15 ~]# devlink dev eswitch show pci/0000:01:00.0
pci/0000:01:00.0: mode legacy
[root@sm15 ~]# devlink dev eswitch show pci/0000:01:00.1
pci/0000:01:00.1: mode legacy

Intel node

[root@nfhhvbcnlb03 ~]# dpdk-devbind.py --status-dev net | grep -i e810
0000:16:00.0 'Ethernet Controller E810-XXV for SFP 159b' numa_node=0 if=ens1f0np0,ens1f0npf0vf0,ens1f0npf0vf1 drv=ice unused=vfio-pci
0000:16:00.1 'Ethernet Controller E810-XXV for SFP 159b' numa_node=0 if=ens1f1np1 drv=ice unused=vfio-pci
0000:40:00.0 'Ethernet Controller E810-XXV for SFP 159b' numa_node=0 if=eno12399np0 drv=ice unused=vfio-pci
0000:40:00.1 'Ethernet Controller E810-XXV for SFP 159b' numa_node=0 if=eno12409np1 drv=ice unused=vfio-pci
[root@nfhhvbcnlb03 ~]# dpdk-devbind.py --status-dev net | grep -i x710
0000:42:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' numa_node=0 if=ens3f0 drv=i40e unused=vfio-pci
0000:42:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' numa_node=0 if=ens3f1 drv=i40e unused=vfio-pci
0000:6a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' numa_node=0 if=ens2f0 drv=i40e unused=vfio-pci
0000:6a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' numa_node=0 if=ens2f1 drv=i40e unused=vfio-pci
[root@nfhhvbcnlb03 ~]# devlink dev eswitch show pci/0000:16:00.0
pci/0000:16:00.0: mode switchdev
[root@nfhhvbcnlb03 ~]# devlink dev eswitch show pci/0000:16:00.1
pci/0000:16:00.1: mode legacy
# x710 has i40e driver which doesn't support switchdev mode
[root@nfhhvbcnlb03 ~]# devlink dev eswitch show pci/0000:42:00.0
pci/0000:42:00.0:

To activate the switchdev mode

pci="0000:16:00.0" # for example
devlink dev eswitch set pci/$pci mode switchdev

# succesful activation
[root@nfhhvbcnlb03 ~]# devlink dev eswitch set pci/0000:16:00.0 mode switchdev
[root@nfhhvbcnlb03 ~]# echo $?

# failed activation. x710 has i40e driver which doesn't support switchdev mode
[root@nfhhvbcnlb03 ~]# devlink dev eswitch set pci/0000:42:00.0 mode switchdev 
kernel answers: Operation not supported

Activating switchdevmode will yield new representors, consider for possible issues with existing PCI management logic

[root@sm15 ~]# devlink dev eswitch show pci/0000:81:00.0
pci/0000:81:00.0: mode switchdev inline-mode link encap-mode basic
[root@sm15 ~]# ls -l /sys/bus/pci/devices/0000\:81\:00.0/net/
total 0
drwxr-xr-x. 5 root root 0 May 14 11:36 eth0
drwxr-xr-x. 5 root root 0 May 14 11:36 eth1
drwxr-xr-x. 5 root root 0 May 14 11:36 eth2
drwxr-xr-x. 5 root root 0 May 14 11:36 eth3
drwxr-xr-x. 5 root root 0 May 14 11:36 eth4
drwxr-xr-x. 5 root root 0 May 14 11:36 eth5
drwxr-xr-x. 5 root root 0 May 14 11:36 eth6
drwxr-xr-x. 5 root root 0 May 14 11:36 eth7
drwxr-xr-x. 5 root root 0 May 13 10:54 pf0
[root@sm15 ~]# dpdk-devbind.py --status-dev net | grep -i pf0
0000:81:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' numa_node=0 if=pf0,eth7,eth5,eth3,eth1,eth6,eth4,eth2,eth0 drv=mlx5_core unused=vfio-pci

When using this mode, hw-tc-offload should be enabled on the pf and the vf representor interfaces

ethtool -K $PF1 hw-tc-offload on
ethtool -K $VF_PR hw-tc-offload on

Some references

• https://www.intel.com/content/www/us/en/support/articles/000098534/ethernet-products.html
• https://docs.kernel.org/networking/switchdev.html#switchdev
• https://docs.kernel.org/networking/representors.html

[sk4zuzu]

@dann1 Hi, I tried to find a way to make this switchdev thing testable without hardware for ansible automation purposes. There is nothing usable besides netdevsim (it seems 🤔), but that's not pci based as it creates another bus (dummy). So I extended the mock https://github.com/sk4zuzu/libvirt-qemu-research/blob/main/files/switchdevmock.sh (best to read with help of tree-sitter 👌 ) which modifies live device structures in sysfs + generically pre/post-processes devlink args and outputs.

Assuming I have two Composable SR-IOV devices:

https://github.com/sk4zuzu/libvirt-qemu-research/blob/b9a89ab848619011eb9cd8298964c24bc9a4fb1d/u1q/Makefile#L98-L116

Then executing:

https://github.com/sk4zuzu/libvirt-qemu-research/blob/b9a89ab848619011eb9cd8298964c24bc9a4fb1d/packer/ubuntu/remote-exec/98-rclocal.sh#L25-L26

and

echo 3 > '/sys/bus/pci/devices/0000:30:00.0/sriov_numvfs'
devlink dev eswitch set 'pci/0000:30:00.0' mode switchdev
echo 3 > '/sys/bus/pci/devices/0000:40:00.0/sriov_numvfs'
devlink dev eswitch set 'pci/0000:40:00.0' mode switchdev

Results in:

root@u1q20:~# devlink dev show
pci/0000:30:00.0
pci/0000:40:00.0

root@u1q20:~# devlink port show
pci/0000:30:00.0/0: type eth netdev eni48np1 flavour physical port 1 splittable false
pci/0000:30:00.0/128: type eth netdev eni48npf0vf0 flavour pcivf controller 0 pfnum 0 vfnum 0 external false splittable false
pci/0000:30:00.0/129: type eth netdev eni48npf0vf1 flavour pcivf controller 0 pfnum 0 vfnum 1 external false splittable false
pci/0000:30:00.0/130: type eth netdev eni48npf0vf2 flavour pcivf controller 0 pfnum 0 vfnum 2 external false splittable false
pci/0000:40:00.0/0: type eth netdev eni64np1 flavour physical port 1 splittable false
pci/0000:40:00.0/128: type eth netdev eni64npf0vf0 flavour pcivf controller 0 pfnum 0 vfnum 0 external false splittable false
pci/0000:40:00.0/129: type eth netdev eni64npf0vf1 flavour pcivf controller 0 pfnum 0 vfnum 1 external false splittable false
pci/0000:40:00.0/130: type eth netdev eni64npf0vf2 flavour pcivf controller 0 pfnum 0 vfnum 2 external false splittable false

root@u1q20:~# lspci -s 0000:30:00.*
30:00.0 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01)
30:00.1 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01)
30:00.2 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01)
30:00.3 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01)
root@u1q20:~# lspci -s 0000:40:00.*
40:00.0 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01)
40:00.1 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01)
40:00.2 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01)
40:00.3 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01)

root@u1q20:~# ls /sys/bus/pci/devices/0000\:[34]0\:00.0/net
'/sys/bus/pci/devices/0000:30:00.0/net':
eni48np1  eni48npf0vf0  eni48npf0vf1  eni48npf0vf2

'/sys/bus/pci/devices/0000:40:00.0/net':
eni64np1  eni64npf0vf0  eni64npf0vf1  eni64npf0vf2

I must admit this is plenty stupid, but I think the whole process is instructive. Do you know any other/better way to do it (not involving C compiler..) ? 🤔

[sk4zuzu]

It seems that this live modification of sysfs structure can cause kernel panic in this specific case, as virtio-pci driver likes to access some wrong places in memory.. (🤣)

I guess the mock should be still usable, but without the "/net" subdir. 👍 😇

[dann1]

Hi @sk4zuzu I also researched a bit how to test this without hardware dependency, and found netdevsim as well.

However I didn't follow it up since during our testing we found out that the switchdev mode is highly dependent on both hardware and firmware. Some cards allowed non legacy mode, while others allowed switchdev, but without hw-offload, others with hw-offload.

Sorry I can't be of more help 😅

[sk4zuzu]

@dann1 That's totally expected, I guess the only good way would be adding or modifying some existing kernel modules. But for a basic mock what I have should be good enough. 🤗