support for Rocky 9.6?

[ysagon]

We installed BeeGFS client 7.4.6 on Rocky9.6 but it seems we have an issue.

• Kernel: 5.14.0-570.26.1.el9_6.x86_64
• doca-ofed-2.9.3-0.2.2.x86_64

Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol __ib_alloc_pd
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol __ib_alloc_pd (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_resolve_addr
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_resolve_addr (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_set_service_type
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_set_service_type (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol ib_dereg_mr_user
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol ib_dereg_mr_user (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_reject
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_reject (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_disconnect
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_disconnect (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol __rdma_create_kernel_id
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol __rdma_create_kernel_id (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_resolve_route
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_resolve_route (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_bind_addr
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_bind_addr (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_create_qp
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_create_qp (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol ib_map_mr_sg
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol ib_map_mr_sg (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_listen
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_listen (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_destroy_qp
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_destroy_qp (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol ib_get_dma_mr (err -2)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol __ib_create_cq
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol __ib_create_cq (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol ib_alloc_mr
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol ib_alloc_mr (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol mlx_backport_dependency_symbol (err -2)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol ib_destroy_cq_user
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol ib_destroy_cq_user (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_connect
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_connect (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_destroy_id
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_destroy_id (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol rdma_accept
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol rdma_accept (err -22)
Jul 25 10:35:46 cpu001 kernel: beegfs: disagrees about version of symbol ib_dealloc_pd_user
Jul 25 10:35:46 cpu001 kernel: beegfs: Unknown symbol ib_dealloc_pd_user (err -22)

Can you confirm it should work?

[iamjoemccormick]

Typically this error means the client module wasn't compiled with the exact kernel and RDMA driver versions you are currently using: https://doc.beegfs.io/latest/trouble_shooting/general.html#loading-the-beegfs-client-module-fails-for-could-not-insert-beegfs-invalid-argument.

While we haven't explicitly tested Rocky 9.6 with 7.4.6 but we did test 9.5 which is also on kernel 5.14 and usually only upstream kernel changes break client builds so the two versions should be close enough.

[ysagon]

Hi, thanks for the info.

We tried again today with the latest kernel from Rocky 9.6 : 5.14.0-570.33.2.el9_6.x86_64 and doca-ofed-2.9.3-0.2.2.x86_64

In /etc/beegfs/beegfs-client-autobuild.conf we have the buildArgs as follow : buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/ofa_kernel/default/include

I forced the beegfs client to rebuild:

/etc/init.d/beegfs-client rebuild home
- BeeGFS module autobuild
$OFED_INCLUDE_PATH = []
$OFED_INCLUDE_PATH = [/usr/src/ofa_kernel/default/include]
$OFED_INCLUDE_PATH = [/usr/src/ofa_kernel/default/include]
Building beegfs client module
feature detection gives: -DKERNEL_HAS_INODE_CTIME -DKERNEL_HAS_INODE_MTIME -DKERNEL_HAS_SCHED_SIG_H -DKERNEL_HAS_LINUX_STDARG_H -DKERNEL_HAS_STATX -DKERNEL_HAS_KREF_READ -DKERNEL_HAS_FILE_D                                                ENTRY -DKERNEL_HAS_SUPER_SETUP_BDI_NAME -DKERNEL_HAS_KERNEL_READ -DKERNEL_HAS_SKWQ_HAS_SLEEPER -DKERNEL_HAS_CURRENT_TIME_SPEC64 -DKERNEL_WAKE_UP_SYNC_KEY_HAS_3_ARGUMENTS -DKERNEL_HAS_IOV_IT                                                ER_KVEC_NO_TYPE_FLAG_IN_DIRECTION -DKERNEL_HAS_PROC_OPS -DKERNEL_HAS_SOCKPTR_T -DKERNEL_HAS_SOCK_SETSOCKOPT_SOCKPTR_T_PARAM -DKERNEL_HAS_TIME64 -DKERNEL_HAS_KTIME_GET_TS64 -DKERNEL_HAS_KTIM                                                E_GET_REAL_TS64 -DKERNEL_HAS_KTIME_GET_COARSE_REAL_TS64 -DKERNEL_HAS_KTIME_GET_TS64 -DKERNEL_HAS_KTIME_GET_REAL_TS64 -DKERNEL_HAS_KTIME_GET_COARSE_REAL_TS64 -DKERNEL_HAS_GENERIC_FILE_SPLICE                                                _READ -DKERNEL_HAS_SETATTR_PREPARE -DKERNEL_HAS_GET_ACL -DKERNEL_HAS_POSIX_GET_ACL_IDMAP -DKERNEL_HAS_GET_INODE_ACL -DKERNEL_HAS_SET_ACL -DKERNEL_HAS_SET_ACL_DENTRY -DKERNEL_HAS_IDMAPPED_MO                                                UNTS -DKERNEL_HAS_FOPS_ITERATE -DKERNEL_HAS_XATTR_HANDLERS_INODE_ARG -DKERNEL_HAS_CPU_IN_THREAD_INFO -DKERNEL_HAS_INODE_GET_SET_CTIME
feature detection gives: -DKERNEL_HAS_INODE_CTIME -DKERNEL_HAS_INODE_MTIME -DKERNEL_HAS_SCHED_SIG_H -DKERNEL_HAS_LINUX_STDARG_H -DKERNEL_HAS_STATX -DKERNEL_HAS_KREF_READ -DKERNEL_HAS_FILE_D                                                ENTRY -DKERNEL_HAS_SUPER_SETUP_BDI_NAME -DKERNEL_HAS_KERNEL_READ -DKERNEL_HAS_SKWQ_HAS_SLEEPER -DKERNEL_HAS_CURRENT_TIME_SPEC64 -DKERNEL_WAKE_UP_SYNC_KEY_HAS_3_ARGUMENTS -DKERNEL_HAS_IOV_IT                                                ER_KVEC_NO_TYPE_FLAG_IN_DIRECTION -DKERNEL_HAS_PROC_OPS -DKERNEL_HAS_SOCKPTR_T -DKERNEL_HAS_SOCK_SETSOCKOPT_SOCKPTR_T_PARAM -DKERNEL_HAS_TIME64 -DKERNEL_HAS_KTIME_GET_TS64 -DKERNEL_HAS_KTIM                                                E_GET_REAL_TS64 -DKERNEL_HAS_KTIME_GET_COARSE_REAL_TS64 -DKERNEL_HAS_KTIME_GET_TS64 -DKERNEL_HAS_KTIME_GET_REAL_TS64 -DKERNEL_HAS_KTIME_GET_COARSE_REAL_TS64 -DKERNEL_HAS_GENERIC_FILE_SPLICE                                                _READ -DKERNEL_HAS_SETATTR_PREPARE -DKERNEL_HAS_GET_ACL -DKERNEL_HAS_POSIX_GET_ACL_IDMAP -DKERNEL_HAS_GET_INODE_ACL -DKERNEL_HAS_SET_ACL -DKERNEL_HAS_SET_ACL_DENTRY -DKERNEL_HAS_IDMAPPED_MO                                                UNTS -DKERNEL_HAS_FOPS_ITERATE -DKERNEL_HAS_XATTR_HANDLERS_INODE_ARG -DKERNEL_HAS_CPU_IN_THREAD_INFO -DKERNEL_HAS_INODE_GET_SET_CTIME
Skipping BTF generation for /opt/beegfs/src/client/client_module_7/build/../source/beegfs.ko due to unavailability of vmlinux
$OFED_INCLUDE_PATH = [/usr/src/ofa_kernel/default/include]
$OFED_INCLUDE_PATH = []

/etc/init.d/beegfs-client start home
Starting BeeGFS Client (home):
- Loading BeeGFS modules
modprobe: ERROR: could not insert 'beegfs': Invalid argument
- BeeGFS module autobuild

I saw something maybe interesting: kmod-knem-1.1.4.90mlnx3-OFED.23.10.0.2.1.1.rhel9u6.x86_64 is for kernel 5.14.0-570.12.1.el9_6.x86_64. When installing this package, it creates an entry in /lib/modules/5.14.0-570.33.2.el9_6.x86_64/weak-updates/knem/knem.ko which should be ok.

[iamjoemccormick]

@ysagon ,

In case you haven't solved this already, have a look at this FAQ and see if it applies: https://doc.beegfs.io/latest/trouble_shooting/general.html#loading-the-beegfs-client-module-fails-for-could-not-insert-beegfs-invalid-argument.

[ysagon]

@iamjoemccormick thanks for the link but it is the same you already posted in this same thread. We checked the content but it didn't helped us, we still have the issue. If you could give a try with OFED LTS and Rocky 9.6 we would be very interested as this prevent us to upgrade our HPC cluster to Rocky 9.6.

[iamjoemccormick]

So I did! I think it is still relevant though. If the client was able to build against the supplied kernel+OFED headers but inserting the module into the kernel fails for Invalid argument and the kernel logs errors like disagrees about version of symbol this usually means the client module was compiled against different kernel and/or OFED headers than the current kernel and/or OFED version.

Usually /usr/src/ofa_kernel/default/include is a symbolic link that is supposed to point to the headers to the currently installed OFED version. Can you check if it is pointing to the headers for the correct OFED version based on ofed_info? You could update the symlink or explicitly specify that as OFED_INCLUDE_PATH.

One other suggestion, if later you get errors building the client module with DOCA OFED support, you might try completely uninstalling everything DOCA/OFED and reinstalling. We have observed bugs in DOCA before where updating via dnf update may result in a broken installation.