[TEST] v1.8.0 release testing - Three nodes upgrade with storage network and RWX volume on RKE2 guest cluster

[TachunLin]

What's the test to develop? Please describe

Perform the Three nodes upgrade with storage network and RWX volume on RKE2 guest cluster for v1.8.0 release testing

Prerequisite and dependency of test

1. Create VLAN 2012 network on mgmt
2. Enable all addons
3. On Harvester, create a cluster network cn and network config nc
4. Create VLAN 2011 network on cn
5. Enable the storage network with vlan ID 2011
6. Enable the Storage Network for RWX Volume setting on the embedded Longhorn UI
7. Create a storage class longhorn-rwx with specific yaml content
8. Check the role-based access control (RBAC) settings are up-to-date.
9. Check the networkfs-manager pods are running.
10. Create Load balancer and PVC in the guest cluster.

Rancher Upgrade Path:

• H1.7.1 + Rv2.13.3 (k1.34) -> H1.7.1 + R2.14.0 (K1.34) -> H1.7.1 + R2.14.0 (K1.35)

Harvester Upgrade Path:

• v1.7.1 -> v1.8.0

OS:

• SLE16
• SLE15 SP7

Pre Upgrade Test steps

1. Import Harvester to Rancher
2. Create a downstream RKE2 guest cluster (Use ubuntu 22.04 image)
3. Specify two vm network NAD including the storage network
4. After RKE2 cluster is running, ensure the harvester-csi-driver version >= 0.1.24
5. Access the RKE2 guest cluster vm
6. Install the NFS client service

7. $ apt-get install -y nfs-common
   

8. Enable the storage network interface and assign IP address to it

9. $ ip link set enp2s0 up
   

10. $ ip a add 10.0.99.100/24 dev enp2s0
    

11. Ensure can ping to the instance-manager pods IP from the RKE2 vm
12. Create a RWX Storage class harvester-rxw on RKE2 cluster
13. Create a volume rwx-vol001 using the harvester-rwx storage class
14. Create a nginx deployments to use above pvc volume
15. Ensure the nginx is running well
16. Confirm all pods are running well on the RKE2 cluster

Upgrade check

1. Can correctly upgrade Harvester from v1.7.1 to v1.8.0-rc5 or later
2. Can correctly upgrade Rancher from v2.13.3 to v2.14.0

Post Upgrade Check

1. Dependencies Check
   • dependency_charts
   • Versions of all the crds
2. Virtual machines are in same state as before and accessible
3. Monitoring chart status
4. Check the RKE2 guest cluster is running well
5. Check the load balancer and PVC state is well
6. Check the nginx-1 and nginx-2 pod with volume rwx-vol001 is runngin well
7. Confirm can access each pod and see the same data
8. Check can write some content to the rwx-vol001 on both pods
9. Create a new rwx volume rwx-vol002
10. Create the nginx-3 and nginx-4 pod and attach the rwx-002 volume to it
11. Check can write some content to the rwx-vol002 on both pods

[TachunLin]

Three node upgrade w/ 2 MGMT/Default Nodes & 1 Witness Node

Test environment

• Harvester: v1.7.1 upgrade to v1.8.0-rc5

• Rancher: v2.13.3

• OS:

  • SLE16
  • SLE15 SP7

• cloud provider

  • 0.2.1000

• csi-driver

  • 0.1.2500

• RKE2: v1.34.5 to v1.35

Prerequisite and dependency setup

${\textsf{✅}}$ Three nodes cluster three management roles $~~$

n1-hp6:~ # kubectl get nodes
NAME      STATUS   ROLES                AGE     VERSION
n1-hp6    Ready    control-plane,etcd   5h11m   v1.34.3+rke2r3
n2-hp10   Ready    control-plane,etcd   4h15m   v1.34.3+rke2r3
n3-hp80   Ready    control-plane,etcd   4h39m   v1.34.3+rke2r3

${\textsf{✅}}$ VLAN 2012 network on mgmt and VLAN 2011 network on other NICs $~~$

${\textsf{✅}}$ Enable the storage network with vlan ID 2011 $~~$

${\textsf{✅}}$ Enable the Storage Network for RWX Volume setting on the embedded Longhorn UI $~~$

${\textsf{✅}}$ Create a storage class `longhorn-rwx` with specific yaml content $~~$

${\textsf{✅}}$ Check the role-based access control (RBAC) settings are up-to-date $~~$

kubectl get clusterrole harvesterhci.io:csi-driver -o yaml

rules:
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  - volumeattachments
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - harvesterhci.io
  resources:
  - networkfilesystems
  - networkfilesystems/status
  verbs:
  - '*'
- apiGroups:
  - harvesterhci.io
  resourceNames:
  - csi-online-expand-validation
  - csi-driver-config
  resources:
  - settings
  verbs:
  - get
  - list
- apiGroups:
  - longhorn.io
  resources:
  - volumes
  - volumes/status
  verbs:
  - get
  - list
- apiGroups:
  - snapshot.storage.k8s.io
  resources:
  - volumesnapshots
  - volumesnapshotcontents
  - volumesnapshotclasses
  - volumesnapshots/status
  - volumesnapshotcontents/status
  verbs:

${\textsf{✅}}$ Check the networkfs-manager pods are running. $~~$

n1-hp6:~ # kubectl get pods -A | grep networkfs-manager
harvester-system                  harvester-networkfs-manager-w8v8h                                 1/1     Running     0               12m
harvester-system                  harvester-networkfs-manager-zhlr4                                 1/1     Running     0               31m
harvester-system                  harvester-networkfs-manager-zpvjd                                 1/1     Running     0               51m

${\textsf{✅}}$ Ensure all instance-manager pods are running and have the storage network IP $~~$

    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "k8s-pod-network",
          "interface": "eth0",
          "ips": [
              "10.52.0.89"
          ],
          "mac": "ee:d7:fa:d5:f7:2c",
          "default": true,
          "dns": {}
      },{
          "name": "harvester-system/storagenetwork-mlgl4",
          "interface": "lhnet1",
          "ips": [
              "172.16.0.4"
          ],
          "mac": "d2:fd:80:52:e6:09",
          "dns": {}
      }]

Test step on Rancher v2.13.3 with Harvester v1.7.1

${\textsf{✅}}$ On Rancher v2.13.3 install UI extension v1.7.1 $~~$

${\textsf{✅}}$ Import Harvester to Rancher $~~$

${\textsf{✅}}$ Create v1.34.5 RKE2 cluster with SLES 15 SP7 and specify two vm network NAD including the storage network $~~$

${\textsf{✅}}$ Ensure RKE2 in Active state $~~$

${\textsf{✅}}$ Copy the kubeconfig and save to your local kubeconfig file and execute k9s command $~~$

localhost:~ # k9s --kubeconfig ./kubeconfig

${\textsf{✅}}$ Search for the csi-driver in daemonsets $~~$

${\textsf{✅}}$ Update the csi-driver image to v0.2.5 $~~$

        env:
        - name: NODE_ID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: CSI_ENDPOINT
          value: unix:///csi/csi.sock
        image: rancher/harvester-csi-driver:v0.2.5
        imagePullPolicy: IfNotPresent

${\textsf{✅}}$ Access to each RKE2 guest cluster VMs and install NFS client service $~~$

sudo zypper install -y nfs-client

Checking for file conflicts: ...........................................................................................................................[done]
(1/5) Installing: keyutils-1.6.3-5.6.1.x86_64 ..........................................................................................................[done]
(2/5) Installing: libevent-2_1-7-2.1.12-150600.1.2.x86_64 ..............................................................................................[done]
Updating /etc/sysconfig/rpcbind ...
(3/5) Installing: rpcbind-0.2.3-5.9.2.x86_64 ...........................................................................................................[done]
(4/5) Installing: libnfsidmap1-1.0-150600.28.12.1.x86_64 ...............................................................................................[done]
Updating /etc/sysconfig/nfs ...
setting /sbin/mount.nfs to root:root 4755. (wrong permissions 0755)
(5/5) Installing: nfs-client-2.6.4-150600.28.12.1.x86_64 ...............................................................................................[done]
sles@rke2-v1345-sles15-pool1-nvlvs-gj2x7:~> 

${\textsf{✅}}$ Enable the storage network interface $~~$

sudo ip link set eth1 up

sles@rke2-v1345-sles15-pool1-sdjnv-f5fzr:~> ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 4e:73:0f:18:89:44 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 10.115.12.157/21 brd 10.115.15.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 06:a0:85:81:18:47 brd ff:ff:ff:ff:ff:ff
    altname enp2s0

${\textsf{✅}}$ Allocate an IP that not exists in the ippools range $~~$

sudo ip a add 172.16.0.50/24 dev eth1

sles@rke2-v1345-sles15-pool1-sdjnv-f5fzr:~> sudo ip a add 172.16.0.50/24 dev eth1
sles@rke2-v1345-sles15-pool1-sdjnv-f5fzr:~> ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 4e:73:0f:18:89:44 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 10.115.12.157/21 brd 10.115.15.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 06:a0:85:81:18:47 brd ff:ff:ff:ff:ff:ff
    altname enp2s0
    inet 172.16.0.50/24 scope global eth1
       valid_lft forever preferred_lft forever

${\textsf{✅}}$ Ensure can ping to the instance-manager pods IP from the RKE2 vm $~~$

sles@rke2-v1345-sles15-pool1-sdjnv-f5fzr:~> ping -c3 172.16.0.4
PING 172.16.0.4 (172.16.0.4) 56(84) bytes of data.
64 bytes from 172.16.0.4: icmp_seq=1 ttl=64 time=11.4 ms
64 bytes from 172.16.0.4: icmp_seq=2 ttl=64 time=0.378 ms
64 bytes from 172.16.0.4: icmp_seq=3 ttl=64 time=0.308 ms

--- 172.16.0.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2018ms
rtt min/avg/max/mdev = 0.308/4.020/11.374/5.200 ms
sles@rke2-v1345-sles15-pool1-sdjnv-f5fzr:~> ping -c3 172.16.0.5
PING 172.16.0.5 (172.16.0.5) 56(84) bytes of data.
64 bytes from 172.16.0.5: icmp_seq=1 ttl=64 time=58.7 ms
64 bytes from 172.16.0.5: icmp_seq=2 ttl=64 time=0.280 ms
64 bytes from 172.16.0.5: icmp_seq=3 ttl=64 time=0.291 ms

--- 172.16.0.5 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2026ms
rtt min/avg/max/mdev = 0.280/19.771/58.743/27.557 ms
sles@rke2-v1345-sles15-pool1-sdjnv-f5fzr:~> ping -c3 172.16.0.6
PING 172.16.0.6 (172.16.0.6) 56(84) bytes of data.
64 bytes from 172.16.0.6: icmp_seq=1 ttl=64 time=0.273 ms
64 bytes from 172.16.0.6: icmp_seq=2 ttl=64 time=0.224 ms
64 bytes from 172.16.0.6: icmp_seq=3 ttl=64 time=0.193 ms

--- 172.16.0.6 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2053ms
rtt min/avg/max/mdev = 0.193/0.230/0.273/0.032 ms

${\textsf{✅}}$ Create a RWX Storage class harvester-rxw on RKE2 cluster use Harvester longhorn-rwx class $~~$

${\textsf{✅}}$ Create a volume rwx-vol001 using the harvester-rwx storage class and set Read Write many mode $~~$

${\textsf{✅}}$ Confirm the rwx-vol001 volume is ready $~~$

${\textsf{✅}}$ Create a nginx1 pod to use the RWX volume rwx-vol001 $~~$

${\textsf{✅}}$ Set the pod -> storage to use existing PVC and mount to /mnt $~~$

${\textsf{✅}}$ Check the nginx1 pod can start in running state $~~$

${\textsf{✅}}$ Access nginx1 pod and write some content $~~$

${\textsf{✅}}$ Create a nginx2 pod to use the RWX volume rwx-vol001 $~~$

${\textsf{✅}}$ Set the pod -> storage to use existing PVC and mount to /mnt $~~$

${\textsf{✅}}$ Check the nginx2 pod can start in running state $~~$

${\textsf{✅}}$ Access nginx2 pod and write some content $~~$

[TachunLin]

Test environment

• Harvester: v1.7.1

• Rancher: v2.13.3

• OS:

  • SLE16
  • SLE15 SP7

• RKE2: v1.34.5 to v1.35

• cloud provider: 0.2.1100

• csi-driver: 0.1.2500

Upgrade to Rancher v2.14.0 with Harvester v1.7.1

${\textsf{✅}}$ Upgrade Rancher from v2.13.3 to v2.14.0 $~~$

${\textsf{✅}}$ Check cloud-provider and csi-driver status on rke2-v1345 RKE2 cluster $~~$

${\textsf{✅}}$ Check can access the rwx-vol001 volume on nginx1 and nginx2 pod $~~$

[TachunLin]

Upgrading Harvester from v1.7.1 to v1.8.0-rc5

${\textsf{✅}}$ VM, Volume and RKE2 cluster state before upgrade $~~$

${\textsf{✅}}$ Trigger upgrade Harvester to v1.8.0-rc5 $~~$

${\textsf{✅}}$ Trigger AutoConverge migration strategy $~~$

n1-hp6:~ # kubectl edit kubevirt -n harvester-system
kubevirt.kubevirt.io/kubevirt edited

      configuration:
        migrations:
          allowAutoConverge: true

${\textsf{✅}}$ Successfully upgrade to v1.8.0-rc5 $~~$

[TachunLin]

Post upgrade check

${\textsf{✅}}$ Virtual machines are in same state as before and accessible $~~$

${\textsf{✅}}$ Check the RKE2 guest cluster is running well $~~$

${\textsf{✅}}$ Check the nginx-1 and nginx-2 pod with volume rwx-vol001 is runngin well $~~$

${\textsf{✅}}$ Check the nginx-1 and nginx-2 pod with volume rwx-vol001 is runngin well $~~$

${\textsf{❌}}$ Confirm can access each pod and see the same data Check can write some content to the rwx-vol001 on both pods $~~$

Encounter known issue:

• [BUG] Guest Cluster to Harvester Longhorn Storage via RWX / NFS connectivty loss during upgrade from 1.7.0 to 1.7.1 harvester#10047

1. Failed to access the rwx-vol001 attached mount path on the nginx1 and nginx2 pod

vokoscreenNG-2026-04-14_16-14-14.mp4

2. The RWX volume is in Bound state

3. The share-manager pod is running with storage network IP on lhnet2

4. The networkfilesystem is ready and have assigned endpoint IP

5. NetworkFilesystem publishes IP 172.16.0.6 to the guest Harvester CSI driver

apiVersion: harvesterhci.io/v1beta1
kind: NetworkFilesystem
metadata:
  creationTimestamp: "2026-04-13T16:21:01Z"
  generation: 2
  name: pvc-77023e96-597b-4eb2-b7d4-9681ff68e4ad
  namespace: harvester-system
  resourceVersion: "1834317"
  uid: 043948a7-ab5d-484c-8dca-b67d2dae1b92
spec:
  desiredState: Enabled
  networkFSName: pvc-77023e96-597b-4eb2-b7d4-9681ff68e4ad
  provisioner: Longhorn
status:
  conditions:
  - lastTransitionTime: "2026-04-13T16:24:41Z"
    message: Endpoint did not contain the corresponding address
    reason: Endpoint is not ready
    status: "True"
    type: NotReady
  - lastTransitionTime: "2026-04-15T00:21:42Z"
    message: Endpoint is changed to 172.16.0.6
    reason: Endpoint is changed
    status: "True"
    type: EndpointChanged
  - lastTransitionTime: "2026-04-15T00:21:42Z"
    message: Endpoint contains the corresponding address
    reason: Endpoint is ready
    status: "True"
    type: Ready
  endpoint: 172.16.0.6
  mountOpts: vers=4.2,noresvport,softerr,timeo=600,retrans=5
  state: Enabled
  status: Ready
  type: NFS

6. However, the guest workloads nginx1 and nginx2 still point to the NFS service at 172.16.0.10

7. Dev team checked the share-manager pod was recreated with a different IP

This is the same situation described in issue #10047