NDM constantly modify blockdevice in cstor

[631068264]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and the sequence of operations performed / commands you ran.]
helm install 3.3.1

kubectl get bd -n openebs
NAME                                           NODENAME       SIZE            CLAIMSTATE   STATUS     AGE
blockdevice-102253af77d45d264eaac2e2d5ba230b   A   2147482582528   Unclaimed    Active     28h
blockdevice-8dd07a04e59af42d034d81f4a20516ab   B    2147482582528   Unclaimed    Inactive   50s

in B

lsblk 
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0    4G  0 disk 
sr0     11:0    1  492K  0 rom  
vda    253:0    0  200G  0 disk 
└─vda1 253:1    0  200G  0 part /
vdb    253:16   0    2T  0 disk 
└─vdb1 253:17   0    2T  0 part 

Follow this https://openebs.io/docs/troubleshooting/ndm I have wipe the disk, restart ndm pod on Node B

wipefs -fa /dev/vdb1
wipefs -fa /dev/vdb

but the block device still Inactive

kubectl get bd -n openebs
NAME                                           NODENAME       SIZE            CLAIMSTATE   STATUS     AGE
blockdevice-102253af77d45d264eaac2e2d5ba230b   A   2147482582528   Unclaimed    Active     29h
blockdevice-a2368ea3ef3a21d2b67d18109f41c6cb  B    2147482582528   Unclaimed    Inactive   107s

I compare the pod log on between A and B

 eventcode=ndm.blockdevice.create.success msg=Created blockdevice object in etcd rname=xxxxx

On A log after success the probe will stop but in B log ndm.blockdevice.deactivate.success or ndm.blockdevice.update.success constantly appear

ndm pod on Node B log openebs-ndm-vl694.log.zip

What did you expect to happen:

The output of the following commands will help us better understand what's going on:
[Pasting long output into a GitHub gist or other pastebin is fine.]

• kubectl get pods -n openebs

kubectl get pods -n openebs

NAME                                              READY   STATUS    RESTARTS   AGE
openebs-cstor-admission-server-664d6d5865-lpq5q   1/1     Running   0          44m
openebs-cstor-csi-controller-0                    6/6     Running   0          44m
openebs-cstor-csi-node-2lkvm                      2/2     Running   0          44m
openebs-cstor-csi-node-8t54j                      2/2     Running   0          44m
openebs-cstor-csi-node-t6l2w                      2/2     Running   0          44m
openebs-cstor-cspc-operator-697b44b857-h7kld      1/1     Running   0          44m
openebs-cstor-cvc-operator-668485f848-lg82n       1/1     Running   0          44m
openebs-localpv-provisioner-5646cc6748-thc4b      1/1     Running   0          44m
openebs-ndm-6j8cl                                 1/1     Running   0          44m
openebs-ndm-jqblp                                 1/1     Running   0          44m
openebs-ndm-operator-65fdff8c8d-nqv6d             1/1     Running   0          44m
openebs-ndm-xpsk5                                 1/1     Running   0          44m

apiVersion: v1
items:
- apiVersion: openebs.io/v1alpha1
  kind: BlockDevice
  metadata:
    annotations:
      internal.openebs.io/uuid-scheme: gpt
    creationTimestamp: "2023-01-17T02:57:50Z"
    generation: 35
    labels:
      kubernetes.io/hostname: A
      ndm.io/blockdevice-type: blockdevice
      ndm.io/managed: "true"
      nodename: A
    name: blockdevice-102253af77d45d264eaac2e2d5ba230b
    namespace: openebs
    resourceVersion: "507570"
    uid: f7f6355d-5544-4a6a-b7dc-70917aac38c3
  spec:
    capacity:
      logicalSectorSize: 512
      physicalSectorSize: 512
      storage: 2147482582528
    details:
      compliance: ""
      deviceType: partition
      driveType: HDD
      firmwareRevision: ""
      hardwareSectorSize: 512
      logicalBlockSize: 512
      model: ""
      physicalBlockSize: 512
      serial: ""
      vendor: ""
    devlinks:
    - kind: by-id
      links:
      - /dev/disk/by-id/virtio-bd6b0058-722d-4fcb-8-part1
    - kind: by-path
      links:
      - /dev/disk/by-path/virtio-pci-0000:00:06.0-part1
      - /dev/disk/by-path/pci-0000:00:06.0-part1
    filesystem: {}
    nodeAttributes:
      nodeName: A
    partitioned: "No"
    path: /dev/vdb1
  status:
    claimState: Unclaimed
    state: Active
- apiVersion: openebs.io/v1alpha1
  kind: BlockDevice
  metadata:
    annotations:
      internal.openebs.io/uuid-scheme: gpt
    creationTimestamp: "2023-01-18T07:56:50Z"
    generation: 63535
    labels:
      kubernetes.io/hostname: B
      ndm.io/blockdevice-type: blockdevice
      ndm.io/managed: "true"
      nodename: B
    name: blockdevice-a2368ea3ef3a21d2b67d18109f41c6cb
    namespace: openebs
    resourceVersion: "556445"
    uid: 8eefafb7-cc39-464a-b861-2663a13aee9f
  spec:
    capacity:
      logicalSectorSize: 512
      physicalSectorSize: 512
      storage: 2147482582528
    details:
      compliance: ""
      deviceType: partition
      driveType: HDD
      firmwareRevision: ""
      hardwareSectorSize: 512
      logicalBlockSize: 512
      model: ""
      physicalBlockSize: 512
      serial: ""
      vendor: ""
    devlinks:
    - kind: by-id
      links:
      - /dev/disk/by-id/virtio-ca6216dc-1b34-4283-b-part1
    - kind: by-path
      links:
      - /dev/disk/by-path/virtio-pci-0000:00:06.0-part1
      - /dev/disk/by-path/pci-0000:00:06.0-part1
    filesystem: {}
    nodeAttributes:
      nodeName: B
    partitioned: "No"
    path: /dev/vdb1
  status:
    claimState: Unclaimed
    state: Active
kind: List
metadata:
  resourceVersion: ""

Environment:

• OpenEBS version 3.3.1
• Kubernetes version (use kubectl version):
• Kubernetes installer & version: 1.23.6
• Cloud provider or hardware configuration:
• Type of disks connected to the nodes (eg: Virtual Disks, GCE/EBS Volumes, Physical drives etc)
• OS (e.g. from /etc/os-release): CentOS Linux 7