I got it working!The driver pod was not failing at all, it was the registrar pod.Problem was the Unity still had old hosts built in it from the failed attempts. Cleared those out and it finally worked. Also other prereqs were not in place (Did not have the iscsi initiators set up on the nodes correctly to begin with) There were other issues as well, I will make a new post detailing tips and tricks for others.
Problem:when deploying the unity-csi driver, running the csi-install.sh script results in pods that are in a CrashLoopBackOff state. unity-controllers = 5/5unity-nodes = 1/2 Error seems to be "Container image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0" already present on machine" How can I trouble shoot this? Environment: - Master and nodes running as virtual machines on vmware 6.7- Running kubernetes 1.18.5- Downloaded the latest driver for csi-unity from github, so I believe this is version 1.5? Trouble shooting performed: - Made sure ./verify.sh showed successes for every check. - Followed all instructions in this link Commands used for verify and install of driver ./verify.sh --namespace unity --values ./values.yaml --node-verify-user nodeadmin./csi-install.sh --namespace unity --values ./values.yaml --node-verify-user nodeadmin My ./values.yaml file used for install csiDebug: "false"volumeNamePrefix: emc-csi-volsnapNamePrefix: emc-csi-snapimagePullPolicy: IfNotPresentcertSecretCount: 1syncNodeInfoInterval: 15controllerCount: 2createStorageClassesWithTopology: trueallowRWOMultiPodAccess: "false"defaultFsType: ext4storageClassProtocols:  - protocol: "iSCSI"  - protocol: "NFS"storageArrayList:  - name: "APM00203025495"    isDefaultArray: "true"    storageClass:    storagePool: "TieredStoragePool_01"    FsType: "ext4"    thinProvisioned: "true"    isDataReductionEnabled: "false"    tieringPolicy: "0"    hostIOLimitName: ""    nasServer: ""    reclaimPolicy: Delete  snapshotClass:    retentionDuration: "" get pods output NAME READY STATUS RESTARTS AGEsnapshot-controller-0 1/1 Running 0 20hunity-controller-78f5d6bc9d-p9r9k 5/5 Running 0 3m23sunity-controller-78f5d6bc9d-s5dgn 5/5 Running 0 3m23sunity-node-89nvr 1/2 CrashLoopBackOff 4 3m23sunity-node-rkbdb 1/2 CrashLoopBackOff 4 3m23sunity-node-tktck 1/2 CrashLoopBackOff 4 3m23sunity-node-wkjpz 1/2 CrashLoopBackOff 4 3m23s Performing a kubectl describe on one of the pods (they all have the same output) Name: unity-node-89nvrNamespace: unityPriority: 0Node: taxmd-k8n03-v/10.104.8.143Start Time: Thu, 01 Apr 2021 08:39:48 -0400Labels: app=unity-nodecontroller-revision-hash=6d957c6896pod-template-generation=1Annotations: Status: RunningIP: 10.104.8.143IPs:IP: 10.104.8.143Controlled By: DaemonSet/unity-nodeContainers:driver:Container ID: docker://f7cc1a40b15f2a5b8ea2503c88f9c072c76594052709a041281d1cd97ad6699cImage: dellemc/csi-unity:v1.5.0Image ID: docker-pullable://dellemc/csi-unity[NAME]:9ab99020e1a5939d6348f488bda0b037ef8c584a44c941ceeedfcee250f15e0dPort: Host Port: Args:--driver-name=csi-unity.dellemc.com--driver-config=/unity-config/configState: RunningStarted: Thu, 01 Apr 2021 08:39:59 -0400Ready: TrueRestart Count: 0Environment:CSI_ENDPOINT: unix:///var/lib/kubelet/plugins/unity.emc.dell.com/csi_sockX_CSI_MODE: nodeX_CSI_UNITY_AUTOPROBE: trueX_CSI_UNITY_ALLOW_MULTI_POD_ACCESS: falseX_CSI_DEBUG: falseX_CSI_PRIVATE_MOUNT_DIR: /var/lib/kubelet/plugins/unity.emc.dell.com/disksX_CSI_EPHEMERAL_STAGING_PATH: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/X_CSI_ISCSI_CHROOT: /noderootX_CSI_UNITY_NODENAME: (v1:spec.nodeName)X_CSI_UNITY_NODENAME_PREFIX:GOUNITY_DEBUG: falseSSL_CERT_DIR: /certsX_CSI_UNITY_SYNC_NODEINFO_INTERVAL: 15Mounts:/certs from certs (ro)/dev from dev (rw)/noderoot from noderoot (rw)/unity-config from unity-config (rw)/var/lib/kubelet/plugins/kubernetes.io/csi from volumedevices-path (rw)/var/lib/kubelet/plugins/unity.emc.dell.com from driver-path (rw)/var/lib/kubelet/pods from pods-path (rw)/var/run/secrets/kubernetes.io/serviceaccount from unity-node-token-cglbm (ro)registrar:Container ID: docker://44914aba36dcd4aa0f63403d0b4d03eadd49f9babb9a1a17258811a9899ef02cImage: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0Image ID: docker-pullable://k8s.gcr.io/sig-storage/csi-node-driver-registrar[NAME]:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5fPort: Host Port: Args:--v=5--csi-address=$(ADDRESS)--kubelet-registration-path=/var/lib/kubelet/plugins/unity.emc.dell.com/csi_sockState: TerminatedReason: ErrorExit Code: 1Started: Thu, 01 Apr 2021 08:41:44 -0400Finished: Thu, 01 Apr 2021 08:41:49 -0400Last State: TerminatedReason: ErrorExit Code: 1Started: Thu, 01 Apr 2021 08:40:58 -0400Finished: Thu, 01 Apr 2021 08:41:03 -0400Ready: FalseRestart Count: 4Environment:ADDRESS: /csi/csi_sockKUBE_NODE_NAME: (v1:spec.nodeName)Mounts:/csi from driver-path (rw)/registration from registration-dir (rw)/var/run/secrets/kubernetes.io/serviceaccount from unity-node-token-cglbm (ro)Conditions:Type StatusInitialized TrueReady FalseContainersReady FalsePodScheduled TrueVolumes:registration-dir:Type: HostPath (bare host directory volume)Path: /var/lib/kubelet/plugins_registry/HostPathType: DirectoryOrCreatedriver-path:Type: HostPath (bare host directory volume)Path: /var/lib/kubelet/plugins/unity.emc.dell.comHostPathType: DirectoryOrCreatevolumedevices-path:Type: HostPath (bare host directory volume)Path: /var/lib/kubelet/plugins/kubernetes.io/csiHostPathType: DirectoryOrCreatepods-path:Type: HostPath (bare host directory volume)Path: /var/lib/kubelet/podsHostPathType: Directorydev:Type: HostPath (bare host directory volume)Path: /devHostPathType: Directorynoderoot:Type: HostPath (bare host directory volume)Path: /HostPathType: Directorycerts:Type: Projected (a volume that contains injected data from multiple sources)SecretName: unity-certs-0SecretOptionalName: unity-config:Type: Secret (a volume populated by a Secret)SecretName: unity-credsOptional: falseunity-node-token-cglbm:Type: Secret (a volume populated by a Secret)SecretName: unity-node-token-cglbmOptional: falseQoS Class: BestEffortNode-Selectors: Tolerations: node.kubernetes.io/disk-pressure:NoExecutenode.kubernetes.io/disk-pressure:NoSchedulenode.kubernetes.io/memory-pressure:NoExecutenode.kubernetes.io/memory-pressure:NoSchedulenode.kubernetes.io/network-unavailable:NoExecutenode.kubernetes.io/network-unavailable:NoSchedulenode.kubernetes.io/not-ready:NoExecutenode.kubernetes.io/pid-pressure:NoSchedulenode.kubernetes.io/unreachable:NoExecutenode.kubernetes.io/unschedulable:NoScheduleEvents:Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled default-scheduler Successfully assigned unity/unity-node-89nvr to taxmd-k8n03-v Normal Pulling 2m5s kubelet, taxmd-k8n03-v Pulling image "dellemc/csi-unity:v1.5.0"Normal Pulled 119s kubelet, taxmd-k8n03-v Successfully pulled image "dellemc/csi-unity:v1.5.0"Normal Created 116s kubelet, taxmd-k8n03-v Created container driverNormal Started 116s kubelet, taxmd-k8n03-v Started container driverNormal Pulling 116s kubelet, taxmd-k8n03-v Pulling image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0"Normal Pulled 114s kubelet, taxmd-k8n03-v Successfully pulled image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0"Normal Started 57s (x4 over 114s) kubelet, taxmd-k8n03-v Started container registrarWarning BackOff 23s (x6 over 104s) kubelet, taxmd-k8n03-v Back-off restarting failed containerNormal Created 11s (x5 over 114s) kubelet, taxmd-k8n03-v Created container registrarNormal Pulled 11s (x4 over 109s) kubelet, taxmd-k8n03-v Container image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0" already present on machine
Unity driver: After running csi-install.sh unity pods show "1/2 
CrashLoopBackOff"