Skip to main content
Pure Technical Services

AVS and CBS Troubleshooting: VMFS Datastore Inaccessible to AVS Host or Cluster

Currently viewing public documentation. Please login to access the full scope of documentation.

Problem

The Azure VMware Solution (AVS) ESXi hosts connect to Pure Cloud Block Store (CBS) via the iSCSI block protocol.  In order for these two endpoints to communicate with one another, one (but ideally multiple) iSCSI paths must exist between each AVS ESXi host iSCSI software adapter inside of the designated AVS Cluster and the Pure CBS controller VMs iSCSI endpoints.  Initializing this connectivity is covered in-depth at the following link.  If the implementation guide has not been followed previously, it is strongly recommended to review the entire Implementation Guide as many of the potential issues shown below can be avoided via a proper environment setup.

Once iSCSI connectivity is established, it is possible that the iSCSI paths between the AVS ESXi hosts and CBS can go offline for a myriad of reasons.  This KB article will cover common ways these paths can go down, common methods to check path status as well as ways to remediate and restore inactive iSCSI paths to an active state. 

To start, we will show and explain the different ways in which a loss of iSCSI connectivity can impact a running AVS and CBS deployment. 

Problem 1:  A VMFS datastore inaccessible to the entire AVS Cluster

From the Azure VMware Solution vCenter console, a CBS-backed VMFS datastore shows as 'inaccessible' from within the GUI, similar to as depicted in the below screenshot.

clipboard_e897fa5cc1c9755b5bf38eb8460317db7.png

In this scenario, the datastore cannot be accessed by the AVS vCenter cluster and all virtual machines that reside on the VMFS volume may also show up as inaccessible or otherwise go offline.

For this example from the CBS GUI, we can see that the VMFS volume has no AVS host group connectivity:

clipboard_e4e49f403c63efaaede4424b8cc4e5a78.png

Problem 2:  VMFS datastore is inaccessible to one or more (but not all) AVS ESXi hosts in a cluster

In this scenario, a VMFS datstore is available to only a few individual AVS ESXi hosts within a cluster, but is not available to all hosts.  The below screenshots show two hosts in a cluster:  one with VMFS datastore connectivity and one without.

clipboard_e173d83398a0821a37108633516f0a5cd.png

 

clipboard_e3adc885a8563369e26a162c567753b7d.png

From the CBS GUI, we can see the the host not showing the VMFS datastore in the vCenter GUI also has no iSCSI paths shown from the CBS GUI.

clipboard_ebd53f43c99f36319dee714a33bf19f27.png

Problem 3:  Individual iSCSI paths down between AVS and CBS but VMFS datastore remains accessible

This problem may not result in an outage but should still be resolved in order to have both the path resiliency as well as path performance afforded by having multiple available paths between AVS and CBS.  Single iSCSI path status can be easily check from Pure CBS.

From the CBS GUI, we can see that the AVS host shown has 'Single Controller' connectivity.  This means that a single iSCSI path is connected from the CBS CT0 controller, but not from CT1.

clipboard_eb47738fb4f9f8b78c26aac88aadf9df8.png

Impact

A VMFS volume may become inaccessible to an entire AVS Cluster, or to a subset of hosts within that cluster.  For the former, that means that any virtual machines hosted on that volume may go into an offline state.  For the latter, that means that vSphere DRS may not be able to automatically migrate VMs hosted on the VMFS volume from one host to another due to resource constraints or if a host experiences a hardware failure or otherwise needs to be taken out of service.

Either scenario can result in virtual machine outages and/or performance degradation and should be mitigated as soon as possible when detected.

Solution

An inaccessible can be attributed to multiple potential causes as denoted in the first section of this KB.  This section of the KB will provide suggested commands to run and components between the AVS and CBS integration to check to confirm connectivity, iSCSI path status and proper AVS ESXi host membership on the CBS array, as well as methods to initialize AVS hosts and rescan host HBAs.

Solution 1:  Confirm AVS ESXi host membership is accurate on Pure Cloud Block Store

AVS ESXi hosts can be automatically replaced for numerous reasons, including ESXi code upgrades or if a hardware failure occurs.  Customer-driven actions such as addition or removal of ESXi hosts from an existing AVS cluster can also account for host membership changes.  It is for this reason that we strongly recommend deploying our host lifecycle management functionality as this will automatically check for AVS ESXi cluster membership updates and if one is detected, will automatically update both CBS as well as the AVS cluster to account for the new host or hosts.

To check that host membership is up to date, login to the Pure CBS GUI, navigate to the impacted host group and review the AVS ESXi hosts listed.  Compare this listing to what is shown in the AVS vCenter, as is depicted in the below screenshot.

In this example, we can see that there is a host mismatch between the AVS Cluster-1 ESXi host membership and what is shown in Pure CBS.  This often leads to the VMFS datastore or datastores utilized by that cluster not being available for the mismatched host.

 

clipboard_e461204df594819ca22f874d311712d71.png

If there is a discrepancy as shown above, proceed to Solution 5 and re-run the build-PCBSCluster command against the impacted cluster and optionally deploy automated AVS host lifecycle management.  The build-pcbscluster command will automatically update the CBS host group membership with the correct AVS ESXi hosts, configure SW iSCSI, populate the host object on CBS with the proper iSCSI IQN and lastly rescan the HBAs on all hosts within that cluster.

Solution 2:  Confirm desired AVS VMFS Volume is connected to the AVS host group in CBS GUI

A simple, but important item to check is to confirm that the VMFS datastore you want to utilize with an AVS cluster is connected to the correct CBS host group.  The new-pcbsvmfsdatastore command will automatically create the volume and connect it to the specified host group for CBS, however, it is possible for the volume to be manually disconnected inside of CBS by a storage administrator so it is important to confirm volume connectivity if there are accessibility issues.

To confirm, login to the Pure CBS GUI, navigate to Storage > Hosts > Hosts Groups and confirm that the desired VMFS volume or volumes are connected to the correct AVS cluster.

clipboard_ec48c533131aa0daa3fa5340d4aaf00bb.png

If the VMFS volume has been disconnected from the desired host group, it can be reconnected by clicking on the three ellipses under Connected Volumes and selecting Connect Volumes.  Note that if the volume has not previously been formatted with VMFS 6, then the datastore will not appear within AVS.  For new VMFS volumes, the new-pbcsvmfsdatastore run command needs to be executed or the restore-pcbsvmfsdatastore can also be run to copy a VMFS snapshot to a volume and automatically connect it to the AVS cluster of your choosing.  All commands shown in this section are detailed in the Storage Management section of the AVS and CBS Implementation guide.

Solution 3:  Confirm AVS ExpressRoute is connected to CBS Azure vNet via Virtual Network Gateway

AVS and CBS network connectivity is accomplished via a virtual network gateway that is connected to the Azure vNet that CBS utilizes.  If that connection is deleted or becomes otherwise unavailable, iSCSI traffic between AVS and CBS will be disrupted.

To confirm connectivity, from the Azure portal, navigate to the Ultra Virtual Network gateway item utilized for AVS and Pure Cloud Block Store connectivity.  Click on 'Connections' and verify that the ExpressRoute connection is in a 'Succeeded' state.  This scenario is shown in the below screenshot.

clipboard_ecec1e748d3f6f8a8eef7fff355249c1f.png

clipboard_e9e7e5e3a9d50d5f3f030525769e067b0.png

 

If no connection to your AVS cluster is shown, or it is not in a Succeeded state, refer to the Configure and Connect section of the Implementation Guide to re-establish an connection between the two solutions.  If you receive an error message when attempting to fix or establish the ExpressRoute connection, open a ticket with Azure support to troubleshoot.

Solution 4:  Verify path connectivity between AVS and CBS

Data path connectivity can be verified from both the Azure VMware Solution vCenter instance as well as Pure Cloud Block Store.  

To view path connectivity from CBS, login to the CBS GUI and navigate to Health > Connections.  You can optionally sort by 'esx' to only show AVS ESXi hosts.

In this example below, we see that two hosts do not have active paths to Azure VMware Solution, while one does.

clipboard_e1e2716cc80f7bffd97c27d091be1ca3a.png

From the AVS vCenter click on the ESXi host of interest.  Select Configure >Storage Adapters > vmhba64 / iSCSI Software Adapter > Paths and observe the path status listed.  The below screenshot depicts a scenario where the paths between the AVS ESXi host and the CBS instance for all LUNs are down/dead.

clipboard_e57a488a5fbcbf43093c46780bf784827.png

If paths are down as shown above, proceed to solution 5 and solution 6 below to attempt to remediate.  If solution 5 and solution 6 fail, open a support request with the Azure VMware Solution team and specifiy that iSCSI paths between AVS and Pure Cloud Block Store appear to be down and need to be looked at in more detail from the AVS ESXi hosts.

Solution 5:  Initialize AVS hosts that have been replaced in AVS cluster and deploy AVS Host Lifecycle Management

If AVS Host Lifecycle Management is not in place, then the build-pcbscluster command must be manually executed against an AVS cluster anytime that the ESXi host membership is updated.  The build command iterates through a cluster and if it has already been executed against an existing AVS ESXi hosts then it is skipped.  This build command is required as any new ESXi host or hosts introduced to the cluster by default do not have the software iSCSI adapter enabled and CBS does not have the host objects with their IQNs listed.  Instructions for executing the build-pcbscluster command can be found in the Storage Management section of the Implementation guide.    

Solution 6:  Rescan AVS ESXi host storage

The VMFS Run Command library in the AVS Azure portal item allows AVS administrators to manually perform a rescan of all ESXi host cluster HBAs to detect new VMFS data volumes connected to the cluster.  How to access the Sync-ClusterVMHostStorage run command is detailed in the below screenshot.

clipboard_e0cb3f04674b72280eefdac955c4d9ebe.png

For the ClusterName field (item 4), make sure to enter the AVS ESXi cluster that you want to be scanned.  That value can be Cluster-1, Cluster-2 and so on.

After clicking Run, the execution status of the run command can be tracked in the 'Run Execution Status' tab within the Run Command Azure blade.