Skip to main content
Contact Us
Register
Pure Technical Services

Troubleshooting: vCenter Storage Provider Errors and Host Protocol Endpoint Connection Errors

  1. Last updated
  2. Save as PDF

Currently viewing public documentation. Please login to access the full scope of documentation.

This is a employee facing KB that outlines troubleshooting steps when working issues with vCenter Servers failing to register the Storage Provider, failure to create the VVol datastore, or connectivity issues with Hosts to the Protocol Endpoint or VVol datastore.

Do not manually reset certificates if the FlashArray is running Purity 5.3.0 or higher in the same method as outlined in this KB for Purity 5.0, 5.1 or 5.2.  All management of the VASA certificates must be done with purecert via the CLI on Purity 5.3.0 and higher.

Any use of the purecert command should not be run by support.  The customer should be the one to run any purecert commands.  Unless explicated outlined in a JIRA or from Engineering, support should not be running any purecert commands on a customer array.

What is the Problem?

There are generally three issues that support has found when helping a customer get started with vVols.

  1. Issues registering the Storage Providers in the vCenter Server
  2. Issues creating the vVol Datastore
  3. Issues with ESXi hosts not mounting to the vVol Datastore or the vVol Datastore showing a Size of 0 MB

What should be checked first?

From Support's perspective, the first step is to ask the right questions. 
From a Customers perspective, the first step is to gather the information needed to build out a complete picture of the issue. 

Here's a list of initial questions to ask and information to gather.

  1. What version of vCenter Server are you using?
    1. Does the customer have multiple vCenters in Enhanced Linked Mode?
  2. What version of ESXi is installed on the Hosts?
  3. Was this array ever connected to a different vCenter?
  4. Were any of the ESXi hosts in a different vCenter Server or previously connected with an array using vVols?
  5. Is NTP configured for ALL ESXi Hosts and vCenter?
    1. From the hosts and vCenter either SSH to the Host/vCenter and check Time with "date" or check in the UI 
      [root@ESXi-1:~] date
Wed Jan 30 18:11:50 UTC 2019

[root@ESXi-2:~] date
Wed Jan 30 18:11:50 UTC 2019

root@prod-vcsa [ ~ ]# date
Wed Jan 30 18:11:50 UTC 2019

root@dev-vcsa [ ~ ]# date
Wed Jan 30 18:11:50 UTC 2019

      esxi ui time.png
      vcsa ui time.png
  6. Are the ESXi Hosts using iSCSI or Fiber Channel HBAs?  
    1. Check if the hosts are on updated Firmware that supports Secondary LUN IDs by running 'esxcli storage core adapter list' live on the ESX host.  
      [root@grv-cpesx03:~] esxcli storage core adapter list 
HBA Name  Driver      Link State  UID                                   Capabilities                         Description
--------  ----------  ----------  ------------------------------------  -----------------------------------  ---------------------------------------------------------------------------------
vmhba0    smartpqi    link-n/a    sas.51402ec01017bf70                                                       (0000:5c:00.0) Microsemi HPE P408i-a SR Gen10
vmhba1    qlnativefc  link-up     fc.51402ec001c67319:51402ec001c67318  Data Integrity, Second Level Lun ID  (0000:12:00.0) QLogic Corp QLE2690 Single Port 16Gb Fibre Channel to PCIe Adapter
vmhba2    qlnativefc  link-up     fc.51402ec001c6731d:51402ec001c6731c  Data Integrity, Second Level Lun ID  (0000:d8:00.0) QLogic Corp QLE2690 Single Port 16Gb Fibre Channel to PCIe Adapter
      1. This is even more important to check with UCS hosts and the fnic driver version.  Please see this KB for more information.
  7. Is the Pure Storage vSphere Plugin being used to register the storage provider or is it being done Manually?
  8. Has the vCenter previously worked with this vVols implementation, but recently stopped working?

The information gathered above will help confirm if the environment has everything ready to use vVols successfully. There are specific requirements, outlined here, that must to be met in order for vVols to work.

Equipped with information above and with requirements verified as compliant, we can now determine what to pursue in troubleshooting for a root cause of the problem.

What are the Troubleshooting Steps?

Depending on the information above paired with the problem being experienced, there are different starting points in troubleshooting. Listed below are some different environment scenarios (with probable symptoms). Then, following those, are general troubleshooting steps for the most common problems seen. Use these scenarios to know what troubleshooting steps to follow.

The vCenter Server & Hosts have never been connected to an Array using vVols

Symptoms

Here, we shouldn't be having any Certificate or keystore / truststore issues. What we look at will depend on where the failure is.
The two probable issues would be a failure to register Storage Provider or hosts failing to mount the vVol Datastore.
More than likely the issue here will be related to a firewall, connectivity issues, or a failure in the process (configuration; IPs, information, username/password, etc).

The vCenter Server has never used vVols but some hosts were previously in another vCenter connected to a vVol datastore

Symptoms

Here, resolution will be a bit trickier due to residual data from the prior vVols. In this situation, we typically see that the customer can register the Storage Provider in vCenter alright. But, when they then create the vVol datastore it will either: fail with a timeout or connection error, or, the datastore will create but only some (or none) of the hosts will have the vVol datastore mounted. More often than not, the customer will need to reset the vvold ssl cert on the individual hosts.

The Pure Storage FlashArray was previously connected to another vCenter

Symptoms

Some possible symptoms here would be that vCenter is unable to register the storage provider with a certificate error or that the storage provider registers, but the hosts have empty VP URLs when looking at the hosts vvold.log.  

General Troubleshooting Steps

The vCenter Server is unable to Register the Storage Provider

  1. Check with the customer that the Pure Storage Array did not have vVols setup with another vCenter.
    1. If the Array was used with another vCenter, please follow the steps here to clear the cached keystore and restart the vasa service on the array.
  2. Check what the error is.  Is it a certificate/authentication error?  Is it a timeout error?  Is it a permissions error?
  3. Have the customer check that the vCenter is able to reach both array controllers eth0 IPs via port 8084
  4. If the array is not on 5.1.6+ or on 5.2.0+ and the Storage Provider Registration process is still failing, please follow the steps here to clear the cached keystore/truststore and restart the vasa service on the array. 
  5. If the array is on Purity 5.1.6+ or 5.2.0+ and this is the first time the array is getting registered, then Support must regenerate the certificate, reload nginx and restart the vasa service.  Be sure to follow this process.
  6. If the array is on Purity 5.1.6, 5.1.7 or 5.2.0 and is using intermediate certs, then you will need to update the nginx vasa conf and reload nginx/restart vasa.  Follow this process for these steps.
  7. If the customer is still unable to register the storage provider after following all of the steps above, please gather logs (as outlined here) and open a JIRA to escalate/investigate further.

Creation of the vVol Datastore is failing

  1. Along with the previous steps, check what the error is.  Is it a timeout error, permissions error, or authentication error?
  2. Was the vVol datastore created manually or with the Plugin?  
  3. Check that the protocol endpoint on the arrays is connected with the Hostgroup or Hosts.
  4. Make sure that the hosts can see the protocol endpoint in vCenter under the hosts Storage Devices.
  5. Ensure that the Hosts Management network is able to reach both Array controllers via port 8084.
    From the Host CLI, run the following: 
    nc -vz <CT0.ETH0 IP address> 8084
nc -vz <CT1.ETH0 IP address> 8084
  6. If everything is passing the checks above and there are still errors trying to create the vVols datastore either through the plugin or manually, please gather logs (as outlined here) and open a JIRA to escalate/investigate further.

Some/All Hosts fail to mount the vVol Datastore or vVol Datastore shows 0 MB size

  1. Along with the previous steps, check what the error is on the host.  Is the host timing out when attempting to mount or are their permission errors?
  2. Double check the Date and NTP Service for the hosts unable to mount/create files on the vVol Datastore.
  3. Here are some useful commands to run if the customer is able to ssh into the esxi host.
    They will help you find out if the host can see it's protocol endpoint and what it's connection status is to the vasa provider.
    1. This will let us see form the host perspective what protocol endpoint/s it sees. 
      esxcli storage vvol protocolendpoint list
      		 
      [root@UCS-Blade-8:~] esxcli storage vvol protocolendpoint list
naa.624a93707dac5a64a4ae4de00004822f
   Host Id: naa.624a93707dac5a64a4ae4de00004822f
   Array Id: com.purestorage:7dac5a64-a4ae-4de0-8e14-1fbbd2accb91
   Type: SCSI
   Accessible: true
   Configured: true
   Lun Id: naa.624a93707dac5a64a4ae4de00004822f
   Remote Host:
   Remote Share:
   NFS4x Transport IPs:
   Server Scope:
   Server Major:
   Auth:
   User:
   Storage Containers: 75a7cd5b-916d-35f0-b9f5-a720a71aaee9
      In the event that this returns nothing, but you see that the other vVol commands (such as vasa provider) are giving responses, then that doesn't mean that the host isn't connected to the protocol endpoint in and of itself.  Rather odds are in the vvold.log you will see that the vp url is empty.  As the PE has not been initialized or authenticated, there won't be a PE that shows up in the GUI or CLI for that host.
       
    2. This will let us see what vasa provider the host is connecting with.  We should see the status as online and match what is in the vCenter storage providers. 
      esxcli storage vvol vasaprovider list
      		 
      [root@UCS-Blade-8:~] esxcli storage vvol vasaprovider list
SLC-m50r2-ct0
   VP Name: SLC-m50r2-ct0
   URL: https://10.204.112.170:8084
   Status: online
   Arrays:
         Array Id: com.purestorage:7dac5a64-a4ae-4de0-8e14-1fbbd2accb91
         Is Active: true
         Priority: 200
    3. This will show us the VASA context (VC UUID) associated with the host.  This is important if we have to look through the vCenter sps log or correlate what we see in the vvold log with the VC UUID. 
      esxcli storage vvol vasacontext get
      		 
      [root@UCS-Blade-8:~] esxcli storage vvol vasacontext get
6e47f6d1-7386-48dc-b119-75abe6f369af
    4. This is going to get us all vVol storage containers that the host can see.  We would expect there to be a size listed and for the accessibility to show as true. 
      esxcli storage vvol storagecontainer list
      		 
      [root@UCS-Blade-8:~] esxcli storage vvol storagecontainer list
SLC-m50r2-vVol-Container
   StorageContainer Name: SLC-m50r2-vVol-Container
   UUID: vVol:75a7cd5b916d35f0-b9f5a720a71aaee9
   Array: com.purestorage:7dac5a64-a4ae-4de0-8e14-1fbbd2accb91
   Size(MB): 8589934592
   Free(MB): 8589863891
   Accessible: true
   Default Policy:
  4. ESXi Host VASA provider in Sync error -- Refreshing the CA Root Certs to the ESXi Hosts

    There is an issue where the CA Root Certs are not correctly pushed to the ESX hosts after the Storage Providers are registered.  A common Symptom that will be observed is that in the vvold.log there will be a null VP url.

    Check the vvold log, specifically check to see if the VP url is getting populated.

    2018-08-15T13:19:42.035Z info vvold[53BCB70] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://10.204.120.210:8084/version.xml
2018-08-15T13:19:42.048Z warning vvold[53BCB70] [Originator@6876 sub=Default] VasaSession::GetEndPoint: failed to get endpoint, err=SSL Exception: Verification parameters:
--> PeerThumbprint: 2A:6E:2B:77:71:8F:1E:9A:06:5B:70:0B:7C:CE:28:BB:57:C8:14:5B
--> ExpectedThumbprint:
--> ExpectedPeerName: 10.204.120.210
--> The remote host certificate has these problems:
-->
--> * unable to get local issuer certificate, using default
2018-08-15T13:19:42.049Z info vvold[53BCB70] [Originator@6876 sub=Default] VasaSession::Initialize url is empty
2018-08-15T13:19:42.050Z warning vvold[53BCB70] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (SLC-m20r2-ct0)!
2018-08-15T13:19:42.050Z info vvold[53BCB70] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://10.204.120.210:8084/version.xml
2018-08-15T13:19:42.051Z error vvold[53BCB70] [Originator@6876 sub=Default] Initialize: Unable to init session to VP SLC-m20r2-ct0 state:

    The first step to take is the refresh the CA root certs to the ESXi hosts in vCenter.  The customer will want to leverage PowerShell and PowerCLI to refresh the CA certs or they can do it manually in the GUI. 

    Here is the PowerShell and PowerCLI workflow used to refresh the CA certs 
    ## Connect to the vCenter Server ##
Connect-VIServer -server prod-vcsa
  
  
## Get the ESXi hosts and set it to a variable ##
$hosts = get-vmhost
  
  
## Start the Service Instance ##
$si = Get-View ServiceInstance
  
  
## Start the certificate Manager view ##
$certMgr = Get-View -Id $si.Content.CertificateManager
  
  
## Using the Cert Manager, refresh the ESXi hosts Certs ##
## This pushes all certificates in the TRUSTED_ROOTS store in the vCenter Server VECS store to the host. ##
$certMgr.CertMgrRefreshCACertificatesAndCRLs($Hosts.ExtensionData.MoRef)
  
  
## Now in vCenter the vVol datastore should be accessible for each of those hosts.  No need to do the ssl_reset and restart on vvold ##
    Here is an example of it being used 
    PS C:\> Connect-VIServer -Server dev-vcsa

Name                           Port  User                          
----                           ----  ----                          
dev-vcsa                       443   ALEX\Carver                   

PS C:\> Get-Cluster -Name "Dev Cluster"

Name                           HAEnabled  HAFailover DrsEnabled DrsAutomationLevel  
                                          Level                                     
----                           ---------  ---------- ---------- ------------------  
Dev Cluster                    True       1          True       FullyAutomated      

PS C:\> $ESXi_Cluster = Get-Cluster -Name "Dev Cluster"

PS C:\> $ESXi_Cluster | Get-VMHost

Name                 ConnectionState PowerState NumCpu CpuUsageMhz CpuTotalMhz   MemoryUsageGB   MemoryTotalGB Version
----                 --------------- ---------- ------ ----------- -----------   -------------   ------------- -------
esxi-7.alex.pures... Connected       PoweredOn      16         151       38304          14.586         255.897   6.7.0
esxi-6.alex.pures... Connected       PoweredOn      20         141       43880          16.166         255.892   6.7.0
esxi-4.alex.pures... Connected       PoweredOn      20          94       43880           8.945         255.892   6.7.0

PS C:\> $hosts = $ESXi_Cluster | Get-VMHost

PS C:\> $hosts

Name                 ConnectionState PowerState NumCpu CpuUsageMhz CpuTotalMhz   MemoryUsageGB   MemoryTotalGB Version
----                 --------------- ---------- ------ ----------- -----------   -------------   ------------- -------
esxi-7.alex.pures... Connected       PoweredOn      16         151       38304          14.586         255.897   6.7.0
esxi-6.alex.pures... Connected       PoweredOn      20         141       43880          16.166         255.892   6.7.0
esxi-4.alex.pures... Connected       PoweredOn      20          94       43880           8.945         255.892   6.7.0

PS C:\> $si = Get-View ServiceInstance

PS C:\> $certMgr = Get-View -Id $si.Content.CertificateManager

PS C:\> $certMgr.CertMgrRefreshCACertificatesAndCRLs($Hosts.ExtensionData.MoRef)

PS C:\>

    In the vCenter Tasks list you will see these refresh processes kick off and complete.  From that point you can check the vvold.log and confirm that everything looks good.  Along with all the other checks.  
    If the Customer wants to use the GUI, then they can.  They would need to right click on each host, go to Certificates and then choose to refresh CA Certs.  This must be done on each host that is having a problem with the Null VP URL in the vvold log.

    Screen Shot 2019-10-03 at 9.50.35 AM.png
  5. ESXi Host VASA provider in Sync error -- vvold ssl_reset process
    1. In the event that the vasa provider is in sync error and the PE is not showing up in the esxcli storage vvol protocolendpoint list , first refresh the storage provider certificates, run through vvold ssl_reset and restart the vvold service.  
      Example of a provider in sync error 
      [root@hostname:~] esxcli storage vvol vasaprovider list
dvqapure01-ct0
   VP Name: arrayname-ct0
   URL: https://10.19.30.32:8084/version.xml
   Status: syncError
   Arrays:
         Array Id: com.purestorage:9b870a5f-23bc-44a3-9101-4f345316b5e6
         Is Active: true
         Priority: 200

    2. First please confirm that the Storage Providers in vCenter for the Array and vVol Datastore in question are both online and in sync.  If the Providers are out offline or out of sync, try to delete/un-register the providers. Then manually register both CT0 and CT1 Storage Providers.  If the Storage Provider registration fails, then refer to the storage provider registration fails section.

      For Example, here is a look at Storage Providers that are both Online
      Storage Provider Page.png

       

    3. One Possibility is that the Date is out of sync on the host.
      The customer needs to make sure that their NTP server is running and they aren't on a manually set date or time and is in sync.

      [root@hostname:~] date
Fri Mar 12 04:54:41 UTC 2010
    4. Check the vvold log, specifically check to see if the VP url is getting populated. 
      2018-08-15T13:19:42.035Z info vvold[53BCB70] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://10.204.120.210:8084/version.xml
2018-08-15T13:19:42.048Z warning vvold[53BCB70] [Originator@6876 sub=Default] VasaSession::GetEndPoint: failed to get endpoint, err=SSL Exception: Verification parameters:
--> PeerThumbprint: 2A:6E:2B:77:71:8F:1E:9A:06:5B:70:0B:7C:CE:28:BB:57:C8:14:5B
--> ExpectedThumbprint:
--> ExpectedPeerName: 10.204.120.210
--> The remote host certificate has these problems:
-->
--> * unable to get local issuer certificate, using default
2018-08-15T13:19:42.049Z info vvold[53BCB70] [Originator@6876 sub=Default] VasaSession::Initialize url is empty
2018-08-15T13:19:42.050Z warning vvold[53BCB70] [Originator@6876 sub=Default] VasaSession::DoSetContext: Empty VP URL for VP (SLC-m20r2-ct0)!
2018-08-15T13:19:42.050Z info vvold[53BCB70] [Originator@6876 sub=Default] Initialize: Failed to establish connection https://10.204.120.210:8084/version.xml
2018-08-15T13:19:42.051Z error vvold[53BCB70] [Originator@6876 sub=Default] Initialize: Unable to init session to VP SLC-m20r2-ct0 state: 0
      If you find this error in vvold, the first thing to do is reset the ssl certs for vvold in order for the host to get in sync with the vCenter Server.
    5. In the vCenter, have the customer refresh the Certificates for both Storage Providers.  In vCenter navigate to the vCenter, then Configure and Storage Providers.
      1. Here is a screen shot to navigate to the page and refresh the Certificate
        Refresh cert.png

        Do this with both CT0 and CT1 Storage Providers
    6. If the refresh fails and the storage providers are in sync error, try removing the storage providers and re-registering them.
    7. The customer can put the host in Maintenance mode if they want, but this command won't be impactful to any existing services.
    8. Have the customer open up two SSH sessions to the host.
    9. In one window/session, tail the vvold.log 
      tail -F /var/log/vvold.log

    10. In the other window/session, reset the vvold ssl cert

      /etc/init.d/vvold ssl_reset && /etc/init.d/vvold restart
    11. From the tailing window, you should see something similar to the following: 
      2018-10-02T16:19:59.933Z [/etc/init.d/vvold[203658]: /etc/init.d/vvold ssl_reset, PID 203658]
2018-10-02T16:19:59.943Z info vvold[42B1B70] [Originator@6876 sub=Default] Reading from pipe returned 1
2018-10-02T16:19:59.943Z warning vvold[42B1B70] [Originator@6876 sub=Default] vvold caught signal 1
2018-10-02T16:19:59.943Z warning vvold[4333B70] [Originator@6876 sub=Default] ReloadSslCert: Resetting all VASA sessions on receiving SIGHUP
2018-10-02T16:19:59.943Z info vvold[4333B70] [Originator@6876 sub=Default] VendorProviderMgr::ResetVPSessions: Resetting only SSL sessions
2018-10-02T16:19:59.943Z warning vvold[4333B70] [Originator@6876 sub=Default] vVolServiceInstance::SetSSLContext: Skipping CRL check as env variable vvold_DO_CRL_CHECK is not set!
2018-10-02T16:19:59.945Z info vvold[4333B70] [Originator@6876 sub=Default] VendorProvider::ResetSession for VP sn1-405-c12-21-ct0 (state: Connected)
2018-10-02T16:19:59.945Z info vvold[4333B70] [Originator@6876 sub=Default] VasaSession::Reset for url https://10.21.88.117:8084/version.xml
2018-10-02T16:19:59.945Z info vvold[4333B70] [Originator@6876 sub=Default] VasaSession::DestroyNolock for url https://10.21.88.117:8084/version.xml
2018-10-02T16:19:59.946Z info vvold[4333B70] [Originator@6876 sub=Default] VasaSession::KillAllConnections VP (sn1-405-c12-21-ct0), purged 3 connections, 0 currently active, new genId (2) (broadcast wakeup to all threads waiting for free connection)
2018-10-02T16:19:59.946Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::AddFirewallRuleForVP port is not initialized
2018-10-02T16:19:59.946Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::GetEndPoint: with url https://10.21.88.117:8084/version.xml
2018-10-02T16:19:59.946Z [/etc/init.d/vvold[203658]: vvold signalled for SSL reset]
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::Parse xml: id=4, loc=/vasa, myVer=4, ver=0
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::ParseVersionXml: Effective VP endpoint is https://10.21.88.117:8084/vasa (version 4)
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::DoSetContext: Setting GUID: 'D2CCA827-514D-E611-0000-78654321009B'  VC UID: 'ed12846d-11f2-4f79-9178-06cb6af644b6' (lastEventId was -1)
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] GetVasaHostInitiators: Adding FC HBA  wwpn: 20:00:00:25:b5:22:00:0d wwnn: 20:00:00:25:b5:44:00:47
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] GetVasaHostInitiators: Adding FC HBA  wwpn: 20:00:00:25:b5:22:00:3d wwnn: 20:00:00:25:b5:44:00:47
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] GetVasaHostInitiators: Adding FC HBA  wwpn: 20:00:00:25:b5:66:00:0d wwnn: 20:00:00:25:b5:44:00:47
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] GetVasaHostInitiators: Adding FC HBA  wwpn: 20:00:00:25:b5:66:00:3d wwnn: 20:00:00:25:b5:44:00:47
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::InitSoap: Master soap client created successfully
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::InitSoap: _masterSoap=00c3aeb8, iomode=33558544
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::InitSoap vvold using 15 secs for soap connect timeout
2018-10-02T16:19:59.985Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::InitSoap vvold using 75 secs for soap receive timeout
2018-10-02T16:20:00.028Z info vvold[416CB70] [Originator@6876 sub=Default] Came to SI::GetvVolContainer: container 916ebef1-ecb5-3355-bea4-0905959c44eb
2018-10-02T16:20:00.028Z info vvold[416CB70] [Originator@6876 sub=Default] StorageArray::GetArrayInfo arrayId com.purestorage:98d1ff12-6d20-469c-b521-c045b19f65fc
2018-10-02T16:20:00.028Z info vvold[416CB70] [Originator@6876 sub=Default] HostManager::GetPEsForStorageArray arrayId: com.purestorage:98d1ff12-6d20-469c-b521-c045b19f65fc, peVec.size: 1, peType: SCSI
2018-10-02T16:20:00.028Z info vvold[416CB70] [Originator@6876 sub=Default] ProtocolEndpoint::GetPEInfo PE info (
--> SCSI PE, ID (host: naa.624a937098d1ff126d20469c0001aad6, vasa: naa.624a937098d1ff126d20469c0001aad6) (accessible, configured))
2018-10-02T16:20:00.801Z warning vvold[3B67B70] [Originator@6876 sub=Default] vVol_ssl_auth_init: Will skip CRL check as env variable vvold_DO_CRL_CHECK is not set!
2018-10-02T16:20:00.801Z warning vvold[3B67B70] [Originator@6876 sub=Default] VasaSession:DoSetContext: Setting VASAVERSION cookie to "3.0"
2018-10-02T16:20:00.801Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::KillAllConnections VP (sn1-405-c12-21-ct0), purged 0 connections, 0 currently active, new genId (1) (broadcast wakeup to all threads waiting for free connection)
2018-10-02T16:20:00.843Z warning vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::SetState: VP sn1-405-c12-21-ct0 [OutOfSync -> Connected], state change locked!
2018-10-02T16:20:00.843Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::DoSetContext: master soap's original keep_alive = 1
2018-10-02T16:20:00.843Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::DoSetContext: VP sn1-405-c12-21-ct0 url https://10.21.88.117:8084/vasa: success eventTimer 5000000  maxBatchsize 1 maxConcurrentRequestsPerSession 5 _sessionId [-4791590017668774193] _genId 1
2018-10-02T16:20:00.843Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::CheckForceParams: Environment variable vvold_FORCE_MAX_CONNECTIONS_PER_SESSION not set, using 5 conns/session!
2018-10-02T16:20:00.843Z info vvold[3B67B70] [Originator@6876 sub=Default]
--> VasaOp::SetPEContextInt [#26852]: ===> Issuing 'setPEContext' to VP sn1-405-c12-21-ct0 (#outstanding 0/5) [session state: Connected]
2018-10-02T16:20:00.843Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::GetFreeConn: VP (sn1-405-c12-21-ct0), added new connection ([14VasaConnection:0x00c678f0]) to the pool (size now 1/5), _genId 1!
2018-10-02T16:20:00.843Z info vvold[416CB70] [Originator@6876 sub=Default] SI:GetvVolContainer successful for sn1-405-c12-21-vVol-container, id=916ebef1-ecb5-3355-bea4-0905959c44eb, maxvVol=0 MB
2018-10-02T16:20:00.844Z info vvold[41EEB70] [Originator@6876 sub=Default]
--> VasaOp::EventPollerCB [#26853]: ===> Issuing 'getEvents' to VP sn1-405-c12-21-ct0 (#outstanding 1/5) [session state: Connected]
2018-10-02T16:20:00.844Z info vvold[41EEB70] [Originator@6876 sub=Default] VasaSession::GetFreeConn: VP (sn1-405-c12-21-ct0), added new connection ([14VasaConnection:0x00c65c28]) to the pool (size now 2/5), _genId 1!
2018-10-02T16:20:00.892Z error vvold[41EEB70] [Originator@6876 sub=Default]
--> VasaOp::ThrowFromSessionError [#26853]: ===> FINAL SUCCESS getEvents VP (sn1-405-c12-21-ct0) Container (sn1-405-c12-21-ct0) timeElapsed=48 msecs (#outstanding 1)
2018-10-02T16:20:00.892Z info vvold[41EEB70] [Originator@6876 sub=Default] VasaSession::EventPollerCB VP sn1-405-c12-21-ct0: connection state changed. Raising alarm to recalculate best VP for all arrays managed by this VP
2018-10-02T16:20:00.892Z info vvold[41EEB70] [Originator@6876 sub=Default] SI:ProcessVpStateChangeEvent: Processing VP State change for sn1-405-c12-21-ct0 (newstate Connected)
2018-10-02T16:20:00.895Z error vvold[3B67B70] [Originator@6876 sub=Default]
--> VasaOp::ThrowFromSessionError [#26852]: ===> FINAL SUCCESS setPEContext VP (sn1-405-c12-21-ct0) Container (sn1-405-c12-21-ct0) timeElapsed=52 msecs (#outstanding 0)
2018-10-02T16:20:00.895Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::SetPEContextInt: for url https://10.21.88.117:8084/vasa (#PEs 1): success
2018-10-02T16:20:00.895Z info vvold[3B67B70] [Originator@6876 sub=Default] VasaSession::SetPEContextInt: SCSI PE (uniqueIdentifier=naa.624a937098d1ff126d20469c0001aad6 lunId=naa.624a937098d1ff126d20469c0001aad6 ipAddress=__none__ serverMount=__none__)
2018-10-02T16:20:00.895Z info vvold[3B67B70] [Originator@6876 sub=Default] Initialize: Success for https://10.21.88.117:8084/vasa
2018-10-02T16:20:00.895Z info vvold[3B67B70] [Originator@6876 sub=Default]
--> VasaOp::EventPollerCB [#26854]: ===> Issuing 'getEvents' to VP sn1-405-c12-21-ct0 (#outstanding 0/5) [session state: Connected]
2018-10-02T16:20:00.895Z info vvold[41EEB70] [Originator@6876 sub=Default] SI:GetBestVpRefForArrayId: VP:sn1-405-c12-21-ct0: priority:200, isActive:\x01, actvReqd:\x01, sessionState:Connected, haStatus:0
2018-10-02T16:20:00.895Z info vvold[41EEB70] [Originator@6876 sub=Default] UpdateVp arrayId com.purestorage:98d1ff12-6d20-469c-b521-c045b19f65fc : old vp
-->  VendorProvider: sn1-405-c12-21-ct0 Vasa SessionInfo:     Vasa Name: Pure Storage VASA Provider
-->     VP Version: 1.0.2    Vasa API Version: 3.0    ActivationReqd: No    Vasa NameSpace: com.purestorage Session Details:
-->     vpId : sn1-405-c12-21-ct0    Session VP URL: https://10.21.88.117:8084/version.xml
-->     Session URL: https://10.21.88.117:8084/vasa State : Connected    Need State change event : 0
-->     SessionId: -4791590017668774193 Timeout: 60    LastEvent: -1 TimerInterval (sec): 5
-->
-->  Refcount: 8 Number of managed Arrays: 1
-->     Managed array: com.purestorage:98d1ff12-6d20-469c-b521-c045b19f65fc priority: 200 isActive: 1
-->  : new vp sn1-405-c12-21-ct0
2018-10-02T16:20:00.896Z info vvold[41EEB70] [Originator@6876 sub=Default] SI:UpdateBestVpRefForArray: array com.purestorage:98d1ff12-6d20-469c-b521-c045b19f65fc new VP sn1-405-c12-21-ct0
2018-10-02T16:20:00.896Z info vvold[41EEB70] [Originator@6876 sub=Default] HostManager::GetContainerPEAccessibility arrayId: com.purestorage:98d1ff12-6d20-469c-b521-c045b19f65fc, cid: 916ebef1-ecb5-3355-bea4-0905959c44eb, accessible: 1,checkAllPEsAccessibility: false containerType: SCSI, APD: 0
2018-10-02T16:20:00.896Z info vvold[42B1B70] [Originator@6876 sub=Default]
--> VasaOp::EventPollerCB [#26855]: ===> Issuing 'getEvents' to VP sn1-405-c12-21-ct0 (#outstanding 1/5) [session state: Connected]
2018-10-02T16:20:00.898Z error vvold[3B67B70] [Originator@6876 sub=Default]
--> VasaOp::ThrowFromSessionError [#26854]: ===> FINAL SUCCESS getEvents VP (sn1-405-c12-21-ct0) Container (sn1-405-c12-21-ct0) timeElapsed=2 msecs (#outstanding 1)
2018-10-02T16:20:00.898Z error vvold[42B1B70] [Originator@6876 sub=Default]
--> VasaOp::ThrowFromSessionError [#26855]: ===> FINAL SUCCESS getEvents VP (sn1-405-c12-21-ct0) Container (sn1-405-c12-21-ct0) timeElapsed=2 msecs (#outstanding 0)
2018-10-02T16:20:20.029Z info vvold[3BA8B70] [Originator@6876 sub=Default] Came to SI::GetvVolContainer: container 916ebef1-ecb5-3355-bea4-0905959c44eb
2018-10-02T16:20:20.029Z info vvold[3BA8B70] [Originator@6876 sub=Default] StorageArray::GetArrayInfo arrayId com.purestorage:98d1ff12-6d20-469c-b521-c045b19f65fc
2018-10-02T16:20:20.029Z info vvold[3BA8B70] [Originator@6876 sub=Default] HostManager::GetPEsForStorageArray arrayId: com.purestorage:98d1ff12-6d20-469c-b521-c045b19f65fc, peVec.size: 1, peType: SCSI
2018-10-02T16:20:20.029Z info vvold[3BA8B70] [Originator@6876 sub=Default] ProtocolEndpoint::GetPEInfo PE info (
--> SCSI PE, ID (host: naa.624a937098d1ff126d20469c0001aad6, vasa: naa.624a937098d1ff126d20469c0001aad6) (accessible, configured))
      The main point to find is that you no longer see the VP URL being null or empty.
    12. Repeat as needed for the other hosts that have an empty VP URL.
    13. If the VP URL was empty and does not populate after the ssl_reset, the customer may need to remove the storage providers in vCenter and then re-register the storage providers.  Then run through the /etc/init.d/vvold ssl_reset process after that. Refer to this KB for reference to how to remove and re-register the storage providers.
    14. VMware has communicated in DCPN 00067311 that the Empty VP URL behavior has been improved in 6.7 P03 and 7.0 U1 and should not occur on these versions or higher.
  6. If everything is passing the checks above and there are still errors trying with hosts unable to mount the vVols datastore, please gather logs (as outlined here) and open a JIRA to escalate/investigate further.

What logs should we request?

Here are the following logs we will want the customer to collect for us.

  1. Complete vCenter Server Logs.
    1. In particular we are interested in the vCenter SPS Logs: /var/log/vmware/vmware-sps/sps.log
    2. At times getting the virgo log will be helpful, more so if there are plugin issues: /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log
  2. Complete Host logs for the Hosts in the vCenter connected to the protocol endpoint.
    1. In particular we are in interested in the hostd, vmkernel and vvold logs: vmkernel.log and vvold.log
  3. Double check that the vasa log is phoned home on fuse.  If this is a darksite, get the array logs for both controllers.
    1. For the array, we will be looking at the vasa log, middleware log and potentially the syslog.

Once you have requested the logs, be sure to have a solid timeline of events so that you can outline the events in the logs and correlate them with what you observed.

Examples of Troubleshooting in JIRA

Here are some JIRAs that have examples of troubleshooting some different vVol issues.

As it's sometimes hard to keep up with all the JIRAs coming through, here are the filters to search for specific issues.

How to check and clear Array Key Store

Sometimes you will need to double check if there is an existing cached key store on the array.  Other times you may need to clear that key and restart the vasa service on the array.  Here are the steps to do this.

Do not manually reset certificates if the FlashArray is running Purity 5.3.0 or higher in the same method as outlined in this KB for Purity 5.0, 5.1 or 5.2.  All management of the VASA certificates must be done with purecert via the CLI on Purity 5.3.0 and higher.

Any use of the purecert command should not be run by support.  The customer should be the one to run any purecert commands.  Unless explicated outlined in a JIRA or from Engineering, support should not be running any purecert commands on a customer array.

Starting in Purity 5.1.6 the truststore and keystore have been moved and are stored in /cache/ssl/vasa.crt.
 

Purity 5.1.5 and Below 
/cache/ui/keystore
/cache/ui/truststore

Purity 5.1.6+ and Purity 5.2

 
/cache/ssl/vasa.crt

When upgrading the Array to Purity 5.1.6+ the stores in /cache/ui will be renamed to include a _backup at the end of them.  

When following the steps below to check the cert, please be sure to change the path to the keystore and truststore appropriately.   

At this time do not try to clear the truststore or keystore on 5.1.6 or above, please open a JIRA if you feel that there is a problem with the stores on the array.

Purity 5.1.5 and Below

  1. Log into the array via Remote Assist with VATS

  2. Peek at the key and trust stores on the both controllers with keytool. Password is "purestorage".

    root@fs64-16-ct1:~# keytool -list -v -keystore /cache/ui/keystore
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: jetty
Creation date: Jun 27, 2017
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=Pure Storage, OU=Pure Storage, O=Pure Storage, C=US
Issuer: OU=VMware Engineering, O=3m-vVol-vc-cert.dev.purestorage.com, ST=California, C=US, DC=local, DC=vsphere, CN=CA
Serial number: ff4ed0d401b85282
Valid from: Mon Jun 26 12:51:30 PDT 2017 until: Wed Jun 27 12:51:30 PDT 2018
Certificate fingerprints:
	 MD5:  14:E9:2C:66:F8:74:95:8B:D6:61:FD:36:AC:AC:96:90
	 SHA1: 9B:02:FC:BD:0B:D2:DF:08:C4:A5:B0:AA:51:EC:58:96:DD:78:8E:23
	 SHA256: E1:54:21:A3:6C:1A:9D:FE:9C:A8:D5:19:32:61:8B:31:52:E7:9A:59:C2:06:85:FD:81:E9:0A:47:39:49:32:A9
	 Signature algorithm name: SHA256withRSA
	 Version: 3
Extensions:
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 80 A3 05 66 4C 5C 84 33   45 A7 99 5F 52 78 10 49  ...fL\.3E.._Rx.I
0010: 79 F9 FC 94                                        y...
]
]
#2: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  IPAddress: 10.14.64.161
]
*******************************************
*******************************************
 
root@fs64-16-ct1:~# keytool -list -v -keystore /cache/ui/truststore
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: vmca_0
Creation date: Jul 12, 2017
Entry type: trustedCertEntry
Owner: OU=VMware Engineering, O=3m-vVol-vc-cert.dev.purestorage.com, ST=California, C=US, DC=local, DC=vsphere, CN=CA
Issuer: OU=VMware Engineering, O=3m-vVol-vc-cert.dev.purestorage.com, ST=California, C=US, DC=local, DC=vsphere, CN=CA
Serial number: da703ac078fc7cf6
Valid from: Sat Dec 17 15:22:09 PST 2016 until: Tue Dec 15 15:22:09 PST 2026
Certificate fingerprints:
	 MD5:  38:F8:04:1B:A3:03:3D:52:1D:AE:FE:B8:5C:EC:CD:1D
	 SHA1: FE:F4:CD:E3:2B:FE:F6:8A:DD:D8:D5:8F:9E:E0:5F:C3:56:13:97:33
	 SHA256: AF:C8:E4:50:5D:B1:8E:D1:68:FF:45:F9:2D:F4:4E:85:6A:C8:FB:E1:F0:4F:5A:86:F6:7C:50:81:08:73:7E:BE
	 Signature algorithm name: SHA256withRSA
	 Version: 3
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:0
]
#2: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  Key_CertSign
  Crl_Sign
]
#3: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  RFC822Name: email@acme.com
  IPAddress: 127.0.0.1
]
#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 80 A3 05 66 4C 5C 84 33   45 A7 99 5F 52 78 10 49  ...fL\.3E.._Rx.I
0010: 79 F9 FC 94                                        y...
]
]
*******************************************
*******************************************

     

  3. Above output is an example of certs installed by VMCA. If the user is upgrading the vCenter/VMCA, installing new CA certs, trying to use the array with a new vCenter, etc. We need to clear the certs installed by the previous VMCA in order to trigger the handshake process to acquire new VMCA certs. To do this, perform the following on both controllers

    rm /cache/ui/keystore /cache/ui/truststore && service vasa restart

     

  4. After this step, check on both controllers that the keystore contains our self-signed cert and truststore is empty

    root@vm-n-ct0:~# keytool -list -v -keystore /cache/ui/keystore
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: jetty
Creation date: Feb 2, 2012
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=Pure Storage, O=Pure Storage, OU=Pure Storage, C=US
Issuer: CN=Pure Storage, O=Pure Storage, OU=Pure Storage, C=US
Serial number: 4f2b166c
Valid from: Thu Feb 02 15:04:12 PST 2012 until: Wed Jan 28 15:04:12 PST 2032
Certificate fingerprints:
	 MD5:  0D:E8:64:78:C5:66:5E:64:8C:4D:D8:AB:EC:78:20:FE
	 SHA1: D8:8F:22:67:B8:9C:30:31:69:66:F4:23:E9:69:94:A4:B2:38:79:21
	 SHA256: E2:3E:10:E5:A7:92:68:F0:63:0C:D2:81:ED:34:2F:97:B6:9E:08:7F:82:F6:E1:53:43:01:27:67:ED:2F:D9:DD
	 Signature algorithm name: SHA1withRSA
	 Version: 3

*******************************************
*******************************************

root@vm-n-ct0:~# keytool -list -v -keystore /cache/ui/truststore
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 0 entries

     

  5. At this point, you should be able to register the VASA providers on the vCenter again.

Purity 5.1.6 through Purity 5.2.x

Do not manually reset certificates if the FlashArray is running Purity 5.3.0 or higher in the same method as outlined in this KB for Purity 5.0, 5.1 or 5.2.  All management of the VASA certificates must be done with purecert via the CLI on Purity 5.3.0 and higher. 

Any use of the purecert command should not be run by support.  The customer should be the one to run any purecert commands.  Unless explicated outlined in a JIRA or from Engineering, support should not be running any purecert commands on a customer array.

In the event that you need to check the cert content and the array is on 5.1.6+ here are some examples:

  1. Here is an example of printing the certificate information 
    openssl x509 -in /cache/ssl/vasa.crt -text -noout

    root@SLC-OOTO-ct1:~# openssl x509 -in /cache/ssl/vasa.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 14093109525121868793 (0xc394c871d19023f9)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=CA, DC=bootcamp-sso-1, DC=dev.purestorage.com, C=US, ST=California, O=Bootcamp-vCSA-1.dev.purestorage.com, OU=VMware Engineering
        Validity
            Not Before: Dec  9 13:47:32 2018 GMT
            Not After : Dec 10 13:47:32 2019 GMT
        Subject: C=US, O=Pure Storage, OU=Pure Storage, CN=10.204.120.145
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:f2:16:fa:b1:c2:11:6f:f5:07:9d:9c:6c:11:d2:
                    0f:2d:a6:ad:66:35:5a:2f:04:17:07:46:98:76:02:
                    6d:33:36:03:ce:aa:c3:a9:5c:b9:e1:79:a1:9a:53:
                    ac:fe:a1:59:ca:26:32:b3:40:49:52:1a:3b:43:f7:
                    dd:9b:27:06:3f:ed:f8:91:b5:af:7f:07:a4:de:fc:
                    b7:d7:63:fc:3f:04:a1:f6:a6:38:94:5c:30:e0:c2:
                    f9:a6:c6:e3:9c:2f:ca:0a:c2:0d:97:72:32:4a:17:
                    02:e4:f3:60:6c:f8:aa:65:08:d0:3b:01:b0:9f:07:
                    3c:fe:4d:cb:4e:ee:93:76:4a:73:7a:9c:2e:63:2e:
                    82:e7:7a:da:5c:1b:50:60:ef:1e:75:8a:b7:c0:4d:
                    f0:da:dc:cd:e9:7b:e0:af:5e:c7:9a:ef:f2:d9:2d:
                    29:77:58:1c:c4:ac:4e:86:3f:70:46:4e:1f:a6:4f:
                    8c:08:0e:2a:05:fd:dd:f5:8d:b9:e8:fe:4e:09:15:
                    63:d8:88:62:f9:80:2a:e1:3a:cf:8b:b1:0e:02:ed:
                    ab:80:cf:25:25:6e:9a:08:d3:f9:66:29:e7:48:75:
                    8a:27:b6:3c:a1:c5:de:6c:40:f8:f4:0f:15:07:81:
                    3e:82:e2:e8:1d:45:da:fb:a7:42:35:80:8e:67:49:
                    96:8b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                IP Address:10.204.120.145
            X509v3 Authority Key Identifier:
                keyid:97:DE:EA:3E:0D:2F:80:B8:E1:C7:E9:FC:27:19:3D:FD:F3:0F:06:F0

            Authority Information Access:
                CA Issuers - URI:https://Bootcamp-vCSA-1.dev.purestorage.com/afd/vecs/ca

    Signature Algorithm: sha256WithRSAEncryption
         46:98:e0:b9:c7:43:9c:8d:3b:d1:8e:37:2f:34:99:81:79:ad:
         c5:cb:cc:57:a3:ea:c0:13:73:8d:02:d4:9e:f1:f0:ef:92:d0:
         f5:45:6d:5b:e7:44:81:c0:d7:0d:72:9e:76:c6:9b:74:a9:b7:
         ee:ab:39:9a:b5:07:1b:63:6f:d9:f5:c7:ae:7e:77:18:e3:4b:
         97:10:bd:a5:bc:89:05:3c:55:4f:58:f0:d2:79:53:19:c2:51:
         9f:96:ab:d4:85:e4:0c:eb:db:a9:29:5e:a2:fa:51:7c:b3:5c:
         5b:b4:40:4c:bc:35:98:43:70:3c:53:f5:b7:b4:79:fe:ba:25:
         4e:3a:3c:43:a8:eb:cf:f6:a4:b3:12:07:64:e1:69:48:17:4c:
         84:68:fa:7b:63:05:96:75:0f:5c:a9:3a:86:99:b2:cc:a1:db:
         5d:b9:be:78:fa:c0:f5:95:eb:71:a2:da:bb:b3:8b:d4:3d:e1:
         86:29:14:e2:f9:e6:ff:3e:2b:63:87:94:7e:c3:1b:12:10:5f:
         6d:60:ac:22:b7:69:8e:e4:91:10:f8:e3:f1:6f:f8:08:fa:2a:
         73:ce:97:a4:21:8f:1a:e4:1c:26:c3:d9:5e:e8:aa:c1:ab:73:
         26:1f:0c:96:4a:a5:8a:e9:8e:33:83:6f:4f:ee:87:13:3c:25:
         0a:af:c3:f4
  2. Then here are the commands you would need to clear the certificate information on the controller 
    openssl genrsa -out /cache/ssl/vasa.key 2048
openssl req -new -key /cache/ssl/vasa.key -out /cache/ssl/vasa.csr -subj "/C=US/O=Pure Storage/OU=Pure Storage/CN=Pure Storage"
openssl x509 -req -days 7300 -in /cache/ssl/vasa.csr -signkey /cache/ssl/vasa.key -out /cache/ssl/vasa.crt
rm /cache/ssl/vasa.csr
service nginx reload
service vasa restart

    This would need to be done for both controllers.

  3. Then you can run the query in step one to confirm that the certificate is cleared to default.