Skip to main content
Pure1 Support Portal

Troubleshooting vCenter Storage Provider Errors and Host Protocol Endpoint Connection Errors

This is a employee facing KB that outlines troubleshooting steps when working issues with vCenter Servers failing to register the Storage Provider, failure to create the VVol datastore, or connectivity issues with Hosts to the Protocol Endpoint or VVol datastore.

What is the Problem?

There are generally three issues that support has found when helping a customer get started with VVols.

  1. Issues registering the Storage Providers in the vCenter Server
  2. Issues creating the VVol Datastore
  3. Issues with ESXi hosts not mounting to the VVol Datastore or the VVol Datastore showing a Size of 0 MB

What should be checked first?

The first step is to ask the right questions. Here's a list of initial questions to ask.

  1. What version of vCenter Server are you using?
    1. Is this Windows vCenter Server or vCenter Server Appliance?
      1. If the customer is using vCSA, do they have multiple vCenters in Enhanced Linked Mode?
  2. What version of ESXi is installed on the Hosts?
  3. Was this array ever connected to a different vCenter for VVols?
  4. Was the vCenter ever connected to another array for VVols, or, was it ever used in the Pure VVol Beta?
  5. Were any of the hosts in a different vCenter Server or previously connected with an array using VVols in another vCenter?
  6. Are the ESXi Hosts using iSCSI or Fiber Channel HBAs?  
    1. If they are using FC Cards, are they on updated Firmware that supports Secondary LUN IDs?
      1. This is even more important to check with UCS hosts and the fnic driver version.  Please see this KB for more information.
  7. Are they using the Pure Storage vSphere Plugin to register the storage provider or are they doing it Manually?
  8. Has the vCenter previously worked with this VVols implementation, but recently stopped working?

The main reason we need to ask these questions is to ensure the customer environment has everything ready to use VVols successfully. There are specific requirements, outlined here, that must to be met in order for VVOLs to work. If those requirements are met, and we have answers to our questions, we're in a better place to troubleshoot.

 Equipped with answers to the above, and with requirements verified as compliant, we can now determine what to pursue in troubleshooting for a root cause of the problem.


What are the Troubleshooting Steps?

Depending on the answers to our above questions paired with the problem being experienced, there are different starting points in troubleshooting. Listed below are some different environment scenarios (with probable symptoms). Then, following those, are general troubleshooting steps for the most common problems seen. Use these scenarios to know what troubleshooting steps to follow.


The vCenter Server & Hosts have never been connected to an Array using VVols

Symptoms

Here, we shouldn't be having any Certificate or key / keystore issues. What we look at will depend on where the failure is.
The two probable issues would be a failure to register Storage Provider or hosts failing to mount the VVol Datastore.
More than likely the issue here will be related to a firewall, connectivity issues, or a failure in the process (configuration; IPs, information, username/password, etc).


The vCenter Server has never used VVols but some hosts were previously in a diff vCenter connected to a VVol datastore

Symptoms

Here, resolution will be a bit trickier due to residual data from the prior VVols. In this situation, we typically see that the customer can register the Storage Provider in vCenter alright. But, when they then create the VVol datastore it will either: fail with a timeout or connection error, or, the datastore will create but only some (or none) of the hosts will have the VVol datastore mounted. More often than not, the customer will need to refresh/renew the individual hosts’ Certificates.


The vCenter Server was previously connected to a VVol Array

Symptoms

There are a few different flavors of this, but really it'll be the same for us. The 1st would be that the vCenter was connected a VVol Beta Array. The 2nd would be that the vCenter was connected to a different Pure Array using VVols, the customer removed original VVols and Storage Provider(s), and now they are trying to register new storage providers. The 3rd is like the 2nd, except instead of a different array it is the same array. In other words, it would be that VVols was connected & functional with a Pure Array previously, the customer removed the Storage Providers, and now they're trying to setup VVols with that same array again.


General Troubleshooting Steps

  • The vCenter Server is unable to Register the Storage Provider

  1. Check with the customer that the Pure Storage Array did not have VVols setup with another vCenter.
    1. If the Array was used with another vCenter, please follow the steps here to clear the cached keystore and restart the vasa service on the array.
  2. Check what the error is.  Is it a certificate/authentication error?  Is it a timeout error?  Is it a permissions error?
  3. Have the customer check that the vCenter is able to reach both array controllers eth0 IPs via port 8084
  4. If everything is passing the checks above and there are still errors trying to register the storage provider either through the plugin or manually, please follow the steps here to clear the cached keystore and restart the vasa service on the array.
    1. If that does not work, please gather logs (as outlined here) and open a JIRA to escalate/investigate further.
  • Creation of the VVol Datastore is failing

  1. Along with the previous steps, check what the error is.  Is it a timeout error, permissions error, or authentication error?
  2. Was the VVol datastore created manually or with the Plugin?  
  3. Check that the protocol endpoint on the arrays is connected with the Hostgroup or Hosts.
  4. Make sure that the hosts can see the protocol endpoint in vCenter under the hosts Storage Devices.
  5. Ensure that the Hosts Management network is able to reach both Array controllers via port 8084.
    1. From the Host CLI, run the following:
      nc -z <CT0.ETH0 IP address> 8084
  6. If everything is passing the checks above and there are still errors trying to create the vvols datastore either through the plugin or manually, please gather logs (as outlined here) and open a JIRA to escalate/investigate further.
  • Some/All Hosts fail to mount the VVol Datastore or VVol Datastore shows 0 MB size

  1. Along with the previous steps, check what the error is on the host.  Is the host timing out when attempting to mount or are their permission errors?
  2. Have the customer follow the following steps for each host (one at a time) connected with the protocol endpoint as outlined in this Guide.
    1. Put the host into maintenance mode
    2. Refresh the certs on the host
    3. Reboot the host
    4. Take the host out of maintenance mode
    5. Repeat the steps on the next host
  3. If everything is passing the checks above and there are still errors trying with hosts unable to mount the vvols datastore, please gather logs (as outlined here) and open a JIRA to escalate/investigate further.

What logs should we request?

Here are the following logs we will want the customer to collect for us.

  1. Complete vCenter Server Logs.
    1. In particular we are interested in the vCenter UI Virgo log: vsphere_client_virgo
  2. Complete Host logs for the Hosts in the vCenter connected to the protocol endpoint.
    1. In particular we are in interested in the vmkernel and vvold logs: vmkernel.log and vvold.log
  3. Double check that the vasa log is phoned home on fuse.  If this is a darksite, get the array logs for both controllers.
    1. For the array, we will be looking at the vasa log, rest log and potentially the syslog.

Once you have requested the logs, be sure to have a solid timeline of events so that you can outline the events in the logs and correlate them with what you observed.


Examples of Troubleshooting in JIRA

Here are some JIRAs that have examples of troubleshooting some different VVol issues.

JIRA Number What is being Troubleshooted? What was the Fix?
ES-41258    
     
     

How to check and clear Array Key Store

Sometimes you will need to double check if there is an existing cached key store on the array.  Other times you may need to clear that key and restart the vasa service on the array.  Here are the steps to do this.

  1. Log into the array via Remote Assist with VATS
  2. Peek at the key and trust stores on the both controllers with keytool. Password is "purestorage".

    root@fs64-16-ct1:~# keytool -list -v -keystore /cache/ui/keystore
    Enter keystore password:
    Keystore type: JKS
    Keystore provider: SUN
    Your keystore contains 1 entry
    Alias name: jetty
    Creation date: Jun 27, 2017
    Entry type: PrivateKeyEntry
    Certificate chain length: 1
    Certificate[1]:
    Owner: CN=Pure Storage, OU=Pure Storage, O=Pure Storage, C=US
    Issuer: OU=VMware Engineering, O=3m-vvol-vc-cert.dev.purestorage.com, ST=California, C=US, DC=local, DC=vsphere, CN=CA
    Serial number: ff4ed0d401b85282
    Valid from: Mon Jun 26 12:51:30 PDT 2017 until: Wed Jun 27 12:51:30 PDT 2018
    Certificate fingerprints:
    	 MD5:  14:E9:2C:66:F8:74:95:8B:D6:61:FD:36:AC:AC:96:90
    	 SHA1: 9B:02:FC:BD:0B:D2:DF:08:C4:A5:B0:AA:51:EC:58:96:DD:78:8E:23
    	 SHA256: E1:54:21:A3:6C:1A:9D:FE:9C:A8:D5:19:32:61:8B:31:52:E7:9A:59:C2:06:85:FD:81:E9:0A:47:39:49:32:A9
    	 Signature algorithm name: SHA256withRSA
    	 Version: 3
    Extensions:
    #1: ObjectId: 2.5.29.35 Criticality=false
    AuthorityKeyIdentifier [
    KeyIdentifier [
    0000: 80 A3 05 66 4C 5C 84 33   45 A7 99 5F 52 78 10 49  ...fL\.3E.._Rx.I
    0010: 79 F9 FC 94                                        y...
    ]
    ]
    #2: ObjectId: 2.5.29.17 Criticality=false
    SubjectAlternativeName [
      IPAddress: 10.14.64.161
    ]
    *******************************************
    *******************************************
     
    root@fs64-16-ct1:~# keytool -list -v -keystore /cache/ui/truststore
    Enter keystore password:
    Keystore type: JKS
    Keystore provider: SUN
    Your keystore contains 1 entry
    Alias name: vmca_0
    Creation date: Jul 12, 2017
    Entry type: trustedCertEntry
    Owner: OU=VMware Engineering, O=3m-vvol-vc-cert.dev.purestorage.com, ST=California, C=US, DC=local, DC=vsphere, CN=CA
    Issuer: OU=VMware Engineering, O=3m-vvol-vc-cert.dev.purestorage.com, ST=California, C=US, DC=local, DC=vsphere, CN=CA
    Serial number: da703ac078fc7cf6
    Valid from: Sat Dec 17 15:22:09 PST 2016 until: Tue Dec 15 15:22:09 PST 2026
    Certificate fingerprints:
    	 MD5:  38:F8:04:1B:A3:03:3D:52:1D:AE:FE:B8:5C:EC:CD:1D
    	 SHA1: FE:F4:CD:E3:2B:FE:F6:8A:DD:D8:D5:8F:9E:E0:5F:C3:56:13:97:33
    	 SHA256: AF:C8:E4:50:5D:B1:8E:D1:68:FF:45:F9:2D:F4:4E:85:6A:C8:FB:E1:F0:4F:5A:86:F6:7C:50:81:08:73:7E:BE
    	 Signature algorithm name: SHA256withRSA
    	 Version: 3
    Extensions:
    #1: ObjectId: 2.5.29.19 Criticality=true
    BasicConstraints:[
      CA:true
      PathLen:0
    ]
    #2: ObjectId: 2.5.29.15 Criticality=true
    KeyUsage [
      Key_CertSign
      Crl_Sign
    ]
    #3: ObjectId: 2.5.29.17 Criticality=false
    SubjectAlternativeName [
      RFC822Name: email@acme.com
      IPAddress: 127.0.0.1
    ]
    #4: ObjectId: 2.5.29.14 Criticality=false
    SubjectKeyIdentifier [
    KeyIdentifier [
    0000: 80 A3 05 66 4C 5C 84 33   45 A7 99 5F 52 78 10 49  ...fL\.3E.._Rx.I
    0010: 79 F9 FC 94                                        y...
    ]
    ]
    *******************************************
    *******************************************

     

  3. Above output is an example of certs installed by VMCA. If the user is upgrading the vCenter/VMCA, installing new CA certs, trying to use the array with a new vCenter, etc. We need to clear the certs installed by the previous VMCA in order to trigger the handshake process to acquire new VMCA certs. To do this, perform the following on both controllers

    root@fs64-16-ct1:~# rm /cache/ui/keystore /cache/ui/truststore && service vasa restart

     

  4. After this step, check on both controllers that the keystore contains our self-signed cert and truststore is empty

    root@vm-n-ct0:~# keytool -list -v -keystore /cache/ui/keystore
    Enter keystore password:
    Keystore type: JKS
    Keystore provider: SUN
    Your keystore contains 1 entry
    Alias name: jetty
    Creation date: Feb 2, 2012
    Entry type: PrivateKeyEntry
    Certificate chain length: 1
    Certificate[1]:
    Owner: CN=Pure Storage, O=Pure Storage, OU=Pure Storage, C=US
    Issuer: CN=Pure Storage, O=Pure Storage, OU=Pure Storage, C=US
    Serial number: 4f2b166c
    Valid from: Thu Feb 02 15:04:12 PST 2012 until: Wed Jan 28 15:04:12 PST 2032
    Certificate fingerprints:
    	 MD5:  0D:E8:64:78:C5:66:5E:64:8C:4D:D8:AB:EC:78:20:FE
    	 SHA1: D8:8F:22:67:B8:9C:30:31:69:66:F4:23:E9:69:94:A4:B2:38:79:21
    	 SHA256: E2:3E:10:E5:A7:92:68:F0:63:0C:D2:81:ED:34:2F:97:B6:9E:08:7F:82:F6:E1:53:43:01:27:67:ED:2F:D9:DD
    	 Signature algorithm name: SHA1withRSA
    	 Version: 3
    
    *******************************************
    *******************************************
    
    root@vm-n-ct0:~# keytool -list -v -keystore /cache/ui/truststore
    Enter keystore password:
    Keystore type: JKS
    Keystore provider: SUN
    Your keystore contains 0 entries

     

  5. At this point, you should be able to register the VASA providers on the vCenter again.

 


If you have any JIRAs you would like to add to this KB, please reach out to Alex Carver, James Griffiths or leave feedback on the KB.