What to expect when upgrading to Purity 5.3 when using vVols

After the upgrade is complete, the vVol Datastore is marked as inaccessible

In the event that the vVol Datastore is marked as "inaccessible" post upgrade, check the following:

1. Are the Storage Providers online with an active and standby Storage Provider?
   1. If the Storage Providers are marked as offline or show up as a sync error
      1. Have Pure Support check that the DB Service and VASA are Online and Health
      2. If VASA and the DB service are healthy and the SPs are offline, then remove and re-register the storage providers. (steps are in the next section)
   2. If the Storage Providers are marked as online and one is active, check if the vVol Datastore is marked inaccessible for all or some hosts
      1. If all the hosts are showing inaccessible, work to remove and re-register the storage providers as outlined below.
2. Is the vVol Datastore marked as inaccessible for one host or all hosts? 
   1. If the vVol Datastore is marked for all or multiple hosts, remove and re-register the storage providers (steps are in the next section)
   2. If the vVol Datastore is only disconnected for one ESXi host, log into the ESXi hosts and restart vvold

      ssh to the ESXi host as root and then run /etc/init.d/vvold restart  - This will be non-disruptive from an IO and Data Path perspective:

      $ ssh root@10.21.165.16 Password: [root@sn1-r720-b05-13:~] [root@sn1-r720-b05-13:~] /etc/init.d/vvold restart /etc/init.d/vvold restart, PID 2661158 Added 2661158 to /var/run/vmware/.vmware-vvol.lock-dir/vvold-lock-dir-pid (1) watchdog-vvold: Terminating watchdog process with PID 2660881 vvold stopped. Successfully cleared vvold memory reservation PID 2661158 removed /var/run/vmware/vvold-done-calling-start Removed /var/run/vmware/.vmware-vvol.lock-dir Added 2661158 to /var/run/vmware/.vmware-vvol.lock-dir/vvold-lock-dir-pid (1) vvold max reserve memory set to 200 WaitVvoldToComeUp /var/run/vmware/.vmware-vvol.started created vvold started successfully! PID 2661158 Created /var/run/vmware/vvold-done-calling-start Removed /var/run/vmware/.vmware-vvol.lock-dir [root@sn1-r720-b05-13:~] exit

3. In the event that the Storage Providers are online/healthy and there are still hosts marked as inaccessible to the vVol Datastore after running an /etc/init.d/vvold restart, then the vCenter Root CA and CRLs need to be refreshed to the ESXi hosts, leverage the following workflow:

   ## Connect to the vCenter Server ##
   Connect-VIServer -server vcenter-server
     
     
   ## Get the ESXi hosts and set it to a variable ##
   $hosts = get-vmhost
     
     
   ## Start the Service Instance ##
   $si = Get-View ServiceInstance
     
     
   ## Start the certificate Manager view ##
   $certMgr = Get-View -Id $si.Content.CertificateManager
     
     
   ## Using the Cert Manager, refresh the ESXi hosts Certs ##
   ## This pushes all certificates in the TRUSTED_ROOTS store in the vCenter Server VECS store to the host. ##
   $certMgr.CertMgrRefreshCACertificatesAndCRLs($Hosts.ExtensionData.MoRef)
     
     
   ## Now in vCenter the vvol datastore should be accessible for each of those hosts.  No need to do the ssl_reset and restart on vvold again.##
   

   Here is an example of running this for a single ESXi Cluster:

   PS C:\> Connect-VIServer -Server dev-vcsa
   
   Name                           Port  User                          
   ----                           ----  ----                          
   dev-vcsa                       443   ALEX\Carver                   
   
   PS C:\> Get-Cluster -Name "Dev Cluster"
   
   Name                           HAEnabled  HAFailover DrsEnabled DrsAutomationLevel  
                                             Level                                     
   ----                           ---------  ---------- ---------- ------------------  
   Dev Cluster                    True       1          True       FullyAutomated      
   
   PS C:\> $ESXi_Cluster = Get-Cluster -Name "Dev Cluster"
   
   PS C:\> $ESXi_Cluster | Get-VMHost
   
   Name                 ConnectionState PowerState NumCpu CpuUsageMhz CpuTotalMhz   MemoryUsageGB   MemoryTotalGB Version
   ----                 --------------- ---------- ------ ----------- -----------   -------------   ------------- -------
   esxi-7.alex.pures... Connected       PoweredOn      16         151       38304          14.586         255.897   6.7.0
   esxi-6.alex.pures... Connected       PoweredOn      20         141       43880          16.166         255.892   6.7.0
   esxi-4.alex.pures... Connected       PoweredOn      20          94       43880           8.945         255.892   6.7.0
   
   PS C:\> $hosts = $ESXi_Cluster | Get-VMHost
   
   PS C:\> $hosts
   
   Name                 ConnectionState PowerState NumCpu CpuUsageMhz CpuTotalMhz   MemoryUsageGB   MemoryTotalGB Version
   ----                 --------------- ---------- ------ ----------- -----------   -------------   ------------- -------
   esxi-7.alex.pures... Connected       PoweredOn      16         151       38304          14.586         255.897   6.7.0
   esxi-6.alex.pures... Connected       PoweredOn      20         141       43880          16.166         255.892   6.7.0
   esxi-4.alex.pures... Connected       PoweredOn      20          94       43880           8.945         255.892   6.7.0
   
   PS C:\> $si = Get-View ServiceInstance
   
   PS C:\> $certMgr = Get-View -Id $si.Content.CertificateManager
   
   PS C:\> $certMgr.CertMgrRefreshCACertificatesAndCRLs($Hosts.ExtensionData.MoRef)
   
   PS C:\>

After the upgrade is complete, the Storage Providers are marked as offline

In the event that the Storage Providers are both marked as offline, the recommendation is to remove and re-register the storage providers for both controllers.

1. Re-Register the Storage Providers with the vSphere Plugin

   Navigate to the Storage Provider screen, locate the correct Storage Provider and then remove the Storage Provider:

   Using the vSphere Plugin, select the correct FlashArray and then Register the Storage Provider:

   Use a FlashArray Array Admin user/password and Register the Storage Providers:

    

2. Re-Register the Storage Providers with PowerShell/PowerCLI

   With PowerShell and PowerCLI you can remove the Storage Providers and then re-register the Storage Providers.  This requires both the Pure Storage PowerShell SDK and the Pure Storage FlashArray VMware module.

   ## Connect to both the vCenter and FlashArray ## PS C:\Users\powercli_1> $FA = New-PfaConnection -endpoint sn1-m20r2-c05-36.purecloud.com -credentials (Get-Credential) -nonDefaultArray cmdlet Get-Credential at command pipeline position 1 Supply values for the following parameters: Credential PS C:\Users\powercli_1> $FA Disposed : False EndPoint : sn1-m20r2-c05-36.purecloud.com UserName : vvol-admin ApiVersion : 1.16 Role : ArrayAdmin PS C:\Users\powercli_1> Connect-VIServer -Server ac-vcenter-1.purecloud.com Name Port User ---- ---- ---- ac-vcenter-1.purecloud.com 443 PURECLOUD\alex ## Then remove the Storage Providers for the Given FlashArray ## PS C:\Users\powercli_1> Remove-PfaVasaProvider -flasharray $FA Confirm Are you sure you want to perform this action? Performing the operation "Unregister FlashArray VASA Provider" on target "sn1-m20r2-c05-36-CT0". [Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): a PS C:\Users\powercli_1>

   ## Connect to both the vCenter and FlashArray ## ## If already connected from the steps to remove the Storage Providers, you can skip this step as you already have the connections ## PS C:\Users\powercli_1> $FA = New-PfaConnection -endpoint sn1-m20r2-c05-36.purecloud.com -credentials (Get-Credential) -nonDefaultArray cmdlet Get-Credential at command pipeline position 1 Supply values for the following parameters: Credential PS C:\Users\powercli_1> $FA Disposed : False EndPoint : sn1-m20r2-c05-36.purecloud.com UserName : vvol-admin ApiVersion : 1.16 Role : ArrayAdmin PS C:\Users\powercli_1> Connect-VIServer -Server ac-vcenter-1.purecloud.com Name Port User ---- ---- ---- ac-vcenter-1.purecloud.com 443 PURECLOUD\alex ## Register the Storage Providers for the given FlashArray and vCenter ## PS C:\Users\powercli_1> New-PfaVasaProvider -flasharray $FA -credentials (Get-Credential) cmdlet Get-Credential at command pipeline position 1 Supply values for the following parameters: Credential Name Status VasaVersion LastSyncTime Namespace Url ---- ------ ----------- ------------ --------- --- sn1-m20r2-c05-36-CT0 online 3.0 1/21/2020 2:59:03 PM com.purestorage https://10.21.149.61:8084 sn1-m20r2-c05-36-CT1 online 3.0 com.purestorage https://10.21.149.62:8084 PS C:\Users\powercli_1>

3. Re-Register the Storage Providers manually in vCenter

   After removing the previous Storage Providers (this is covered in step 1), you will manually add them by clicking the +Add option.  From there, enter the Storage Provider information for VASA-CT0 and then for VASA-CT1:

   Note that the port :8084 must be used in the URL and then both -ct0 and -ct1 are both being set with an Array Admin user.

    

If re-registering the storage providers fails, then do the following:

1. Log into the FlashArray as a array admin user, and run purecert list

   Running purecert list should get you the output for the current status of the VASA certificates for each controller.  You should see the common name set to the IP address for ct0.eth0 and ct1.eth0.  The org and org unit should both be Pure Storage as well.  If you are seeing this, then your certificates should be in a good state for registering the Storage Providers.

   pureuser@sn1-405-c12-21> purecert list Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name management imported 2048 sn1-405-c12-21.purecloud.com Online-Sub-CA-2 2019-12-10 23:45:36 UTC 2021-12-09 23:45:36 UTC US California Mountain View Pure Storage Solutions Engineering administrator@sso.alex.purestorage.com sn1-405-c12-21.purecloud.com vasa-ct0 imported 2048 10.21.88.117 CA 2020-01-07 00:02:02 UTC 2021-01-07 00:02:02 UTC US - - Pure Storage Pure Storage - 10.21.88.117 vasa-ct1 imported 2048 10.21.88.118 CA 2020-01-07 00:02:06 UTC 2021-01-07 00:02:06 UTC US - - Pure Storage Pure Storage - 10.21.88.118

2. Try to re-register the storage providers after a few minutes after confirming purecert list looks healthy.
3. In the event that purecert is not returning anything for vasa-ct0 or vasa-ct1, confirm that your array is running Purity //FA 5.3.  From there, follow this KB for resetting your VASA certificates.  Please be sure to reset the certificate, not creating a CSR and importing a certificate.  The purpose for this is to first get everything into a health spot.  You can generate a CSR and import a signed certificate at a later time.
4. In the event that the purecert command is failing to update/reset the certificates or the Storage Providers continue to fail, Open a Remote Assist session on the FlashArray and have Pure Support check that the internal database service and VASA are running healthy on both controllers.

After the upgrade is complete, vSphere vMotion, Clone, Backup or other vVols related jobs are failing

These types of symptoms post upgrade have been the most uncommon, but given the impact that this could cause, it's something that needs to be reviewed on how to check, troubleshoot and remediate the issue. 

First, what could possibly be the issue?  When the Primary Controller is rebooted and the Secondary Controller takes over as primary, this will trigger a Storage Provider failover in the vSphere environment.  When this happens, a request, setPEContext, is issued from the ESXi hosts to the VASA Provider.  This request wants to get information about the Protocol Endpoints and the VASA Provider.  In the event that the backend database services are taking longer than expected, this request will then fail.  In the /var/log/vvold.log file on the ESXi host, the following logging can be observed:

2020-01-21T20:55:20.207Z info vvold[2660891] [Originator@6876 sub=Default] HostManager::RefreshPEMap needSetPEContext=true
--> VasaOp::SetPEContextInt [#1]: ===> Issuing 'setPEContext' to VP sn1-m20r2-c05-36-ct0 (#outstanding 0/5) [session state: Connected]
--> VasaOp::ThrowFromSessionError [#1]: ===> FINAL FAILURE setPEContext, error (STORAGE_FAULT / Error setting PE Context org.apache.http.ConnectionClosedException: Connection closed / ) VP (sn1-m20r2-c05-36-ct0) Container (sn1-m20r2-c05-36-ct0) timeElapsed=14 msecs (#outstanding 0)
2020-01-21T20:55:21.167Z error vvold[2660906] [Originator@6876 sub=Default] VasaSession::SetPEContextInt: for url https://10.21.149.61:8084/version.xml (#PEs 1) VP failed to setPEContext: STORAGE_FAULT (Error setting PE Context org.apache.http.ConnectionClosedException: Connection closed / )
2020-01-21T20:55:21.167Z info vvold[2660906] [Originator@6876 sub=Default] VasaSession::SetPEContextInt: SCSI PE (uniqueIdentifier=__none__ lunId=naa.624a9370ef9d69657e164d4600011535 ipAddress=__none__ serverMount=__none__)

The problem here is that natively in ESXi, this request will not be retried for this VASA session.  Unless the ESXi host's vvold service is restarted, another Storage Provider failover occurs in vCenter or the Storage Providers are removed and re-registered.  In which case in the vvold.log the following logging should show that the SetPEContext is successful:

2020-01-21T21:02:15.086Z info vvold[2661223] [Originator@6876 sub=Default] HostManager::RefreshPEMap needSetPEContext=true
--> VasaOp::SetPEContextInt [#1]: ===> Issuing 'setPEContext' to VP sn1-m20r2-c05-36-ct0 (#outstanding 0/5) [session state: Connected]
--> VasaOp::ThrowFromSessionError [#1]: ===> FINAL SUCCESS setPEContext VP (sn1-m20r2-c05-36-ct0) Container (sn1-m20r2-c05-36-ct0) timeElapsed=16 msecs (#outstanding 0)
2020-01-21T21:02:15.839Z info vvold[2661231] [Originator@6876 sub=Default] VasaSession::SetPEContextInt: for url https://10.21.149.61:8084/vasa (#PEs 1): success
2020-01-21T21:02:15.839Z info vvold[2661231] [Originator@6876 sub=Default] VasaSession::SetPEContextInt: SCSI PE (uniqueIdentifier=__none__ lunId=naa.624a9370ef9d69657e164d4600011535 ipAddress=__none__ serverMount=__none__)

Likely the quickest way to ensure that the request is getting reissued to all ESXi hosts in the vCenter would be to remove and re-register the storage providers.  These steps are outlined above. In the event that only one or two hosts have this issue, then you could just ssh as root to each host and restart the vvold service.  Overall, it may just be easier to re-register the storage providers.  Both steps are covered in the previous section.  Before re-registering the storage providers, check with Pure Support to ensure that both the backend database service and VASA are both running healthy on both controllers.