Skip to main content
Contact Us
Register
Pure Technical Services

Troubleshooting: PowerShell "Object Reference Not Set" Failure with PowerCLI 12

  1. Last updated
  2. Save as PDF

Currently viewing public documentation. Please login to access the full scope of documentation.

What is the Issue?

Pure Storage has identified an issue using PowerCLI versions 12.0 and 12.1 when running commands that query the Storage Providers that are registered in vCenter.

Pure Storage is working with VMware to root cause and fix the issue for a future release of PowerCLI.

For example, cmdlets such as Get-VasaStorageArray and Get-SpbmReplicationGroup will fail with the error "Object reference not set to an instance of an object."

> Get-VasaStorageArray
Get-VasaStorageArray:        1/5/2021 4:48:07 PM    Get-VasaStorageArray            Object reference not set to an instance of an object.

> Get-SpbmReplicationGroup
Get-SpbmReplicationGroup:    1/5/2021 2:58:44 PM    Get-SpbmReplicationGroup        Object reference not set to an instance of an object.

In order to trigger this issue a vCenter Server will have a FlashArray's VASA providers registered as storage providers, but there can not have been a storage provider failover that has occurred after these providers have been registered. 

The issue appears to be tied with the queryProviders call that is made to vCenter as part of these cmdlets.  Specifically PowerCLI is not populating details for the standby storage provider.  As PowerCLI is leaving values null on the standby provider, PowerShell is unable to parse the provider response and the null values as part of the query which will cause an failure in PowerShell with the object reference not set.  This can be checked with PowerShell by issuing a Get-VasaProvider and pipe it to format-list. 

Let's compare what we see with PowerCLI 11.1 and then PowerCLI 12.1:

PowerCLI 11.1

> Get-VasaProvider | format-list

Version : 1.1.1
Description :
LastSyncTime : 1/5/2021 4:33:43 PM
Namespace : com.purestorage
ProviderId : d79af7f6-ee6a-4be4-8d8a-06ae62a67212-0
Status : online
SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile}
Url : https://10.21.149.61:8084/version.xml
VasaVersion : 3.0
ProviderType : Persistence
Name : sn1-m20r2-c05-36-ct0
Id : VasaProvider-vasaProvider-158
Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-158/

Version : 1.1.1
Description :
LastSyncTime :
Namespace : com.purestorage
ProviderId : d79af7f6-ee6a-4be4-8d8a-06ae62a67212-1
Status : online
SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile}
Url : https://10.21.149.62:8084/version.xml
VasaVersion : 3.0
ProviderType : Unknown
Name : sn1-m20r2-c05-36-ct1
Id : VasaProvider-vasaProvider-159
Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-159/

PowerCLI 12.1

> Get-VasaProvider | format-list

Version : 1.1.1
Description :
LastSyncTime : 1/5/2021 4:33:43 PM
Namespace : com.purestorage
ProviderId : d79af7f6-ee6a-4be4-8d8a-06ae62a67212-0
Status : online
SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile}
Url : https://10.21.149.61:8084/version.xml
VasaVersion : 3.0
ProviderType : Persistence
Name : sn1-m20r2-c05-36-ct0
Id : VasaProvider-vasaProvider-158
Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-158/

Version :
Description :
LastSyncTime :
Namespace :
ProviderId :
Status :
SupportedProviderProfile :
Url :
VasaVersion :
ProviderType :
Name :
Id : VasaProvider-vasaProvider-159
Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-159/

Notice that with PowerCLI 11.1 that the standby provider still has the information populated whereas with PowerCLI 12.1 only the Id and Uid have values populated.

Workarounds or Fixes?

There are two ways that this issue can be corrected and one of them is not recommended:

  1. Use PowerCLI 11 until a fix is provided in PowerCLI 12 (recommended)
  2. Disable ETH0 on the active storage providers controller to force a provider failover
  3. Un-Register the Standby storage provider (not recommended)

Pure Storage does not recommend to un-register the standby provider as that will remove the HA aspect of VASA and if the active provider's controller goes down (rebooted or is powered off) then vSphere is unable to reach an active storage provider and the management path is now down from that vCenter to the FlashArray VASA service.  

At this time Pure Storage recommends to either hold off using PowerCLI 12 until this is fixed or to force a storage provider failover.  The reason that a storage provider failover will help is because this forces the standby provider to become active and when the provider becomes active the information will be populated when queryProviders is issued from PowerCLI in the future. 

Here is an example where CT0 is currently the active provider and CT1 is the standby provider.

PS /Users/alex.carver> Get-VasaProvider | where {$_.Namespace -ne 'IOFILTERS' -and $_.Namespace -ne 'VSAN'}

Name                 Status       VasaVersion LastSyncTime           Namespace            Url
----                 ------       ----------- ------------           ---------            ---
sn1-x70-b05-33-ct0   online       3.0         1/8/2021 2:53:29 PM    com.purestorage      https://10.21.149.40:8084/version.…



PS /Users/alex.carver> Get-VasaProvider | where {$_.Namespace -ne 'IOFILTERS' -and $_.Namespace -ne 'VSAN'} | format-list

Version                  : 1.1.1
Description              :
LastSyncTime             : 1/8/2021 2:53:29 PM
Namespace                : com.purestorage
ProviderId               : 395a60c2-5803-40be-95b7-029b1b3ffc3e-0
Status                   : online
SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile}
Url                      : https://10.21.149.40:8084/version.xml
VasaVersion              : 3.0
ProviderType             : Persistence
Name                     : sn1-x70-b05-33-ct0
Id                       : VasaProvider-vasaProvider-164
Uid                      : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-164/

Version                  :
Description              :
LastSyncTime             :
Namespace                :
ProviderId               :
Status                   :
SupportedProviderProfile :
Url                      :
VasaVersion              :
ProviderType             :
Name                     :
Id                       : VasaProvider-vasaProvider-165
Uid                      : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-165/

While logged in as an array admin, disable the active providers ETH0 interface (in this case it's ct0) and this can be done from the GUI as well if the CLI is not preferred.

pureuser@sn1-x70-b05-33> purenetwork disable ct0.eth0
Name      Enabled  Subnet  Address       Mask           Gateway      MTU   MAC                Speed      Services    Subinterfaces
ct0.eth0  False    -       10.21.149.40  255.255.255.0  10.21.149.1  1500  24:a9:37:01:cc:f6  1.00 Gb/s  management  -

Please note that the storage provider failure due to network connectivity failures can take between 15 and 30 minutes to complete.  Check the make sure that the standby provider has taken over as active before re-enabling ETH0.  Also try to plan this activity when a 30 minute management path failure can be tolerated.

Then check for when the standby provider becomes the active provider, this can be done with the get-VasaProvider cmdlet previously used.

PS /Users/alex.carver> Get-VasaProvider | where {$_.Namespace -ne 'IOFILTERS' -and $_.Namespace -ne 'VSAN'}

Name                 Status       VasaVersion LastSyncTime           Namespace            Url
----                 ------       ----------- ------------           ---------            ---
sn1-x70-b05-33-ct0   offline      3.0         1/8/2021 3:23:30 PM    com.purestorage      https://10.21.149.40:8084/version.…
sn1-x70-b05-33-ct1   online       3.0         1/8/2021 4:17:25 PM    com.purestorage      https://10.21.149.41:8084/version.…

PS /Users/alex.carver> Get-VasaProvider | where {$_.Namespace -ne 'IOFILTERS' -and $_.Namespace -ne 'VSAN'} | format-list

Version                  : 1.1.1
Description              :
LastSyncTime             : 1/8/2021 3:23:30 PM
Namespace                : com.purestorage
ProviderId               : 395a60c2-5803-40be-95b7-029b1b3ffc3e-0
Status                   : offline
SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile}
Url                      : https://10.21.149.40:8084/version.xml
VasaVersion              : 3.0
ProviderType             : Persistence
Name                     : sn1-x70-b05-33-ct0
Id                       : VasaProvider-vasaProvider-164
Uid                      : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-164/

Version                  : 1.1.1
Description              :
LastSyncTime             : 1/8/2021 4:17:25 PM
Namespace                : com.purestorage
ProviderId               : 395a60c2-5803-40be-95b7-029b1b3ffc3e-1
Status                   : online
SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile}
Url                      : https://10.21.149.41:8084/version.xml
VasaVersion              : 3.0
ProviderType             : Persistence
Name                     : sn1-x70-b05-33-ct1
Id                       : VasaProvider-vasaProvider-165
Uid                      : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-165/

pureuser@sn1-x70-b05-33> purenetwork enable ct0.eth0
Name      Enabled  Subnet  Address       Mask           Gateway      MTU   MAC                Speed      Services    Subinterfaces
ct0.eth0  True     -       10.21.149.40  255.255.255.0  10.21.149.1  1500  24:a9:37:01:cc:f6  1.00 Gb/s  management  -

Okay, now you will notice that the vasaProvider-165 is populated in we see that ct0 is offline and ct1 is now appearing in the provider list.  You can re-enable ETH0 from the FlashArray at this point.  Then you can check Get-VasaStorageArray and Get-SpbmReplicationGroup to see if everything is corrected.

PS /Users/alex.carver> Get-VasaStorageArray | where {$_.ModelId -eq 'FlashArray'}

Name                           ModelId              VendorId
----                           -------              --------
sn1-x70-b05-33                 FlashArray           PURE

PS /Users/alex.carver> Get-SpbmReplicationGroup -VasaProvider 'sn1-x70-b05-33-ct1'

Name                      ReplicationState
----                      ----------------
sn1-x70-b05-33:x70-1-pol… Source
sn1-x70-b05-33:x70-1-pol… Source
sn1-x70-b05-33:x70-1-pol… Source
sn1-x70-b05-33:x70-1-pol… Source
sn1-x70-b05-33:x70-1-pol… Source
sn1-x70-b05-33:x70-1-pol… Source
sn1-x70-b05-33:x70-1-pol… Source
sn1-x70-b05-33:x70-1-pol… Source
sn1-x70-b05-33:VMFS-WSFC… Source
sn1-x70-b05-33:VMFS-WSFC… Source
sn1-x70-b05-33:pwsh-test… Source
d79af7f6-ee6a-4be4-8d8a-… Target
d79af7f6-ee6a-4be4-8d8a-… Target
d79af7f6-ee6a-4be4-8d8a-… Target
35770c78-edaf-4afc-9b75-… Target
35770c78-edaf-4afc-9b75-… Target
35770c78-edaf-4afc-9b75-… Target
35770c78-edaf-4afc-9b75-… Target
35770c78-edaf-4afc-9b75-… Target
35770c78-edaf-4afc-9b75-… Target
35770c78-edaf-4afc-9b75-… Target
35770c78-edaf-4afc-9b75-… Target
35770c78-edaf-4afc-9b75-… Target

Recommendation 

Truthfully if the management network path failure scenario failed over storage providers in a quick and efficient manner, the recommended action would be to force the storage provider failover.  However, because of how long the provider failover can take, the best recommendation Pure can make at this time is to not use PowerCLI 12 until a fix for this has been released.