Troubleshooting: PowerShell "Object Reference Not Set" Failure with PowerCLI 12
What is the Issue?
Pure Storage has identified an issue using PowerCLI versions 12.0 and 12.1 when running commands that query the Storage Providers that are registered in vCenter.
Pure Storage is working with VMware to root cause and fix the issue for a future release of PowerCLI.
For example, cmdlets such as Get-VasaStorageArray
and Get-SpbmReplicationGroup
will fail with the error "Object reference not set to an instance of an object."
> Get-VasaStorageArray Get-VasaStorageArray: 1/5/2021 4:48:07 PM Get-VasaStorageArray Object reference not set to an instance of an object. > Get-SpbmReplicationGroup Get-SpbmReplicationGroup: 1/5/2021 2:58:44 PM Get-SpbmReplicationGroup Object reference not set to an instance of an object.
In order to trigger this issue a vCenter Server will have a FlashArray's VASA providers registered as storage providers, but there can not have been a storage provider failover that has occurred after these providers have been registered.
The issue appears to be tied with the queryProviders call that is made to vCenter as part of these cmdlets. Specifically PowerCLI is not populating details for the standby storage provider. As PowerCLI is leaving values null on the standby provider, PowerShell is unable to parse the provider response and the null values as part of the query which will cause an failure in PowerShell with the object reference not set. This can be checked with PowerShell by issuing a Get-VasaProvider and pipe it to format-list.
Let's compare what we see with PowerCLI 11.1 and then PowerCLI 12.1:
PowerCLI 11.1
> Get-VasaProvider | format-list Version : 1.1.1 Description : LastSyncTime : 1/5/2021 4:33:43 PM Namespace : com.purestorage ProviderId : d79af7f6-ee6a-4be4-8d8a-06ae62a67212-0 Status : online SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile} Url : https://10.21.149.61:8084/version.xml VasaVersion : 3.0 ProviderType : Persistence Name : sn1-m20r2-c05-36-ct0 Id : VasaProvider-vasaProvider-158 Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-158/ Version : 1.1.1 Description : LastSyncTime : Namespace : com.purestorage ProviderId : d79af7f6-ee6a-4be4-8d8a-06ae62a67212-1 Status : online SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile} Url : https://10.21.149.62:8084/version.xml VasaVersion : 3.0 ProviderType : Unknown Name : sn1-m20r2-c05-36-ct1 Id : VasaProvider-vasaProvider-159 Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-159/
PowerCLI 12.1
> Get-VasaProvider | format-list Version : 1.1.1 Description : LastSyncTime : 1/5/2021 4:33:43 PM Namespace : com.purestorage ProviderId : d79af7f6-ee6a-4be4-8d8a-06ae62a67212-0 Status : online SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile} Url : https://10.21.149.61:8084/version.xml VasaVersion : 3.0 ProviderType : Persistence Name : sn1-m20r2-c05-36-ct0 Id : VasaProvider-vasaProvider-158 Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-158/ Version : Description : LastSyncTime : Namespace : ProviderId : Status : SupportedProviderProfile : Url : VasaVersion : ProviderType : Name : Id : VasaProvider-vasaProvider-159 Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-159/
Notice that with PowerCLI 11.1 that the standby provider still has the information populated whereas with PowerCLI 12.1 only the Id and Uid have values populated.
Workarounds or Fixes?
There are two ways that this issue can be corrected and one of them is not recommended:
- Use PowerCLI 11 until a fix is provided in PowerCLI 12 (recommended)
- Disable ETH0 on the active storage providers controller to force a provider failover
- Un-Register the Standby storage provider (not recommended)
Pure Storage does not recommend to un-register the standby provider as that will remove the HA aspect of VASA and if the active provider's controller goes down (rebooted or is powered off) then vSphere is unable to reach an active storage provider and the management path is now down from that vCenter to the FlashArray VASA service.
At this time Pure Storage recommends to either hold off using PowerCLI 12 until this is fixed or to force a storage provider failover. The reason that a storage provider failover will help is because this forces the standby provider to become active and when the provider becomes active the information will be populated when queryProviders is issued from PowerCLI in the future.
Here is an example where CT0 is currently the active provider and CT1 is the standby provider.
PS /Users/alex.carver> Get-VasaProvider | where {$_.Namespace -ne 'IOFILTERS' -and $_.Namespace -ne 'VSAN'} Name Status VasaVersion LastSyncTime Namespace Url ---- ------ ----------- ------------ --------- --- sn1-x70-b05-33-ct0 online 3.0 1/8/2021 2:53:29 PM com.purestorage https://10.21.149.40:8084/version.… PS /Users/alex.carver> Get-VasaProvider | where {$_.Namespace -ne 'IOFILTERS' -and $_.Namespace -ne 'VSAN'} | format-list Version : 1.1.1 Description : LastSyncTime : 1/8/2021 2:53:29 PM Namespace : com.purestorage ProviderId : 395a60c2-5803-40be-95b7-029b1b3ffc3e-0 Status : online SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile} Url : https://10.21.149.40:8084/version.xml VasaVersion : 3.0 ProviderType : Persistence Name : sn1-x70-b05-33-ct0 Id : VasaProvider-vasaProvider-164 Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-164/ Version : Description : LastSyncTime : Namespace : ProviderId : Status : SupportedProviderProfile : Url : VasaVersion : ProviderType : Name : Id : VasaProvider-vasaProvider-165 Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-165/
While logged in as an array admin, disable the active providers ETH0 interface (in this case it's ct0) and this can be done from the GUI as well if the CLI is not preferred.
pureuser@sn1-x70-b05-33> purenetwork disable ct0.eth0 Name Enabled Subnet Address Mask Gateway MTU MAC Speed Services Subinterfaces ct0.eth0 False - 10.21.149.40 255.255.255.0 10.21.149.1 1500 24:a9:37:01:cc:f6 1.00 Gb/s management -
Please note that the storage provider failure due to network connectivity failures can take between 15 and 30 minutes to complete. Check the make sure that the standby provider has taken over as active before re-enabling ETH0. Also try to plan this activity when a 30 minute management path failure can be tolerated.
Then check for when the standby provider becomes the active provider, this can be done with the get-VasaProvider cmdlet previously used.
PS /Users/alex.carver> Get-VasaProvider | where {$_.Namespace -ne 'IOFILTERS' -and $_.Namespace -ne 'VSAN'} Name Status VasaVersion LastSyncTime Namespace Url ---- ------ ----------- ------------ --------- --- sn1-x70-b05-33-ct0 offline 3.0 1/8/2021 3:23:30 PM com.purestorage https://10.21.149.40:8084/version.… sn1-x70-b05-33-ct1 online 3.0 1/8/2021 4:17:25 PM com.purestorage https://10.21.149.41:8084/version.… PS /Users/alex.carver> Get-VasaProvider | where {$_.Namespace -ne 'IOFILTERS' -and $_.Namespace -ne 'VSAN'} | format-list Version : 1.1.1 Description : LastSyncTime : 1/8/2021 3:23:30 PM Namespace : com.purestorage ProviderId : 395a60c2-5803-40be-95b7-029b1b3ffc3e-0 Status : offline SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile} Url : https://10.21.149.40:8084/version.xml VasaVersion : 3.0 ProviderType : Persistence Name : sn1-x70-b05-33-ct0 Id : VasaProvider-vasaProvider-164 Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-164/ Version : 1.1.1 Description : LastSyncTime : 1/8/2021 4:17:25 PM Namespace : com.purestorage ProviderId : 395a60c2-5803-40be-95b7-029b1b3ffc3e-1 Status : online SupportedProviderProfile : {ProfileBasedManagementProfile, ReplicationProfile} Url : https://10.21.149.41:8084/version.xml VasaVersion : 3.0 ProviderType : Persistence Name : sn1-x70-b05-33-ct1 Id : VasaProvider-vasaProvider-165 Uid : /VIServer=purecloud\alex@10.21.202.43:443/VasaProvider=VasaProvider-vasaProvider-165/ pureuser@sn1-x70-b05-33> purenetwork enable ct0.eth0 Name Enabled Subnet Address Mask Gateway MTU MAC Speed Services Subinterfaces ct0.eth0 True - 10.21.149.40 255.255.255.0 10.21.149.1 1500 24:a9:37:01:cc:f6 1.00 Gb/s management -
Okay, now you will notice that the vasaProvider-165 is populated in we see that ct0 is offline and ct1 is now appearing in the provider list. You can re-enable ETH0 from the FlashArray at this point. Then you can check Get-VasaStorageArray and Get-SpbmReplicationGroup to see if everything is corrected.
PS /Users/alex.carver> Get-VasaStorageArray | where {$_.ModelId -eq 'FlashArray'} Name ModelId VendorId ---- ------- -------- sn1-x70-b05-33 FlashArray PURE PS /Users/alex.carver> Get-SpbmReplicationGroup -VasaProvider 'sn1-x70-b05-33-ct1' Name ReplicationState ---- ---------------- sn1-x70-b05-33:x70-1-pol… Source sn1-x70-b05-33:x70-1-pol… Source sn1-x70-b05-33:x70-1-pol… Source sn1-x70-b05-33:x70-1-pol… Source sn1-x70-b05-33:x70-1-pol… Source sn1-x70-b05-33:x70-1-pol… Source sn1-x70-b05-33:x70-1-pol… Source sn1-x70-b05-33:x70-1-pol… Source sn1-x70-b05-33:VMFS-WSFC… Source sn1-x70-b05-33:VMFS-WSFC… Source sn1-x70-b05-33:pwsh-test… Source d79af7f6-ee6a-4be4-8d8a-… Target d79af7f6-ee6a-4be4-8d8a-… Target d79af7f6-ee6a-4be4-8d8a-… Target 35770c78-edaf-4afc-9b75-… Target 35770c78-edaf-4afc-9b75-… Target 35770c78-edaf-4afc-9b75-… Target 35770c78-edaf-4afc-9b75-… Target 35770c78-edaf-4afc-9b75-… Target 35770c78-edaf-4afc-9b75-… Target 35770c78-edaf-4afc-9b75-… Target 35770c78-edaf-4afc-9b75-… Target 35770c78-edaf-4afc-9b75-… Target
Recommendation
Truthfully if the management network path failure scenario failed over storage providers in a quick and efficient manner, the recommended action would be to force the storage provider failover. However, because of how long the provider failover can take, the best recommendation Pure can make at this time is to not use PowerCLI 12 until a fix for this has been released.