Web Guide: Managing the VASA Certificates with purecert via the CLI
Overview
Now that an FlashArray Admin user can manage the VASA Certificates with the purecert CLI command, there is a need for documentation about how to manage those certificates and what can be done with purecert. There is an existing KB that details how to setup Multi vCenter with the VASA Provider, as well as covering how to import CA signed certificates to the VASA Providers. This KB will go over some of the purecert command examples and use cases outside of the workflow needed to import certificates for Multi vCenter support.
Both management interfaces must be configured on both controllers and both arrays with enabled and active links. The management interfaces are as follows:
- FlashArray//XR4 - ct0.eth4, ct0.eth5, ct1.eth4, ct1.eth5
- All other FlashArray Models - ct0.eth0, ct0.eth1, ct1.eth0
Commands and Examples
Here is a quick look at the common purecert commands used to manage the VASA certificate. We will then look at some use cases for managing the VASA certificate with purecert.
purecert list
Using purecert list will help show what certificates are currently in use for both the management certificates and VASA certificates. Let's look at some examples of different certificates listed with purecert list.
- Here is an example of imported CA signed certificates for VASA-CT0 and VASA-CT1 in use on the FlashArray:
purecert list Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name management imported 2048 10.21.88.112 Sub-CA 2019-09-26 12:45:28 PDT 2021-09-25 12:45:28 PDT US California Mountain View Pure Storage Solutions Engineering administrator@sso.alex.purestorage.com 10.21.88.112 vasa-ct0 imported 2048 10.21.88.113 CA 2019-08-09 14:08:52 PDT 2021-08-08 14:08:52 PDT US California Mountain View Pure Storage Solutions Engineering - 10.21.88.113 vasa-ct1 imported 2048 10.21.88.114 CA 2019-08-09 14:09:17 PDT 2021-08-08 14:09:17 PDT US California Mountain View Pure Storage Solutions Engineering - 10.21.88.114
- Here is an example of local self signed certificates for VASA-CT0 and VASA-CT1 on the FlashArray:
purecert list Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name management imported 2048 10.21.88.112 Sub-CA 2019-09-26 12:45:28 PDT 2021-09-25 12:45:28 PDT US California Mountain View Pure Storage Solutions Engineering administrator@sso.alex.purestorage.com 10.21.88.112 vasa-ct0 self-signed 4096 10.21.88.113 10.21.88.113 2019-10-28 07:37:42 PDT 2029-10-25 07:37:42 PDT US California Mountain View Pure Storage Pure Storage - 10.21.88.113 vasa-ct1 self-signed 4096 10.21.88.114 10.21.88.114 2019-10-28 07:37:59 PDT 2029-10-25 07:37:59 PDT US California Mountain View Pure Storage Pure Storage - 10.21.88.114
- Then here is an example of VMCA signed certificates in use for VASA-CT0 and VASA-CT1 on the FlashArray:
purecert list Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name management imported 2048 10.21.88.112 Sub-CA 2019-09-26 12:45:28 PDT 2021-09-25 12:45:28 PDT US California Mountain View Pure Storage Solutions Engineering administrator@sso.alex.purestorage.com 10.21.88.112 vasa-ct0 imported 2048 10.21.88.113 CA 2019-10-27 07:39:49 PDT 2020-10-27 07:39:49 PDT US - - Pure Storage Pure Storage - 10.21.88.113 vasa-ct1 imported 2048 10.21.88.114 CA 2019-10-27 07:50:06 PDT 2020-10-27 07:50:06 PDT US - - Pure Storage Pure Storage - 10.21.88.114
Using purecert list allows you to quickly look at the certificates currently in used on the FlashArray. You can also use --certificate with purecert list to show the certificate itself should you want to inspect it:
purecert list --certificate vasa-ct0 -----BEGIN CERTIFICATE----- MIIEBzCCAu+gAwIBAgIJAO8DQ2CZ/J4OMA0GCSqGSIb3DQEBCwUAMIGoMQswCQYDVQQDDAJDQTEX MBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZFgVsb2NhbDELMAkGA1UEBhMC VVMxEzARBgNVBAgMCkNhbGlmb3JuaWExKjAoBgNVBAoMIXN0YWdpbmctdmNzYS5hbGV4LnB1cmVz dG9yYWdlLmNvbTEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVyaW5nMB4XDTE5MTAyNzE0Mzk0OVoX DTIwMTAyNzE0Mzk0OVowUjELMAkGA1UEBhMCVVMxFTATBgNVBAoTDFB1cmUgU3RvcmFnZTEVMBMG A1UECxMMUHVyZSBTdG9yYWdlMRUwEwYDVQQDEwwxMC4yMS44OC4xMTMwggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQCo26bbs43ZgAEqOD17j5F1P1Q+5lNZUmTaADxaXQkgyVx8LAI+Ldwm 12dIO+4auHzRCqa54Hzv9cNtUH/46wMQA2t1y4LvDAB1/82J3jdSoze2TjfLKXkrsKP/i63B8L6s H4iYn7ZgvdtlCzw6eKLLX+jQM/JVYxU0wPc31PxR7DNt+dzHCelShbXwckJTrggh8jXJGLitAqXA JurDz/4kS9lGzKsV5Qy9eSsQGftcmzaQkIpfAfXLGDy3ja0A/psttzPB5t4pfqo2eI0/iVZ4Oetr 1Qs5qzBPh60/5kAycX9Of7GY7B4nfqQJsKRSDH1QjtCt+FPaN3hT5H+xTz5tAgMBAAGjgYgwgYUw DwYDVR0RBAgwBocEChVYcTAfBgNVHSMEGDAWgBQl8TMF+lV/HZgGvz9oV0wkZSHaLzBRBggrBgEF BQcBAQRFMEMwQQYIKwYBBQUHMAKGNWh0dHBzOi8vc3RhZ2luZy12Y3NhLmFsZXgucHVyZXN0b3Jh Z2UuY29tL2FmZC92ZWNzL2NhMA0GCSqGSIb3DQEBCwUAA4IBAQBVl5fY7ogjxAUeH4cTGRMOEcHZ 4kxXc08+MBznq63I9Gw2IWfGj08UP4KJsQUwZ7WwFcFJmOAFdlk/uVK9QzRLfQPq2Iz+AGpYiXs3 k4NkoSWnk18YbYFaw9ItVgsor5XwoIVwhQvOypOivBXZGfDgORFWXXJYIzhbTWWO3bIxaqu+AexW ddNWYdHFvCP9Kn4yWWOdpuW+7Pi1nUKk1cqBBcbn7lftHWd6la7VRYg4/Y9w69jlgvo4TDIzI/r4 oe+rbh92/Z5MKuDD3rCf8Qw7jtyte7PR6hoPrfda2FMwXi0X8XX2UNjp64jwOM9HnWs/JGne1ARW OIJ6MteEwmwu -----END CERTIFICATE-----
purecert delete
In the event that you need to re-create the certificate for vasa-ct0 or vasa-ct1, you will first need to delete the certificate in use. (Note: If you are deleting VASA certs for the purpose of not receiving further certificate expiration alerts, follow 1-3 of the "Resetting the VASA Certificates with purecert" section below instead).
Delete the certificate in use:
# purecert list Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name management imported 2048 10.21.88.112 Sub-CA 2019-09-26 12:45:28 PDT 2021-09-25 12:45:28 PDT US California Mountain View Pure Storage Solutions Engineering administrator@sso.alex.purestorage.com 10.21.88.112 vasa-ct0 imported 2048 10.21.88.113 CA 2019-08-09 14:08:52 PDT 2021-08-08 14:08:52 PDT US California Mountain View Pure Storage Solutions Engineering - 10.21.88.113 vasa-ct1 imported 2048 10.21.88.114 CA 2019-08-09 14:09:17 PDT 2021-08-08 14:09:17 PDT US California Mountain View Pure Storage Solutions Engineering - 10.21.88.114 # purecert delete vasa-ct0 Name vasa-ct0 # purecert delete vasa-ct1 Name vasa-ct1 # purecert list Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name management imported 2048 10.21.88.112 Sub-CA 2019-09-26 12:45:28 PDT 2021-09-25 12:45:28 PDT US California Mountain View Pure Storage Solutions Engineering administrator@sso.alex.purestorage.com 10.21.88.112
The only time that purecert delete should need to be used is in the event that you need to generate a new self-signed certificate for vasa-ct0 and vasa-ct1. Otherwise, you can import a signed certificate or self signed certificate with purecert as well.
purecert create
With purecert create, you will have the ability to create both self signed certificates as well as Certificate Signing Requests (CSRs). One thing to note about the CSR is that currently Pure Storage does not support creating CSRs with custom Subject Alternative Name (SAN) entries. This means that the CA that is signing the request will need to have the ability to have SAN entries provided as part of issuing the certificate.
Please note that the common name should be the IP address for CT0.ETH0 for VASA-CT0 and the IP Address for CT1.ETH0 for VASA-CT1. There are instances where you may want to use the IP address for CT0.ETH1 for VASA-CT0 or the IP address for CT1.ETH1 for VASA-CT1. This can be done as well. The IP addresses can be found by running a "purenetwork list".
Here is an example of creating new self signed certificates for vasa-ct0 and vasa-ct1:
# purecert create --common-name 10.21.88.113 --country US --self-signed --key-size 4096 --locality 'Mountain View' --organization 'Pure Storage' --organizational-unit 'Pure Storage' --state California vasa-ct0 Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name vasa-ct0 self-signed 4096 10.21.88.113 10.21.88.113 2019-10-28 07:37:42 PDT 2029-10-25 07:37:42 PDT US California Mountain View Pure Storage Pure Storage - 10.21.88.113 # purecert create --common-name 10.21.88.114 --country US --self-signed --key-size 4096 --locality 'Mountain View' --organization 'Pure Storage' --organizational-unit 'Pure Storage' --state California vasa-ct1 Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name vasa-ct1 self-signed 4096 10.21.88.114 10.21.88.114 2019-10-28 07:37:59 PDT 2029-10-25 07:37:59 PDT US California Mountain View Pure Storage Pure Storage - 10.21.88.114
The reason to do this for vasa-ct0 and vasa-ct1 would be in the case that previous certificate is in question or a bad format that is causing vCenter to fail registering the storage providers. To rule out the current certificate you would delete the existing certificate, generate a self signed cert with the Organization and Organizational unit set to 'Pure Storage', and then re-register the storage providers in the desired vCenter Server.
Resetting the VASA Certificates with purecert
Here is the workflow that you can follow to reset the VASA certificate for vasa-ct0 and vasa-ct1.
- Delete the existing VASA certificates:
# purecert delete vasa-ct0 Name vasa-ct0 # purecert delete vasa-ct1 Name vasa-ct1 # purecert list Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name management imported 2048 10.21.88.112 Sub-CA 2019-09-26 12:45:28 PDT 2021-09-25 12:45:28 PDT US California Mountain View Pure Storage Solutions Engineering administrator@sso.alex.purestorage.com 10.21.88.112
- Create self signed VASA Certificates that have the Organization and Organizational Unit set to 'Pure Storage':
# purecert create --common-name 10.21.88.113 --country US --self-signed --key-size 4096 --locality 'Mountain View' --organization 'Pure Storage' --organizational-unit 'Pure Storage' --state California vasa-ct0 Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name vasa-ct0 self-signed 4096 10.21.88.113 10.21.88.113 2019-10-28 07:37:42 PDT 2029-10-25 07:37:42 PDT US California Mountain View Pure Storage Pure Storage - 10.21.88.113 # purecert create --common-name 10.21.88.114 --country US --self-signed --key-size 4096 --locality 'Mountain View' --organization 'Pure Storage' --organizational-unit 'Pure Storage' --state California vasa-ct1 Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name vasa-ct1 self-signed 4096 10.21.88.114 10.21.88.114 2019-10-28 07:37:59 PDT 2029-10-25 07:37:59 PDT US California Mountain View Pure Storage Pure Storage - 10.21.88.114
When resetting the VASA certificate the O and OU must be set to "Pure Storage" for both. This is a requirement for resetting the certificate.
This is due to how the VASA service determines if the certificate is a custom/imported certificate or if it is a default certificate that will allow a vCenter Server to import it's own certificate to the VASA service. Should the O and OU be anything else than "Pure Storage" this will not create the correct default certificate in VASA and vCenter will fail when registering the storage Provider. - Check that the new certificates show up in purecert list:
purecert list Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name management imported 2048 10.21.88.112 Sub-CA 2019-09-26 12:45:28 PDT 2021-09-25 12:45:28 PDT US California Mountain View Pure Storage Solutions Engineering administrator@sso.alex.purestorage.com 10.21.88.112 vasa-ct0 self-signed 4096 10.21.88.113 10.21.88.113 2019-10-28 07:37:42 PDT 2029-10-25 07:37:42 PDT US California Mountain View Pure Storage Pure Storage - 10.21.88.113 vasa-ct1 self-signed 4096 10.21.88.114 10.21.88.114 2019-10-28 07:37:59 PDT 2029-10-25 07:37:59 PDT US California Mountain View Pure Storage Pure Storage - 10.21.88.114
- Register the Storage Providers in vCenter:
Follow the section "Registering FlashArray VASA Providers with the Plugin" from this KB to register the Storage Providers in vCenter.Note that the initial attempt to register the Storage Provider may fail. This could be due to the VASA Provider needing to update the Self Signed Certificate to include the SAN entry for the IP address. Generally speaking, registering the Storage Provider will work on the first attempt; however, there is a chance that you may want to wait 5 to 10 minutes. The job that updates the VASA certificates SAN entries is scheduled to run every 5 minutes.
In the event that the Storage Provider needs to be re-registered as soon as possible, please open up a support case and the TSE working the case can restart the VASA service on each controller to force the SAN entry to be updated in the certificate stored in VASA. - Confirm that the VMCA signed certificate is imported to vCenter:
purecert list Name Status Key Size Issued To Issued By Valid From Valid To Country State/Province Locality Organization Organizational Unit Email Common Name management imported 2048 10.21.88.112 Sub-CA 2019-09-26 12:45:28 PDT 2021-09-25 12:45:28 PDT US California Mountain View Pure Storage Solutions Engineering administrator@sso.alex.purestorage.com 10.21.88.112 vasa-ct0 imported 2048 10.21.88.113 CA 2019-10-27 07:39:49 PDT 2020-10-27 07:39:49 PDT US - - Pure Storage Pure Storage - 10.21.88.113 vasa-ct1 imported 2048 10.21.88.114 CA 2019-10-27 07:50:06 PDT 2020-10-27 07:50:06 PDT US - - Pure Storage Pure Storage - 10.21.88.114
Keep in mind that this does not configure and setup the VASA Providers for multi vCenter support. These VMCA signed certificates will only allow one VMware SSO to register and authenticate with the VASA Providers.
Null VP URL on ESXi Hosts after Re-Registering the Storage Provider
When resetting the VASA Provider Certificate, Pure has noticed that the vvold service on the ESXi hosts will have the old certificate cached and will fail to authenticate with the VASA Provider. Essentially, the vvold service needs to have the ssl_reset ran against that. The easiest way to do that is to refresh the CA certs from vCenter to the ESXi hosts. As part of that process, the ESXi hosts will update the ssl certs on the ESXi host's services. Here is an example of using PowerShell and PowerCLI to refresh the CA Root Certs on the ESXi hosts in vCenter.
Leverage the following workflow:
## Connect to the vCenter Server ## Connect-VIServer -server vcenter-server ## Get the ESXi hosts and set it to a variable ## $hosts = get-vmhost ## Start the Service Instance ## $si = Get-View ServiceInstance ## Start the certificate Manager view ## $certMgr = Get-View -Id $si.Content.CertificateManager ## Using the Cert Manager, refresh the ESXi hosts Certs ## ## This pushes all certificates in the TRUSTED_ROOTS store in the vCenter Server VECS store to the host. ## $certMgr.CertMgrRefreshCACertificatesAndCRLs($Hosts.ExtensionData.MoRef) ## Now in vCenter the vvol datastore should be accessible for each of those hosts. No need to do the ssl_reset and restart on VVold ##
Here is an example of this workflow:
PS C:\> Connect-VIServer -Server dev-vcsa
Name Port User
---- ---- ----
dev-vcsa 443 ALEX\Carver
PS C:\> Get-Cluster -Name "Dev Cluster"
Name HAEnabled HAFailover DrsEnabled DrsAutomationLevel
Level
---- --------- ---------- ---------- ------------------
Dev Cluster True 1 True FullyAutomated
PS C:\> $ESXi_Cluster = Get-Cluster -Name "Dev Cluster"
PS C:\> $ESXi_Cluster | Get-VMHost
Name ConnectionState PowerState NumCpu CpuUsageMhz CpuTotalMhz MemoryUsageGB MemoryTotalGB Version
---- --------------- ---------- ------ ----------- ----------- ------------- ------------- -------
esxi-7.alex.pures... Connected PoweredOn 16 151 38304 14.586 255.897 6.7.0
esxi-6.alex.pures... Connected PoweredOn 20 141 43880 16.166 255.892 6.7.0
esxi-4.alex.pures... Connected PoweredOn 20 94 43880 8.945 255.892 6.7.0
PS C:\> $hosts = $ESXi_Cluster | Get-VMHost
PS C:\> $hosts
Name ConnectionState PowerState NumCpu CpuUsageMhz CpuTotalMhz MemoryUsageGB MemoryTotalGB Version
---- --------------- ---------- ------ ----------- ----------- ------------- ------------- -------
esxi-7.alex.pures... Connected PoweredOn 16 151 38304 14.586 255.897 6.7.0
esxi-6.alex.pures... Connected PoweredOn 20 141 43880 16.166 255.892 6.7.0
esxi-4.alex.pures... Connected PoweredOn 20 94 43880 8.945 255.892 6.7.0
PS C:\> $si = Get-View ServiceInstance
PS C:\> $certMgr = Get-View -Id $si.Content.CertificateManager
PS C:\> $certMgr.CertMgrRefreshCACertificatesAndCRLs($Hosts.ExtensionData.MoRef)
PS C:\>
Once this is all completed, the vvold service should be able to authenticate with the VASA Provider.
Checking or Inspecting the VASA Certificate
There may be times that part of the troubleshooting storage providers failing to register needs to include inspecting the VASA Certificate. vSphere will not be able to authenticate and connect to a VASA Provider if the certificate is not following x509 security standards with the certificate. The most common issues that are seen with certificate issues are actually on the vSphere and VMCA side of things and not with VASA. This is not to say that issues with the VASA certificate can't happen though.
Here are a couple issues to check when inspecting the VASA Certificate:
- Does the Common Name match the Subject Alternative Name (SAN)?
- When resetting the certificate, does the O and OU match "Pure Storage" for both?
The easiest way to inspect the certificate would be by navigating to the VASA Provider URL in a web browser and then inspecting the certificate. The example that is shown below is of an imported CA Signed certificate to VASA-CT0.
From a Chrome Browser on either Mac or Windows, follow these steps:
Using Chrome on Windows
|
Using Chrome on Mac
|
From a Firefox Browser on either Mac or Windows, follow these steps:
Using Firefox on Windows
|
Using Firefox on Mac
|
The point of checking the certificate is really to make sure that the Organization and Organizational Unit match what was provided and that there is a SAN Entry for the Common Name given. This helps confirm that the certificate was regenerated or imported correctly and that vCenter should not have problems with the certificate for the VASA provider.
Related KBs and References
Here are some KBs or blog posts that are related to managing the VASA certificates or using purecert.