Troubleshooting: SRA 3.0.154 Fails to Authenticate Array Managers with Signed Certificates
Overview
Here is what the issue is with the certain types of certificates.
This issue is with dotnet trying to utilize libcurl attempting to pull the ssl/cert information from the FlashArray when the certificates have certain CDP and AIA configurations.
By default a Certification Authority with Microsoft has a file path, ldap path and http path to find the CDP and AIA. In some cases environments we've seen that the ldap path has been removed or omitted and only the http paths for the CRL distribution Point and Authority Information Access exists.
For example, how a default setup looks:
CRL Distribution Point [1]CRL Distribution Point Distribution Point Name: Full Name: URL=ldap:///CN=Sub-CA,CN=Sub-CA,CN=CDP,CN=Public Key Services,CN=Services,CN=Configuration,DC=alex,DC=purestorage,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint (ldap:///CN=Sub-CA,CN=Sub-CA,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=alex,DC=purestorage,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint) Authority Information Access [1]Authority Info Access Access Method=Certification Authority Issuer (1.3.6.1.5.5.7.48.2) Alternative Name: URL=ldap:///CN=Sub-CA,CN=AIA,CN=Public Key Services,CN=Services,CN=Configuration,DC=alex,DC=purestorage,DC=com?cACertificate?base?objectClass=certificationAuthority (ldap:///CN=Sub-CA,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=alex,DC=purestorage,DC=com?cACertificate?base?objectClass=certificationAuthority)
How a setup with only http setup looks:
CRL Distribution Point [1]CRL Distribution Point Distribution Point Name: Full Name: URL=http://Online-Sub-CA-1.alex.purestorage.com/CertEnroll/Online-Sub-CA-1.crl URL=http://Online-Sub-CA-2.alex.purestorage.com/CertEnroll/Online-Sub-CA-2.crl Authority Information Access [1]Authority Info Access Access Method=Certification Authority Issuer (1.3.6.1.5.5.7.48.2) Alternative Name: URL=http://online-sub-ca-1.alex.purestorage.com/CertEnroll/Online-Sub-CA-1.alex.purestorage.com_Online-Sub-CA-1.crt [2]Authority Info Access Access Method=Certification Authority Issuer (1.3.6.1.5.5.7.48.2) Alternative Name: URL=http://online-sub-ca-2.alex.purestorage.com/CertEnroll/Online-Sub-CA-2.alex.purestorage.com_Online-Sub-CA-2.crt
What we see from the SRA when only having the http path for the CDP, is it attempts to reach out to grab the ssl/cert information from the flasharray, dotnet tries to use libcurl, but our SRA image does not have curl installed by default and it fails with the following type of error.
Inner Exception='The SSL connection could not be established, see inner exception.' Inner Exception='The type initializer for 'Http' threw an exception.' Inner Exception='The type initializer for 'HttpInitializer' threw an exception.' Inner Exception='Unable to load shared library 'System.Net.Http.Native' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library libSystem.Net.Http.Native: No such file or directory'.
This looks to be related to some issues others have seen in containerized setups with dotnet: https://github.com/dotnet/corefx/issues/30003
Workaround
Here is the work around for SRA 3.0.154
This is resolved in the SRA 4.1 release.
With admin/root access to the SRM server, connect to the SRA and install curl.
There is a chance that even after enabling sshd on the SRM server that you are unable to access the SRM server via SSH. If that is the case refer to this post to update the sshd config to allow ssh access to the admin user.
Here is how this is done and what it looks like:
- SSH as admin to the SRM Server.
ssh admin@srm.server.ip.addy admin@UCS-VCSA-SRM-67 [ ~ ]$
- Find the docker container ID for the Pure SRA. There may be two SRA containers deployed, you'll want to make the changes to both containers.
admin@UCS-VCSA-SRM-67 [ ~ ]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a676c004546d a7d1c6b2045f "/srm/sra/command" 5 weeks ago Up 5 weeks romantic_jones admin@UCS-VCSA-SRM-67 [ ~ ]$
- Connect to the SRA container
admin@UCS-VCSA-SRM-67 [ ~ ]$ sudo docker exec -it a676c004546d /bin/sh /sra #
- Update the apt package and then install curl
/sra # apk update && apk add curl fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz v3.7.3-145-g40640b4440 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main] v3.7.3-146-g4f0b80be4d [http://dl-cdn.alpinelinux.org/alpine/v3.7/community] OK: 9061 distinct packages available (1/3) Installing libssh2 (1.8.2-r0) (2/3) Installing libcurl (7.61.1-r3) (3/3) Installing curl (7.61.1-r3) Executing busybox-1.27.2-r11.trigger OK: 16 MiB in 31 packages /sra #
- Exit the CLI and then run a device discovery
/sra # exit admin@UCS-VCSA-SRM-67 [ ~ ]$ exit
After getting curl installed, the array discover, device discovery, etc should all work.
Conclusion
Thus far in the investigation and reproduction this has only reproduced when the CRL Distribution Point and Authority Information Access only provide an https or http url in the certificate. This causes the SRA to attempt to reach out with libcurl which by default is not installed in the version of dotnet that the container is using. This is resolved in the SRA 4.1 release.
Please let Pure Support know or leave a comment in the KB if you have any questions or concerns with the process outlined in this KB.