vSphere Plugin Troubleshooting Guide
The purpose of this KB is to give a general troubleshooting workflow of Pure's vSphere HTML plugin (locally installed and remotely installed).
Helpful Plugin KBs
- Installing the local vSphere Plugin
- Installing the remote vSphere Plugin
- vSphere Client Plugin user guides
- Plugin workflow guides
- Connecting the vSphere Client Plugin to Pure1
- vSphere Client Plugin release notes
- RBAC support in the plugin
Terminology
It can be helpful to get an understanding of the language used with the vSphere plugin before diving in to an issue.
| Component | Description |
|---|---|
| Local vSphere Plugin (HTML) | Managed through Powershell, the FlashArray GUI or vRealize Orchestrator (vRO). |
| Remote vSphere Plugin | Managed through the Pure VMware appliance. VMware will eventually not support vSphere plugins being installed locally on the vCenter; because of this, all new vSphere Plugin features will be included for the remote plugin and not the local plugin. |
| Flash vSphere Plugin | Currently unsupported version of the vSphere Plugin. Flash by Adobe is End of Life (EOL) as of December 2020, so it is strongly recommended to use the Remote or Local HTML plugin. |
Remote Plugin Log Investigation
From the VMware appliance, run the command below and follow the instructions at the prompt:
puresupport bundle
Local Plugin Log Investigation
Virgo logs are where plugin log information is stored for the vCenter.
Virgo logs for the HTML plugin are located at /var/log/vmware/vsphere-ui/logs live on the vCenter and at the same path in the support bundle from this KB.
The most recent log file is vsphere_client_virgo.log. Log files are rolled and compressed periodically and will be renamed to vsphere_client_virgo_x.log.zip.
When searching the virgo logs for issues related to the Pure Plugin, there can be a lot of information that isn't related to the Pure plugin. Using less, searching for com.purestorage.purestoragehtml is helpful because this will pull up information related to the Pure vCenter plugin.
Common Issues and Resolutions
It is recommended to be on the latest version of the plugin before troubleshooting any issues. Every plugin release has new fixes and enhancements; see this KB for more information.
- Remote Plugin: Unable to add/view arrays or "Authenticate with Pure1 to use this feature" returned despite successful Pure1 Connection
-
There are three possible symptoms of this issue:
- In the FlashArray list page, the error "Error retrieving array list. Please try again later." is returned.
- When trying to import arrays via Pure1, the error "Authenticate with Pure1 to use this feature" is returned despite previously successful registration with Pure1 through the plugin.
- When adding an array manually, a "no permissions" error is returned.
Resolutions:
These errors can indicate two issues:- A .local domain is being used by the vCenter (vcenter.purestorage.local) and this is preventing the OVA from being able to make requests via the vCenter's DNS name. To resolve this, follow step 15 from the Online Deployment Procedure for the remote plugin.
- Pure's vSphere plugin on Pure's VMware appliance has been registered with more than one vCenter without Enhanced Linked Mode (ELM) connecting the two vCenters. Deploy one of Pure's VMware appliance for each of the vCenters in the environment.
- In the FlashArray list page, the error "Error retrieving array list. Please try again later." is returned.
- Installation Failing
-
If you are having issues getting the plugin installed onto your vCenter, it's a great idea to check the virgo logs from above. You might see something like the below:
DOWNLOAD_FAILED: Error downloading plugin package com.purestorage.purestoragehtml:4.2.0 from https://ArrayManagementIPAddress/download/purestorage-vsphere-plugin.zip?version=4.2.0. Reason: Download error. Make sure that the URL is reachable and the thumbprint is correct. java.net.SocketTimeoutException: connect timed out
This log line often means there is a communication issue between wherever the plugin is being downloaded from (array or where the PowerShell files are hosted) and going to the vCenter.
Resolution:
Port 443 needs to be opened up between the vCenter and the location it is being downloaded from. If you are installing with PowerShell, you'll need to open up 443 between the vCenter's management interface and pure-vmware-plugin-repository.s3-us-west-1.amazonaws.com; if you are installing from the array, you'll need to open up 443 between the vCenter's management interface and the array's management IP addresses. - Plugin Not Visible in vCenter
-
After successfully installing the plugin, it is not visible in vCenter.
In the virgo log, you should see this after a successful plugin installation:
[2020-01-27T20:54:38.869Z] [INFO ] vc-extensionmanager-pool-70 com.vmware.vise.vim.extension.PluginStatusTaskManager DEPLOYED: Deployed plugin package com.purestorage.purestoragehtml:4.2.0 successfully.
If you see the above log line, after 15 minutes or less the Pure Storage link should show up in the main Menu in vCenter:
If the Pure Storage link does not show up after waiting for the vCenter to fully deploy the plugin, there might be a need to restart the vsphere-ui service on the vCenter or, in extremely rare circumstances, the entire vCenter. To validate the need for this, first ensure that the MOB recognizes the plugin as installed. Go to https://vCenterURL/mob and log in with the normal vCenter credentials.
Click content:
Click ExtensionManager:
Click (more...):
Finally, we can validate that we have successfully installed the HTML Plugin on this page if you see this entry. If you click on extensionList["com.purestorage.purestoragehtml"] it will show you the version of the plugin that was installed (in our case, 4.3.1):
Resolution:
If you see the expected verison in the MOB and the Pure Storage link is not showing up in the vCenter menu, you will likely need to restart the vsphere-ui service on the vCenter by running the command service-control --restart vsphere-ui on the vCenter's CLI. Although this service restart should not be disruptive, it should be checked with VMware support.If restarting the vsphere-ui service doesn't resolve the issue, on rare occasions, the entire vCenter has needed to be restarted. This recommendation should come from VMware support.
- Cannot Create Datastores
-
When trying to create a datastore through the plugin, you get a error like Error loading array list or it will hang for a long time trying to select a compute resource in the 4 Compute Resource step and never populate any information:
Resolution:
This issue has been fixed in Pure vCenter Plugin version 4.4 and later. - Performance Charts Require Credentials or Showing Incorrect Data
-
In the Monitor section of a datastore, there is a Pure Storage section for Capacity and Performance; sometimes these will show incorrect data or show a login prompt:
Or fail to connect all together:
Resolution: In Purity, the URLs on the array that the vSphere Plugin uses to populate data into the plugin have changed, causing the charts in the vSphere plugin to stop working in some circumstances. Pure has resolved this in Pure vCenter plugin version 4.5.0 and later. Unable to find host or host group: Unable to find corresponding hostgroupwhen trying to add a datastore-
The cause of this error is a mismatch between the Cluster/Host arrangement on vCenter vs. the hostgroup/host arrangement on the FlashArray.Ideally, a one-to-one relationship will exist between the Cluster on vSphere to a hostgroup on the FlashArray. All hosts within the Cluster will be represented on the FlashArray with a host entry, with each FlashArray host entry containing the IQN or WWN of the ESX host.If there are ESX hosts within the Cluster on vSphere which do not have a host entry within the corresponding hostgroup, you will receive the error mentioned above.Example:In the vCenter server, if you have a cluster with:ClusterA with:HostA (iqn-1)HostB (iqn-2)On the FlashArray, you need to make sure there is a hostgroup that has a host entry corresponding to each ESX host in vSphere:HostGroupA with:Host1 (iqn-1)Host2 (iqn-2)
The name of the hosts and cluster do not matter -- Only IQNs and WWNs are referenced.
- Adding an Array Fails in vCenter When an Underscore is Used in Username
-
vCenter does not allow underscore usage in either host names or domain names, and this restriction also carries over to user names.
If a user name is created with an underscore in the "add array" section of the plugin it will error out with an error similar to below in the virgo logs:
https://VCENTER_PROD.EXAMPLE.COM:12443>/vmwb/logbrowser: Host name may not be null ERROR [ERROR] http-bio-9443-exec-8 System.err java.lang.IllegalArgumentException: Host name may not be null ERROR [ERROR] http-bio-9443-exec-8 com.vmware.loganalyzer.provider.LogBrowserServiceImpl Error when searching service in LookupService, urn:logbrowser:logbrowser java.lang.Exception: https://VCENTER_PROD.EXAMPLE.COM:12443/vmwb/logbrowser: Host name may not be nullThis is an inherited issue, and to get the user name to successfully work in the 'Add Array' section, omit the underscore.
Fore more information, please see this VMware KB.
- vSphere Plugin 4.2.0 and 3.1.3 does not allow adding arrays in certain versions of vCenter
-
The purpose of this KB is to give a general Pure's vSphere plugin (HTML) troubleshooting workflow.
After the plugin is successfully installed, if you aren't able to add arrays to the plugin, you might have an issue that is fixed in plugin version 4.2.1. This issue has been seen on vCenter 5.5, 6.0 and 6.5 with the plugin versions 4.2 and 3.1.3.
In the HTML GUI, this is the error you'll see:
Whereas in FLEX/FLASH you'll see something different:
In this latter case it's vague so you'll need to work further through the Troubleshooting steps to verify the presence of SocketFactory errors in vsphere_client_virgo.log to know whether you've hit this issue for sure or not.
To confirm the issue, please follow the below steps.
1. Go to the virgo logs on the vCenter; if the customer is using HTML, they'll be at this path:
/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.logIf the customer is using the Flex (Flash) client, they'll be at this path:
/var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log2. You're going to look for this error:
errorMessage='javax/net/SocketFactory'Here is an example of the full body of the error on an HTML client:
[2019-12-13T02:23:56.831Z] [INFO ] SsoTokenRenewalExecutor-223 ######## 100029 200004 com.vmware.vsphere.client.security.sso.SsoTokenLifetimeManager sessionId 100029, clientId 200004, Token expiration time: Sun Jan 12 02:23:56 UTC 2020 Token renewal scheduled approximately for: Sun Jan 12 02:18:56 UTC 2020 [2019-12-13T02:25:37.205Z] [WARN ] http-bio-5090-exec-7 com.purestorage.purestoragehtml.services.FlashArrayServiceImpl Caught throwable. java.lang.NoClassDefFoundError: javax/net/SocketFactory at org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:525) at org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:387) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:168) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:605) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:440) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.purestorage.purestoragehtml.services.util.FlashArrayHelper.checkConnection(Unknown Source) at com.purestorage.purestoragehtml.services.FlashArrayServiceImpl$1.invoke(Unknown Source) at com.purestorage.purestoragehtml.services.FlashArrayServiceImpl$1.invoke(Unknown Source) at com.purestorage.purestoragehtml.services.logging.TypedPureOperationLogger.invoke(Unknown Source) at com.purestorage.purestoragehtml.services.FlashArrayServiceImpl.createFlashArray(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at com.sun.proxy.$Proxy778.createFlashArray(Unknown Source) at com.purestorage.purestoragehtml.mvc.controller.FlashArrayServiceController.createFlashArray(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:100) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:604) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:565) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.vmware.vise.security.SessionManagementFilter.doFilter(SessionManagementFilter.java:134) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:689) at org.eclipse.virgo.web.tomcat.support.ApplicationNameTrackingValve.invoke(ApplicationNameTrackingValve.java:33) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1137) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)[2019-12-13T02:25:37.207Z] [ERROR] http-bio-5090-exec-7 com.purestorage.purestoragehtml.services.FlashArrayServiceImpl createFlashArray : error = TypedPureResponse{error=PureResponseError{code=-4, errorMessage='javax/net/SocketFactory', detailMessage='null'}, result=null} [2019-12-13T02:25:37.208Z] [INFO ] http-bio-5090-exec-7 com.purestorage.purestoragehtml.services.FlashArrayServiceImpl createFlashArray : duration = 4 ms, params = CreateFlashArrayRequest(name=420-1, url=flasharray-m20-1.purecloud.com, username=pureuser, password=pureuser), result = TypedPureResponse{error=PureResponseError{code=-4, errorMessage='javax/net/SocketFactory', detailMessage='null'}, result=null} ^CResolution:
There are two options:1. If you are using HTML, update to plugin version 4.2.1 or newer (install 4.2.1 directly to your vCenter by using this guide for Powershell).
2. If you are using Flash/Flex, uninstall 3.1.3 and install 3.1.1.