How To: Increasing the Socket Read Timeout in vRO
What is the Issue?
At times steps in a vRO workflow may take some time to complete. For example, a rescan job during the "Create VMFS Datastore on a new FlashArray Volume" Workflow.
Should the rescan take longer than 1 minute (the default read timeout) to complete, then the Workflow will fail. For example:
[2019-01-15 11:29:22.521] [I] Volume object "bootcamp-alex-vro-ds-1" created successfully! [2019-01-15 11:29:22.967] [I] Volume: 'bootcamp-alex-vro-ds-1' Connected to Host Group: 'bootcamp-alex-esxi-cluster-1' [2019-01-15 11:30:23.078] [E] (com.vmware.library.vc.storage/rescanAll) Error in (Dynamic Script Module name : rescanAll#1) java.net.SocketTimeoutException: Read timed out [2019-01-15 11:30:23.577] [I] Volume "bootcamp-alex-vro-ds-1" deleted successfully [2019-01-15 11:30:23.955] [I] Volume object "bootcamp-alex-vro-ds-1" eradicated successfully! [2019-01-15 11:30:23.988] [E] Workflow execution stack: *** item: 'Create VMFS Datastore on new FlashArray Volume./item11', state: 'failed', business state: 'null', exception: 'java.net.SocketTimeoutException: Read timed out (Dynamic Script Module name : rescanAll#1)' workflow: 'Create VMFS Datastore on new FlashArray Volume.' (9b3e69aa-8e5b-4545-8316-166eb5ef0b28) | 'attribute': name=hosts type=Array/VC:HostSystem value=#{#VC:HostSystem#dunes://service.dunes.ch/CustomSDKObject?id='Bootcamp-vcsa-1.dev.purestorage.com%2Cid:host-38'&dunesName='VC:HostSystem'#;#VC:HostSystem#dunes://service.dunes.ch/CustomSDKObject?id='Bootcamp-vcsa-1.dev.purestorage.com%2Cid:host-30'&dunesName='VC:HostSystem'#}# | 'attribute': name=result type=Array/PS:Host value=#{#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-1-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-1-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-2-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-2-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-3-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-3-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-4-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-4-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-5-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-5-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-6-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-6-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-7-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-7-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-8-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-8-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-eddie-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-eddie-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-eddie-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-eddie-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-brent-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-brent-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-brent-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-brent-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-shawn-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-shawn-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-shawn-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-shawn-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-jlarson-esx-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-jlarson-esx-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-jlarson-esx-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-jlarson-esx-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Mike-ESXi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Mike-ESXi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Mike-ESXi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Mike-ESXi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-philip-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-philip-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-philip-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-philip-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-hatchett-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-hatchett-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-hatchett-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-hatchett-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-ryan-ESXi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-ryan-ESXi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-ryan-ESXi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-ryan-ESXi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-johnathan-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-johnathan-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-johnathan-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-johnathan-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-luis-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-luis-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-luis-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-luis-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Darren-ESXi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Darren-ESXi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Darren-ESXi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Darren-ESXi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Rika-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Rika-esxi-02'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Rika-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Rika-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Matt-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Matt-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Matt-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Matt-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-nelson-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-nelson-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-nelson-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-nelson-esxi-4'&dunesName='PS:Host'#}# | 'attribute': name=hostResults type=Array/string value=#{#string#bootcamp-alex-esxi-2#;#string#bootcamp-alex-esxi-1#}# | 'attribute': name=flashArrayHosts type=Array/PS:Host value=#{#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-1'&dunesName='PS:Host'#}# | 'attribute': name=hgroupName type=string value=bootcamp-alex-esxi-cluster-1 |
The Key part is this Error Line:
[2019-01-15 11:30:23.078] [E] (com.vmware.library.vc.storage/rescanAll) Error in (Dynamic Script Module name : rescanAll#1) java.net.SocketTimeoutException: Read timed out |
Should this be causing workflows to fail, a possible solution is to increase the SocketReadTimeout value from the default 1 minute to something sufficiently long, such as 5 minutes.
How to Address the Issue?
The setting that will need to have a value set is com.vmware.vmo.plugin.vi4.socketReadTimeout
. This can be done via the GUI and adding this advanced setting or it can be added to the vmo.properties file via ssh. The method that will be shown here is how to update it via SSH.
- First, ensure that SSH is enabled on the vRO server.
- SSH as root to the vRO Server.
- Update the
/etc/vco/app-server/vmo.properties
with the increased timeout. This can be done with vim or echoing the line to the end of that file. - Restarting the vco-server and vco-configurator services.
- Running through the workflow once the vRO Server is back up.
Here is an example of running through this process
ssh root@vro.server.ip.address cat /etc/vco/app-server/vmo.properties echo "com.vmware.vmo.plugin.vi4.socketReadTimeout = 180000" >> /etc/vco/app-server/vmo.properties cat /etc/vco/app-server/vmo.properties service vco-server restart service vco-configurator restart |
---|
$ ssh root@10.21.203.40 VMware vRealize Appliance root@10.21.203.40's password: Last login: Tue Jan 15 19:02:09 UTC 2019 from 192.168.3.4 on ssh vRO:~ # vRO:~ # cat /etc/vco/app-server/vmo.properties #Wed Sep 19 14:12:54 UTC 2012 database.username=vmware database.url=jdbc:postgresql://localhost:5433/vmware?sslmode=verify-ca&sslrootcert=/var/vmware/vpostgres/current/.postgresql/root.crt encoded.vco.db.password=vcoencrypted:AZ8xK4Nemtbr+1LRL/CnLQ== database.driverClassName=org.postgresql.Driver database.hibernate.dialect = org.hibernate.dialect.PostgreSQLDialect vco.vsphere.lookup-service.url = https://prod-vcsa.alex.purestorage.com/lookupservice/sdk vco.vsphere.lookup-service.cert.alias = vco.vsphere.lookup-service.ssl.certificate ch.dunes.net.jboss-server.port = 8230 ch.dunes.authentication.provider = vsphere ch.dunes.vso.client-keep-alive-disable = true ch.dunes.task.max-workflow-tokens = 100 com.vmware.o11n.log-events-expiration-days = 15 vRO:~ # vRO:~ # echo "com.vmware.vmo.plugin.vi4.socketReadTimeout = 180000" >> /etc/vco/app-server/vmo.properties vRO:~ # vRO:~ # cat /etc/vco/app-server/vmo.properties #Wed Sep 19 14:12:54 UTC 2012 database.username=vmware database.url=jdbc:postgresql://localhost:5433/vmware?sslmode=verify-ca&sslrootcert=/var/vmware/vpostgres/current/.postgresql/root.crt encoded.vco.db.password=vcoencrypted:AZ8xK4Nemtbr+1LRL/CnLQ== database.driverClassName=org.postgresql.Driver database.hibernate.dialect = org.hibernate.dialect.PostgreSQLDialect vco.vsphere.lookup-service.url = https://prod-vcsa.alex.purestorage.com/lookupservice/sdk vco.vsphere.lookup-service.cert.alias = vco.vsphere.lookup-service.ssl.certificate ch.dunes.net.jboss-server.port = 8230 ch.dunes.authentication.provider = vsphere ch.dunes.vso.client-keep-alive-disable = true ch.dunes.task.max-workflow-tokens = 100 com.vmware.o11n.log-events-expiration-days = 15 com.vmware.vmo.plugin.vi4.socketReadTimeout = 180000 vRO:~ # vRO:~ # service vco-server restart Stopping tomcat instance Instance is running as PID=16139, shutting down... Instance is running PID=16139, sleeping for up to 30 seconds waiting for shutdown Instance shut down gracefully starting tomcat instance Using CATALINA_BASE: /var/lib/vco/app-server Using CATALINA_HOME: /usr/share/tomcat Using CATALINA_TMPDIR: /var/lib/vco/app-server/temp Using JRE_HOME: /usr/java/jre-vmware Using CLASSPATH: /usr/lib/vco/app-server/bin/notsoserial.jar:/usr/lib/vco/app-server/bin/o11n-notsoserial-extension.jar:/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar Using CATALINA_PID: /var/lib/vco/app-server/logs/tomcat.pid Tomcat started. Status: RUNNING as PID=51442 vRO:~ # service vco-configurator restart Stopping tomcat instance Instance is running as PID=9207, shutting down... Instance is running PID=9207, sleeping for up to 30 seconds waiting for shutdown Instance shut down gracefully starting tomcat instance Using CATALINA_BASE: /var/lib/vco/configuration Using CATALINA_HOME: /usr/share/tomcat Using CATALINA_TMPDIR: /var/lib/vco/configuration/temp Using JRE_HOME: /usr/java/jre-vmware Using CLASSPATH: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar Using CATALINA_PID: /var/lib/vco/configuration/logs/tomcat.pid Tomcat started. Status: RUNNING as PID=51783 vRO:~ # |
Then running through the Workflow again will finish successfully. Notice that the time it took for the rescan took about 3 minutes. [2019-01-15 11:55:26.629] [I] Volume object "bootcamp-alex-vro-ds-1" created successfully! [2019-01-15 11:55:27.141] [I] Volume: 'bootcamp-alex-vro-ds-1' Connected to Host Group: 'bootcamp-alex-esxi-cluster-1' [2019-01-15 11:58:32.744] [I] The new device NAA is naa.624a9370f75f6261e61c40f2000df623 [2019-01-15 11:58:32.748] [I] Found matching device PURE iSCSI Disk (naa.624a9370f75f6261e61c40f2000df623) [2019-01-15 11:58:32.853] [I] PURE iSCSI Disk (naa.624a9370f75f6261e61c40f2000df623) [2019-01-15 11:58:32.858] [I] 1 [2019-01-15 11:58:32.863] [I] PURE iSCSI Disk (naa.624a9370f75f6261e61c40f2000df623) [2019-01-15 11:58:32.866] [I] Disk name naa.624a9370f75f6261e61c40f2000df623 [2019-01-15 11:58:32.869] [I] Device path /vmfs/devices/disks/naa.624a9370f75f6261e61c40f2000df623 [2019-01-15 11:58:32.875] [I] Disk capacity 1024 GB [2019-01-15 11:58:33.136] [I] Disk total number of blocks 1048576 [2019-01-15 11:58:33.139] [I] Disk block size 1048576 [2019-01-15 11:58:33.142] [I] Number of blocks required for the new partition 1048576 [2019-01-15 11:58:33.148] [I] Partition format gpt [2019-01-15 11:58:34.726] [I] Added datastore bootcamp-alex-vro-ds-1 |
This change helped out with the environment that was encountering long rescan times. This could be from iSCSI config, a lot of hosts in the environment, etc. There could be other workflows that have steps that take longer than a minute to complete as well. This was a specific example of the rescans taking a long time to complete.
Some of the steps on how to address and/or workaround these timeouts was gathered from this article.