Skip to main content
Pure Technical Services

How To: Increasing the Socket Read Timeout in vRO

Currently viewing public documentation. Please login to access the full scope of documentation.

KP_Ext_Announcement.png
When running through vRO Workflows there are times that the default Socket Read Timeout of One Minute will trigger before a step in that workflow completes.  This will trigger a "java.net.SocketTimeoutException: Read timed out" and the Workflow will fail.  This KB will cover how to increase the Socket Read Timeout on the vRO Server to address the workflow failure.

What is the Issue?

At times steps in a vRO workflow may take some time to complete.  For example, a rescan job during the "Create VMFS Datastore on a new FlashArray Volume" Workflow.

Should the rescan take longer than 1 minute (the default read timeout) to complete, then the Workflow will fail.  For example:

[2019-01-15 11:29:22.521] [I] Volume object "bootcamp-alex-vro-ds-1" created successfully!
[2019-01-15 11:29:22.967] [I] Volume: 'bootcamp-alex-vro-ds-1' Connected to Host Group: 'bootcamp-alex-esxi-cluster-1'
[2019-01-15 11:30:23.078] [E] (com.vmware.library.vc.storage/rescanAll) Error in (Dynamic Script Module name : rescanAll#1) java.net.SocketTimeoutException: Read timed out
[2019-01-15 11:30:23.577] [I] Volume "bootcamp-alex-vro-ds-1" deleted successfully
[2019-01-15 11:30:23.955] [I] Volume object "bootcamp-alex-vro-ds-1" eradicated successfully!
[2019-01-15 11:30:23.988] [E] Workflow execution stack:
***
item: 'Create VMFS Datastore on new FlashArray Volume./item11', state: 'failed', business state: 'null', exception: 'java.net.SocketTimeoutException: Read timed out (Dynamic Script Module name : rescanAll#1)'
workflow: 'Create VMFS Datastore on new FlashArray Volume.' (9b3e69aa-8e5b-4545-8316-166eb5ef0b28) 
|  'attribute': name=hosts type=Array/VC:HostSystem value=#{#VC:HostSystem#dunes://service.dunes.ch/CustomSDKObject?id='Bootcamp-vcsa-1.dev.purestorage.com%2Cid:host-38'&dunesName='VC:HostSystem'#;#VC:HostSystem#dunes://service.dunes.ch/CustomSDKObject?id='Bootcamp-vcsa-1.dev.purestorage.com%2Cid:host-30'&dunesName='VC:HostSystem'#}#
|  'attribute': name=result type=Array/PS:Host value=#{#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-1-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-1-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-2-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-2-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-3-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-3-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-4-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-4-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-5-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-5-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-6-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-6-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-7-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-7-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-8-FC'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::UCS-BLADE-8-iSCSI'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-eddie-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-eddie-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-eddie-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-eddie-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-brent-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-brent-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-brent-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-brent-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-shawn-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-shawn-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-shawn-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-shawn-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-jlarson-esx-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-jlarson-esx-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-jlarson-esx-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-jlarson-esx-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Mike-ESXi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Mike-ESXi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Mike-ESXi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Mike-ESXi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-philip-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-philip-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-philip-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-philip-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-hatchett-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-hatchett-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-hatchett-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-hatchett-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-ryan-ESXi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-ryan-ESXi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-ryan-ESXi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-ryan-ESXi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-johnathan-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-johnathan-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-johnathan-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-johnathan-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-luis-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-luis-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-luis-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-luis-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Darren-ESXi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Darren-ESXi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Darren-ESXi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Darren-ESXi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Rika-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Rika-esxi-02'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Rika-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Rika-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Matt-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Matt-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Matt-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::Bootcamp-Matt-esxi-4'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-nelson-esxi-1'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-nelson-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-nelson-esxi-3'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-nelson-esxi-4'&dunesName='PS:Host'#}#
|  'attribute': name=hostResults type=Array/string value=#{#string#bootcamp-alex-esxi-2#;#string#bootcamp-alex-esxi-1#}#
|  'attribute': name=flashArrayHosts type=Array/PS:Host value=#{#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-2'&dunesName='PS:Host'#;#PS:Host#dunes://service.dunes.ch/CustomSDKObject?id='SLC-OOTO::bootcamp-alex-esxi-1'&dunesName='PS:Host'#}#
|  'attribute': name=hgroupName type=string value=bootcamp-alex-esxi-cluster-1

The Key part is this Error Line:

[2019-01-15 11:30:23.078] [E] (com.vmware.library.vc.storage/rescanAll) Error in (Dynamic Script Module name : rescanAll#1) java.net.SocketTimeoutException: Read timed out

Should this be causing workflows to fail, a possible solution is to increase the SocketReadTimeout value from the default 1 minute to something sufficiently long, such as 5 minutes.


How to Address the Issue?

The setting that will need to have a value set is com.vmware.vmo.plugin.vi4.socketReadTimeout.  This can be done via the GUI and adding this advanced setting or it can be added to the vmo.properties file via ssh.  The method that will be shown here is how to update it via SSH.

  1. First, ensure that SSH is enabled on the vRO server.
  2. SSH as root to the vRO Server.
  3. Update the /etc/vco/app-server/vmo.properties with the increased timeout.  This can be done with vim or echoing the line to the end of that file.
  4. Restarting the vco-server and vco-configurator services.
  5. Running through the workflow once the vRO Server is back up.

Here is an example of running through this process

ssh root@vro.server.ip.address
cat /etc/vco/app-server/vmo.properties
echo "com.vmware.vmo.plugin.vi4.socketReadTimeout = 180000" >> /etc/vco/app-server/vmo.properties
cat /etc/vco/app-server/vmo.properties
service vco-server restart
service vco-configurator restart
$ ssh root@10.21.203.40
VMware vRealize Appliance
root@10.21.203.40's password:
Last login: Tue Jan 15 19:02:09 UTC 2019 from 192.168.3.4 on ssh
vRO:~ # 
vRO:~ # cat /etc/vco/app-server/vmo.properties
#Wed Sep 19 14:12:54 UTC 2012
database.username=vmware
database.url=jdbc:postgresql://localhost:5433/vmware?sslmode=verify-ca&sslrootcert=/var/vmware/vpostgres/current/.postgresql/root.crt
encoded.vco.db.password=vcoencrypted:AZ8xK4Nemtbr+1LRL/CnLQ==
database.driverClassName=org.postgresql.Driver
database.hibernate.dialect = org.hibernate.dialect.PostgreSQLDialect
vco.vsphere.lookup-service.url = https://prod-vcsa.alex.purestorage.com/lookupservice/sdk
vco.vsphere.lookup-service.cert.alias = vco.vsphere.lookup-service.ssl.certificate
ch.dunes.net.jboss-server.port = 8230
ch.dunes.authentication.provider = vsphere
ch.dunes.vso.client-keep-alive-disable = true
ch.dunes.task.max-workflow-tokens = 100
com.vmware.o11n.log-events-expiration-days = 15
vRO:~ #
vRO:~ # echo "com.vmware.vmo.plugin.vi4.socketReadTimeout = 180000" >> /etc/vco/app-server/vmo.properties
vRO:~ #
vRO:~ # cat /etc/vco/app-server/vmo.properties
#Wed Sep 19 14:12:54 UTC 2012
database.username=vmware
database.url=jdbc:postgresql://localhost:5433/vmware?sslmode=verify-ca&sslrootcert=/var/vmware/vpostgres/current/.postgresql/root.crt
encoded.vco.db.password=vcoencrypted:AZ8xK4Nemtbr+1LRL/CnLQ==
database.driverClassName=org.postgresql.Driver
database.hibernate.dialect = org.hibernate.dialect.PostgreSQLDialect
vco.vsphere.lookup-service.url = https://prod-vcsa.alex.purestorage.com/lookupservice/sdk
vco.vsphere.lookup-service.cert.alias = vco.vsphere.lookup-service.ssl.certificate
ch.dunes.net.jboss-server.port = 8230
ch.dunes.authentication.provider = vsphere
ch.dunes.vso.client-keep-alive-disable = true
ch.dunes.task.max-workflow-tokens = 100
com.vmware.o11n.log-events-expiration-days = 15
com.vmware.vmo.plugin.vi4.socketReadTimeout = 180000
vRO:~ #
vRO:~ # service vco-server restart
Stopping tomcat instance
Instance is running as PID=16139, shutting down...
Instance is running PID=16139, sleeping for up to 30 seconds waiting for shutdown
Instance shut down gracefully
starting tomcat instance
Using CATALINA_BASE:   /var/lib/vco/app-server
Using CATALINA_HOME:   /usr/share/tomcat
Using CATALINA_TMPDIR: /var/lib/vco/app-server/temp
Using JRE_HOME:        /usr/java/jre-vmware
Using CLASSPATH:       /usr/lib/vco/app-server/bin/notsoserial.jar:/usr/lib/vco/app-server/bin/o11n-notsoserial-extension.jar:/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar
Using CATALINA_PID:    /var/lib/vco/app-server/logs/tomcat.pid
Tomcat started.
Status:                RUNNING as PID=51442
vRO:~ # service vco-configurator restart
Stopping tomcat instance
Instance is running as PID=9207, shutting down...
Instance is running PID=9207, sleeping for up to 30 seconds waiting for shutdown
Instance shut down gracefully
starting tomcat instance
Using CATALINA_BASE:   /var/lib/vco/configuration
Using CATALINA_HOME:   /usr/share/tomcat
Using CATALINA_TMPDIR: /var/lib/vco/configuration/temp
Using JRE_HOME:        /usr/java/jre-vmware
Using CLASSPATH:       /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar
Using CATALINA_PID:    /var/lib/vco/configuration/logs/tomcat.pid
Tomcat started.
Status:                RUNNING as PID=51783
vRO:~ #

 Then running through the Workflow again will finish successfully.  Notice that the time it took for the rescan took about 3 minutes.  

[2019-01-15 11:55:26.629] [I] Volume object "bootcamp-alex-vro-ds-1" created successfully!
[2019-01-15 11:55:27.141] [I] Volume: 'bootcamp-alex-vro-ds-1' Connected to Host Group: 'bootcamp-alex-esxi-cluster-1'
[2019-01-15 11:58:32.744] [I] The new device NAA is naa.624a9370f75f6261e61c40f2000df623
[2019-01-15 11:58:32.748] [I] Found matching device PURE iSCSI Disk (naa.624a9370f75f6261e61c40f2000df623)
[2019-01-15 11:58:32.853] [I] PURE iSCSI Disk (naa.624a9370f75f6261e61c40f2000df623)
[2019-01-15 11:58:32.858] [I] 1
[2019-01-15 11:58:32.863] [I] PURE iSCSI Disk (naa.624a9370f75f6261e61c40f2000df623)
[2019-01-15 11:58:32.866] [I] Disk name naa.624a9370f75f6261e61c40f2000df623
[2019-01-15 11:58:32.869] [I] Device path /vmfs/devices/disks/naa.624a9370f75f6261e61c40f2000df623
[2019-01-15 11:58:32.875] [I] Disk capacity 1024 GB
[2019-01-15 11:58:33.136] [I] Disk total number of blocks 1048576
[2019-01-15 11:58:33.139] [I] Disk block size 1048576
[2019-01-15 11:58:33.142] [I] Number of blocks required for the new partition 1048576
[2019-01-15 11:58:33.148] [I] Partition format gpt
[2019-01-15 11:58:34.726] [I] Added datastore bootcamp-alex-vro-ds-1

This change helped out with the environment that was encountering long rescan times.  This could be from iSCSI config, a lot of hosts in the environment, etc.  There could be other workflows that have steps that take longer than a minute to complete as well.  This was a specific example of the rescans taking a long time to complete.


Some of the steps on how to address and/or workaround these timeouts was gathered from this article.