Skip to main content
Pure Technical Services

VMware vSphere: Log Analysis

Currently viewing public documentation. Please login to access the full scope of documentation.

Intro

This article is meant to help guide you through locating the most commonly needed information to gather from ESXi host logging during an investigation. The following information assumes you are looking through a host bundle that's been pulled from an ESXi host. Please feel free to contact me in slack at jhughes or email me at jhughes@purestorage.com if you have a suggestion/request.

ESXi Host Logging

First Steps

Before you dive head first into ESX host logging you'll want to be familiar with the common log locations. This is already detailed in depth at the following link.

https://support.purestorage.com/Solutions/VMware_Platform_Guide/General_Troubleshooting_KBs/VMware_vSphere%3A_ESXi_Host_Troubleshooting#Log_File_Types

A common first step is to identify which Pure volumes you are needing to look at on the ESXi host end.

Identifying the Pure luns is also detailed here: 

https://support.purestorage.com/Solutions/VMware_Platform_Guide/General_Troubleshooting_KBs/VMware_vSphere%3A_ESXi_Host_Troubleshooting#Identifying_PURE_LUNs

Associating ESXi Host Paths to Array Ports

When looking for path related problems or evidence in the ESXi host logging you'll need to know what labels ESXi is using for the path that has been created with the various Pure Array ports. Once you have found the specific path(s) that are listed on the host end for the Pure Array it makes finding events from vobd.log or vmkernel.log a lot easier by only searching for those path(s).

Fibre Channel

First, gather the path information from esxcfg-mpath_-b.txt from the commands folder in the ESX logging for the volume.

VMware references the vmhbaxx:xx:xx:L[lunid] within their own logging and is what you'll want to reference in the vmkernel, vobd, etc. logging.

Host side WWN's are shown in green. Array side WWN's are shown in red.

 

esxi-host/commands/esxcfg-mpath_-b.txt

naa.624a937013e8daec958141dd000125f9 : PURE Fibre Channel Over Ethernet Disk (naa.624a937013e8daec958141dd000125f9)
   vmhba64:C0:T0:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:db WWPN: 10:00:72:33:8e:d0:00:da FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:d8:98 VLAN ID: 4064 Underlying PNIC: vmnic2  Target: WWNN: 52:4a:93:7b:8a:95:d5:01 WWPN: 52:4a:93:7b:8a:95:d5:01
   vmhba64:C0:T2:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:db WWPN: 10:00:72:33:8e:d0:00:da FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:d8:98 VLAN ID: 4064 Underlying PNIC: vmnic2  Target: WWNN: 52:4a:93:7b:8a:95:d5:11 WWPN: 52:4a:93:7b:8a:95:d5:11
   vmhba64:C0:T3:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:db WWPN: 10:00:72:33:8e:d0:00:da FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:d8:98 VLAN ID: 4064 Underlying PNIC: vmnic2  Target: WWNN: 52:4a:93:7b:8a:95:d5:13 WWPN: 52:4a:93:7b:8a:95:d5:13
   vmhba64:C0:T1:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:db WWPN: 10:00:72:33:8e:d0:00:da FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:d8:98 VLAN ID: 4064 Underlying PNIC: vmnic2  Target: WWNN: 52:4a:93:7b:8a:95:d5:03 WWPN: 52:4a:93:7b:8a:95:d5:03
   vmhba65:C0:T1:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:dd WWPN: 10:00:72:33:8e:d0:00:dc FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:58:9a VLAN ID: 4065 Underlying PNIC: vmnic3  Target: WWNN: 52:4a:93:7b:8a:95:d5:12 WWPN: 52:4a:93:7b:8a:95:d5:12
   vmhba65:C0:T2:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:dd WWPN: 10:00:72:33:8e:d0:00:dc FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:58:9a VLAN ID: 4065 Underlying PNIC: vmnic3  Target: WWNN: 52:4a:93:7b:8a:95:d5:10 WWPN: 52:4a:93:7b:8a:95:d5:10
   vmhba65:C0:T3:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:dd WWPN: 10:00:72:33:8e:d0:00:dc FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:58:9a VLAN ID: 4065 Underlying PNIC: vmnic3  Target: WWNN: 52:4a:93:7b:8a:95:d5:02 WWPN: 52:4a:93:7b:8a:95:d5:02
   vmhba65:C0:T0:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:dd WWPN: 10:00:72:33:8e:d0:00:dc FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:58:9a VLAN ID: 4065 Underlying PNIC: vmnic3  Target: WWNN: 52:4a:93:7b:8a:95:d5:00 WWPN: 52:4a:93:7b:8a:95:d5:00

 

The WWN's can be correlated on the array via pureport list and with purehost list.

$ purehost list | grep -A 1 hostname
esx01psynvdilfll  10:00:72:33:8E:D0:00:DA  -    LAF-VDI
                  10:00:72:33:8E:D0:00:DC
(Tue Jul 30 20:41:59 UTC) jhughes@cloudfuse-prod-07f2658fac6abe231:/logs/fmolhs.org/dev01ppurelfll-ct1/2019_07_30$ pureport list
 
------------------------------------------------------------------------
Jul 30 14:17:17 pureport list
------------------------------------------------------------------------
Name     WWN                      Portal  IQN  Failover
CT0.FC0  52:4A:93:7B:8A:95:D5:00  -       -    -
CT0.FC1  52:4A:93:7B:8A:95:D5:01  -       -    -
CT0.FC2  52:4A:93:7B:8A:95:D5:02  -       -    -
CT0.FC3  52:4A:93:7B:8A:95:D5:03  -       -    -
CT1.FC0  52:4A:93:7B:8A:95:D5:10  -       -    -
CT1.FC1  52:4A:93:7B:8A:95:D5:11  -       -    -
CT1.FC2  52:4A:93:7B:8A:95:D5:12  -       -    -
CT1.FC3  52:4A:93:7B:8A:95:D5:13  -       -    -

Now that we know what are paths are you can use the vobd.log and the vmkernel.log to determine issues for those paths.

If you are looking for just up and down events, I'd recommend looking at the vobd.log and searching out the specific path you are looking at. If you need all paths search for simply the LUN id listed in the path in this case L254.

~/var/run/log$ zgrep -i vmhba64:C0:T0:L254 vobd.log
2018-12-10T15:15:41.460Z: [scsiCorrelator] 3640800142356us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba64:C0:T0:L254 changed state from on
2018-12-10T15:15:41.491Z: [scsiCorrelator] 3640800145888us: [vob.scsi.scsipath.remove] Remove path: vmhba64:C0:T0:L254
2019-07-17T13:04:50.964Z: [scsiCorrelator] 8557726246035us: [vob.scsi.scsipath.add] Add path: vmhba64:C0:T0:L254
2019-07-17T13:04:50.978Z: [scsiCorrelator] 8557726260253us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba64:C0:T0:L254 changed state from dead
2019-07-22T14:44:10.910Z: [scsiCorrelator] 8995700627433us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba64:C0:T0:L254 changed state from on
2019-07-22T14:44:10.911Z: [scsiCorrelator] 8995700627478us: [vob.scsi.scsipath.remove] Remove path: vmhba64:C0:T0:L254
2019-07-22T14:44:10.915Z: [scsiCorrelator] 8995700631851us: [vob.scsi.scsipath.add] Add path: vmhba64:C0:T0:L254
2019-07-22T14:44:10.922Z: [scsiCorrelator] 8995700639019us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba64:C0:T0:L254 changed state from dead
~/var/run/log$ zgrep -i L254 vobd.log
2019-07-17T13:04:50.946Z: [scsiCorrelator] 8557726227948us: [vob.scsi.scsipath.add] Add path: vmhba65:C0:T0:L254
2019-07-17T13:04:50.946Z: [scsiCorrelator] 8557726228125us: [vob.scsi.scsipath.add] Add path: vmhba65:C0:T1:L254
.
.
.
2019-07-17T13:05:13.296Z: [scsiCorrelator] 8557748578552us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba64:C0:T2:L254 changed state from dead
2019-07-17T13:05:13.296Z: [scsiCorrelator] 8557748579162us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba64:C0:T3:L254 changed state from dead
2019-07-17T13:05:13.297Z: [scsiCorrelator] 8557748579997us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba64:C0:T1:L254 changed state from dead
iSCSI

For iSCSI it's more difficult to associate paths with a specific array port. Again you can check the esxcfg-mpath_-b.txt in the commands folder and look for the NAA number of the volume you want to know about.

esxcfg-mpath_-b.txt
 
naa.624a93708b6243d446c34f98000121af : PURE iSCSI Disk (naa.624a93708b6243d446c34f98000121af)
   vmhba64:C2:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf  Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000004 PortalTag=155
   vmhba64:C3:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf  Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000005 PortalTag=155
   vmhba64:C4:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf  Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000006 PortalTag=155
   vmhba64:C5:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf  Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000007 PortalTag=155
   vmhba64:C0:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf  Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000002 PortalTag=155
   vmhba64:C6:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf  Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000008 PortalTag=155
   vmhba64:C1:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf  Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000003 PortalTag=155
   vmhba64:C7:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf  Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000001 PortalTag=155
 

Notice in the case of ISCSI the ESXi host only note the target IQN of the array. To find what ETH port we are dealing with for each path, you'll have to check into vmware-vimdump_-o----U-dcui.txt file in the commands folder. You are looking for the vim.host.MultipathInfo. Path for the associated vmhbaxx:xx:xx:L[lunid]. The associated IP should be listed at the bottom of this output.

$ pureport list
------------------------------------------------------------------------
Jul 30 22:17:14 pureport list
------------------------------------------------------------------------
Name          WWN  Portal              IQN                                                      Failover
CT0.ETH4.154  -    192.168.10.10:3260  iqn.2010-06.com.purestorage:flasharray.512495d090d847c9  -
CT0.ETH5.154  -    192.168.10.11:3260  iqn.2010-06.com.purestorage:flasharray.512495d090d847c9  -
CT1.ETH4.154  -    192.168.10.12:3260  iqn.2010-06.com.purestorage:flasharray.512495d090d847c9  -
CT1.ETH5.154  -    192.168.10.13:3260  iqn.2010-06.com.purestorage:flasharray.512495d090d847c9  -
esxi-host/commands/vmware-vimdump_-o----U-dcui.txt

(vim.host.MultipathInfo.Path) {
                                 dynamicType = <unset>,
                                 dynamicProperty = (vmodl.DynamicProperty) [],
                                 key = 'key-vim.host.MultipathInfo.Path-vmhba64:C2:T0:L253',
                                 name = 'vmhba64:C2:T0:L253',
                                 pathState = 'active',
                                 state = 'active',
                                 isWorkingPath = true,
                                 adapter = 'key-vim.host.InternetScsiHba-vmhba64',
                                 lun = 'key-vim.host.MultipathInfo.LogicalUnit-0200fd0000624a93708b6243d446c34f98000121af466c61736841',
                                 transport = (vim.host.InternetScsiTargetTransport) {
                                    dynamicType = <unset>,
                                    dynamicProperty = (vmodl.DynamicProperty) [],
                                    iScsiName = 'iqn.2010-06.com.purestorage:flasharray.512495d090d847c9',
                                    iScsiAlias = '',
                                    address = (str) [
                                       '192.168.10.11:3260'

Associating VVOL RFC Label with Array Volume

For VVOL volumes ESXi labels each volume with a unique RFC number. This is separate from the NAA number as we are utilizing a single array volume with the protocol endpoint. Vmkernel.log and vvold.log in the ESXi host logging will refer to this number when reporting issues for that volume.

If you are new to VVOLs and need a better understanding I recommend starting here:

https://support.purestorage.com/Solutions/VMware_Platform_Guide/003Virtual_Volumes_-_VVols/Troubleshooting/Understanding_Virtual_Volumes_and_Failure_Scenarios

VVOL volumes are listed in the vmfs/volumes/ folder on the host logging. They are labeled with an RFC number. This won't correlate directly to any volume ID on the array.

~hostlogs ls -lah vmfs/volumes/vvol*
total 136K
drwxrwxrwx 34 dschaat fuseuser 6.0K May 29 19:58 .
drwxrwxrwx  4 dschaat fuseuser 6.0K May 29 19:59 ..
drwxrwxrwx  2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.2064fb10-e9d1-41ea-b586-e5cf78e0e266
drwxrwxrwx  2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.233032b9-11d3-4f58-9bcd-e43c1e03169a
drwxrwxrwx  2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.25f0ceb9-765b-453e-b06d-516fbd3fbaf4
drwxrwxrwx  2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.38cae476-5d65-4465-9bbc-a37c64a80ef6
drwxrwxrwx  2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.3d4f3a13-31f9-4d91-9c6d-b99e100cfc5f
drwxrwxrwx  2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.455fb510-d36a-4adf-b6ed-b800c1c2216b
drwxrwxrwx  2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.54ca118e-5062-4b39-97d3-99d6755341f7
drwxrwxrwx  2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.562743c3-e37b-4425-8ea4-e4ab34e7c23f
drwxrwxrwx  2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.57af1a31-fbbc-4ed2-9768-11e60730bce3
drwxrwxrwx  2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.5d7188cf-dc5a-4cdd-8fc6-0f57de70c3c3

We can still determine which volumes on the array the RFC number associates to through the diagnostics log. This lists all associated data, config and swap volumes of the VVOL.

fuse~$ zgrep -i rfc4122.2064fb10-e9d1-41ea-b586-e5cf78e0e266 diagnostics.log-2019073001.gz
vvol-vmwinbuild-2064fb10-vg/Config-196e57ab                            PURE_VVOL_ID                                   rfc4122.2064fb10-e9d1-41ea-b586-e5cf78e0e266                                                                                                                                                                                                                                                                                                -
vvol-vmwinbuild-2064fb10-vg/Data-61d6cabc                              VMW_VVolNamespace                              /vmfs/volumes/vvol:77596d73ea8c3096-8279bf44f15a6156/rfc4122.2064fb10-e9d1-41ea-b586-e5cf78e0e266                                                                                                                                                                                                                                           -
vvol-vmwinbuild-2064fb10-vg/Swap-34c1d0b5                              VMW_VVolNamespace                              /vmfs/volumes/vvol:77596d73ea8c3096-8279bf44f15a6156/rfc4122.2064fb10-e9d1-41ea-b586-e5cf78e0e266

When investigating VVOL issues generally, look to the vvold.log in var/run/log of the host logging. The following link has information on failure scenarios to look for:

https://support.purestorage.com/Solutions/VMware_Platform_Guide/003Virtual_Volumes_-_VVols/Troubleshooting/Understanding_Virtual_Volumes_and_Failure_Scenarios

 

Correlating Pure Storage Plugin Events with Vcenter Logging

When looking for events that involve the Pure Storage plugin you'll need to look into the Vcenter logging specifically. This is not included in a host bundle unless you specifically select to include them so make sure to ask for Vcenter logging if it is needed.

Events that are intiated with the Pure Storage plugin on Vcenter will register in the vsphere_client_virgo.log which is located in the /var/log/vmware/vsphere-client/logs folder.

Below is an example of a datastore creation being created with the Pure Storage plugin. On the array end we'll see the associated API call within the rest.log.

root@bootcamp-johnathan-vcsa-1 [ /var/log/vmware/vsphere-client/logs ]# less vsphere_client_virgo.log
[2019-07-25T03:25:08.948Z] [INFO ] http-nio-9090-exec-2          com.purestorage.PureDatastoreImpl                                 createHandler : duration = 307 ms, params = ASObject(593708984){activityId=710AEDA3-DF7A-EF97-C17A-27289BEF62B2, sizeUnits=4, flashArrayId=3a
cc8acd-9257-4a15-8bef-6a001e0f0cf9/fa-0a769403-6af1-4bd6-a59a-98e4db2b8918, volumeName=jhughes-test-3, serverGuid=2bd52d7e-2c5b-4cd2-b39f-144afeb64229, snapshotName=, action=createVolume, volumeSize=1}, result = {volumeSerial=DA3369DA7755463B0039E358}
[2019-07-25T03:25:08.971Z] [INFO ] http-nio-9090-exec-10         org.springframework.flex.servlet.MessageBrokerHandlerAdapter      Channel endpoint amf received request.
[2019-07-25T03:25:09.172Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: slc-wfh
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group UCS-ESXi-67-Cluster-1-FC
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group UCS-ESXi-67-Cluster-1-iSCSI
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group UCS-ESXi-67-Cluster-2-FC
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group UCS-ESXi-67-Cluster-2-iSCSI
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-alex-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-alex-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-eddie-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-eddie-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-brent-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-brent-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-shawn-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-shawn-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-jlarson-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-jlarson-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group Bootcamp-Mike-ESXi-Cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-philip-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-philip-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group Bootcamp-Mike-ESXi-Cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-hatchett-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-hatchett-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group Bootcamp-ryan-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group Bootcamp-ryan-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10         com.purestorage.FlashArrayClient                                  findHostGroupName: Checking host group bootcamp-johnathan-esxi-cluster-1
[2019-07-25T03:25:09.564Z] [INFO ] http-nio-9090-exec-10         com.purestorage.PureDatastoreImpl                                 createHandler : duration = 592 ms, params = ASObject(1793856952)
{activityId=710AEDA3-DF7A-EF97-C17A-27289BEF62B2, flashArrayId=3acc8acd-9257-4a15-8bef-6a001e0f0cf9/fa-0a769403-6af1-4bd6-a59a-98e4db2b8918, volumeName=jhughes-test-3, serverGuid=2bd52d7e-2c5b-4cd2-b39f-144afeb64229, host={"hbas":{},"iqns":[],"children":[{"hbas":{},"iqns":[],"productVersion":"6.7.0","children":[],"isCluster":false,"portWorldWideNamesHex":[],"isInMaintenanceMode":false,"moName":"host-11","isFolder":false,"name":"bootcamp-johnathan-esxi-1.dev.purestorage.com"},{"hbas":{},"iqns":[],"productVersion":"6.7.0","children":[],"isCluster":false,"portWorldWideNamesHex":[],"isInMaintenanceMode":false,"moName":"host-33","isFolder":false,"name":"bootcamp-johnathan-esxi-2.dev.purestorage.com"}],"isCluster":true,"portWorldWideNamesHex":[],"isInMaintenanceMode":false,"moName":"domain-c7","isFolder":false,"name":"ESXi 6.7 Cluster 1"}, action=connectVolume}, result = {}

This will be a Post result in the rest logging. 

2019-07-25 03:24:49.231 [INFO] 10.204.114.155 >> POST http://10.204.120.159/api/1.13/volume/jhughes-test-3
{
  "size":1099511627776
}
2019-07-25 03:24:49.453 [INFO] << 200 OK
{
  "created":"2019-07-25T03:24:49Z",
  "name":"jhughes-test-3",
  "serial":"DA3369DA7755463B0039E356",
  "size":1099511627776,
  "source":null
}

Miscellaneous Host Information

This section includes common information that you'll need to gather from the host. Most of this should be found with the mr_vmware tool output; however, if you have to manually search, these are some of the common points.

Host Local Date and Time Zone

Found in the commands/date.txt file.

$hostlog/commands$ cat date.txt
Tue Oct  9 14:37:29 UTC 2018
Host Hardware Information

Found in commands/esxcfg-info_-a.txt. 

(Sat Aug 03 02:10:39 UTC) jhughes@cloudfuse-prod-07f2658fac6abe231:/support/ES-51882/VMware-Logs/ESXi_Host-Logs/esx-DLESX023.jdadelivers.com-2018-10-09--14.37-1547173/commands$ grep -E "Product Name|Vendor Name|Serial Number|Hardware Uptime" esxcfg-info_-a.txt
      |----Product Name.............................................PowerEdge M620
      |----Vendor Name..............................................Dell Inc.
      |----Serial Number............................................C8ZYJ02
      |----Hardware Uptime..........................................3645770847531
               |----Vendor Name.....................................Intel Corporation
ESXi Build Version

Found in commands/vmware_-vl.txt

(Sat Aug 03 02:13:13 UTC) jhughes@cloudfuse-prod-07f2658fac6abe231:/support/ES-51882/VMware-Logs/ESXi_Host-Logs/esx-DLESX023.jdadelivers.com-2018-10-09--14.37-1547173/commands$ cat vmware_-vl.txt
VMware ESXi 6.5.0 build-7967591
VMware ESXi 6.5.0 Update 1
ESXi SCSI FNIC Driver Version

Found in commans/localcli_software-vib-list.txt

(Sat Aug 03 02:19:39 UTC) jhughes@cloudfuse-prod-07f2658fac6abe231:/support/ES-51882/VMware-Logs/ESXi_Host-Logs/esx-DLESX023.jdadelivers.com-2018-10-09--14.37-1547173/commands$ grep -E "Name|fnic" localcli_software-vib-list.txt
Name                           Version                              Vendor   Acceptance Level  Install Date
scsi-fnic                      1.5.0.45-3vmw.650.0.0.4564106        VMW      VMwareCertified   2018-08-28
ESXi HBA/NIC Information

Found in commands/localcli_storage-core-adapter-list.txt. Further information can be found in either commands/nicinfo.sh.txt for ISCSI or commands/vmkmgmt_keyval_-a.txt for FC.

(Sat Aug 03 02:28:53 UTC) jhughes@cloudfuse-prod-07f2658fac6abe231:/support/ES-51882/VMware-Logs/ESXi_Host-Logs/esx-DLESX023.jdadelivers.com-2018-10-09--14.37-1547173/commands$ cat localcli_storage-core-adapter-list.txt
HBA Name  Driver      Link State  UID                                   Capabilities         Description
--------------------------------------------------------------------------------------------------------
vmhba0    vmw_ahci    link-n/a    sata.vmhba0                                                (0000:00:1f.2) Intel Corporation Patsburg 6 Port SATA AHCI Controller
vmhba1    qlnativefc  link-up     fc.20000024ff531ff0:21000024ff531ff0  Second Level Lun ID  (0000:03:00.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba2    qlnativefc  link-up     fc.20000024ff531ff1:21000024ff531ff1  Second Level Lun ID  (0000:03:00.1) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba32   vmkusb      link-n/a    usb.vmhba32                                                () USB
vmhba33   bnx2i       unbound     iscsi.vmhba33                         Second Level Lun ID  QLogic 57810 10 Gigabit Ethernet Adapter
vmhba34   bnx2i       unbound     iscsi.vmhba34                         Second Level Lun ID  QLogic 57810 10 Gigabit Ethernet Adapter
APD vs PDL

APD generally means we have lost access to all storage paths. A timer is started and once that timer is exceeded then paths are marked permanently lost. This means we can recover paths in an APD event, but we can still have an outage from a production standpoint in the time frame of a APD. Once a path has been marked PDL, then a host reboot is required to bring them back. Review the following VMware KB, which goes over some of the failure symptoms and examples of vmkernel logging:

https://kb.vmware.com/s/article/2004684