VMware vSphere: Log Analysis
Intro
This article is meant to help guide you through locating the most commonly needed information to gather from ESXi host logging during an investigation. The following information assumes you are looking through a host bundle that's been pulled from an ESXi host. Please feel free to contact me in slack at jhughes or email me at jhughes@purestorage.com if you have a suggestion/request.
ESXi Host Logging
First Steps
Before you dive head first into ESX host logging you'll want to be familiar with the common log locations. This is already detailed in depth at the following link.
A common first step is to identify which Pure volumes you are needing to look at on the ESXi host end.
Identifying the Pure luns is also detailed here:
Associating ESXi Host Paths to Array Ports
When looking for path related problems or evidence in the ESXi host logging you'll need to know what labels ESXi is using for the path that has been created with the various Pure Array ports. Once you have found the specific path(s) that are listed on the host end for the Pure Array it makes finding events from vobd.log or vmkernel.log a lot easier by only searching for those path(s).
Fibre Channel
First, gather the path information from esxcfg-mpath_-b.txt from the commands folder in the ESX logging for the volume.
VMware references the vmhbaxx:xx:xx:L[lunid] within their own logging and is what you'll want to reference in the vmkernel, vobd, etc. logging.
Host side WWN's are shown in green. Array side WWN's are shown in red.
esxi-host/commands/esxcfg-mpath_-b.txt naa.624a937013e8daec958141dd000125f9 : PURE Fibre Channel Over Ethernet Disk (naa.624a937013e8daec958141dd000125f9) vmhba64:C0:T0:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:db WWPN: 10:00:72:33:8e:d0:00:da FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:d8:98 VLAN ID: 4064 Underlying PNIC: vmnic2 Target: WWNN: 52:4a:93:7b:8a:95:d5:01 WWPN: 52:4a:93:7b:8a:95:d5:01 vmhba64:C0:T2:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:db WWPN: 10:00:72:33:8e:d0:00:da FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:d8:98 VLAN ID: 4064 Underlying PNIC: vmnic2 Target: WWNN: 52:4a:93:7b:8a:95:d5:11 WWPN: 52:4a:93:7b:8a:95:d5:11 vmhba64:C0:T3:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:db WWPN: 10:00:72:33:8e:d0:00:da FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:d8:98 VLAN ID: 4064 Underlying PNIC: vmnic2 Target: WWNN: 52:4a:93:7b:8a:95:d5:13 WWPN: 52:4a:93:7b:8a:95:d5:13 vmhba64:C0:T1:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:db WWPN: 10:00:72:33:8e:d0:00:da FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:d8:98 VLAN ID: 4064 Underlying PNIC: vmnic2 Target: WWNN: 52:4a:93:7b:8a:95:d5:03 WWPN: 52:4a:93:7b:8a:95:d5:03 vmhba65:C0:T1:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:dd WWPN: 10:00:72:33:8e:d0:00:dc FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:58:9a VLAN ID: 4065 Underlying PNIC: vmnic3 Target: WWNN: 52:4a:93:7b:8a:95:d5:12 WWPN: 52:4a:93:7b:8a:95:d5:12 vmhba65:C0:T2:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:dd WWPN: 10:00:72:33:8e:d0:00:dc FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:58:9a VLAN ID: 4065 Underlying PNIC: vmnic3 Target: WWNN: 52:4a:93:7b:8a:95:d5:10 WWPN: 52:4a:93:7b:8a:95:d5:10 vmhba65:C0:T3:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:dd WWPN: 10:00:72:33:8e:d0:00:dc FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:58:9a VLAN ID: 4065 Underlying PNIC: vmnic3 Target: WWNN: 52:4a:93:7b:8a:95:d5:02 WWPN: 52:4a:93:7b:8a:95:d5:02 vmhba65:C0:T0:L254 LUN:254 state:active fcoe Adapter: WWNN: 10:00:72:33:8e:d0:00:dd WWPN: 10:00:72:33:8e:d0:00:dc FCF MAC: 00:00:b8:17:c0:0c VNPort MAC: 80:41:00:00:58:9a VLAN ID: 4065 Underlying PNIC: vmnic3 Target: WWNN: 52:4a:93:7b:8a:95:d5:00 WWPN: 52:4a:93:7b:8a:95:d5:00
The WWN's can be correlated on the array via pureport list and with purehost list.
$ purehost list | grep -A 1 hostname esx01psynvdilfll 10:00:72:33:8E:D0:00:DA - LAF-VDI 10:00:72:33:8E:D0:00:DC
(Tue Jul 30 20:41:59 UTC) jhughes@cloudfuse-prod-07f2658fac6abe231:/logs/fmolhs.org/dev01ppurelfll-ct1/2019_07_30$ pureport list ------------------------------------------------------------------------ Jul 30 14:17:17 pureport list ------------------------------------------------------------------------ Name WWN Portal IQN Failover CT0.FC0 52:4A:93:7B:8A:95:D5:00 - - - CT0.FC1 52:4A:93:7B:8A:95:D5:01 - - - CT0.FC2 52:4A:93:7B:8A:95:D5:02 - - - CT0.FC3 52:4A:93:7B:8A:95:D5:03 - - - CT1.FC0 52:4A:93:7B:8A:95:D5:10 - - - CT1.FC1 52:4A:93:7B:8A:95:D5:11 - - - CT1.FC2 52:4A:93:7B:8A:95:D5:12 - - - CT1.FC3 52:4A:93:7B:8A:95:D5:13 - - -
Now that we know what are paths are you can use the vobd.log and the vmkernel.log to determine issues for those paths.
If you are looking for just up and down events, I'd recommend looking at the vobd.log and searching out the specific path you are looking at. If you need all paths search for simply the LUN id listed in the path in this case L254.
~/var/run/log$ zgrep -i vmhba64:C0:T0:L254 vobd.log 2018-12-10T15:15:41.460Z: [scsiCorrelator] 3640800142356us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba64:C0:T0:L254 changed state from on 2018-12-10T15:15:41.491Z: [scsiCorrelator] 3640800145888us: [vob.scsi.scsipath.remove] Remove path: vmhba64:C0:T0:L254 2019-07-17T13:04:50.964Z: [scsiCorrelator] 8557726246035us: [vob.scsi.scsipath.add] Add path: vmhba64:C0:T0:L254 2019-07-17T13:04:50.978Z: [scsiCorrelator] 8557726260253us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba64:C0:T0:L254 changed state from dead 2019-07-22T14:44:10.910Z: [scsiCorrelator] 8995700627433us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba64:C0:T0:L254 changed state from on 2019-07-22T14:44:10.911Z: [scsiCorrelator] 8995700627478us: [vob.scsi.scsipath.remove] Remove path: vmhba64:C0:T0:L254 2019-07-22T14:44:10.915Z: [scsiCorrelator] 8995700631851us: [vob.scsi.scsipath.add] Add path: vmhba64:C0:T0:L254 2019-07-22T14:44:10.922Z: [scsiCorrelator] 8995700639019us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba64:C0:T0:L254 changed state from dead
~/var/run/log$ zgrep -i L254 vobd.log 2019-07-17T13:04:50.946Z: [scsiCorrelator] 8557726227948us: [vob.scsi.scsipath.add] Add path: vmhba65:C0:T0:L254 2019-07-17T13:04:50.946Z: [scsiCorrelator] 8557726228125us: [vob.scsi.scsipath.add] Add path: vmhba65:C0:T1:L254 . . . 2019-07-17T13:05:13.296Z: [scsiCorrelator] 8557748578552us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba64:C0:T2:L254 changed state from dead 2019-07-17T13:05:13.296Z: [scsiCorrelator] 8557748579162us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba64:C0:T3:L254 changed state from dead 2019-07-17T13:05:13.297Z: [scsiCorrelator] 8557748579997us: [vob.scsi.scsipath.pathstate.on] scsiPath vmhba64:C0:T1:L254 changed state from dead
iSCSI
For iSCSI it's more difficult to associate paths with a specific array port. Again you can check the esxcfg-mpath_-b.txt in the commands folder and look for the NAA number of the volume you want to know about.
esxcfg-mpath_-b.txt
naa.624a93708b6243d446c34f98000121af : PURE iSCSI Disk (naa.624a93708b6243d446c34f98000121af)
vmhba64:C2:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000004 PortalTag=155
vmhba64:C3:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000005 PortalTag=155
vmhba64:C4:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000006 PortalTag=155
vmhba64:C5:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000007 PortalTag=155
vmhba64:C0:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000002 PortalTag=155
vmhba64:C6:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000008 PortalTag=155
vmhba64:C1:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000003 PortalTag=155
vmhba64:C7:T0:L253 LUN:253 state:active iscsi Adapter: iqn.1998-01.com.vmware:adcnetsec-vmcore01-541dadcf Target: IQN=iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 Alias= Session=00023d000001 PortalTag=155
Notice in the case of ISCSI the ESXi host only note the target IQN of the array. To find what ETH port we are dealing with for each path, you'll have to check into vmware-vimdump_-o----U-dcui.txt file in the commands folder. You are looking for the vim.host.MultipathInfo. Path for the associated vmhbaxx:xx:xx:L[lunid]. The associated IP should be listed at the bottom of this output.
$ pureport list
------------------------------------------------------------------------
Jul 30 22:17:14 pureport list
------------------------------------------------------------------------
Name WWN Portal IQN Failover
CT0.ETH4.154 - 192.168.10.10:3260 iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 -
CT0.ETH5.154 - 192.168.10.11:3260 iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 -
CT1.ETH4.154 - 192.168.10.12:3260 iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 -
CT1.ETH5.154 - 192.168.10.13:3260 iqn.2010-06.com.purestorage:flasharray.512495d090d847c9 -
esxi-host/commands/vmware-vimdump_-o----U-dcui.txt
(vim.host.MultipathInfo.Path) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
key = 'key-vim.host.MultipathInfo.Path-vmhba64:C2:T0:L253',
name = 'vmhba64:C2:T0:L253',
pathState = 'active',
state = 'active',
isWorkingPath = true,
adapter = 'key-vim.host.InternetScsiHba-vmhba64',
lun = 'key-vim.host.MultipathInfo.LogicalUnit-0200fd0000624a93708b6243d446c34f98000121af466c61736841',
transport = (vim.host.InternetScsiTargetTransport) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
iScsiName = 'iqn.2010-06.com.purestorage:flasharray.512495d090d847c9',
iScsiAlias = '',
address = (str) [
'192.168.10.11:3260'
Associating VVOL RFC Label with Array Volume
For VVOL volumes ESXi labels each volume with a unique RFC number. This is separate from the NAA number as we are utilizing a single array volume with the protocol endpoint. Vmkernel.log and vvold.log in the ESXi host logging will refer to this number when reporting issues for that volume.
If you are new to VVOLs and need a better understanding I recommend starting here:
VVOL volumes are listed in the vmfs/volumes/ folder on the host logging. They are labeled with an RFC number. This won't correlate directly to any volume ID on the array.
~hostlogs ls -lah vmfs/volumes/vvol* total 136K drwxrwxrwx 34 dschaat fuseuser 6.0K May 29 19:58 . drwxrwxrwx 4 dschaat fuseuser 6.0K May 29 19:59 .. drwxrwxrwx 2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.2064fb10-e9d1-41ea-b586-e5cf78e0e266 drwxrwxrwx 2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.233032b9-11d3-4f58-9bcd-e43c1e03169a drwxrwxrwx 2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.25f0ceb9-765b-453e-b06d-516fbd3fbaf4 drwxrwxrwx 2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.38cae476-5d65-4465-9bbc-a37c64a80ef6 drwxrwxrwx 2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.3d4f3a13-31f9-4d91-9c6d-b99e100cfc5f drwxrwxrwx 2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.455fb510-d36a-4adf-b6ed-b800c1c2216b drwxrwxrwx 2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.54ca118e-5062-4b39-97d3-99d6755341f7 drwxrwxrwx 2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.562743c3-e37b-4425-8ea4-e4ab34e7c23f drwxrwxrwx 2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.57af1a31-fbbc-4ed2-9768-11e60730bce3 drwxrwxrwx 2 dschaat fuseuser 6.0K May 29 19:59 rfc4122.5d7188cf-dc5a-4cdd-8fc6-0f57de70c3c3
We can still determine which volumes on the array the RFC number associates to through the diagnostics log. This lists all associated data, config and swap volumes of the VVOL.
fuse~$ zgrep -i rfc4122.2064fb10-e9d1-41ea-b586-e5cf78e0e266 diagnostics.log-2019073001.gz vvol-vmwinbuild-2064fb10-vg/Config-196e57ab PURE_VVOL_ID rfc4122.2064fb10-e9d1-41ea-b586-e5cf78e0e266 - vvol-vmwinbuild-2064fb10-vg/Data-61d6cabc VMW_VVolNamespace /vmfs/volumes/vvol:77596d73ea8c3096-8279bf44f15a6156/rfc4122.2064fb10-e9d1-41ea-b586-e5cf78e0e266 - vvol-vmwinbuild-2064fb10-vg/Swap-34c1d0b5 VMW_VVolNamespace /vmfs/volumes/vvol:77596d73ea8c3096-8279bf44f15a6156/rfc4122.2064fb10-e9d1-41ea-b586-e5cf78e0e266
When investigating VVOL issues generally, look to the vvold.log in var/run/log of the host logging. The following link has information on failure scenarios to look for:
https://support.purestorage.com/Solutions/VMware_Platform_Guide/003Virtual_Volumes_-_VVols/Troubleshooting/Understanding_Virtual_Volumes_and_Failure_Scenarios
Correlating Pure Storage Plugin Events with Vcenter Logging
When looking for events that involve the Pure Storage plugin you'll need to look into the Vcenter logging specifically. This is not included in a host bundle unless you specifically select to include them so make sure to ask for Vcenter logging if it is needed.
Events that are intiated with the Pure Storage plugin on Vcenter will register in the vsphere_client_virgo.log which is located in the /var/log/vmware/vsphere-client/logs folder.
Below is an example of a datastore creation being created with the Pure Storage plugin. On the array end we'll see the associated API call within the rest.log.
root@bootcamp-johnathan-vcsa-1 [ /var/log/vmware/vsphere-client/logs ]# less vsphere_client_virgo.log
[2019-07-25T03:25:08.948Z] [INFO ] http-nio-9090-exec-2 com.purestorage.PureDatastoreImpl createHandler : duration = 307 ms, params = ASObject(593708984){activityId=710AEDA3-DF7A-EF97-C17A-27289BEF62B2, sizeUnits=4, flashArrayId=3a
cc8acd-9257-4a15-8bef-6a001e0f0cf9/fa-0a769403-6af1-4bd6-a59a-98e4db2b8918, volumeName=jhughes-test-3, serverGuid=2bd52d7e-2c5b-4cd2-b39f-144afeb64229, snapshotName=, action=createVolume, volumeSize=1}, result = {volumeSerial=DA3369DA7755463B0039E358}
[2019-07-25T03:25:08.971Z] [INFO ] http-nio-9090-exec-10 org.springframework.flex.servlet.MessageBrokerHandlerAdapter Channel endpoint amf received request.
[2019-07-25T03:25:09.172Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: slc-wfh
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group UCS-ESXi-67-Cluster-1-FC
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group UCS-ESXi-67-Cluster-1-iSCSI
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group UCS-ESXi-67-Cluster-2-FC
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group UCS-ESXi-67-Cluster-2-iSCSI
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-alex-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-alex-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-eddie-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-eddie-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-brent-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-brent-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-shawn-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-shawn-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-jlarson-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-jlarson-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group Bootcamp-Mike-ESXi-Cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-philip-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-philip-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group Bootcamp-Mike-ESXi-Cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-hatchett-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-hatchett-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group Bootcamp-ryan-esxi-cluster-1
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group Bootcamp-ryan-esxi-cluster-2
[2019-07-25T03:25:09.260Z] [INFO ] http-nio-9090-exec-10 com.purestorage.FlashArrayClient findHostGroupName: Checking host group bootcamp-johnathan-esxi-cluster-1
[2019-07-25T03:25:09.564Z] [INFO ] http-nio-9090-exec-10 com.purestorage.PureDatastoreImpl createHandler : duration = 592 ms, params = ASObject(1793856952)
{activityId=710AEDA3-DF7A-EF97-C17A-27289BEF62B2, flashArrayId=3acc8acd-9257-4a15-8bef-6a001e0f0cf9/fa-0a769403-6af1-4bd6-a59a-98e4db2b8918, volumeName=jhughes-test-3, serverGuid=2bd52d7e-2c5b-4cd2-b39f-144afeb64229, host={"hbas":{},"iqns":[],"children":[{"hbas":{},"iqns":[],"productVersion":"6.7.0","children":[],"isCluster":false,"portWorldWideNamesHex":[],"isInMaintenanceMode":false,"moName":"host-11","isFolder":false,"name":"bootcamp-johnathan-esxi-1.dev.purestorage.com"},{"hbas":{},"iqns":[],"productVersion":"6.7.0","children":[],"isCluster":false,"portWorldWideNamesHex":[],"isInMaintenanceMode":false,"moName":"host-33","isFolder":false,"name":"bootcamp-johnathan-esxi-2.dev.purestorage.com"}],"isCluster":true,"portWorldWideNamesHex":[],"isInMaintenanceMode":false,"moName":"domain-c7","isFolder":false,"name":"ESXi 6.7 Cluster 1"}, action=connectVolume}, result = {}
This will be a Post result in the rest logging.
2019-07-25 03:24:49.231 [INFO] 10.204.114.155 >> POST http://10.204.120.159/api/1.13/volume/jhughes-test-3
{
"size":1099511627776
}
2019-07-25 03:24:49.453 [INFO] << 200 OK
{
"created":"2019-07-25T03:24:49Z",
"name":"jhughes-test-3",
"serial":"DA3369DA7755463B0039E356",
"size":1099511627776,
"source":null
}
Miscellaneous Host Information
This section includes common information that you'll need to gather from the host. Most of this should be found with the mr_vmware tool output; however, if you have to manually search, these are some of the common points.
Host Local Date and Time Zone
Found in the commands/date.txt file.
$hostlog/commands$ cat date.txt Tue Oct 9 14:37:29 UTC 2018
Host Hardware Information
Found in commands/esxcfg-info_-a.txt.
(Sat Aug 03 02:10:39 UTC) jhughes@cloudfuse-prod-07f2658fac6abe231:/support/ES-51882/VMware-Logs/ESXi_Host-Logs/esx-DLESX023.jdadelivers.com-2018-10-09--14.37-1547173/commands$ grep -E "Product Name|Vendor Name|Serial Number|Hardware Uptime" esxcfg-info_-a.txt
|----Product Name.............................................PowerEdge M620
|----Vendor Name..............................................Dell Inc.
|----Serial Number............................................C8ZYJ02
|----Hardware Uptime..........................................3645770847531
|----Vendor Name.....................................Intel Corporation
ESXi Build Version
Found in commands/vmware_-vl.txt
(Sat Aug 03 02:13:13 UTC) jhughes@cloudfuse-prod-07f2658fac6abe231:/support/ES-51882/VMware-Logs/ESXi_Host-Logs/esx-DLESX023.jdadelivers.com-2018-10-09--14.37-1547173/commands$ cat vmware_-vl.txt VMware ESXi 6.5.0 build-7967591 VMware ESXi 6.5.0 Update 1
ESXi SCSI FNIC Driver Version
Found in commans/localcli_software-vib-list.txt
(Sat Aug 03 02:19:39 UTC) jhughes@cloudfuse-prod-07f2658fac6abe231:/support/ES-51882/VMware-Logs/ESXi_Host-Logs/esx-DLESX023.jdadelivers.com-2018-10-09--14.37-1547173/commands$ grep -E "Name|fnic" localcli_software-vib-list.txt Name Version Vendor Acceptance Level Install Date scsi-fnic 1.5.0.45-3vmw.650.0.0.4564106 VMW VMwareCertified 2018-08-28
ESXi HBA/NIC Information
Found in commands/localcli_storage-core-adapter-list.txt. Further information can be found in either commands/nicinfo.sh.txt for ISCSI or commands/vmkmgmt_keyval_-a.txt for FC.
(Sat Aug 03 02:28:53 UTC) jhughes@cloudfuse-prod-07f2658fac6abe231:/support/ES-51882/VMware-Logs/ESXi_Host-Logs/esx-DLESX023.jdadelivers.com-2018-10-09--14.37-1547173/commands$ cat localcli_storage-core-adapter-list.txt HBA Name Driver Link State UID Capabilities Description -------------------------------------------------------------------------------------------------------- vmhba0 vmw_ahci link-n/a sata.vmhba0 (0000:00:1f.2) Intel Corporation Patsburg 6 Port SATA AHCI Controller vmhba1 qlnativefc link-up fc.20000024ff531ff0:21000024ff531ff0 Second Level Lun ID (0000:03:00.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA vmhba2 qlnativefc link-up fc.20000024ff531ff1:21000024ff531ff1 Second Level Lun ID (0000:03:00.1) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA vmhba32 vmkusb link-n/a usb.vmhba32 () USB vmhba33 bnx2i unbound iscsi.vmhba33 Second Level Lun ID QLogic 57810 10 Gigabit Ethernet Adapter vmhba34 bnx2i unbound iscsi.vmhba34 Second Level Lun ID QLogic 57810 10 Gigabit Ethernet Adapter
APD vs PDL
APD generally means we have lost access to all storage paths. A timer is started and once that timer is exceeded then paths are marked permanently lost. This means we can recover paths in an APD event, but we can still have an outage from a production standpoint in the time frame of a APD. Once a path has been marked PDL, then a host reboot is required to bring them back. Review the following VMware KB, which goes over some of the failure symptoms and examples of vmkernel logging: