How To: Adjusting the UCS nfnic queue depth on ESXi 6.7
Cisco has a published Bug outlining the issue, CSCvo09082, which can be referred to as needed. With the default queue depth with the nfnic driver set to 32, that in turn limits all Maximum Device Queue Depth and DSRNO (Disk Schedule Number Requests Outstanding) to 32. This would apply to device backings for VMFS Datastores, RDMs and Protocol Endpoints. This has primarily been an issue when using vSphere Virtual Volumes as the default device queue depth for a Protocol Endpoint should be 128. This could also affect any high workload and IOPs VMs that need higher queue depths across the board.
Cisco released an updated version of the nfnic driver in 4.0.0.35 to adjust "lun_queue_depth_per_path". This is not the default nfnic driver for ESXi 6.7 U2 and under. This means that this driver will need to be manually installed until a Cisco image or an image from VMware are released with an updated nfnic driver.
Pure Storage does recommend using the nfnic driver version 4.0.0.52 or higher if the hosts are running ESXi 6.7 or higher.
This can be downloaded from VMware here.
You can get the updated nfnic driver from Cisco's UCS Blade Software page (the 4.0.4 release which can be found here) and from VMware directly as well here.
In this article the method of updating the nfnic driver, setting the nfnic "lun_queue_depth_per_path" parameter and updating the Protocol Endpoint's PESRNO (Protocol Endpoint Schedule Number Requests Outstanding) will be outlined and demonstrated.
Updating the nfnic Driver
- Confirm what version is currently running on the ESXi host.
[root@ESXi-3:~] esxcli software vib list |grep nfnic nfnic 4.0.0.17-0vmw.670.2.48.13006603 VMW VMwareCertified 2019-04-15 [root@ESXi-3:~] esxcli storage core device list -p naa.624a937016d9d3754f8a461800011016 Display Name: PURE Fibre Channel Disk (naa.624a937016d9d3754f8a461800011016) ... Device Max Queue Depth: 32 No of outstanding IOs with competing worlds: 32
The device max queue depth and outstanding IOs for the PE is currently set to 32. -
For this ESXi Host the nfnic driver 4.0.0.17 is installed which does not have any supported module parameters.
[root@ESXi-3:~] esxcli system module parameters list -m nfnic [root@ESXi-3:~]
-
The nfnic driver 4.0.0.35 has been copied out from the 4.0.4 VMware related drivers ISO that can be downloaded from Cisco. The nfnic package can also be downloaded from VMware. The nfnic folder was copied to a datastore in the VMware environment that each of the hosts were connected to, in this case an ISO Repository.
[root@ESXi-3:~] ls /vmfs/volumes/sn1-m20-c12-25-ISO-Repository/VMware/Drivers/nfnic_ESXi_6.7U2/ README.html VMW-ESX-6.7.0-nfnic-4.0.0.35-offline_bundle-13303543.zip nfnic-4.0.0.35-1OEM.670.0.0.8169922.x86_64.vib Release_Notes_nfnic-4.0.0.35.txt doc
-
Here is the workflow for upgrading the nfnic driver:
The host is put into maintenance mode (either from the CLI or UI) [root@ESXi-3:~] esxcli system maintenanceMode set --enable true
The nfnic driver is upgraded via esxcli next[root@ESXi-3:~] esxcli software vib install -d "/vmfs/volumes/sn1-m20-c12-25-ISO-Repository/VMware/Drivers/nfnic_ESXi_6.7U2/VMW-ESX-6.7.0-nfnic-4.0.0.35-offline_bundle-13303543.zip"
The host will need to be rebooted after the nfnic driver is upgraded[root@ESXi-3:~] esxcli system shutdown reboot --reason "nfnic driver upgrade"
-
Once the reboot is complete the nfnic queue depth parameter can be set.
Setting the nfnic Queue Depth Parameter
- Unfortunately after upgrading the nfnic driver the option to set that parameter won't be available, it will only be configurable after the first reboot completes. Once the ESXi hosts is back online reconnect via CLI and check that the nfnic module parameters are available.
[root@ESXi-3:~] esxcli system module parameters list -m nfnic Name Type Value Description ------------------------ ----- ----- -------------------------------------------------------------- lun_queue_depth_per_path ulong nfnic lun queue depth per path: Default = 32. Range [1 - 1024]
- Note that the default value is 32, but can be configured up to 1024. Set the value to 128 in the event that vSphere Virtual Volumes are being used in the environment. The value can be set higher or lower as needed, but 128 is recommended when using VVols.
[root@ESXi-3:~] esxcli system module parameters set -m nfnic -p lun_queue_depth_per_path=128 [root@ESXi-3:~] esxcli system module parameters list -m nfnic Name Type Value Description ------------------------ ----- ----- -------------------------------------------------------------- lun_queue_depth_per_path ulong 128 nfnic lun queue depth per path: Default = 32. Range [1 - 1024] [root@ESXi-3:~] esxcli storage core device list -p naa.624a937016d9d3754f8a461800011016 Display Name: PURE Fibre Channel Disk (naa.624a937016d9d3754f8a461800011016) ... Device Max Queue Depth: 32 No of outstanding IOs with competing worlds: 32 [root@ESXi-3:~] esxcli storage core device set -O 128 -d naa.624a937016d9d3754f8a461800011016 Unable to set device's sched-num-req-outstanding. Error was: Cannot set device queue depth parameter. sched-num-req-outstanding should be <= 32
- The ESXi host will need to be rebooted for the parameter change to take effect. As shown above the device setting can not have the setting increased until after the reboot.
[root@ESXi-3:~] esxcli system shutdown reboot --reason "nfnic parameter update"
- After the ESXi host is back online, the PE Device Queue Depth can be increased.
Updating the Protocol Endpoint No of Outstanding IOs
- Once the ESXi host is back online check the Device Max Queue Depth setting on the PE to confirm that it is updated.
[root@ESXi-3:~] esxcli storage core device list -p naa.624a937016d9d3754f8a461800011016 Display Name: PURE Fibre Channel Disk (naa.624a937016d9d3754f8a461800011016) ... Device Max Queue Depth: 128 No of outstanding IOs with competing worlds: 32
- Now the Outstanding IO value can be set on the PE Device, as it's default will still be 32.
[root@ESXi-3:~] esxcli storage core device set -O 128 -d naa.624a937016d9d3754f8a461800011016 [root@ESXi-3:~] [root@ESXi-3:~] esxcli storage core device list -p naa.624a937016d9d3754f8a461800011016 Display Name: PURE Fibre Channel Disk (naa.624a937016d9d3754f8a461800011016) ... Device Max Queue Depth: 128 No of outstanding IOs with competing worlds: 128
- Note that even though the ESXi PE has the default number of Outstanding Requests set to 128, when connecting any additional PEs the default will still be set at 32. The device setting will need to be updated manually. We are working with VMware to investigate why the default setting is not getting applied to additional PEs connected.