Skip to main content
Pure1 Support Portal

How To: Adjusting the UCS nfnic queue depth on ESXi 6.7

With ESXi 6.7 the Cisco fnic driver was replaced with the Cisco nfnic driver for UCS Blades running ESXi.  With this update the default queue depth with the nfnic driver is set  to 32 and initially there was not a module parameter to adjust the nfnic queue depth.  Cisco released an updated version of the nfnic driver (4.0.0.28+) with a configurable module parameter, "lun_queue_depth_per_path".  This KB will cover updating the nfnic driver, adjusting the nfnic queue depth and device queue depth.

Cisco has a published Bug outlining the issue, CSCvo09082, which can be referred to as needed.  With the default queue depth with the nfnic driver set to 32, that in turn limits all Maximum Device Queue Depth and DSRNO (Disk Schedule Number Requests Outstanding) to 32.  This would apply to device backings for VMFS Datastores, RDMs and Protocol Endpoints.  This has primarily been an issue when using vSphere Virtual Volumes as the default device queue depth for a Protocol Endpoint should be 128.  This could also affect any high workload and IOPs VMs that need higher queue depths across the board.

Cisco released an updated version of the nfnic driver in 4.0.0.35 to adjust "lun_queue_depth_per_path".  This is not the default nfnic driver for ESXi 6.7 U2 and under.  This means that this driver will need to be manually installed until a Cisco image or an image from VMware are released with an updated nfnic driver.

You can get the updated nfnic driver from Cisco's UCS Blade Software page (the 4.0.4 release which can be found here) and from VMware directly as well here.

In this article the method of updating the nfnic driver, setting the nfnic "lun_queue_depth_per_path" parameter and updating the Protocol Endpoint's PESRNO (Protocol Endpoint Schedule Number Requests Outstanding) will be outlined and demonstrated.


Updating the nfnic Driver

  1. Confirm what version is currently running on the ESXi host.
    [root@ESXi-3:~] esxcli software vib list |grep nfnic
    nfnic                          4.0.0.17-0vmw.670.2.48.13006603       VMW     VMwareCertified   2019-04-15
    
    [root@ESXi-3:~] esxcli storage core device list -p
    naa.624a937016d9d3754f8a461800011016
       Display Name: PURE Fibre Channel Disk (naa.624a937016d9d3754f8a461800011016)
    ...
       Device Max Queue Depth: 32
       No of outstanding IOs with competing worlds: 32
    
    The device max queue depth and outstanding IOs for the PE is currently set to 32.
  2. For this ESXi Host the nfnic driver 4.0.0.17 is installed which does not have any supported module parameters.

    [root@ESXi-3:~] esxcli system module parameters list -m nfnic
    [root@ESXi-3:~]
    
  3. The nfnic driver 4.0.0.35 has been copied out from the 4.0.4 VMware related drivers ISO that can be downloaded from Cisco.  The nfnic package can also be downloaded from VMware.  The nfnic folder was copied to a datastore in the VMware environment that each of the hosts were connected to, in this case an ISO Repository. 

    [root@ESXi-3:~] ls /vmfs/volumes/sn1-m20-c12-25-ISO-Repository/VMware/Drivers/nfnic_ESXi_6.7U2/
    README.html                                               VMW-ESX-6.7.0-nfnic-4.0.0.35-offline_bundle-13303543.zip  nfnic-4.0.0.35-1OEM.670.0.0.8169922.x86_64.vib
    Release_Notes_nfnic-4.0.0.35.txt                          doc
    
  4. Here is the workflow for upgrading the nfnic driver:

    The host is put into maintenance mode (either from the CLI or UI)
    [root@ESXi-3:~] esxcli system maintenanceMode set --enable true
    
    The nfnic driver is upgraded via esxcli next
    [root@ESXi-3:~] esxcli software vib install -d "/vmfs/volumes/sn1-m20-c12-25-ISO-Repository/VMware/Drivers/nfnic_ESXi_6.7U2/VMW-ESX-6.7.0-nfnic-4.0.0.35-offline_bundle-13303543.zip"
    
    The host will need to be rebooted after the nfnic driver is upgraded
    [root@ESXi-3:~] esxcli system shutdown reboot --reason "nfnic driver upgrade"
    
  5. Once the reboot is complete the nfnic queue depth parameter can be set.


Setting the nfnic Queue Depth Parameter

  1. Unfortunately after upgrading the nfnic driver the option to set that parameter won't be available, it will only be configurable after the first reboot completes.  Once the ESXi hosts is back online reconnect via CLI and check that the nfnic module parameters are available.
    [root@ESXi-3:~] esxcli system module parameters list -m nfnic
    Name                      Type   Value  Description
    ------------------------  -----  -----  --------------------------------------------------------------
    lun_queue_depth_per_path  ulong         nfnic lun queue depth per path: Default = 32. Range [1 - 1024]
    
  2. Note that the default value is 32, but can be configured up to 1024.  Set the value to 128 in the event that vSphere Virtual Volumes are being used in the environment.  The value can be set higher or lower as needed, but 128 is recommended when using VVols.
    [root@ESXi-3:~] esxcli system module parameters set -m nfnic lun_queue_depth_per_path=128
    
    [root@ESXi-3:~] esxcli system module parameters list -m nfnic
    Name                      Type   Value  Description
    ------------------------  -----  -----  --------------------------------------------------------------
    lun_queue_depth_per_path  ulong  128    nfnic lun queue depth per path: Default = 32. Range [1 - 1024]
    
    [root@ESXi-3:~] esxcli storage core device list -p
    naa.624a937016d9d3754f8a461800011016
       Display Name: PURE Fibre Channel Disk (naa.624a937016d9d3754f8a461800011016)
    ...
       Device Max Queue Depth: 32
       No of outstanding IOs with competing worlds: 32
       
    [root@ESXi-3:~] esxcli storage core device set -O 128 -d naa.624a937016d9d3754f8a461800011016
    Unable to set device's sched-num-req-outstanding. Error was: Cannot set device queue depth parameter. sched-num-req-outstanding should be <= 32
    
  3. The ESXi host will need to be rebooted for the parameter change to take effect.  As shown above the device setting can not have the setting increased until after the reboot.
    [root@ESXi-3:~] esxcli system shutdown reboot --reason "nfnic parameter update"
    
  4. After the ESXi host is back online, the PE Device Queue Depth can be increased.

Updating the Protocol Endpoint No of Outstanding  IOs

  1. Once the ESXi host is back online check the Device Max Queue Depth setting on the PE to confirm that it is updated.
    [root@ESXi-3:~] esxcli storage core device list -p
    naa.624a937016d9d3754f8a461800011016
       Display Name: PURE Fibre Channel Disk (naa.624a937016d9d3754f8a461800011016)
    ...
       Device Max Queue Depth: 128
       No of outstanding IOs with competing worlds: 32
    
  2. Now the Outstanding IO value can be set on the PE Device, as it's default will still be 32.
    [root@ESXi-3:~] esxcli storage core device set -O 128 -d naa.624a937016d9d3754f8a461800011016
    [root@ESXi-3:~]
    
    [root@ESXi-3:~] esxcli storage core device list -p
    naa.624a937016d9d3754f8a461800011016
       Display Name: PURE Fibre Channel Disk (naa.624a937016d9d3754f8a461800011016)
    ...
       Device Max Queue Depth: 128
       No of outstanding IOs with competing worlds: 128
    
  3. Note that even though the ESXi PE has the default number of Outstanding Requests set to 128, when connecting any additional PEs the default will still be set at 32.  The device setting will need to be updated manually.  We are working with VMware to investigate why the default setting is not getting applied to additional PEs connected.