iSCSI Best Practices for Windows Server and FlashArray
This article will cover some of the iSCSI best practice recommendations for Windows Server and the Pure Storage FlashArray. This article will be updated as new or revised configurations are introduces, tested, and validated.
iSCSI and Windows Server
These are some best practices that should be implemented for the versions of Windows Server that are mentioned for each setting.
These are recommendations by Pure Storage and Microsoft and should be thoroughly tested before being implemented in a production environment.
Some of these changes require altering the Windows registry. It is highly recommended to create a backup of the registry before making changes.
Disabling Delayed ACK and Nagle
Applicable to Pure supported Windows Server versions 2012, 2012R2, 2016, and 2019.
This Best Practice has not yet been tested with bonded (aka "teamed") network interfaces.
The Windows operating system incorporates a setting for TCPIP called the Delayed Acknowledgement feature. By changing the default settings, you could possibly reduce the amount of overall network latency when using iSCSI connections. Along with this setting, the registry entry that enables Nagle should also be set to disabled. These settings should be applied to only network interface instances that are used for iSCSI traffic. All iSCSI adapters should be set to the same settings. If the entries do not exist, they should be created.
Altering these settings may have a negative effect on other services and protocols used on the server. While this is a recommended practice, caution is given that the setting changes should be tested in your environment before altering a production system. Please consult official Microsoft documentation for more details.
Registry Keys
Subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\<Interface GUID>
Entry: TcpAckFrequency
Value Type: REG_DWORD
Valid Range: 0-255
Default: 2
New Setting: 1
Subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\<Interface GUID>
Entry: TcpNoDelay
Value Type: REG_DWORD
Valid Range: 0-1
Default: 0
New Setting: 1
This image illustrates the new settings.
The machine should be rebooted after making any of these registry changes.
PowerShell to Add or Change Registry Keys
This is a snippet of a example script that will enumerate all of the network adapters on a system and require the user to enter the adapter name(s) they wish to apply the settings to. This script is not all-inclusive to every environment and should be tested thoroughly before running in production.
The individual script is available in the Pure PowerShell Example Scripts repository. The Pure Storage PowerShell Toolkit also contains a cmdlet to perform this configuration. The machine should be rebooted after making any of these registry changes.
<# Script to disable Nagle (TcpNoDelay) and TcpAckFrequency on a per adapter basis.#> $AdapterNames = @() Write-Host "All available adapters: " Write-Host " " $adapters = Get-NetAdapter | Sort-Object Name | Format-Table -Property "Name", "InterfaceDescription", "MacAddress", "Status" $adapters Write-Host " " $AdapterNames = Read-Host "Please enter all iSCSI adapter names to be tested. Use a comma to seperate the names - ie. NIC1,NIC2,NIC3" $AdapterNames = $AdapterNames.Split(',') Write-Host " " Write-Host "Adapter names being configured: " $AdapterNames Write-Host "===============================" foreach ($adapter in $AdapterNames) { $adapterGuid = (Get-NetAdapterAdvancedProperty -Name $adapter -RegistryKeyword "NetCfgInstanceId" -AllProperties).RegistryValue $RegKeyPath = "HKLM:\system\currentcontrolset\services\tcpip\parameters\interfaces\$adapterGuid\" $TAFRegKey = "TcpAckFrequency" $TNDRegKey = "TcpNoDelay" ## TcpAckFrequency if ((Get-ItemProperty $RegkeyPath).$TAFRegKey -eq "1") { Write-Host ": TcpAckFrequency is set to disabled (1). No action required." } if (-not (Get-ItemProperty $RegkeyPath $TAFRegKey -ErrorAction SilentlyContinue)) { Write-Host ": TcpAckFrequency key does not exist." Write-Host "REQUIRED ACTION: Set the TcpAckFrequency registry value to 1 for $adapter ?" -NoNewline $resp = Read-Host -Prompt "Y/N?" if ($resp.ToUpper() -eq 'Y') { Write-Host "Creating Registry key and setting to disabled..." New-ItemProperty -Path $RegKeyPath -Name 'TcpAckFrequency' -Value '1' -PropertyType DWORD -Force -ErrorAction SilentlyContinue } else { Write-Host "WARNING" -ForegroundColor Yellow -NoNewline Write-Host ": TcpAckFrequency registry key exists but is enabled. Changing to disabled." Set-ItemProperty -Path $RegKeyPath -Name 'TcpAckFrequency' -Value '1' -Type DWORD -Force -ErrorAction SilentlyContinue } } if ($resp.ToUpper() -eq 'N') { Write-Host "ABORTED" -ForegroundColor Yellow -NoNewline Write-Host ": Registry key not created or altered by request of user." } ## TcpNoDelay if ((Get-ItemProperty $RegkeyPath).$TNDRegKey -eq "1") { Write-Host ": TcpNoDelay (Nagle) is set to disabled (1). No action required." } if (-not (Get-ItemProperty $RegkeyPath $TNDRegKey -ErrorAction SilentlyContinue)) { Write-Host "REQUIRED ACTION: Set the TcpNodelay (Nagle) registry value to 1 for $adapter ?" -NoNewline $resp = Read-Host -Prompt "Y/N?" if ($resp.ToUpper() -eq 'Y') { Write-Host "TcpNoDelay registry key does not exist. Creating..." New-ItemProperty -Path $RegKeyPath -Name 'TcpNoDelay' -Value '1' -PropertyType DWORD -Force -ErrorAction SilentlyContinue } else { Write-Host "WARNING" -ForegroundColor Yellow -NoNewline Write-Host ": TcpNoDelay registry key exists. Setting value to 1." Set-ItemProperty -Path $RegKeyPath -Name 'TcpNoDelay' -Value '1' -Type DWORD -Force -ErrorAction SilentlyContinue } } if ($resp.ToUpper() -eq 'N') { Write-Host "ABORTED" -ForegroundColor Yellow -NoNewline Write-Host ": TcpNoDelay registry key not created or altered by request of user." } }
Setting Power Plan to High Performance
Applicable to Pure supported Windows Server versions 2012, 2012R2, 2016, and 2019.
Use these commands to set the Power Plan to High Performance on Windows Server. The Pure Storage PowerShell Toolkit also contains a cmdlet to perform this configuration.
PS >$p = Get-CimInstance -Name root\cimv2\power -Class win32_PowerPlan -Filter "ElementName = 'High Performance'" Caption : Description : Favors performance, but may use more energy. ElementName : High performance InstanceID : Microsoft:PowerPlan\{8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c} IsActive : False PSComputerName : PS >powercfg /setactive ([string]$p.InstanceID).Replace("Microsoft:PowerPlan\{","").Replace("}","") Caption : Description : Favors performance, but may use more energy. ElementName : High performance InstanceID : Microsoft:PowerPlan\{8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c} IsActive : True PSComputerName :
Disable Network Adapter Power Management
Applicable to Pure supported Windows Server versions 2012, 2012R2, 2016, and 2019.
Warning - This command will cause a network adapter reset unless the -NoRestart
switch is used. If -NoRestart
is used, you must manually restart the network adapter via the Windows UI or script to reset the network adapter and activate the power management setting. For more information, refer to this Microsoft documentation.
Alter the -Name
parameter to reflect the name of the adapter to be modified.
PS C:\> Disable-NetAdapterPowerManagement -Name "Ethernet1"
Header and Data Digest Settings
The Microsoft Windows implementation of iSCSI includes the capability to enable Header and/or Data Digests. These protocols provide an extra measure of data integrity to the standard TCP checksums that are present in the iSCSI traffic flow. A Header Digest will perform a checksum of the iSCSI Protocol Data Units (PDUs) at the 48-byte header of the iSCSI TCP packet. A Data Digest will perform a checksum against each data segment included in the iSCSI TCP packet that is attached to the PDU.
Enabling either or both of these Digests does require additional CPU usage on the initiator as well as the target to calculate and append checksums. This process occurs bi-directionally (incoming and outgoing) on the initiator and the target. Every storage environment is unique and when altering settings such as Digests, it must first be fully tested to accurately assess overall impacts on IOPS and CPU performance on the iSCSI network and participating devices. In our lab testing, we found slight to moderate increases in CPU overhead based on load when these options were enabled, with a minor effect on IOPS. The overall recommendation from Pure is to keep these options disabled if they are not required in your environment. This recommendation should not be considered a "best practice" configuration as the decision to enable these options are based on the specific environments and workloads and would need to be tested.