Considerations for Deploying Hyper-V Hosts Using FlashArray

BIOS/UEFI Power Settings for Processors

Hardware vendors have settings in BIOS/UEFI to regulate how much power each processor core can use. By default, the settings usually allow the server to reduce core speed to save power. The setting(s) that control this behavior vary from manufacturer to manufacturer. For example, this setting is sometimes referred to as a C-State, while others may call it something different. Note that sometimes to ensure full power, multiple related settings may need to be configured. Vendors may also have recommended profiles for virtualization.

Refer to the hardware vendor’s documentation for the server’s particular CPU configuration to determine the appropriate CPU power profile configuration.

Windows Server Power Plan

By default, Windows Server’s power plan is set to Balanced. Set it to High performance for SQL Server installations, as shown in Figure 1. If this setting cannot be set in Windows Server, check Group Policy to ensure that it is not set somewhere that is not obvious and more challenging to alter.

Figure 1. Power plan in Windows Server

Network Card Power Management

Disable any network adapter power management to ensure consistent performance, especially if using network-based storage such as iSCSI.

Settings are often specific to a vendor's network adapter. Use the documentation in Microsoft learn for the PowerShell cmdlet Disable-NetAdapterPowerManagement to see which options are available for disabling any power management settings present. Note that the use of this command could temporarily restart the network adapter and interrupt server connectivity. Below is an example execution that disables all power management features on a NIC named Ethernet 1.

Disable-NetAdapterPowerManagement -Name "Ethernet 1"

Single Root I/O Virtualization

Single Root I/O Virtualization (SR-IOV) is an extension of the PCI Express (PCIe) specification that allows a device, including network adapters, to separate access to its resources among various hardware functions and differentiate between different traffic streams. It allows traffic streams to be delivered directly to the Hyper-V or VM layers separately. It enables networking traffic to bypass the software-defined virtual networking switch layer of Hyper-V and directly into the guest OS to provide for bare-metal networking stack performance. Physical server firmware, physical NIC, and network drivers must all support enabling SR-IOV.

SR-IOV must be enabled in UEFI/BIOS. To check that this has been set, issue the following PowerShell command on each Hyper-V host:

(Get-VMHost).IovSupport

This should return a value of True if it is available and False if it is not or just not enabled.

Enabling SR-IOV on a NIC can be done via PowerShell. An example is shown below.

Enable-NetAdapterSriov -Name "Ethernet 1"

Figure 3 also shows it enabled in the driver for a NIC.

Figure 3. SR-IOV enabled on a NIC

To enable SR-IOV on the Hyper-V vSwitch, open the Virtual Switch Manager. Select or create the virtual switch, and next to the External network connection type, select “Enable single-root I/O virtualization (SR-IOV)”, as shown in Figure 4.

Figure 4. Single-Root I/O Virtualization in Hyper-V Manager

Alternatively, use PowerShell to enable SR-IOV when creating a new virtual switch. An example is shown below.

New-VMSwitch <virtual-switch-name> -NetAdapterName <network-adapter-name> -EnableIov $true

Finally, Figure 5 shows enabling SR-IOV on a VM's vNIC.

Figure 5. Single-Root I/O Virtualization for a vNIC in Hyper-V Manager

Remote Direct Memory Access

The use of Remote Direct Memory Access, or RDMA, is highly encouraged for high-speed interconnects between Hyper-V hosts for functionality such as Live Migration. RDMA requires both RDMA-enabled NICs and switches that support RDMA to exchange data in main memory without requiring the CPU or operating system to act as an intermediary. RDMA-enabled server adapters boost performance because it reduces the overhead of resource consumption (mainly CPU) and increases network throughput rates and lower latency.

Receive Side Scaling

Receive Side Scaling (RSS) enables NICs to provide more consistent resource consumption across multiple CPUs on multiprocessor platforms. By default, Windows Server usually picks a single CPU to process network traffic handling on. With RSS enabled, Windows Server will more widely distribute the I/O data across more than one deferred procedure call (DPC) for more concurrent traffic handling.

To enable RSS for all physical adapters in the Hyper-V host, the following script can be used. First, RSS is enabled for use globally, then enabled per physical network adapter.

Command Line example enabling RSS globally.

netsh interface tcp set global rss=enabled

Using PowerShell to enable RSS on each adapter.

foreach($NIC in (Get-NetAdapter -Physical)) {

Enable-NetAdapterRSS -name $NIC.name

}
 

Figure 6 shows RSS enabled on a NIC.

Figure 6. RSS enabled

Virtual Machine Queue

Virtual Machine Queue (VMQ) can be enabled on the physical NICs on the Hyper-V host. VMQ is a network adapter offload technology that can extend Native RSS to the Hyper-V layer. The goal is to allow for Hyper-V to manage the network traffic processing queues in the host hypervisor layer and both reduce host CPU consumption and improve guest network throughput by spreading out the CPU load across multiple host processors and by using direct memory access to transfer network packets directly into a VM’s shared memory. This feature must be supported by the physical network adapter, and once enabled at the host as shown in Figure 7, can be enabled per vNIC.

Figure 7. VMQ enabled on a NIC

NIC Teaming

Microsoft has natively supported NIC teaming, or the ability to combine multiple NICs as a single NIC, since Windows Server 2012. Prior to that, teaming was achieved with proprietary software and did not always work well with features like a WSFC. Teaming can enhance both performance and availability depending on the configuration if employed. To understand the teaming in Windows Server, reference  “Windows Supported Networking Scenarios” at Microsoft Learn.

There are two types of teaming:

●        Switch independent – This type is independent of the network switch.

●        Switch dependent – This type requires a network switch configuration. Link Aggregation Control Protocol (LACP) is a core aspect of switch dependent. This means LACP must be configured on the physical switch. Consult the switch vendor’s documentation for information on how to achieve this task.

For Windows Server 2016 and above, Switch Embedded Teaming (SET) is recommended to be enabled for Hyper-V hosts. SET was introduced with Windows Server and System Center Virtual Machine Manager (SCVMM) 2016 as a new method to simplify the teaming of multiple network adapters. SET is managed at the Hyper-V switch level and not at a network team level. Physical NIC teaming is no longer required to allow more concurrent traffic across multiple network adapters. SET is specifically integrated with Packet Direct, Converged RDMA vNICs, and software defined networking (SDN) quality of service (QoS) configurations. SET can be enabled within SCVMM by configuring a logical switch with an embedded team uplink mode.

Examples of enabling SET using PowerShell on vSwitch are below. Change the names of your network adapters to match your hardware.

New Switch

New-VMSwitch -Name "VMSETSwitch" -NetAdapterName "Ethernet 3", "Ethernet 4" -EnableEmbeddedTeaming $true

Existing Switch

Get-VMSwitch | FL Name, EmbeddedTeamingEnabled, NetAdapterInterfaceDescriptions, SwitchType

Multiple Paths to FlashArray

Both FC and iSCSI support the Multipath IO (MPIO) feature of Windows Server. This feature allows a Hyper-V host to communicate to and from storage arrays across multiple paths to provide improved performance, load balancing, and resiliency for storage connections. Each physical HBA should have ports connected via redundant switching to different controller interconnects in the storage array. FlashArray has multiple interconnects on each redundant controller. An example of multiple hosts connected with active multipathing to a FlashArray is shown in Figure 8.

Figure 8. Active multipathing

A physical host should be connected to more than a single switch to ensure redundancy in and out of the host.  These redundant switches are used to connect each physical server and devices like shared storage in a redundant manner. Physical interfaces, either network adapters or Host Bus Adapters (HBAs), or both, connect the physical server to the switches, and should be physically redundant within the host. One or more logical virtual switches at the Hyper-V layer act as an intermediary to pass through and route VM traffic to the physical adapter(s).

The VM’s vNIC or virtual Fibre Channel Adapter (vHBA) is then bound to a virtual switch port to complete the end-to-end connection to the physical network. A number of redundancy and performance-oriented factors need to be reviewed to ensure the network architecture is optimal for Hyper-V and storage.

Fibre Channel Best Practices

A Host Bus Adapters (HBA) is a dedicated fiber-optic transceiver adapter to connect to Fibre Channel (FC)-based storage networks. A FlashArray can be connected to a Hyper-V host via FC.

Physical HBAs have a tunable setting on the physical device called queue depth. The queue depth value specifies the number of I/O requests that the host will put into a queue for a target storage port. If the maximum queue depth is reached, the device will reject and discard additional commands, forcing the Hyper-V host to reissue the I/O request a short time later. This results in latency to the host and/or the VM. If this discard and retry occurs, the observed storage latency inside the Hyper-V VM will increase, potentially causing performance degradation for the workloads.

When switching to faster storage such as FlashArray, default queue depth values may need to be changed. If necessary, adjusting the queue depth to a higher vale can help boost performance levels from the Hyper-V host and VM workloads. Increasing it too high can result in performance degradation from overloading the SAN controllers, as the ports might be unable to handle such a high rate of concurrent commands. All Hyper-V hosts connected to these ports could be negatively impacted.

Check with your HBA hardware vendor for the value of your HBA queue depths and adjustment process, and test accordingly with a combination of your application workload and synthetic benchmarking tools to find the right queue depth balance for your storage array.

iSCSI Best Practices

iSCSI uses NICs to connect to FlashArray to be able to presented storage as block to the Hyper-V host.

To properly implement iSCSI and MPIO on Hyper-V hosts with FlashArray, read the document “iSCSI Best Practices for Windows Server and FlashArray”, “Setup iSCSI on Windows Server”, and the MPIO articles in the Microsoft Platform Guide. This includes information on configuring settings such as Delayed Acknowledgement (Delayed Ack for short) which can potentially reduce end-to-end latency as well as TcpAckFrequency and TcpNoDelay, which is covered in the iSCSI best practices document above.

For iSCSI-based Pure Storage connections, bi-directional flow control should be enabled for all switch ports that are used for iSCSI traffic.

At least one dedicated isolated network or VLAN should be configured for iSCSI traffic. If this is a VLAN, unicast storm control, multicast, and broadcasts should be disabled. Storm and broadcast control prevents traffic on the VLAN from being disrupted by a broadcast, unicast, or multicast storm on a physical network interface. These storms are disruptive from other devices flooding the network with packets that create excessive traffic and disrupt communications. The group responsible for networking can implement these changes.

If more than one network is used for iSCSI and you are clustering the Hyper-V hosts as a WSFC, each iSCSI network must have its own subnet. A WSFC requires a distinct network for each network that will show up in the WSFC which means a separate subnet for each. There cannot be just one subnet for iSCSI. This will also affect how things are configured on the FlashArray as well.

Jumbo frames is a setting that may improve performance for some configurations that use iSCSI. Before configuring jumbo frames, evaluate whether or not the effort and the cost of managing jumbo frames outweighs the potential performance gain. If the performance gain will be large, the endeavor may be worth it. Tuning jumbo frames involves changing the Maximum Transmission Unit (MTU) which usually defaults to 1500 and can be set as high as 9000. While this allows iSCSI network communication to fully leverage the full available bandwidth of the physical network, there are caveats and challenges. All switch port hops and end-to-end connections between the Hyper-V hosts, physical and virtual switches, and enterprise storage must be fully configured for jumbo frames to achieve the full networking rate of speed. If anything in that chain is misconfigured, payload mismatches will result in a severe degradation of performance or at worst, no connectivity to storage.

As part of the MPIO configuration, use either the Round Robin (RR) or Least Queue Depth (LQD) load balancing policies to improve the performance and utilization of all available paths to the underlying storage. For hosts with 10 or fewer paths to a FlashArray volume, LQD is recommended for use. Test each algorithm to determine which works more effectively for your workload. Follow the best practices documented in the Microsoft Platform Guide for MPIO and iSCSI.