Web Guide: FlashBlade VMware vSphere PoC Guide

Executive Summary

FlashBlade, the high performance, massively scalable, highly-available Network Attached Storage solution from Pure Storage represents a technologically significant breakthrough in the flash-based storage segment. While the Pure Storage FlashArray offers the best platform for block storage, significant number of customers is interested in deploying Network Attached Storage as their primary means for data storage in physical and virtual environments.

This document focuses on FlashBlade in VMware ESXi deployments and describes how to build a Proof of Concept environment, execute workload tests, and to collect performance data.

Audience

The primary audience for this paper are system engineers, solution architects, vSphere and storage administrators, network administrators, and field engineers interested in understanding benefits and technical details of FlashBlade implementation in virtualized, software defined data centers. The essential understanding and knowledge of ESXi, network, and storage is required.

Introduction

Network File System, the widely-adopted distributed file system that has become an open standard, provides means for shared access to files stored on the NFS server. The NFS datastore support introduced with ESX version 3, greatly extended vSphere’s storage capabilities opening the entire new type of the devices to be deployed in virtualized environments. NFS datastores, conceptually resemble traditional VMFS datastores and may be easily deployed and integrated into existing network infrastructure.

Test Objectives

The primary goal of the testing was to demonstrate FlashBlade’s superior scalability, throughput, and performance when deployed as an NFS-backed datastore in virtualized data center.

FlashBlade

Test Bed Infrastructure

The test bed infrastructure designed for this test included the following hardware and software components – see Figure 5:

1. Pure Storage FlashBlade
   1. Single chassis
      1. 2+2 redundant power supplies
      2. 1+1 Elastic Fabric Modules
      3. Seven DirectFlash 8TB blades
   2. Two Fabric Modules
2. Four active 40Gb/s QSFP+ Ethernet ports
3. Servers:
   1. Four Cisco UCS B200 M4 2 socket with Intel Xeon E5-2600 series processor @2.40 GHz, 128 GB of memory, five 10GB/S vNICs
   2. Supermicro X8DTG-D 2 socket with Intel Xeon E5606 @ 2.13 GHz, 96GB of memory, two 1Gb/s and four 10Gb/s NICs – see Appendix A for VMware Load Balancing (not shown in Figure 3)
4. Network Switches
   1. Two Cisco Nexus 9396PX
   2. Network Topology – see Figure 5.
5. Software:
   1. FlashBlade Purity version 2.1.3
   2. ESXi verion 6.5
   3. vCenter Server version 6.5
   4. Windows Server 2016
   5. VDBench version 5.04.06
   6. Pure Storage VDBench GoKit 1.1

Figure 5

FlashBlade Configuration

FlashBlade may be configured using web-based HTML 5 user interface (no client installation required), command line or RestAPI.

puresubnet create --prefix<subnet/mask> --vlan <vland_id> <vlan_name>

puresubnet create --prefix 10.25.0.0/16 --vlan vlan202

purenetwork create vip --address <IP_address> --servicelist data name

purenetwork create vip --address 10.25.0.10 --servicelist data subnet25_NIC

  purefs create --rules <rules> --size <size> File_System

purefs create --rules '*(rw,no_root_squash)' --size 78GB DS10

purefs setattr --rules <rules> File_System  

purefs setattr --rules '*(rw,no_root_squash)' DS10

  purefs add --protocol <protocol> File_System

purefs add –protocol nfs DS10

purefs enable --fast-remove-dir --snapshot-dir File_System  

ESXi Host Configuration

To create a Virtual switch and NFS based datastores using vSphere Web Based client follow the steps below:

1. Create a vSwitch – see Figure 11

a. Select the hosts tab, Host Let me know if this sounds reasonable➤Configure (tab)➤Virtual switches-➤Add host networking icon

b. Select connection type: Select VMkernel Network Adapter – see Figure 12.

Figure 12

c. Select target device: New standard switch – see Figure 13.

Figure 13

d. Create a Standard Switch: Assign free physical network adapters to the new switch (click green ‘+’ sign and select an available active adapter (vmnic)) – see Figure 14.

Figure 14

e. Select Next when finished assigning adapters

f. Port properties – see Figure 15.

i. Network label (for example: VMkernelNFS)

ii. VLAN ID: leave at default (0) if you are not planning to tag the outgoing network frames

iii. TCP/IP stack: Default

iv. Available service: all disabled

Figure 15

g. IPv4 settings – see Figure 16.

i. IPv4 settings: Use static IPv4 settings (provide IP address and subnet mask)

ii. Provided the IP address and the corresponding subnet mask

iii. Review settings and finish creating the vSwitch

Figure 16

2. Verify connectivity from ESXi host to FlashBlade
   1. Login as root to the ESXi host
   2. Issue vmkping command  

vmkping <destination_ip>

ESXi NFS Datastore Configuration Settings

As the number of NFS datastores connected with each host increases, it will be necessary to adjust the following ESXi parameters on each ESXi server (see also Table 1):

• NFS.MaxVolumes – Maximum number of NFS mounted volumes (per host)
• Net.TcpipHeapSize - Initial TCP/IP heap size in MB
• Net.TcpipHeapMax – maximum TCP/IP heap size in MB
• SunRPC.MaxConnPerIp – maximum number of unique TCP connections per IP address

Table 1

The SunRPC.MaxConnPerIp should be increased to avoid sharing the host to NFS datastore connections. There is a maximum of 256 NFS datastores with 128 unique TCP connections, therefore forcing connection sharing when the NFS datastore limit is reached.

The settings listed in Table 1 must be adjusted on each ESXi host using vSphere Web Client (Advanced System Settings) or command line and may require a reboot.

esxcli system settings set -o “/SectionName/OptionName” --int-value=<value>

esxcli system settings set -o “/NFS/MaxVolumes” --int-value=16 

Test Environment Preparation

c:\windows\system32\netsh advfirewall set allprofiles off

c:\vdbench\vdbench -tf

********************************************************************************
*Global General Definition section
dedupratio=1
dedupunit=4k
compratio=3
********************************************************************************

****************************************************************************
*Global Host Definition section
hd=default,shell=vdbench
**************************************************************************
* DS01
* hd name for the slave node (label), system=IP_address or hostname of the slave node
hd=d1001,system=172.16.1.1
hd=d1002,system=172.16.1.2
hd=d1003,system=172.16.1.3
***********************************************

**************************************************************************
* Global Storage Definition section
* 50GB e:\vd-fb-nfstest.pdf file will be created
sd=default,openflags=directio,align=4k,threads=1
sd=sd1_$host,host=$host,LUN=E:\vd-fb-nfstest.pdf,size=50g,threads=8
**************************************************************************

*************************************************
*Global Workload Definition section
*Storage Workload Defaults
wd=wd1_$host,sd=sd1_$host,host=$host,rdpct=66,xfersize=(4k,12,8k,17,16k,21,32k,30,64k,20)
*************************************************

*************************************************
*Global Run Definition section
rd=rd1,wd=wd1_*,iorate=400, elapsed=3600,interval=30
*************************************************

************************************************************
*Load Other Configuration Files
include=global_general_definition.vdb
include=global_host_definition1.vdb
include=global_storage_definition.vdb
include=global_workload_definition.vdb
include=global_run_definition.vdb
************************************************************

set hr=%time:~0,2%
set hr=%hr: =0%
set start_time=%date:~-4,4%%date:~-10,2%%date:~-7,2%_%hr%%time:~3,2%%time:~6,2%
start /WAIT "FlashBlade" cmd /c ..\vdbench -f workload_fb.vdb -o reports\fb\2DS_maxIOPS_16VMs

@echo off
REM Import registry branch to enable AutoLogon
C:\windows\regedit.exe /S c:\<location_of_registry_file\name_of_registry_file>
REM Disable firewall
c:\windows\system32.netsh.exe advfirewall set allprofiles off
REM Start vdbench listener (remote shell)
c:\vdbench\vdbench.bat rsh

C:\Windows\system32\netsh advfirewall show allprofiles

netstat -an | Select-String 5560

Test Methodology and Execution

The tests were executed Vdbench with the following parameters:

• I/O Rate: variable
• Average IO Size: 27 KB
• Compression Ratio: 3:1
• Read Percentage: 20, 66, 80
• Unique Data: 15 GB
• IO Pattern: Random
• Test Execution Interval: 3600 seconds (1 hour)
• Data Collection Interval: 30 seconds

The average IO size of 27KB has been empirically selected based on all Pure Storage production arrays by average IO size with the following distribution – see Figure 31.

Figure 31

Testing Procedure

All tests were started on the master node by executing the batch file created in step Test Environment Preparation/Master Node.

Under some circumstances, when executing tests with large number of slave virtual machines, the master node fails to connect to some of the slaves. As a workaround, it may be necessary to initiate the test by starting two instances of Vdbench on the master node. Simply, copy the c:\vdbench directory and its subdirectories to another location on the same server, modify the test parameters (divide the number of hosts –  see global_host_defiinition.vdb file) and execute the startup scripts from different directories.

The tests were executed in four groups.

Group 1:  128 virtual machines, 100 IOPS per VM, variable read – write ratio.

Group 2: 128 virtual machines, 200 IOPS per VM, variable read – write ratio.

Group 3: 128 virtual machines, 400 IOPS per VM, variable read – write ratio. 

Group 4: 66/34 percent read/write ratio, 400 IOPS per virtual machine, variable number of virtaul machines.

Test Results

Conclusion

The test results show that as the workload and the number of virtual machines increase FlashBlade maintains high throughput coupled with excellent, nearly linear scale-out characteristics. As the customer’s environment expands, it is safe to predict that FlashBlade will deliver consistent and robust IO performance. Additionally, the multiple chassis online expansion without the need for manual data redistribution combined with exceptional scalability makes the FlashBlade an ideal storage device for high bandwidth, high demand virtual environments capable of supporting large number of virtual machines and ESXi hosts with minimal administrative overhead and seamless storage capacity and performance growth.

Appendix A

VMware supports several load balancing algorithms for virtual switches:

1. Route based on originating virtual port – network uplinks are selected based on the virtual machine port id - this is the default routing policy
1. Route based on source MAC hash – network uplinks are selected based on the virtual machine MAC address
2. Route based on IP hash – network uplinks are selected based on the source and destination IP address of each datagram
3. Route based on physical NIC load – uplinks are selected based on the load evaluation performed by the virtual switch; this algorithm is available only on vSphere Distributed Switch
4. Explicit failover – uplinks are selected based on the order defined in the list of Active adapters; no load balancing

The route based on originating virtual port and route based on source MAC hash routing teaming and failover policies require virtual machine to vSwitch connection therefore, they are not appropriate for VMkernel Virtual switches and NFS datastores. The Route based on IP hash is the only applicable teaming option.

Route based on IP hash load balancing ensures the egress network traffic is directed through one vmnic and ingress through another.

The route based on IP hash teaming policy also requires configuration changes of the network switches. The procedure to properly setup link aggregation is beyond the scope of this document. The following VMware Knowledge Base article provides additional details and examples regarding EtherChannel / Link Aggregation Control Protocol (LACP) with ESXi/ESX and Cisco/HP switches configuration:

https://kb.vmware.com/s/article/1004048

Changing network load balancing policy – see Figure A1 and Figure A2.

To change network load balancing policy using vSphere Web Client:

1. Select Host (tab)➤Host➤Configure-➤Virtual switches
2. Select the switch➤Edit (Pencil)
3. Virtual switch Edit Setting dialog➤Teaming and failover➤Load Balancing➤Route based on IP hash

To change network load balancing policy using command line:

esxcli network vswitch standard policy failover set -l iphash -v <vswitch-name>

Example:

esxcli network vswitch standard policy failover set -l iphash -l iphash -v vSwitch1

Figure A1

Appendix B

The typical Ethernet (IEEE 802.3 Standard) Maximum Transmission Unit is 1500 bytes, however larger MTU values are also available. Both FlashBlade and ESXi provide support for jumbo frames with MTU of 9000 bytes. To enable jumbo frame on FlashBlade and ESXi host follow steps below, for the switch configuration consult the switch’s manufacturer documentation.

  puresubnet create --prefix <subnet/mask> --vlan <vlan_id> --mtu <mtu> vlan_name

puresubnet create –prefix 10.25.64.0/21 –vlan vlan2064

  esxcfg-vswitch -m <MTU> <vSwitch>

esxcfg-vswitch -m 9000 vSwitch2

esxcfg-vmknic -m <MTU> <portgroup_name>  

esxcfg-vmknic -m 9000 VMkernel2vs

vmkping -s 8784 -d <destination_ip>

vmkping -s 8784 -d 192.168.1.10

Appendix C

A sample PowerShell CLI script to provision multiple virtual machines using template.

# Connect to vCenter server
Connect-VIServer -Server <vCenter_IP_address> -User <”user_name”> -Password <password>    
# Define the naming convention for the virtual machines.
$vmNameTemplate = "d1{0:D3}"
# write-host $vmNameTemplate
# Save the cluster in which the virtual machines should be created into a variable.
$cluster = Get-Cluster fb
# Save the template on which the virtual machines should be based into a variable.
$template = Get-Template w2k16_1
# Create the virtual machines.
$vmList = @()    
for ($i = 1; $i -le 4; $i++) {
    $vmName = $vmNameTemplate -f $i
    $vmList += New-VM -Name $vmName -ResourcePool $cluster -Template $template -VMhost "hc-b200m4-27.purelocal.purestorage.com" -Datastore "DS01"
}
# Save the static IP addresses from the stored CSV file into a variable.
$staticIpList = Import-CSV C:\vms\1wlist.csv
# Create the customization specification.
$winSpec = New-OSCustomizationSpec -Name wdeleteme1 -Workgroup pure -NamingScheme VM -OSType Windows -FullName Administrator -Orgname pure -ChangeSid:$true -AdminPassword "Osmium76!" -TimeZone 004
# Clone the customization specification to a nonpersistent type.
$specClone = New-OSCustomizationSpec -Spec $winSpec -Type NonPersistent
# Apply the customization specification to each virtual machine.
for ($i = 0; $i -lt $vmList.Count; $i++) {
    $ip = $staticIpList[$i].IP
    $nicMapping = Get-OSCustomizationNicMapping -OSCustomizationSpec $specClone
    $nicMapping | Set-OSCustomizationNicMapping -IpMode UseStaticIP -IpAddress $ip -SubnetMask "255.255.0.0" -DefaultGateway "172.16.0.1" -Dns "172.16.0.41"
#$nicMapping | Set-OSCustomizationNicMapping -IpMode UseDhcp
    Set-VM -VM $vmList[$i] -OSCustomizationSpec $specClone -Confirm:$false
    # New-HardDisk -VM $vmList[$i] -Datastore Vol01 -CapacityGB 20
}

A sample IP address list file for virtual machine assignment: 

name,IP
000,172.16.1.1
001,172.16.1.2
002,172.16.1.3
003,172.16.1.4