Web Guide: Implementing vSphere Virtual Volumes with FlashArray

Purity Version

While vVols support was first introduced with Purity 5.0.0, there have been significant fixes and enhancements to the VASA provider in later releases of Purity. Because of this, Pure has set the required Purity version for vVols to a later release.

• For general vVols use, while Purity 5.3 and 5.1 can support vVols both Purity release are end of life.  As such, the minimum target Purity version should be Purity//FA 6.1.8.

vSphere Version

While vSphere Virtual Volumes 2.0 was released with vSphere 6.0, the Pure Storage FlashArray only supports vSphere Virtual Volumes 3.0, which was release with vSphere 6.5.  As such, the minimum required vSphere version is 6.5 GA release.  That being said, there are significant fixes specific to vVols so the required versions and recommended versions are as follows:

• Requirement: vSphere Version is 6.5 U3 or higher (vSphere 6.5 and 6.7 are both be end of life since October of 2022 and will be at end of technical guidance in October of 2023)
• Recommended: vSphere Version 7.0 Update 3f or higher

vSphere 7.0 Update 3 released several improvements for vVols and running vVols at scale.  vSphere 7.0 Update 3f should be the minimum vSphere version to target for running vVols with Pure Storage in your vSphere environment.

vSphere Environment

With regards to the vSphere environment, there are some networking requirements and some strong recommendations from Pure Storage when implementing vVols in your vSphere Environment.

• Requirement: NTP must be configured the same across all ESXi hosts and vCenter Servers in the environment.  The time and data must be configured to the current date/time.
• Recommended: Configure Syslog forwarding for vSphere environment.
• Requirement: Network port 8084 must be open and accessible from vCenter Servers and ESXi hosts to the FlashArray that will be used for vVols.
• Recommended: Use Virtual Machine Hardware version 14 or higher.
• Requirement: Do not run vCenter servers on vVols.
  • While a vCenter server can run on vVols, in the event of any failure on the VASA Management Path combined with a vCenter server restart, the environment could enter a state where vCenter Server may not be able to boot or start.  Please see the failure scenerio KB for more detail on this.
• Recommended: Either configured a SPBM policy to snapshot all of the vVol VM's Config vVols or manually put Config vVols in a FlashArray protection group with snapshot scheduled enabled.
  • A snapshot of the Config vVol is required for the vSphere Plugin's VM undelete feature.  Having a backup of the Config vVol also helps the recovery process or roll back process for the VM in the event that there is an issue.  There is a detailed KB that outlines some of these workflows that can be found here.

FlashArray Environment

Here is some more detail and color for the requirements and recommendations with the FlashArray:

• Requirement: The FlashArray Protocol Endpoint object 'pure-protocol-endpoint' must exist. The FlashArray admin must not rename, delete or otherwise edit the default FlashArray Protocol Endpoint.
  • Currently, Pure Storage stores important information for the VASA Service with the pure-protocol-endpoint namespace.  Destroying or renaming this object will cause VASA to be unable to forward requests to the database service in the FlashArray.  This effectively makes the VASA Provider unable to process requests and the Management Path to fail.  Pure Storage is working to correct this and improve this implementation in a future Purity release.
• Recommendation:  Create a local array admin user when running Purity 5.1 and higher.  This user should then be used when registering the storage providers in vCenter.
• Recommendation: Following vSphere Best Practices with the FlashArray, ESXi clusters should map to FlashArray host groups and ESXi hosts should map to FlashArray hosts.  
• Recommendation: The protocol endpoint should be connected to host groups on the FlashArray and not to individual hosts.
• Recommendation: While multiple protocol endpoints can be created manually, the default device queue depth for protocol endpoints is 128 in ESXi and can be configured up to 4096.  This generally means adding additional protocol endpoints is often unnecessary.

VASA Provider/Storage Provider

The FlashArray has a storage provider running on each FlashArray controller called the VASA Service. The VASA Service is part of the core Purity Service, meaning that it automatically starts when Purity is running on that controller.  In vSphere, the VASA Providers will be registered as Storage Providers.  While Storage Providers/VASA Providers can manage multiple Storage Arrays, the Pure VASA Provider will only manage the FlashArray that it is running on.  Even though the VASA Service is running and active on both controllers, vCenter will only use one VASA Provider as the active Storage Provider and the other VASA Provider will be the Standby Provider.

Here are some requirements and recommendations when working with the FlashArray VASA Provider.

• Requirement: Register both VASA Providers, CT0 and CT1, respectively.
  • While it's possible to only register a single VASA provider, this leaves a single point of failure in your management path.
• Recommendation: Do not use a Active Directory user to register the storage providers. 
  • Should the AD service/server be running on vVols, Pure Storage strongly recommends not to use an AD user to register the storage providers.  This leaves a single point of failure on the management path in the event that the AD User have permissions changed, password changed or the account is deleted.
• Recommendation: User a local array admin created to register the storage providers.
• Recommendation: Should the FlashArray be running Purity 5.3.6 or higher, Import CA signed certificates to VASA-CT0 and VASA-CT1
  • Follow the KB that covers setting up VASA for multiple vCenters and managing the VASA certificate with purecert cli.
  • The Best Practice would be to have a different certificate expiration date for each VASA certificate. 
    For example, VASA-CT0 expires on Nov 14 and VASA-CT1 expires on Dec 14.

Managed Snapshots for vVols based VMs

One of the core benefits of using vVols is the integration with storage and vSphere Manage Snapshots.  The operations of the managed snapshot are offloaded to the FlashArray and there is no performance penalty for keeping the managed snapshots.  When the operations behind managed snapshot are offloaded to VASA and the FlashArray, this creates additional work being done on the FlashArray that is not there with managed snapshots on VMFS. 

Here are some points to keep in mind when using Managed Snapshots with vVols based VMs.

• Managed Snapshots for vVols based VMs create volumes for each Data vVol on that VM that have a -snap suffix in their naming.
  • The process of taking a managed snapshot for a vVol based VM will first issue a Prepare Snapshot Virtual Volume operation which will cause VASA to create placeholder data-snap volumes.  Once completed vSphere will then send the Snapshot Virtual Volume request after stunning the VM.  VASA will then take consistent point in time snapshots of each data vVol and copy them out to the placeholder volumes previously created.  Once the requests complete for each virtual disk the VM is unstunned and the snapshot is completed.
  • With FA volumes being created for the managed snapshot, this directly impacts the volume count on the FlashArray.  For example, a vVol VM with 5 VMDK (Data vVols) will create 5 new volumes on the FA for each managed snapshot.  If 3 managed snapshots are taken, then this VM has a volume count on the FA of 22 volumes (1 Config and 20 Data vVols while powered off; 1 additional Swap vVol while powered on). 
• Managed Snapshots only trigger Point in Time snapshots of the Data vVols and not the Config vVol.  In the event that the VM is deleted and a recovery of the VM is desired, it will manually have to be done from a pgroup snapshot.
• The process of VMware taking a managed snapshot is fairly serialized; specifically, the snapshotVirtualVolume operations are serialized.  This means that if a VM has 3 VMDKs (Data vVols), the snapshotVIrtualVolume request will be issued for one VMDK and after it's complete the next VMDK will have the operation issued against it. The more VMDKs a VM has, the larger the impact to how long the managed snapshot will take to complete. This could increase the stun time for that VM.  
  • VMware has committed to improveing the performance of these calls from vSphere.  In vSphere 7.0 U3 they have updated snapshotVirtualVolume to use the max batch size advertised by VASA to issue snapshotVirtualVolume calls with multiple data vVols.  Multiple snapshotVirtualVolume calls for the same VM will be issued close to the same time now as well in the event that the number of virutal disks is greater than the max batch size.
• Recommendation:  Plan accordingly when setting up managed snapshots (scheduled or manual) and configuring backup software which leverages managed snapshots for incremental backups.  The size of the Data vVols and the amount of Data vVols per VM can impact how long the snapshot virtual volume op takes and how long the stun time can be for the VM.

Storage Policy Based Management (SPBM)

There are a few aspects of utilizing Storage Policies with vVols and the FlashArray to keep in mind when managing your vSphere Environment.

• Storage Policies can be compatible with one or multiple replication groups (FlashArray protection groups).
  • While storage policies can be compatible with multiple replication groups, when applying the policy to a VM, mutliple replication groups should not be used.  The VM should be part of a single consistency group.
• SPBM Failover workflow APIs are ran against the replication group and not the storage policy itself.
• Recommendation: Attempt to keep replication groups under 100 VMs.  This will assist with the VASA Ops being issued against the policies and replication groups and the time it takes to return these queries.
  • This includes both Snapshot and Replication enabled protection groups.  These VASA Ops, such as queryReplicationGroup, will look up all objects in both local replication and snapshot pgroups, as well as target protection groups.  The more protection groups and the more objects in protection groups will inherently cause these queries to take longer.  Please see vVols Deep Dive: Lifecycle of a VASA Operation for more information.
• Recommendation: Do not change the default storage policy with the vVols Datastore.  This could cause issues in the vSphere UI when provisioning to the vVols Datastore.

FlashArray SafeMode with vVols

For FlashArrays with SafeMode enabled additional considerations and planning will be required for the best experience.  As the management of storage is done through VASA, the VASA service frequently will create new volumes, destroy volumes, eradicate volumes, place volumes in FlashArray protection groups, remove volumes from FlashArray protection groups and disable snapshot/replication schedules.

For more detailed information on SafeMode with vVols see the User Guide.  Here is a quick summary of recommendations when running vVols with SafeMode enabled on the FlashArray.

• Any FlashArray should be running Purity 6.1.8 or higher when using vVols before enabling SafeMode.
• vSphere Environment running 7.0 U1 or higher is ideal to leverage the allocated bitmap hint as part of VASA 3.5.
• Object count, object count, object count.  Seriously, the biggest impact that enabling SafeMode will have is on object count.  Customers that want to enable SafeMode must plan to always be monitoring the object counts for volumes, volume groups, volumes snapshots and pgroup snapshots.  Do not just monitor current object counts but all pending eradication object counts as well.
• The use of Auto-RG for SPBM when assigning replication groups to a VM should not be used.
• Once a VM has a storage policy replication group assigned, VASA will be unable to assign a different replication group.  Plan that once a storage policy and replication group are assigned, that the vSphere admin will be unable to change that with SafeMode enabled.
• Failover replication group workflows will not be able to disable replication group schedules.  Nor will cleanup workflows be able to eradicate objects.  Users must plan for higher object counts after any tests or failover workflows.  
• Environments that are frequently powering on/off VMs or vMotioning between hosts will have higher amounts of swap vVols pending eradication.  Should the eradication timer be changed to be longer than 24hr, then they will be pending eradication for longer time.  Storage and vSphere admins will have to plan around higher object counts with these environments.
  • In some cases, vSphere Admins may want to configure a VMFS Datastore that is shared between all hosts to be the target for VMs Swap.
• When changed block tracking (CBT) is enabled the first time, this will increase the amount of volume snapshots pending eradication.  Backup workflows that periodically refresh CBT (disable and re-enable CBT) will increase the amount of this volume diffs that are issued.  Pure does not recommend to frequently refresh CBT.  Once enabled, CBT should not normally need to be refreshed.

Using the FlashArray UI to Manage the Protocol Endpoint

When its first VASA provider is registered, a FlashArray automatically creates a PE called pure-protocol-endpoint.  The pure-protocol-endpoint can be filtered in the Volumes view.  A PE can be connected from the PE volume view or from a Host/Host Group view in the FlashArray UI.

From the Storage -> Volumes view Click on the options and select Show Protocol Endpoints

This view will display the Protocol Endpoints for the FlashArray

From the PE View the PE can be connected to a Host or Host Group Best Practice is to connect the PE to a Host Group and not Hosts individually.

From the Connect Host Groups page you can select one or multiple Host Groups to connect the PE to

Using the FlashArray CLI to Manage the Protocol Endpoint

From the FlashArray CLI a storage admin can manage the Protocol Endpoint.  This includes listing/viewing, creating, connecting, disconnecting or destroying a protocol endpoint.

pureuser@sn1-x50r2-b12-36> purevol list --protocol-endpoint
Name                    Source  Created                  Serial
pure-protocol-endpoint  -       2020-12-02 12:28:08 PST  F4252922ADE248CF000113E6

pureuser@sn1-x50r2-b12-36> purevol create --protocol-endpoint prod-protocol-endpoint
Name                    Source  Created                  Serial
prod-protocol-endpoint  -       2020-12-02 12:29:21 PST  F4252922ADE248CF000113E7

pureuser@sn1-x50r2-b12-36> purevol list --protocol-endpoint
Name                    Source  Created                  Serial
prod-protocol-endpoint  -       2020-12-02 12:29:21 PST  F4252922ADE248CF000113E7
pure-protocol-endpoint  -       2020-12-02 12:28:08 PST  F4252922ADE248CF000113E6

pureuser@sn1-x50r2-b12-36> purevol connect --hgroup Prod-Cluster-FC --lun 10 prod-protocol-endpoint
Name                    Host Group       Host       LUN
prod-protocol-endpoint  Prod-Cluster-FC  ESXi-3-FC  10
prod-protocol-endpoint  Prod-Cluster-FC  ESXi-2-FC  10
prod-protocol-endpoint  Prod-Cluster-FC  ESXi-1-FC  10

pureuser@sn1-x50r2-b12-36> purevol list --connect
Name                                Size  LUN  Host Group       Host
prod-protocol-endpoint              -     10   Prod-Cluster-FC  ESXi-1-FC
prod-protocol-endpoint              -     10   Prod-Cluster-FC  ESXi-2-FC
prod-protocol-endpoint              -     10   Prod-Cluster-FC  ESXi-3-FC

pureuser@sn1-x50r2-b12-36> purevol connect --hgroup Prod-Cluster-FC --lun 11 pure-protocol-endpoint
Name                    Host Group       Host       LUN
pure-protocol-endpoint  Prod-Cluster-FC  ESXi-3-FC  11
pure-protocol-endpoint  Prod-Cluster-FC  ESXi-2-FC  11
pure-protocol-endpoint  Prod-Cluster-FC  ESXi-1-FC  11
pureuser@sn1-x50r2-b12-36> purevol disconnect --hgroup Prod-Cluster-FC pure-protocol-endpoint
Name                    Host Group       Host
pure-protocol-endpoint  Prod-Cluster-FC  -

pureuser@sn1-x50r2-b12-36> purevol create --protocol-endpoint dr-protocol-endpoint
Name                  Source  Created                  Serial
dr-protocol-endpoint  -       2020-12-02 14:15:23 PST  F4252922ADE248CF000113EA

pureuser@sn1-x50r2-b12-36> purevol destroy dr-protocol-endpoint
Name
dr-protocol-endpoint

A FlashArray’s performance is independent of the number of volumes it hosts; array’s full performance capability can be delivered through a single PE. PEs are not performance bottlenecks for vVols, so a single PE per array is all that is needed.

Configuring a single PE per array does not restrict multi-tenancy. Sub-lun connections are host-specific.

More than one PE can be configured, but is seldom necessary.

As is typical for the FlashArray architecture, vVol support, and in particular, the PE implementation are as simple as it is possible for them to be.

Using the Plugin to Mount vVol Datastore

Mounting vVol Datastores Manually: FlashArray Actions 

Alternatively, vVol datastores can be provisioned by connecting the PE to the hosts or host group, rescanning each host’s I/O interconnects, and mounting the vVol datastore to each host. These operations require both FlashArray and VMware involvement, however. Array administrators can use the CLI, REST, or REST interfaces, or tools such as PowerShell. VMware administrators can use the Web Client, the VMware CLI, or the VMware SDK and SDK-based tools like PowerCLI.

Pure Storage recommends using the Plugin to provision PEs to hosts.  Keep in mind that the FlashArray UI does not allow creation of Protocol Endpoints.  The FlashArray UI does allow finding the Protocol Endpoint and connecting them to Hosts and Host Groups.

A Protocol Endpoint on the FlashArray can be viewed and connected from either the FlashArray UI or CLI.

Using the FlashArray UI to Manage the Protocol Endpoint

When its first VASA provider is registered, a FlashArray automatically creates a PE called pure-protocol-endpoint.  The pure-protocol-endpoint can be filtered in the Volumes view.  A PE can be connected from the PE volume view or from a Host/Host Group view in the FlashArray UI.

From the Storage -> Volumes view Click on the options and select Show Protocol Endpoints

This view will display the Protocol Endpoints for the FlashArray

From the PE View the PE can be connected to a Host or Host Group Best Practice is to connect the PE to a Host Group and not Hosts individually.

From the Connect Host Groups page you can select one or multiple Host Groups to connect the PE to

Using the FlashArray CLI to Manage the Protocol Endpoint

From the FlashArray CLI a storage admin can manage the Protocol Endpoint.  This includes listing/viewing, creating, connecting, disconnecting or destroying a protocol endpoint.

Protocol endpoints that have been created can be listed with purevol list --protocol-endpoint pureuser@sn1-x50r2-b12-36> purevol list --protocol-endpoint Name Source Created Serial pure-protocol-endpoint - 2020-12-02 12:28:08 PST F4252922ADE248CF000113E6

A protocol endpoint can be created with purevol create --protocol-endpoint pureuser@sn1-x50r2-b12-36> purevol create --protocol-endpoint prod-protocol-endpoint Name Source Created Serial prod-protocol-endpoint - 2020-12-02 12:29:21 PST F4252922ADE248CF000113E7 pureuser@sn1-x50r2-b12-36> purevol list --protocol-endpoint Name Source Created Serial prod-protocol-endpoint - 2020-12-02 12:29:21 PST F4252922ADE248CF000113E7 pure-protocol-endpoint - 2020-12-02 12:28:08 PST F4252922ADE248CF000113E6

To connect a protocol endpoint use either purehgroup connect or purevol connect pureuser@sn1-x50r2-b12-36> purevol connect --hgroup Prod-Cluster-FC --lun 10 prod-protocol-endpoint Name Host Group Host LUN prod-protocol-endpoint Prod-Cluster-FC ESXi-3-FC 10 prod-protocol-endpoint Prod-Cluster-FC ESXi-2-FC 10 prod-protocol-endpoint Prod-Cluster-FC ESXi-1-FC 10 pureuser@sn1-x50r2-b12-36> purevol list --connect Name Size LUN Host Group Host prod-protocol-endpoint - 10 Prod-Cluster-FC ESXi-1-FC prod-protocol-endpoint - 10 Prod-Cluster-FC ESXi-2-FC prod-protocol-endpoint - 10 Prod-Cluster-FC ESXi-3-FC

A protocol endpoint can be disconnected from a host and host group with purevol disonnect. However, if there are any active sub-lun connections this operation will fail as disconnecting the PE would cause a sev-1 and data path failure to that ESXi host. pureuser@sn1-x50r2-b12-36> purevol connect --hgroup Prod-Cluster-FC --lun 11 pure-protocol-endpoint Name Host Group Host LUN pure-protocol-endpoint Prod-Cluster-FC ESXi-3-FC 11 pure-protocol-endpoint Prod-Cluster-FC ESXi-2-FC 11 pure-protocol-endpoint Prod-Cluster-FC ESXi-1-FC 11 pureuser@sn1-x50r2-b12-36> purevol disconnect --hgroup Prod-Cluster-FC pure-protocol-endpoint Name Host Group Host pure-protocol-endpoint Prod-Cluster-FC -

A disconnected Protocol Endpoint can be destroyed with purevol destroy. DO NOT DESTROY THE DEFAULT PURE-PROTOCOL-ENDPOINT! pureuser@sn1-x50r2-b12-36> purevol create --protocol-endpoint dr-protocol-endpoint Name Source Created Serial dr-protocol-endpoint - 2020-12-02 14:15:23 PST F4252922ADE248CF000113EA pureuser@sn1-x50r2-b12-36> purevol destroy dr-protocol-endpoint Name dr-protocol-endpoint

A FlashArray’s performance is independent of the number of volumes it hosts; array’s full performance capability can be delivered through a single PE. PEs are not performance bottlenecks for vVols, so a single PE per array is all that is needed.

Configuring a single PE per array does not restrict multi-tenancy. Sub-lun connections are host-specific.

A FlashArray automatically creates a default pure-protocol-endpoint PE when its first VASA provider is registered. If necessary, additional PEs can also be created manually.  However, in most cases the default pure-protocol-endpoint is fine to use.  There is no additional HA value added by connecting a host to multiple protocol endpoints.

Do not rename, destroy or eradicate the pure-protocol-endpoint PE on the FlashArray.  This namespace is required for VASA to be able to storage required metadata for VASA to correctly work with the FlashArray. 

BEST PRACTICE: Use one PE per vVol container. All hosts should share the same PE and vVol to host bindings are host-specific, so multi-tenancy is inherently supported.

More than one PE can be configured, but is seldom necessary.

As is typical for the FlashArray architecture, vVol support, and in particular, the PE implementation are as simple as it is possible for them to be.

Mounting vVol Datastores Manually: Web Client Actions

Navigate to the vCenter UI once the PE is connected to the FlashArray Host Group that corresponds to the vSphere ESXi Cluster.

Although the PE volumes are connected to the ESXi hosts from a FlashArray standpoint, ESXi does not recognize them until an I/O rescan occurs. (This is partially correct.  If you are on a recent version of Purity and ESXi, a Unit Attention will be issued to the ESXi hosts when the PE is connected to the hosts.  At this time, the ESXi host will dynamically update the devices that are presented via the SAN).  In the event that the FlashArray is not on a recent release of Purity  (Purity 5.1.15+, 5.3.6+ or 6.0.0+), a storage rescan from the ESXi hosts will be required for the PE to show up in the ESXi hosts connected devices.

To display a provisioned PE, select the host in the inventory pane, select the Configure tab, and click Storage Devices. The PE appears as a 1 megabyte device.

The screen is useful to find the PEs that have been successfully connected via a SAN transport method.  Multipathing can be configured on the PE from this view as well.

Note that in this example there are three PEs from three different arrays.  When navigating to the Storage -> Protocol Endpoints Screen the PEs that are used as a vVol Datastore mount are displayed.  In this example we only have two that show, as there are currently only two vVol Datastores (from two different arrays) created.

The expected behavior is that the ESXi host will only display connected PEs that are currently being used as mounts for a vVol Datastore.

To mount a vVol datastore, right-click the target host or cluster, select Storage from the dropdown menu, and select New Datastore from the secondary dropdown to launch the New Datastore wizard.

Click the vVol Type

Enter in a friendly name for the datastore and select the vVol container in the Backing Storage Container list.

This is how the storage container list looks on Purity//FA 6.4.1 and higher.  The default container will show up as the default_storage_container (red box) and all others will show up with the pod name as the storage container name (orange box).

This is how the storage container list looks in Purity//FA 6.4.0 or earlier.  The default container for an array will only be shown as Vvol container.

Clicking a container displays the array that hosts it in the lower Backing Storage Container panel.

No Backing Storage listing typically indicates either that the array’s VASA providers have not been registered or that vCenter cannot communicate with them.

Select the host(s) on which to mount the vVol datastore.  Best Practice would be to connect the vVol Datastore to all hosts in that ESXi Cluster.

Review the configuration details and then click Finish.

Once a vVol datastore is mounted, the Configure tab for any ESXi host to which it is mounted lists the PEs available from the array that the host is connected via SAN transport.  Note now that the PE LUN 253 is now listed as a PE for the ESXi host.

Mounting a vVol Datastore to Additional Hosts

In the event that an ESXi host has been added to a Cluster or the vVol Datastore was only mounted to some hosts in the cluster there is a workflow to mount additional hosts to the vVol Datastore.

To mount the vVol datastore to additional hosts, right-click on the vVol Datastore and select Mount Datastore to Additional Hosts from the dropdown menu to launch the Mount Datastore to Additional Hosts wizard.

Select the hosts to which to mount the vVol datastore by checking their boxes and click Finish.

FlashArray Organization of vVols

FlashArrays organize the vVols associated with each vVol-based VM as a volume group. Each time VMware administrator creates a vVol-based VM, the hosting FlashArray creates a volume group with the following naming schema: vvol-{VM Name}-{unique 8 character string}-vg

Volume Groups Area of GUI Volumes Tab

To list the volumes associated with a vVol-based VM, select the Storage view Volumes tab. In the Volume Groups area, select the volume group name containing the VM name from the list or enter the VM name in the search box.

The Volumes area of the pane lists the volumes associated with the VM.

GUI View of Volume Group Membership

Clicking a volume name displays additional detail about the selected volume.

GUI View of a vVol's Details

Note:
Clicking the volume group name in the navigation breadcrumbs returns to the volume groups display.

As with all FlashArray data objects, destroying a volume group moves it to the array’s Destroyed Volume Groups folder for 24 hours before eradicating it permanently.

To recover or eradicate a destroyed volume group, click the respective icons in the Destroyed Volume Groups pane.  A destroyed volume group can only be eradicated if all objects in that volume group have already been eradicated.

FlashArray GUI Destroyed Volume Groups View

The FlashArray CLI and REST interfaces can also be used to manage volume groups of vVols.

VM Datastore Structures

vVols do not change the fundamental VM architecture:

• Every VM has a configuration file (a VMX file) that describes its virtual hardware and special settings
• Every powered-on VM has a swap file.
• Each virtual disk added to a VM is implemented as a storage object that limits guest OS disk capacity.
• Every VM has a memory (vmem) file used to store snapshots of its memory state.

Conventional VM Datastores

Every VM has a home directory that contains information, such as:

Virtual hardware descriptions 

Guest operating system version and settings, BIOS configuration, virtual SCSI controllers, virtual NICs, pointers to virtual disks, etc.

Logs

Information used during VM troubleshooting

VMDK files 

Files that correspond to the VM’s virtual disks, whether implemented as NFS, VMFS, physical and virtual mode RDMs (Raw Device Mappings), or vVols. VMDK files indicate  where the ESXi vSCSI layer should send each virtual disk’s I/O.

For complete list VM home directory contents refer to VMware's documentation that covers Virtual Machine Files.

When a VMware administrator creates a VM based on VMFS or NFS, VMware creates a directory in its home datastore.

vCenter UI View - VM Settings - VM Options

 

Web Client File Browser View of a VM's Home Directory

With vVol-based VMs, there is no file system, but VMware makes the structure appear to be the same as that of a conventional VM. What occurs internally is quite different, however.

Config vVols 

When a VMware administrator creates a vVol-based VM, vCenter creates a 4-gigabyte thin-provisioned configuration vVol (config vVol) on the array, which ESXi formats with VMFS. A VM’s config vVol stores the files required to build and manage it: its VMX file, logs, VMDK pointers, etc. To create a vVol-based VM, right-click any inventory pane object to launch the New Virtual Machine wizard and specify that the VM’s home directory be created on a vVol datastore.

vCenter UI View - New VM

vCenter UI View - VM Hardware Settings

When VM creation is complete, a directory with the name of the VM appears in the array’s vVol datastore. The directory contains the VM’s vmx file, log file and an initially empty vmsd file used to store snapshot information.

vCenter UI View - vVol DS File Browser

In the Web Client, a vVol datastore appears as a collection of folders, each representing a mount point for the mini-file system on a config vVol. The Web Client GUI Browse Datastore function and ESXi console cd operations work as they do with conventional VMs. Rather than traversing one file system, however, they transparently traverse the file systems hosted on all of the array’s config vVols.

A FlashArray creates a config vVol for each vVol-based VM. Arrays name config vVols by concatenating the volume group name with config-<UUID>. Arrays generate UUIDs randomly; an array administrator can change them if desired.

An array administrator can search for volumes containing a vVol-based VM name to verify that its volume group and config vVol have been created.

FlashArray UI View - Volumes List

As objects are added to a vVol-based VM, VMware creates pointer files in its config vVol; these are visible in its directory. When a VM is deleted, moved to another array, or moved to a non-vVol datastore, VMware deletes its config vVol.

Data vVols

Each data vVol on an array corresponds to a virtual disk. When a VMware administrator creates a virtual disk in a vVol datastore, VMware directs the array to create a volume and creates a VMDK file pointing to it in the VM’s config vVol. Similarly, to resize or delete a virtual disk, VMware directs the array to resize or destroy the corresponding volume.

Creating a Data vVol

vVol-based virtual disk creation is identical to conventional virtual disk creation. To create a vVol-based virtual disk using the Web Client, for example, right-click a VM in the Web Client inventory pane and select Edit Settings from the dropdown menu to launch the Edit Settings wizard.

vCenter UI View - VM Edit Settings

Select New Hard Disk in the New device dropdown and click Add.

vCenter UI View - New Hard Disk Selection

Enter configuration parameters. Select the VM’s home datastore (Datastore Default) or a different one for the new virtual disk, but to ensure that the virtual disk is vVol-based, select a vVol datastore.

vCenter UI View - Specifying Data vVol Parameters

Click OK to create the virtual disk. VMware does the following:

1. For a VM’s first vVol on a given array, directs the array to create a volume group and a config vVol for it.
2. Directs the array to create a volume in the VM’s volume group.
3. Creates a VMDK pointer file in the VM’s config vVol to link the virtual disk to the data vVol on the array.
4. Adds the new pointer file to the VM’s VMX file to enable the VM to use the data vVol.

The FlashArray GUI Storage view Volumes tab lists data vVols in the Volumes pane of the volume group display.

FlashArray UI View - Volume Group Volume Objects View

Resizing a Data vVol

A VMware administrator can use any of several management tools expand a data vVol to a maximum size of 62 terabytes while it is online. Although FlashArrays can shrink volumes as well, vSphere does not support that function.

vCenter UI View - vSphere Disallows Volume Shrinking

Note:
VMware enforces the 62 terabyte maximum to enable vVols to be moved to VMFS or NFS, both of whose maximum virtual disk size is 62 terabytes.

At this time VMware does not support expanding a Volume that is configured with a SCSI controller that is enabled with sharing.

To expand a data vVol using the Web Client, right-click the VM in the inventory pane, select Edit Settings from the dropdown menu, and select the virtual disk to be expanded from the dropdown. The virtual disk’s current capacity is displayed. Enter the desired capacity and click OK, and use guest operating system tools to expose the additional capacity to the VM. 

vCenter UI View - Entering Expanded Data vVol Capacity

FlashArray UI View - Updated Capacity Size of the Data vVol

Deleting a Data vVol

Deleting a data vVol is identical to deleting any other type of virtual disk. When a VMware administrator deletes a vVol-based virtual disk from a VM, ESXi deletes the reference VMDK file and directs the array to destroy the underlying volume.

To delete a vVol-based virtual disk, right-click the target VM in the Web Client inventory pane, select Edit Settings from the dropdown menu to launch the Edit Settings wizard. Select the virtual disk to be deleted, hover over the right side of its row and click the    symbol when it appears.

vCenter UI View - Selecting Data vVol for Deletion

To remove the vVol from the VM, click the OK button. To remove it from the VM and destroy it on the array, check the Delete files from datastore checkbox and click OK.

vCenter UI View - Destroying the Volume on the Array

Note:
Delete files from datastore is not a default—if it is not selected, the vVol is detached from the VM, but remains on the array. A VMware administrator can reattach it with the Add existing virtual disk Web Client command.

The ESXi host deletes the data vVol’s VMDK pointer file and directs the array to destroy the volume (move it to its Destroyed Volumes folder for 24 hours.

FlashArray UI View - Deleted Data vVol in Destroyed Volumes Objects

An array administrator can recover a deleted vVol-based virtual disk at any time during the 24 hours following deletion. After 24 hours, the array permanently eradicates the volume and it can no longer be recovered.

Swap vVols

VMware creates swap files for VMs of all types when they are powered on, and deletes them at power-off. When a vVol-based VM is powered on, VMware directs the array to create a swap vVol, and creates a swap (.vswp) file in the VM’s config vVol that points to it.

vCenter UI View - vVol DS Browser - Powered Off VM Files Illustrates the components of a powered-off vVol-based VM. There is no vswp file.

FlashArray UI View - Volumes for Powered Off vVol based VM The VM’s volume group does not include a swap volume.

To power on a vVol-based VM, right-click it in the Web Client inventory pane, select Power from the dropdown menu, and Power On from the secondary dropdown. 

vCenter UI View - Powering On the VM

 When a VM is powered on, the Web Client file navigator lists two vswp files in its folder.

vCenter UI View - vVol DS Browser - Powered On VM Files

The swap vVol’s name in the VM’s volume group on the array is Swap- concatenated with a unique identifier. The GUI Volumes tab shows a volume whose size is the VM’s memory size. 

FlashArray UI View - Swap Volume for Powered On VM

vCenter UI View - VMs Virtual Memory Size

Like all FlashArray volumes, swap vVols are thin-provisioned—they occupy no space until data is written to them.

To power off a vVol-based VM, right-click it in the Web Client inventory pane, select Power from the dropdown menu, and Shut Down Guest OS from the secondary dropdown.

vCenter UI View - Command to Shutdown the Guest OS

When a VM is powered off, its vswp file disappears from the Web Client file navigator, and the FlashArray GUI Volumes tab no longer shows a swap volume on the array.

FlashArray UI View - Powered Off VM's Volumes (note there is no swap)

VMware destroys and immediately eradicates swap vVols from the array. (They do not remain in the Destroyed Volumes folder for 24 hours.)

FlashArray UI View - Audit log of operations to destroy and eradicate Swap

Memory vVols

VMware creates memory vVols for two reasons:

VM suspension

When a VMware administrator suspends a VM, VMware stores its memory state in a memory vVol. When the VM resumes, its memory state is restored from the memory vVol, which is then deleted.

VM snapshots

When a VMware management tool creates a snapshot of a vVol-based VM with the “store memory state” option, VMware creates a memory vVol. Memory vVols that contain VM snapshots are deleted when the snapshots are deleted. They are described in the section titled Creating a VM Snapshot with Saved Memory.

To suspend a running VM, right-click its entry in the Web Client inventory pane, select Power from the dropdown menu, and Suspend from the secondary dropdown.

vCenter UI View - Command to Suspend the VM

VMware halts the VM’s processes, creates a memory vVol and a vmss file to reference it, de-stages (writes) the VM’s memory contents to the memory vVol, and directs the array to destroy and eradicate its swap vVol.

FlashArray UI View - Memory vVol in the VM Volume Group

vCenter UI View - vVol DS Browser - Memory vVol File

When the VM’s memory has been written, the ESXi host unbinds its vVols. They are bound again when it is powered on.

To resume a suspended VM, right-click it in the Web Client inventory pane, select Power from the dropdown menu, and Power On from the secondary dropdown.

vCenter UI View - Powering on the Suspended VM

Powering on a suspended VM binds its vVols, including its memory vVol, to the ESXi host, and loads its memory state is from the memory vVol. Once loading is complete, VMware unbinds the memory vVol and destroys the volume and the VASA provider will automatically eradicate it on the FlashArray.

FlashArray UI View - Eradicated Memory vVol

Recovering Deleted vVols

Deleted data and config vVols are both recoverable within 24 hours of deletion.

Throughout a VM’s life, it has a config vVol in every vVol datastore it uses. The config vVol hosts the VM’s home folder which contains its VMX file, logs, swap pointer file, and data vVol (VMDK) and snapshot pointer files. Restoring a config vVol from a snapshot and the corresponding data and snapshot vVols effectively restores a deleted VM.

vCenter UI View - vVol DS Browser - Typical VM Home Directory

Creating a Config vVol FlashArray Snapshot

As there needs to be a snapshot of the Config vVol in order to run through this recovery workflow Pure has provided several ways to snapshot the Config vVol.

FlashArray UI View - Taking an array based snapshot of the Config vVol

vCenter UI View - Pure vSphere Plugin - Create Config Snapshot from VM Overiew Page

vCenter UI View - Pure vSphere Plugin - Create VM Home (config vVol) Snapshot from Pure Storage Snapshot Management Page

There are other ways to do this, including the FlashArray CLI, having the confg vVol be part of a FlashArray protection group, using storage policies with snapshot rulesets, etc.  The main thing is that by default there are no array snapshots taken for any of the vVols.  Pure encourages the use of Storage Policies to leverage array based snapshots to help protect the vms from accidental deletion.

Here the Config vVol now shows three volume snapshots that were taken using the above three methods.

FlashArray UI View - Config vVol that has volume snapshots on the FlashArray

Manually Restoring a Deleted vVol VM

To delete a VM, VMware deletes the files in its config vVol and directs the array to destroy the config vVol and any of its data vVols that are not shared with other VMs.

vCenter UI View - Destroying a Powered Off VM

An array administrator can recover destroyed vVols at any time within 24 hours of their destruction. But because the config vVol’s files are deleted before destruction, recovering a VM’s config vVol results in an empty folder. A recovered config vVol must be restored from its most recent snapshot.

Recovering a config vVol requires at least one pre-existing array-based snapshot. Without a config vVol snapshot, a VM can be recovered, but its configuration must be recovered manually.

When a VMware administrator deletes a VM, VMware directs the array to destroy its config vVol, data vVols, and any snapshots. The array moves the objects to its destroyed objects folders for 24 hours.

FlashArray UI View - Destroyed Volumes and Volume Group for the VM that was Deleted

To recover a deleted VM, recover its volume group first, followed by its config and data vVols. To recover a single object on the array, click the   icon next to it.

To recover multiple objects of the same type with a single action, click the vertical ellipsis and select Recover… to launch the Recover Volumes wizard. Select the config vVol and the data vVols to be recovered by checking their boxes and click the Recover button.

FlashArray UI View - Command to Recover Destroyed Volumes

FlashArray UI View - Selecting Volumes to Recover

While the VMs Volumes and Volume Group were restored, recall that during the VM deletion process that the Config vVol is first erased at the VMFS level from vSphere.  When navigating to the VMs Directory it will be empty.

vCenter UI View - Empty Directory of the Recovered Config

In the GUI Snapshots pane, click the vertical ellipsis to the right of the snapshot from which to restore, and select Restore from the dropdown menu.

FlashArray UI View - Restoring Config vVol from Volume Snapshot - 1

When the Restore Volume from Snapshot wizard appears, click the Restore button.

FlashArray UI View - Restoring Config vVol from Volume Snapshot - 2

Restoring the config vVol from a snapshot recreates the pointer files it contains. In the Web Client file navigator, right-click the vmx file and select Register VM… from the dropdown menu to register the VM.

vCenter UI View - Registering the Recovered VM

After registration, all data vVols, snapshots, and the VM configuration are as they were when the snapshot of the config vVol was taken.

Creating a Managed Snapshot Without Saving Memory

If neither Snapshot the virtual machine’s memory nor Quiesce guest file system is selected, VMware directs the array to create snapshots with no pre-work. All FlashArray snapshots are crash consistent, so snapshots of vVol based-VMs that they host are likewise at least crash consistent.

The managed snapshot process for vVols comes in two parts:  Prepare to Snapshot Virtual Volume (prepareToSnapshotVirtualVolume) and then Snapshot Virtual Volume (snapshotVirtualVolume).  When a vSphere user initiates a managed snapshot operation vSphere will communicate with the array's VASA Provider to in the following method:

1. vSphere issues a Prepare to Snapshot Virtual Volume request to the VASA Provider
2. The VASA Provider ensures that the virtual volume is ready to have a managed snapshot taken
   1. On the FlashArray this is the step that the volumes are created with the data-vvol name with a -snap suffix
   2. If the VM has a policy and replication group associated with it, the data-snap volumes are placed in the protection group associated with that policy and replication group
3. The VASA Provider responds back to vSphere that the prepare operation has completed and provides a uuid for the snapshot
4. vSphere pauses the VM to ensure that no outstanding activity happens (Usually referred to as VM Stun time)
5. vSphere issues a Snapshot Virtual Volume request to the VASA Provider for each Data vVol that the VM has
6. The VASA provider copies out the data vVols to the managed snapshot
   1. On the FlashArray a purevol copy is issued where the data-vvol is copied out and overwrites the data-vvol-snap that was created during the prepare phase
   2. In vSphere 7.0 U3 and higher the Snapshot Virtual Volume request passes multiple vVol uuids as part of the request, which helps improve batching and performance at scale
7. The VASA provider responds back to vSphere that Snapshot Virtual Volume is complete
8. vSphere unpauses the VM (the VM is unstunned) and the managed snapshot operation is complete 

While this seems like a lot of steps and process at first, they typically will complete under a ms.  The biggest goal that both VMware and Pure Storage has is to decrease the amount of calls and time it takes between when the VM is stunned, snapshot virtual volume is issued and then completed.

Here is a view from the Snapshot Management for a normal snapshot that had completed.

vSphere Client View - Snapshot Management View - Normal Snapshot

Here is a view from the FlashArray that shows the new volume objects created for a managed snapshot.

FlashArray GUI View - Managed Snapshot for vVols based VM

Note:
FlashArray volume names are auto-generated, but VMware tools list the snapshot name supplied by the VMware administrator.

Creating a Managed Snapshot with Saved Memory

If the VMware administrator selects Store the Virtual Machine’s Memory State, the underlying snapshot process is more complex.

Memory snapshots generally take somewhat longer than non-memory ones because the ESXi host directs the array to create a memory vVol to which it writes the VM’s entire memory image. Creation time is proportional to the VM’s memory size.

vSphere Client View - Managed Snapshot Wizard

Normal Snapshots, Memory Snapshots and File Quiesced Snapshots will all cause a VM to pause briefly.  A normal snapshot causing the smallest amount of "VM Stun" during the snapshot process.  File Quiesced and Memory based snapshots can have a VM "Stun Time" that varies depending on how busy the VM is, how large the VM is sized (Memory or Storage) and the version of vSphere.  Typically the stun time for a VM snapshot will be seconds, but a memory snapshot could vastly increase the amount of stun time if the VM is sized large and is very busy.  

The memory vVol in a VM’s volume group created as a consequence of a memory snapshot stores the VM’s active state (memory image). ArrayView 25 shows the volume group of a VM with a memory snapshot (vvol-test-a-VM-light-0011-92ecaac2-vg/Memory-f60d917b). The size of the memory vVol is the memory size of the VM’s memory image.

FlashArray GUI View - Memory Managed Snapshot Object

VMware flags a memory snapshot with a green play icon to indicate that it includes the VM’s memory state.

vSphere Client View - Snapshot Management View - Memory Managed Snapshot

Reverting a VM to a Managed Snapshot

VMware management tools can revert VMs to snapshots taken by VMware. As with snapshot creation, reverting is identical for conventional and vVol-based VM snapshots.

To restore a VM from a snapshot, from the Web Client Hosts & Clusters or VMs and Templates view, select the VM to be restored and click the Snapshots tab in the adjacent pane to display a list of the VM’s snapshots.

Select the snapshot from which to revert, click the All Actions button, and select Revert to from the dropdown menu.

vSphere Client View - Reverting a VM to a Managed Snapshot

Subsequent steps differ slightly for non-memory and memory snapshots.

Reverting a VM from a Non-memory Managed Snapshot

The Revert to command displays a confirmation dialog. Click Yes to revert the VM to the selected snapshot.

The array overwrites the VM’s data vVols from their snapshots. Any data vVols added to the VM after the snapshot was taken are unchanged.

Before reverting a VM from a non-memory snapshot, VMware shuts the VM down. Thus, reverted VMs are initially powered off.

Reverting a VM from Memory Managed Snapshot

To revert a VM to a memory snapshot, the ESXi host first directs the array to restore the VM’s data vVols from their snapshots, and then binds the VM’s memory vVol and reloads its memory. Reverting a VM to a memory snapshot takes slightly longer and results in a burst of read activity on the array.

A VM reverted to a memory snapshot can be reverted either suspended or to a running state. Check the Suspend this virtual machine when reverting to selected snapshot box in the Confirm Revert to Snapshot wizard to force the reverted VM to be powered off initially. If the box is not checked, the VM is reverted into its state at the time of the snapshot.

FlashArray GUI View - Memory Snapshot Read IO

Deleting a Managed Snapshot

Snapshots created with VMware management tools can be deleted with those same tools. VMware administrators can only delete snapshots taken with VMware tools.

To delete a VM snapshot from the Web Client Host and Clusters or VMs and Templates view, select the target VM and click the Snapshots tab in the adjacent pane to display a list of its snapshots.

Select the snapshot to be deleted, click the All Actions button, and select Delete Snapshot from the dropdown menu to launch the Confirm Delete wizard. Click Yes to confirm the deletion.

VMware removes the VM’s snapshot files from the vVol datastore and directs the array to destroy the snapshot. Depending on whether FlashArray Safemode is enabled or not will have differing outcomes on the array.

When Safemode is disabled the VASA Provider will destroy and automatically eradicate the deleted managed snapshot.  This helps reduce object count churn in environments that leverage managed snapshots as part of backup workflows.

One of the options when Safemode is enabled is disabling the manual eradication on the array.  This means that the VASA Provider is no longer able to automatically eradicate the deleted managed snapshot.  Keep this in mind when planning object count headroom when using vVols with FlashArray Safemode.

FlashArray GUI View - Deleted Managed Snapshot objects are eradicated

FlashArray GUI View - Safemode is enabled - Deleted Managed Snapshot objects are not eradicated

When VMware deletes a conventional VM snapshot, it reconsolidates (overwrites the VM’s original VMDKs with the data from the delta VMDKs). Depending on the amount of data changed after the snapshot, this can take a long time and have significant performance impact. With FlashArray based snapshots of vVols, however, there is no reconsolidation. Destroying a Flasharray snapshot is essentially instantaneous. Any storage reclamation occurs after the fact during the normal course of the array’s periodic background garbage collection (GC).

Viewing VM vVol details

Creating a FlashArray Snapshot of a vVol Disk 

Restoring a vVol from a FlashArray Snapshot

Creating a vVol Copy 

Storage Capability Compliance

Administrators can specify values for some or all of these capabilities when creating storage policies. VMware performs two types of policy compliance checks:

• If a vVol were to be created on the array, could it be configured with the feature?
• Is a vVol in compliance with its policy? For example, a vVol with a policy of hourly snapshots must be (a) on FlashArray that hosts a protection group with hourly snapshots enabled and (b) a member of that protection group.

The following table lists the circumstances under which a policy offers each capability, and under which a vVol is in or out of compliance with it. 

Combining Capabilities and Storage Compliance

This section describes an example of combining capabilities into a policy. Storage policies are a powerful method of assuring specific configuration control, but they affect how vVol compliance is viewed. For an array or vVol to be compliant with a policy:

1. The array or vVol must comply with all of the policy’s capabilities
2. For snapshot and replication capabilities, the array must have at least one protection group that offers all of the policy’s capabilities. For example, if a policy requires hourly local snapshots and replication every 5 minutes, a protection group with a hourly snapshots and a different protection group with 5 minute replication do not make the array compliant. VMware requires that volumes be in a single group during policy configuration, so to be compliant for this example, an array would require at least one protection group with hourly snapshots and 5 minute replication.
3. Some combinations of capabilities cannot be compliant. For example, setting an array’s Local Snapshot Policy Capable capability to No and specifying a policy that includes snapshots means that no storage compliant with the policy can be hosted on that array.

Creating Custom Storage Policies

Click the home icon at the top of the Web Client home screen, and select Policies and Profiles from the dropdown menu to display the VM Storage Policies pane.

Policies and Profiles Command

Select the VM Storage Policies tab and click the Create VM Storage Policy button  to launch the Create New VM Storage Policy wizard.

Create VM Storage Policy Button

Select a vCenter from the dropdown and enter a name and description for the policy.

Create New VM Storage Policy Wizard

It is a best practice to use a naming convention that is operationally meaningful. For example the name above suggests that the policy will have local snapshot protection with a one hour interval and 15 minute replication.

Select com.purestorage.storage.policy in the <Select provider> dropdown to use the FlashArray VASA provider rules (com.purestorage.storage.policy) to create the storage policy.

Before Purity 6.2.6 and VASA Provider 2.0.0 the FlashArray VASA Provider rulesets showed up as a string like this.  Moving forward the rules will have a friendlier name displaying as just "Pure Storage"

Rule-set Page of the Create New VM Storage Policy Wizard

A storage policy requires at least one rule. To locate all VMs and virtual disks to which this policy will be assigned on FlashArrays, click the <Add rule> dropdown and select the Pure Storage FlashArray capability.

Adding a Storage Policy Rule

At least one Placement rule needs to be provided in order to create a storage policy with the Pure Storage rules.

Error if no placement rule is provided

The selected rule name appears above the <Add rule> dropdown, and a dropdown list of valid values appears to the right of it. Select Yes and click Next to create the policy. As defined thus far, the policy requires that VMs and vVols to which it is assigned be located on FlashArrays, but they are not otherwise constrained. When a policy is created, the Plugin checks registered arrays for compliance and displays a list of vVol datastores on arrays that support it.

List of Arrays Compatible with a New Storage Policy

The name assigned to the policy (FlashArray-1hrSnap15minReplication) suggests that it should specify hourly snapshots and 15-minute replications of any VMs and virtual volumes to which it is assigned. Click Back to edit the rule-set.

FlashArray replication and snapshot (only required for Purity//FA versions below 6.2.6) capabilities require component rules. Click Custom and select Replication from the dropdown to display the Replication component rule pane.

Selecting Replication Capabilities for the Policy

Click the Add Rule dropdown again, select Remote Replication Interval, enter 15 in the text box, select Minutes as the unit and click Next to display the list of registered arrays that are compatible with the augmented policy.

Specifying Replication Interval Rule

In vSphere there is the ability to create pre-defined replication rule requirements.  These can be created in the Storage Policy Components tab under Policies and Profiles.

Creating a new Storage Policy Component

After selecting the Pure Storage Replication Provider the same replication rules are available to be used.  Here we create a component ruleset that matches the custom one we provided in the storage policy above.

Storage Policy Component with Replication rules

If we had wanted to use a replication component in the new storage policy we would have selected it from a drop down instead of choosing the custom option.

Selecting a Storage Policy Component

With Purity//FA 6.2.6 and VASA Provider 2.0.0 local snapshot protection is provided through a placement capability.  To provide local snapshot protection the "Local Snapshot Protection" rule will need to be selected.  While the local snapshot rules can be added to a replication component, we recommend using local snapshot placement rules for this moving forward.  Here is what the local snapshot placement rules look like.

Selecting Local Snapshot Placement Capability

With local snapshot placement VASA will be creating a new FlashArray protection group that will be mapped one to one with this storage policy.  As such all protection group settings will be required as part of enabling that capability ruleset.  Additionally help tooltips are provided for each rule.

Snapshot placement rule hints

In the example provided here we have a Storage Policy that has both local snapshot placement enabled along with replication rules.

Storage Policy review and finish

A VMware administrator can edit a storage policy that no longer fulfills the needs of the VMs assigned to make it fulfill current needs.

To change a policy’s parameters from the Policies and Profiles page in the Web Client, select VM Storage Policies, select the policy to be changed, and click the Edit button to display a list of the policy’s rules. Make the needed rule changes and click OK.

Edit Settings Button

Changing or adding a Policy Rule

Clicking OK launches the VM Storage Policy in Use wizard, offering two options for resolution:

Manually later 

Flags all VMs and virtual disks to which the changed policy is assigned as Out of Date.

Now

Assigns the changed policy to all VMs and virtual disks assigned to the original policy.

VM Storage Policy in Use Wizard

Should Manually Later have been chosen the VM compliance can be checked by selecting the storage policy and then looking at the VM Compliance tab.

Out of Date Storage Policies

If Manually later is selected, VMs and vVols show Out of Date compliance status. Update the policies for the affected VMs by clicking the Reapply from the VM Storage Policy header bar.

Reapply Storage Policy Button

Confirm the Reapply Action

Now that the Policy has been reapplied the VMs policy status will be in compliance.  

Assigning a Storage Policy to New VM

A VMware administrator can assign a storage policy to a new VM created using the Deploy from Template wizard. (The procedure is identical to policy assignment with the Create New Virtual Machine and Clone Virtual Machine wizards.)

Right-click the target template in the Web Client inventory pane’s VMs and Templates list, and select New VM from This Template.

New VM from Template Command

From the Storage selection page there is a section for VM Storage policy that allows the selectin of any policies that have been created.  

Select Storage Step of Template

Setting a Policy for an Entire VM

In the Select Storage pane, select Thin Provision from the Select virtual disk format dropdown (FlashArrays not running on Purity//FA 6.2.0 or later only support thin provisioned volumes; selecting other options causes VM creation to fail), and either select a datastore (VMFS, NFS or vVol) from the list or a policy from the VM storage policy dropdown.

Selecting a policy filters the list to include only compliant storage. For example, selecting the built-in vVol No Requirements Policy, would filter the list to show only vVol datastores.

Selecting the FlashArray-15MinReplication-Component policy filters out datastores on arrays that do not have protection groups with those properties.

Selecting a Storage Policy

A storage policy that includes local snapshots or remote replication requires a replication group. An existing group can be assigned, or, if Automatic is selected, VMware directs the array to create a protection group with the specified capabilities.

Whichever option is chosen, the VM’s config vVol and all of its data vVols are assigned the same policy. (Swap vVols are never assigned a storage policy.) Click Finish to complete the wizard. The VM is created and its data and config vVols are placed in the assigned protection group.

Assign an Existing Replication Group

The screenshot below shows the FlashArray GUI view of a common storage policy for an entire vVol-based VM.

FlashArray GUI View of a VM-wide Storage Policy

Assigning a Policy to Each of VM's Virtual Disks 

In most cases, VMware administrators put all of a VM’s volumes in the same protection group, thereby assigning the same storage policy to them.

Alternatively, there is the option to apply the storage policy at a per virtual disk basis.  Select configure Per Disk.

Configure storage policy per Disk

In this view, a separate storage policy can be specified for for the VM’s config vVol as well as for each virtual disk (data vVol).

The Configuration File line refers to the VM’s config vVol. The remaining lines enumerate its data vVols (Hard Disk 1 in the example).

Configure Per Disk - Configuring the Configuration File

The objects can be configured individual or by selecting multiple virtual disks or configuration file.

Configure Per Disk - Selecting one virtual disk

Configure Per Disk - Selecting two virtual disks

Selecting a policy from the VM storage policy dropdown filters the list to include only compliant datastores. For example, selecting the vVol No Requirements Policy lists only vVol datastores. 

A storage policy that includes local snapshots or remote replication requires a replication group. An existing group can be assigned (for example, sn1-x70-c05-33:FlashArray-SPBM-15minReplication).

Alternatively, if Automatic is selected, the array creates a protection group with the capabilities specified by the policy. Whichever option is chosen, the policy is assigned to the vVol.

Select VM Storage Policy - Storage - Replication Group

For example, a VM’s config vVol might be assigned a FlashArray-1hrLocalSnap storage policy, which uses Local Snapshot Protection rules.  Whereas its the boot data vVol might be assigned the FlashArray-15minReplication policy, corresponding to the sn1-x70-c05-33:FlashArray-SPBM-15minReplication replication group.

Separate Storage Policies for Config and Data vVols

Here are the screenshots from the Array that list the contents of the two protection groups that correspond to the policies chosen for the config and data.

Config vVol in the VASA managed protection group for the given Storage Policy

Data vVol in the 15 Minute Replication Array Protection Group

Changing a VM's Storage Policy

To change a VM’s storage policy, a VMware administrator assigns a new policy to it. VMware directs the array to reconfigure the affected vVols. If the change makes the VM or any of its virtual disks non-compliant, the VMware administrator must adjust their policies.

To change a VM’s storage policy, select the VMs and Templates view in the Web Client inventory pane, (1) right-click the target VM, (2) select VM Policies from the dropdown menu, and (3) select Edit VM Storage Policies from the secondary dropdown to launch the Edit VM Storage Policies wizard.

Edit VM Storage Policies Command

The VM Storage Policies can be edied a per disk level or all at the same time.  Same with the replication group selection.

Edit VM Storage Policies Wizard

Edit VM Storage Policy Replication Group

To change the storage policy assigned to a VM’s config vVol or a single data vVol, select a policy from the dropdown in the VM Storage Policy column of its row in the table.

Selecting a policy that is not valid for the array that hosts a vVol displays a Datastore does not match current VM policy error message. To satisfy the selected policy, the VM would have to be moved to a different array (reconfiguration would not suffice).

A storage policy change may require that the replication groups for one or more vVols be changed.

One or More Replication Groups not Configured

This warning typically appears for one of two reasons:

1. One or more vVols are in replication groups (FlashArray protection groups) do not comply with the new storage policy.
2. The new storage policy requires that vVols be in a replication group, and one or more vVols are not.

When the policy and replication groups (if required) are set then the Policy summary page will show the state and compliance for each object.

One or More Replication Groups not Configured

Assigning a Policy during Storage Migration

Compliance with an existing or newly assigned storage policy may require migrating a VM to a different array. For example, VM migration is required if:

• A policy specifying a different array than the current VM or virtual disk location is assigned
• A policy requiring QoS (or not) is assigned to a VM or virtual disk located on an array with the opposite QoS setting.
• A policy specifying snapshot or replication parameters not available with any protection group on a VM or virtual disk’s current array is assigned.
• Some of these situations can be avoided by array reconfiguration, for example by creating a new protection group or inverting the array’s QoS setting. Others, such as a specific array requirement, cannot. If an array cannot be made to meet a policy requirement, the VMware administrator must use Storage vMotion to move the VM or virtual disk to one that can satisfy the requirement. The administrator can select a new storage policy during Storage vMotion.

Here is what the process would look like when migrating a VM from VMFS to vVols.  During the storage selection process you would be able to choose the storage policy and what compatible storage is available.  After choosing to migrate the VM, select to change storage only.  

Migrate VM - Storage Only

From the select storage page in the wizard the option to select storage policy is there.  When selecting a storage policy the compatible storage will be shown at the top of the storage list.  If choosing a policy that requires a replciation group there is a replication group selection box at the bottom.

Migrate VM - Storage and Policy Selection

Once the desired policy and replication group (if needed) is selected the migration wizard can be completed.  

vVols Replication with Site Recovery Manager

vVols Replication PowerCLI Commands

> Get-SpbmFaultDomain

Name                 VasaProvider
----                 ------------
sn1-m20r2-c05-36     sn1-m20r2-c05-36-ct0
sn1-x70-b05-33       sn1-x70-b05-33-ct0
sn1-x70-c05-33       sn1-x70-c05-33-ct0

> Get-SpbmReplicationGroup

Name                                      ReplicationState
----                                      ----------------
sn1-x70-b05-33:vVols-Replication          Source
sn1-x70-b05-33:x70-1-policy-ac1-light-001 Source

> Get-SpbmReplicationPair

Source Group                              Target Group
------------                              ------------
sn1-x70-b05-33:vVols-Replication          395a60c2-5803-40be-95b7-029b1b3ffc3e:62
sn1-x70-c05-33:x70-2-policy-ac2-light-001 35770c78-edaf-4afc-9b75-f3fb5c2acee9:9

> Get-SpbmPointInTimeReplica

Name  CreationTime         ReplicationGroup
----  ------------         ----------------
      8/25/2020 3:34:25 PM 395a60c2-5803-40be-95b7-029b1b3ffc3e:62
PiT-1 8/25/2020 3:33:48 PM 395a60c2-5803-40be-95b7-029b1b3ffc3e:62

> Get-SpbmStoragePolicy

Name                                     Description                                                                                         Rule Sets
----                                     -----------                                                                                         ---------
Pure-Demo                                                                                                                                    {(com.purestorage.storage.policy.PureFlashArray=True) AND (com.purestorage.storage.replication.RemoteReplicationInterval=00:05:00…
VVol No Requirements Policy              Allow the datastore to determine the best placement strategy for storage objects
FlashArray Snap 1 DAYS                   FlashArray Storage Policy. Snapshot every 1 Days, retained for 7 Days.                              {(com.purestorage.storage.policy.PureFlashArray=True) AND (com.purestorage.storage.replication.LocalSnapshotInterval=1.00:00:00) …
FlashArray Replication 8 HOURS           FlashArray Storage Policy. Remote Replication every 8 Hours, retained for 1 Days.                   {(com.purestorage.storage.policy.PureFlashArray=True) AND (com.purestorage.storage.replication.RemoteReplicationInterval=08:00:00…

> Sync-SpbmReplicationGroup -ReplicationGroup '395a60c2-5803-40be-95b7-029b1b3ffc3e:62' -PointInTimeReplicaName 'PiT-3'
Sync-SpbmReplicationGroup: 8/25/2020 3:48:04 PM Sync-SpbmReplicationGroup               Error doing 'Sync' on replication group '30488813-7524-3538-868d-66c8037a6d39/395a60c2-5803-40be-95b7-029b1b3ffc3e:62'. Reason:                                                         
Error 1: Sync of the replication group is ongoing. Ongoing task ID: 'SmsTask-SmsTask-90'

> Sync-SpbmReplicationGroup -ReplicationGroup '395a60c2-5803-40be-95b7-029b1b3ffc3e:62' -PointInTimeReplicaName 'PiT-4'
Sync-SpbmReplicationGroup: 8/25/2020 3:56:26 PM Sync-SpbmReplicationGroup               Error doing 'Sync' on replication group '30488813-7524-3538-868d-66c8037a6d39/395a60c2-5803-40be-95b7-029b1b3ffc3e:62'. Reason:                                                         
Error 1: Sync of the replication group is ongoing. Ongoing task ID: 'SmsTask-SmsTask-92'

> Start-SpbmReplicationTestFailover -ReplicationGroup '395a60c2-5803-40be-95b7-029b1b3ffc3e:62'

[FlashArray-B-vVol-DS] rfc4122.918928d8-01aa-47f1-80cd-f31e66d5eac7/vVols-Rep-VM-1.vmx
[FlashArray-B-vVol-DS] rfc4122.fa025596-332f-4e39-82e8-8055f7b589fb/vVols-Rep-VM-2.vmx
[FlashArray-B-vVol-DS] rfc4122.f24fc678-26a4-4234-9356-3b712abbc20b/vVols-Rep-VM-3.vmx

> Get-SpbmReplicationGroup -ReplicationGroup '395a60c2-5803-40be-95b7-029b1b3ffc3e:62'

Name                                      ReplicationState
----                                      ----------------
395a60c2-5803-40be-95b7-029b1b3ffc3e:62   InTest

> Stop-SpbmReplicationTestFailover -ReplicationGroup '395a60c2-5803-40be-95b7-029b1b3ffc3e:62'

Name                      ReplicationState
----                      ----------------
395a60c2-5803-40be-95b7-… Target

> Start-SpbmReplicationPrepareFailover -ReplicationGroup 'sn1-x70-b05-33:vVols-Replication'

> Start-SpbmReplicationFailover -ReplicationGroup '395a60c2-5803-40be-95b7-029b1b3ffc3e:62'

Confirm
Are you sure you want to perform this action?
Performing the operation "Starting failover on" on target "Replication group '395a60c2-5803-40be-95b7-029b1b3ffc3e:62'".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

[FlashArray-B-vVol-DS] rfc4122.f44d3e0f-f25d-4107-bbd6-9a8c2940720b/vVols-Rep-VM-1.vmx
[FlashArray-B-vVol-DS] rfc4122.6a3f3e3c-755d-4c00-a63c-1fd4a69b1476/vVols-Rep-VM-2.vmx
[FlashArray-B-vVol-DS] rfc4122.995eff3c-630c-4ea1-bf33-2eb0f06de84d/vVols-Rep-VM-3.vmx

> Get-SpbmReplicationGroup -ReplicationGroup '395a60c2-5803-40be-95b7-029b1b3ffc3e:62'

Name                                      ReplicationState
----                                      ----------------
395a60c2-5803-40be-95b7-029b1b3ffc3e:62   FailedOver

> Start-SpbmReplicationReverse -ReplicationGroup '395a60c2-5803-40be-95b7-029b1b3ffc3e:62'

Name                      ReplicationState
----                      ----------------
sn1-x70-c05-33:r-vVols-R… Source

vVols Replication with vRealize Orchestrator