Skip to main content
Pure Technical Services

Demoting an ActiveDR Pod in a VMware Environment

Currently viewing public documentation. Please login to access the full scope of documentation.

KP_Ext_Announcement.png

Prior to the demotion of any pod (a source pod or a target pod) any storage in that pod must be removed from use first. Any VM sitting on a VMFS datastore in that pod must be shutdown and unregistered. Any VMFS must be unmounted. Any RDM must be moved from active VMs, or those VMs must be shutdown and unregistered. Follow the steps described in this post to do so. The overall process to demote a pod is:

  1. Shutdown and VMs running on datastores or using RDMs from the source pod.
  2. Unregister those virtual machines.
  3. Unmount the datastores
  4. Demote the source pod

In general, Pure Storage recommends investing in VMware's Site Recovery Manager as it automates all of these steps in a simple and integrated way. Furthermore it provides run book tracking and resource mapping. Not using SRM is fully supported, but it is important to track changes, VM network mappings, RDM usage, and VM ordering manually to make sure a DR event is smoothly handled.

Preparing the VMware Environment for a Pod Demotion

A pod demotion makes all of the volumes in the pod unable to accept new writes ("demoting" it from an active state). Therefore, it is important to make sure the hosts using that storage are prepared for the sudden loss of write access to the storage. A pod demotion does not disconnect the volumes from a host--it instead only changes their read/write state. In the case of a VMware environment the "hosts" using the storage are of two types; the virtual machines, and the ESXi hosts.

Therefore the process to prepare a VMware environment involves shutting down virtual machines running off of the ActiveDR-protected storage and informing the ESXi hosts that this storage will go away.

For virtual machines that use ActiveDR storage but for whatever reason need to continue running after a demotion you should either:

  1. Storage vMotion the virtual machine(s) or the affected virtual disk(s) to storage that is not in the to-be-demoted pod.
  2. Remove virtual disk(s) or RDMs that are stored in the to-be-demoted pod from those virtual machine(s).

Instructions on these two particular steps are well-documented in VMware collateral and no special process needs to be followed in the case of ActiveDR. Therefore instructions on performing either of these tasks are beyond the scope of this document.

Identify Storage with the Pure Storage Plugin for the vSphere Client

The first step is the identify the storage that is in the pod. Identify the name of the pod you plan to demote. If you know a datastore that is in the pod, you can click on the datastore and review the summary tab for the exact name of the pod (hosting pod will be the pod name before the directional arrow in the pod property):

clipboard_ec86b85f98630590475ee0d94309e2608.png

The pod name in this case is ActiveDRpodA and the FlashArray is flasharray-m20-1. Armed with this information, click on the top menu in the vSphere Client and choose Pure Storage.

clipboard_e3e2f18b4f51ca7236921f8ff9c45faad.png

Choose the FlashArray in the list and then click on the Volume Group tab below:

1) Select your array 2) Click on the Volume Groups tab.
clipboard_ec498e2b946f317aa1fb6426574c1a02c.png clipboard_e9a37170ff764a75480735ed721fd42a8.png

 

In the search bar under the Volume Groups tab you can search for datastore names OR FlashArray volume names. In the case of a volume in a pod, all of their names are preceded with the name of the pod. Simply type in the name of the pod to find all of the volumes that are in use as datastores.

clipboard_e0113afd9f879c3ae61ce9f08482e34c0.png

If nothing appears then verify the pod name. If the name is correct this means that there are no VMFS datastores from that pod in the VMware environment. 

This search function will not list volumes in use as RDMs, nor volumes that are presented to other environments, nor volumes presented to the VMware environment but not in use at all. If there are more volumes in the pod that do not show up here it is because of one of those three reasons.

Click on each volume to see the volume and datastore details and then click on the Go to Datastore link to prepare the datastore for demotion. That process is discussed below.

Identify Storage with the FlashArray UI and the vSphere Client

If the Pure Storage Plugin for the vSphere Client is not in use and/or there are additional volumes to identify you can use the FlashArray UI tools (CLI, GUI, or REST) in combination with the standard features in the vSphere Client. 

The first step is the identify the storage that is in the pod.

This can be show in the FlashArray GUI or the CLI (or scripted via tools like PowerShell). Identify the serial numbers of the volumes in the pod.

The FlashArray GUI:

1) List the volumes in the pod 2) Find volume details  3) Record the serial number of the volume
clipboard_e1d47c11f76898f092290ad1c1b7cba8d.png clipboard_e23e79dad9fe2e5eaac2b5deb8d385d4d.png clipboard_edcfe5e94db36b5296102148cca169202.png

The FlashArray CLI:

clipboard_e5bea137d138c047e018e53bb745ce914.png

In this case, I have four volumes which have the following serial numbers:  

5EE86996F8334FA00006E860, 5EE86996F8334FA00006E864, 5EE86996F8334FA00006ED79, 5EE86996F8334FA00006EDA4

Now login to the vSphere Client, click on a host, then the Configure tab, then Storage Devices, then use the filter object to search for the serial number.

1) Click on a host 2) Choose Configure then Storage Devices 3) On the Name column filter by the desired serial number
clipboard_e1f3ca02d1a1a66b578cf864a7fe086ed.png clipboard_e52b23e53cbc550ac5c2db38b98b20959.png clipboard_e1af031c1f1362d423b59a62c2c4835c1.png

If the volume is connected to that host, it will be listed. If the volume is in use as a VMFS datastore it will be linked in the datastore column. If that column is empty it either means that volume is un-used (but present) or in as use as an RDM.

clipboard_e064a53a0afb0e341b590fbaf664b567f.png

Shutdown and Unregister Virtual Machines

Once you have identified the datastore(s), the next step is to shutdown the virtual machines and then unregister them from inventory. Because the datastore will go offline, it is important to remove any objects accessing files (configuration or virtual disks) on the datastore. 

First click on the datastore, then its VMs tab.

1) Click on the VMFS datastore 2) Click on the VMs tab
clipboard_e7a094a886df440378db7f1ede84f2077.png clipboard_eaa4b65d2b390937ac0994217deeaa300.png

Before shutting down VMs ensure that you are ready to do so! Shutting down virtual machines means terminating applications--so it is critical to be sure you are shutting down the right applications and it is the correct time to do so. Also the below process assumes all VMs can be shut down at once--if you have specific power-down ordering follow that process. The below is just an EXAMPLE. The key is to shut down and unregister all VMs on the specific datastore(s) before proceeding.

 Next click on the column heading State and sort so all of the powered-on VMs are listed consecutively. Then on the first VM in the list and then hold down the shift key and select the last VM in the list that is powered-on. Choose the menu option Shut Down Guest OS (this is the preferred method, choosing power-off is not graceful and should be avoided when possible).

1) Sort by power state 2) Select all powered on VMs 3) Shut down the VMs. Preferably use Shut Down Guest OS
clipboard_e3d5a243ba54e3692b138297dd9c4b8ea.png clipboard_e818e2f68c27c007138f5361f4f9b2a2d.png clipboard_e2d179b6da73ada69a7ab7c6317b422a1.png

Once the virtual machines are shut down, unregister those VMs from vCenter. Select all of the VMs on the datastore, right-click any one of them, and choose Remove from Inventory.

1) Select all of the VMs 2) Right-click and choose Remote from Inventory 3) Confirm operation(s)
clipboard_efd501b57a9eba69dc2be8cff7c21244f.png clipboard_ef316419974ff284ef591f561cc1d2952.png clipboard_e74c713ed675ffd81efeea1cef453de37.png

Prepare VMFS datastores for ActiveDR Demotion

Verify that the VMs are removed--also click on the VM Templates tab and ensure that there are no VMs reported there. If there are, repeat the Remove from Inventory process for them.

1) Ensure the virtual machines are full unregistered. 2) Ensure all templates are unregistered.
clipboard_edc3393e31be2bef816ea33a25a16f830.png clipboard_ef68b95ccbc02eabeb952df1f73e95e18.png

Once the datastore workloads are cleaned out--put them in maintenance mode. This ensures that no new VMs get placed on them from this point on. Right-click on the datastore, choose Maintenance Mode, then Enter Maintenance Mode.

1) Choose datastore. 2) Enabled maintenance mode 3) Ensure completion.
clipboard_e119a0aa2c72a6723d4df8707c2d89559.png clipboard_e36b9a0aa34c371b8737d5b5473b4c900.png clipboard_ec09b87e56111755bb1e564bf751308f5.png

If maintenance mode fails, it means a workload is still active on the datastore. Note this step is not required, but recommended.

The last step is to unmount the datastore--this removes the datastore from inventory of the hosts and vCenter. Right-click on the datastore and click Unmount Datastore. Choose all hosts in the dialog box and click OK.

1) Right-click datastore 2) Choose unmount datastore. 3) Unmount the datastore from all hosts.
clipboard_e486fdc7fb34d30df9e89eb3ff36373bf.png clipboard_e3029a1aa874a2230a9d891cc039a36d5.png clipboard_e75d67442b198612a245ce023d431d64b.png

This will then unmount the datastore from all of the hosts:

clipboard_e80f8ce07df271d04ce59983708a13e66.png

The datastore will go to inaccessible in the vSphere Client.

clipboard_ef645e54bbd0165b673b2dbd3220e459e.png

The next step is to detach the volume hosting the datastore. Click on a host, then Configure tab, and Storage Device panel. Select the device hosting the datastore and choose Detach. In newer release of vSphere, this will allow to detach from multiple hosts at once. In older releases you will need to repeat this process for each host that see the datastore.

1) Choose a host then Configure > Storage Devices. 2) Select the device and choose Detach 3) Select all hosts and choose Yes.
clipboard_ed9ea5f0c15d1fe46d522c02a13cd8c02.png clipboard_e97e9f27e8736d82b1623f1a94af48be5.png clipboard_e2afb666dca2b38d7a3296fe2313e5399.png

The datastore will be removed from the list upon the next rescan, or you can perform a manual rescan of the cluster(s) to force the update. You may want to do this to ensure that no host still sees it and it is indeed detached from all existing hosts. If the datastore disappears upon rescan the volume has been completely removed.

1) Rescan storage on hosts, clusters, or datacenters. 2) Verify the datastore is no longer listed.
clipboard_e046d9f16a8659825ef03fd95597c3c07.png clipboard_e5961c7f897d74c4848f3b6b7f9756f60.png

Repeat the above process for each VMFS datastore.

Prepare RDMs for ActiveDR Demotion

When it comes to RDMs it is a bit more difficult to find what VMs are using RDMs and what volumes are in use as RDMs. The simplest option at scale is to use VMware PowerCLI. 

First you need to identify the FlashArray volume that is used an RDM, or in the vSphere Client find the device. Eitherway retrieve the serial number of the volume.

In the FlashArray UI:

1) Go to the volume in the FlashArray 2) Copy the serial number
clipboard_ee40539002bacba905d8029ca4a37faa5.png clipboard_e34db26feb0d9937bab18c7cc1034a2ba.png

In the vSphere Client:

1) Go to a host that sees the RDM 2) Select the device 3) Copy the NAA
clipboard_ec89cf7550f136a171fdf8323ef9a5124.png clipboard_ec6267b81a7d01d5f009a92b8c9e34c1e.png clipboard_e6887a447d9561a78be7b9a6dfcd398a6.png

The next step is to use PowerCLI. If you do not have it installed, install it from the PowerShell Gallery:

1) Launch PowerShell (Core or Desktop) 2) Install VMware.PowerCLI 3) Verify installation
clipboard_ea58ab9ebe6b37b6ddc31c5b9c85122aa.png clipboard_e7d114fd071de21266a986a7177df7992.png clipboard_edb22bc5cbffe80c1bcfe20ed133bd156.png

If you pulled the serial number from the FlashArray run the following code (supplying your vCenter address and credentials and the serial number):

connect-viserver -Server <vCenter address>
$serial = <serial number in quotes>
Get-vm | where-object {$_ |Get-HardDisk -DiskType "RawPhysical","RawVirtual" | where-object {$_.ExtensionData.Backing.LunUuid.substring(10).substring(0,32) -eq $serial}}

Example: 

clipboard_e089be6973711a23a531702baebff448b.png

If you pulled the NAA from the vSphere Client, run this code:

connect-viserver -Server <vCenter address>
$naa = <NAA in quotes>
Get-vm | where-object {$_ |Get-HardDisk -DiskType "RawPhysical","RawVirtual" | where-object {("naa." + $_.ExtensionData.Backing.LunUuid.substring(10).substring(0,32)) -eq $naa}}

Example:

clipboard_e1d67460082dea55d482ed4591e114174.png

This will return any VM using that volume as an RDM.

Now go to that VM and either shut it down and unregister or remove the RDM from the VM. You do not need to do both.

Remove the RDM:

1) Edit VM settings 2) Remove the disk from the VM 3) Select Delete Files from Datastore and click OK.
clipboard_e4be7a350c9dbf70a83dac8e25a98c7da.png clipboard_e650f7b1fe3a2d7c4e2a063bf0f3d6cbc.png clipboard_e4dc777e36ed9e01a301e2808a362f492.png

Or...

Shutdown and unregister the VM:

1) Shut the VM down with Shut Down Guest OS 2) Right-click and choose Remove from Inventory
clipboard_ee02ab4a4b1f550bc2a83903c636b4d85.png clipboard_e8d90771c84de30425a89c5390e7a47a7.png

The next step is to detach the volume hosting the datastore. Click on a host, then Configure tab, and Storage Device panel. Select the device hosting the datastore and choose Detach. In newer release of vSphere, this will allow to detach from multiple hosts at once. In older releases you will need to repeat this process for each host that see the datastore.

1) Choose a host then Configure > Storage Devices. 2) Select the device and choose Detach 3) Select all hosts and choose Yes.
clipboard_ed9ea5f0c15d1fe46d522c02a13cd8c02.png clipboard_e20c95d0387e73c2f5cdf3a214ce78710.png clipboard_e9543dcd161f79251fa74b82f59d7b796.png

Repeat for all RDMs. 

Demoting a Source ActiveDR Pod

Once all workloads are shutdown to the pod you can commence with a demotion. A good way to confirm this is to check the performance statistics on the pod. It should all be zero:

clipboard_ea60bd79feee43c110ff40b29ab9ef0c2.png

Once confirmed, navigate in the FlashArray GUI to the pod you would like to demote. In this case it is a source pod. You would demote a source pod when you are about to failover the workload to the target site and this ensures that the data is fully synchronized to the target site and that the source becomes inaccessible on the source. This prevents the same workloads from running in production at once.

ActiveDR, however, does allow you to run the workload on both the source pod and the target pod at the same time--this is generally referred to as a test recovery and is discussed here: Promoting an ActiveDR Pod in a VMware environment

Go to Storage then Pods, then click on the pod.

clipboard_ebe26acc0d0de225a892ea5cdcd9152fc.png

In the upper right-hand corner, click on the vertical ellipsis and choose Demote. Choose Quiesce and click Demote.

DO NOT DEMOTE A POD if there are any volumes in that pod still connected to ESXi and the host personality of those FlashArray hosts is not set to ESXi. See here for more detail: Guidelines for ActiveDR in VMware Environments

1) Choose Demote from the pod management menu. 2) Choose Quiesce and Demote to complete the process.
clipboard_ea82a3fdb5743813d0620c496fad5be99.png clipboard_e62baaed5408c8ac8f3fa9ad6819de989.png

Is there a time you wouldn't want to choose "Quiesce"? Possibly. If there was some type of transient failure on source site and you have already failed the workload over to the remote site and restarted the virtual machines. In this scenario production data has been written to the target site that is newer than what is on the original source. Therefore replicating any non-replicated changes from the original source to the original target site is not necessary.

During the demotion process, the state of the replica link will go to quiescing and then to quiesced when the synchronization is fully complete.

clipboard_e10d058d13a11176e0f1e9d2a6da31913.png

The overall pod state will be demoted.

The full state of the pod is preserved upon the demotion (volumes, data, snapshots, protection groups) and is stored in an "undo" pod for 24 hours.

clipboard_e8c9ce14af5b8ff027ff5f07efb153584.png

This will provide an option to restore the environment exactly to the way it was if the new target site begins replicating over the original pod and it is deemed those changes are a mistake. To keep that point-in-time of the pod longer you can copy it to a new pod--though it should be noted the copied pod will provide new serial numbers to a volumes and snapshots, but the data will be identical to the original pod.

Once the demotion occurs the volumes will go NOT READY and not respond to reads or writes from a host. The volumes will only respond to VPD inquiries. So VMware will see the device, but will not know if there is a VMFS on them, and they will not be able to be re-attached. The pod hosting the volumes must be in the promoted state in order for it to be used by ESXi.

Note the below image shows demoted devices and the Attach button is grayed out.

clipboard_e496e510ab77932c32bce1577c3efbf7c.png

 

Demoting a Target ActiveDR Pod

The process of a demotion of a target pod is another way to say that you are ending an ActiveDR test. The implication is that the source pod is still promoted and in-use. 

You can verify this by examining your pod replica link state. The current pod being view (activeDRpodB) is promoted, and also the target for the replication:

clipboard_e640c65dc4ef6098059cce075dd81c7fc.png

Prior to demotion you should verify two things:

  1. Is there active I/O going to the pod? If so, examine the workloads and stop them. If they should/cannot be stopped, move the workload to a volume not in the target pod.
  2. Are there additional volumes in the target pod as compared to the source pod? If so, it is possible someone accidentally provisioned a new volume into the target pod while it was demoted. Verify the "additional" volume usage prior to demotion.

The simplest way to ensure the pod is not actively serving I/O is to check the performance statistics.

Navigate to Analysis -> Performance -> Pods, then select the target pod.

clipboard_e38ef9db42bdb742422023e1053ef6338.png

If there is detectable I/O move to the Volumes tab and search for volumes with the pod name:

clipboard_e08d02190111ab04d0ecf70cdc53eecbe.png

Investigate the volumes with active I/O and determine if they should be terminated or moved out of the target pod.

Once the pod has been cleared...

clipboard_e028b3106286ba35281b5b73959ad200f.png...it is time to verify that no unexpected volumes are in the pod. Once the pod is promoted, it not only makes the replicated objects available, but also allows new objects to be created within the pod.

If a user provisions a new volume into that pod, the volume will be removed entirely once the pod has been demoted, as a promoted target pod is seen as only having temporary changes that do not need to persist.

A simple way to ensure you do not accidentally remove a volume, check that the volume count in the source pod is the same as the target pod. In the source pod, look at the volumes box and in the upper right hand corner there will be a count of volumes:

clipboard_ee3b1375c786d92eb656426f2b1d870a3.png

This is showing two volumes.

In the target pod, it has the same.

clipboard_e46e7b07e8486508151e134f55930400e.png

Once you have confirmed that no unexpected volumes are in the pod, you can proceed to demote:

clipboard_ecb2412d3391426f1d7816b3f047fd593.png

If it is determined that after the pod is demoted that some data from the point-in-time immediately prior to demotion is needed, the contents of the pod at that time is protected temporarily in an "undo" pod. This undo pod is kept in the "destroyed pods" area for a period designated by the eradication timer which defaults to 24 hours, but can be increased upon request to Pure Storage support.

clipboard_eb5ce7eacd103a3bd6f32bf128a8bd83b.png

This undo pod can be copied to a new pod so the contents can be used/manipulated.