Troubleshooting ActiveDR Test Recovery Errors in Site Recovery Manager

A Non-SRM controlled Volume is added to the source pod

As of the 4.0 release of the Pure Storage SRA, volumes not belonging to the VMware environment (meaning in use as a VMFS or RDM) that is controlled by SRM is not supported. In other words, for a given ActiveDR pod, either all of the volumes must be in use as an VMFS or RDM and protected by the same SRM protection group, or none of them. Supporting "non-VMware" volumes is under investigation for a future release of the SRA.

If a volume is added to the source pod prior to a test:

And the 4th volume (in this case) is not in the protection group:

The test recovery will still succeed, but the cleanup will fail. The test recovery will tell the SRA to recover the 3 volumes SRM knows about, and the SRA will tag them:

During the cleanup, the SRA will see a 4th volume which is untagged. The cleanup will fail with the following error:

 Failed to delete snapshots of replica devices. SRA command 'testFailoverStop' failed. More than one type of devices found in the pod. Please verify that all the volumes are either untagged or tagged with same value in the "puresra" namespace.

This failure can also occur if someone manually deleted the tags on one of the expected volumes. The SRA will fail in this situation because it does not want to demote the pod when an in-use volume is in it--the demotion process causes all volumes to go NOT READY, meaning they will not be able to be written to or read from. If some environment is using that 4th volume, it will lose access. In this situation, manual intervention is required.

The options are:

• Delete the non-VMware volume
• Terminate the pod replica link, move the volume out of the pod, and then relink the pods (note this will cause replication to stop for the duration of the process!).

You may either do this, then re-run test recovery cleanup or run test recovery cleanup with the force cleanup option selected to have the SRA skip the demotion step. The force option will require manual intervention to demote the pod.

Volume added to target pod during test

When the target pod is promoted during a test, the pod is now available for provisioning. While it is possible to do so, it is not recommended to provision from a pod that is promoted for the use of a test as upon demotion those volumes will be destroyed as the pod will sync back up with the source pod which does not have the volume that was created in the temporarily promoted target pod.

In this case there are two pods linked for ActiveDR, a source pod called srm-podA and a target pod srm-podB. During an SRM test, someone added a volume to srm-podB called volcreatedintarget:

This volume is not in the source pod:

The SRM protection group is not aware of it either:

A cleanup will fail because that volume is unknown and therefore not tagged as being in test. The SRA does not want to continue on and demote the pod because that volume could be serving up production data--someone might have accidentally provisioned it to the target pod when it was promoted. While doing so is a mistake--accidents happen and the SRA does not want to make it worse. Therefore, the cleanup will fail:

 Failed to delete snapshots of replica devices. SRA command 'testFailoverStop' failed. More than one type of devices found in the pod. Please verify that all the volumes are either untagged or tagged with same value in the "puresra" namespace.

The options are:

• Move the data off of the volume to a new FlashArray volume in a different pod via some host-based tool and then delete the original volume.
• Terminate the pod replica link, move the volume out of the pod, and then relink the pods (note this will cause replication to stop for the duration of the process!).

You may either do this, then re-run test recovery cleanup or run test recovery cleanup with the force cleanup option selected to have the SRA skip the demotion step. The force option will require manual intervention to demote the pod.

A Test Recovery tag was accidentally deleted

It is important to remember: DO NOT manually edit or remove tags in the puresra namespace unless you understand fully what you are doing. If some rogue admin or typo'ed script deletes the tags during a test recovery it will be necessary to manually intervene.

During the test recovery, all volumes in the source pod get tagged with three pieces of information:

• The operation (like testFailover) 
• Timestamp of the test
• UUID of the pod they belong to.

The tags look like this:

If someone deletes one of the tags, the cleanup process will fail. So in this case, my 3rd volume (activeDR-SRM3-RDM) was untagged:

 Failed to delete snapshots of replica devices. SRA command 'testFailoverStop' failed. More than one type of devices found in the pod. Please verify that all the volumes are either untagged or tagged with same value in the "puresra" namespace.

 If you log into the target array CLI, you can query for the tags of the volumes in the pod:

If you add the target pod name with an asterisk at the end, you will only see volumes in that pod.

Compare that list to the list of volumes in the SRM protection group:

If there is a volume in the protection group that is missing in the tagged volume list (and that volume is in fact in the target pod), you know the tag was for some reason deleted.

If they are the same list of volumes, then it means someone added a volume to the target pod during the test. Look at the above section for handling that.

If a volume is missing the tag, you can either add the tag or re-run the test recovery with the force option and manually demote the pod. This will erase all of the tags on the target volumes. Generally the latter is the preferred option as tagging can be error prone.

To add the tag:

Look at the tags in the pod, copy the key and the value. Identify the volume that needs to be tagged and tag it as follows:

purevol tag SRM-podB::activeDR-SRM3-RDM --key puresra-testFailover_1597691959294 --value e184134e-5a94-9f76-5c53-ae33e6df93d5 --namespace puresra

The above shows listing the tags, adding the tag, then listing the tags again. You must use the puresra namespace--otherwise the tag will not work.

Once done, re-run the test recovery cleanup.

 

The alternative (let's assume the same scenario as above) is to force the test recovery cleanup and then manually demote the pod.

Let the cleanup complete, then login to the target FlashArray and identify the target pod. Click on the vertical ellipsis and choose Demote. CONFIRM that this is indeed the target pod, not the source pod. The replication direction should be coming towards this pod like in the screenshot below:

Click Demote.

The tags will be removed:

Once you confirm that no issues were caused by the demote, you can eradicate the undo pod (if you need to run another test) or if not, let the eradication timer remove it in due time.