Skip to main content
Pure1 Support Portal

Volume Shadow Copy Management

Overview 

There have been discussions in various Microsoft forums regarding vulnerabilities with Volume Shadow Copies (VSS) that are stored on the host which initiated the shadow copy using either diskshadow or vssadmin utilities included in Windows Server. For full context see https://www.bleepingcomputer.com/news/security/why-everyone-should-disable-vssadmin-exe-now/ and  https://social.technet.microsoft.com/Forums/Lync/en-US/7f13731c-c59c-4cc9-84fc-29a450dde2fd/is-this-safe-to-deal-with-shadow-copies-?forum=winserverfiles.

Pure Storage supports the creation of Volume Shadow Copies using our VSS Hardware Provider (PureProvider). PureProvider supports the creation of application consistent snapshots for Microsoft SQL Server, Microsoft Exchange and Microsoft SharePoint. If unfamiliar with how Volume Shadow Copies work please refer to http://www.purepowershellguy.com/?p=2131 which explains the VSS Workflow and differences between application and crash consistent snapshot’s. Today, Pure Storage does not have a standalone VSS Requester and recommends the use of diskshadow (native Windows Server requester) to initiate shadow copy operations.

This article will explain in detail how to configure the PureProvider settings to protect the FlashArray volumes and snapshots. 

Working with Shadow Copies

The diskshadow and vssadmin utilities have different sets of commands that can be executed to create and manage shadow copies. Let's first start with how a shadow copy is created. The below screenshot shows a volume (Server05-SQLData01) connected to the host (Server05). The Server05 host is running Microsoft SQL Server 2017 and all of the data files (MDF/LDF) are stored on the Server05-SQLData01 volume.

VSS_VolumeConnection.png

From the Windows Server host the volume can be seen using the Disk Management utility.

VSS_VolumeConnect_DiskMgrView.png

Create Shadow Copy

To create a shadow copy of the Server05-SQLData01 volume the following commands would be executed from diskshadow. The "add volume F:" command is adding the mapped drive as shown in the Disk Management screenshot above. Server05 is running SQL Server with an attached database named, AdventureWorks2017. As shown below the SqlServerWriter service is called to take a shadow copy of the databases.

DISKSHADOW> set context persistent
DISKSHADOW> set option transportable
DISKSHADOW> set verbose on
DISKSHADOW> set metadata c:\temp\vss_metadata.cab
DISKSHADOW> begin backup
DISKSHADOW> add volume F: 
DISKSHADOW> create

* Including writer "SqlServerWriter":
        + Adding component: \SERVER05\master
        + Adding component: \SERVER05\model
        + Adding component: \SERVER05\msdb
        + Adding component: \SERVER05\AdventureWorks2017

Alias VSS_SHADOW_1 for shadow ID {e2cabff8-686d-4156-8806-cbc3def7a1ca} set as environment variable.
Alias VSS_SHADOW_SET for shadow set ID {06e6a2e9-8276-4b06-a33f-b445f2c5bead} set as environment variable.
Inserted file Manifest.xml into .cab file vss_metadata.cab
Inserted file BCDocument.xml into .cab file vss_metadata.cab
Inserted file WM0.xml into .cab file vss_metadata.cab
Inserted file WM1.xml into .cab file vss_metadata.cab
Inserted file WM2.xml into .cab file vss_metadata.cab
Inserted file WM3.xml into .cab file vss_metadata.cab
Inserted file WM4.xml into .cab file vss_metadata.cab
Inserted file WM5.xml into .cab file vss_metadata.cab
Inserted file WM6.xml into .cab file vss_metadata.cab
Inserted file WM7.xml into .cab file vss_metadata.cab
Inserted file WM8.xml into .cab file vss_metadata.cab
Inserted file WM9.xml into .cab file vss_metadata.cab
Inserted file WM10.xml into .cab file vss_metadata.cab
Inserted file DisFB54.tmp into .cab file vss_metadata.cab

DISKSHADOW> end backup

Best Practice

When defining the metadata file it is recommended to use a file name that has meaning so that any previously created metadata files do not get overwritten. Below is an example of specifying a SQL Server shadow copy that was taken at 9am. 

SET METADATA C:\MyShadowsCopies\SQL-9am.cab

After the shadow copy has been created it is listed on the initiating host and in the FlashArray Snapshots as shown below. The snapshot that is created has a unique name that includes the shadow set ID. In the below example the snapshot name is Server05-SQLData01.VSS-06E6A2E982764B06A33FB445F2C5BEAD. Looking at the shadow copy creation above the snapshot set ID is {06e6a2e9-8276-4b06-a33f-b445f2c5bead}. 

VSS_VolumeSnapshot.png

Importing & Exposing a Shadow Copy 

To use a shadow copy (FlashArray Snapshot) there are a few steps that include:

  • Loading the metadata (vss_metadata.cab) from the create shadow copy process.
  • Importing the shadow copy.
  • Exposing the shadow copy to the Windows Server host.
DISKSHADOW> load metadata c:\temp\vss_metadata.cab
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabManifest.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabBCDocument.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM0.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM1.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM2.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM3.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM4.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM5.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM6.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM7.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM8.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM9.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabWM10.xml.
Extracted C:\Users\ADMINI~1.MSL\AppData\Local\Temp\vss_metadata.cabDis68A9.tmp.
Alias VSS_SHADOW_1 for value {e2cabff8-686d-4156-8806-cbc3def7a1ca} set as an environment variable.
Alias VSS_SHADOW_SET for value {3cd498ab-953f-41d5-aa1a-a1732c59d9cb} set as an environment variable.

DISKSHADOW> import

Once the shadow copy has been imported the system can be queried for the available shadow copies.

DISKSHADOW> list shadows all

Querying all shadow copies on the computer ...
        * Shadow copy ID = {84c7b3fc-d073-4572-8384-c1a48afdf975}               %VSS_SHADOW_1%
                - Shadow copy set: {06e6a2e9-8276-4b06-a33f-b445f2c5bead}       %VSS_SHADOW_SET%
                - Original count of shadow copies = 1
                - Original volume name: \\?\Volume{7da2264a-803c-4c0e-b629-91a9530ded55}\ [F:\]
                - Creation time: 1/11/2018 2:51:03 PM
                - Shadow copy device name: \\?\Volume{9b2895cb-f718-11e7-836f-0025b543001f}
                - Originating machine: SERVER05.mslab.purestorage.com
                - Service machine: SERVER05.mslab.purestorage.com
                - Not exposed
                - Provider ID: {781c006a-5829-4a25-81e3-d5e43bd005ab}
                - Attributes:  Transportable No_Auto_Release Persistent Hardware Imported
                
Number of shadow copies listed: 1

Finally the shadow copy can be exposed to the Windows Server host. To expose the shadow copy the Shadow copy ID or the Alias (%VSS_SHADOW_1%) is required. 

DISKSHADOW> expose {84c7b3fc-d073-4572-8384-c1a48afdf975} H:
The shadow copy was successfully exposed as H:\.

Looking at the Disk Management utility the shadow copy is now exposed as drive H:\.

VSS_ShadowCopy_Exposed.png

The final step when done working with an exposed shadow copy is to unexpose. The volume Server05-SQLData01 (H:) is the shadow copy that will be unexposed. There are foru different methods to unexpose

  • <shadowID>  -- The shadow copy ID or a currently defined alias (eg. %VSS_SHADOW_1%).
  • <drive letter> -- Exposed to a drive letter like "H:".
  • <mount point> -- An exposed mountpoint, in the form "C:\MyMounts\VSS\" or "H:\".
  • <share> -- Remotely exposed to a share like "ShareName" .
DISKSHADOW> UNEXPOSE H:
Shadow copy ID {84c7b3fc-d073-4572-8384-c1a48afdf975} is no longer exposed.

At this point we have covered the following:

  • Creating a shadow copy.
  • Loading and importing a shadow copy.
  • Exposing a shadow copy to the Windows Server host.
  • Unexposing a shadow copy.

In the next section the topic of how to manage shadow copies with diskshadow and some custom settings with the Pure Storage VSS Hardware Provider.

When customizing the settings of the PureProvider it requires editing of the Windows Server registry. Please backup the Windows Server registry before implementing any of these customizations. The below example shows how to perform a backup.

REG EXPORT HKLM PureStorage-RegistryBackup.reg /y

Managing Shadow Copies

When deleting (aka destroyed) a volume or snapshot from the FlashArray they are not immediately eradicated from the system. There is a 24hr waiting period before they are automatically eradicated from the FlashArray. Depending on what has been deleted they are placed in Destroyed Volumes or Destroyed Snapshots. Shown below is an example of several VSS snapshots in the Destroyed Snapshots. To immediately delete and/or eradicate a volume or snapshot from the FlashArray it must be done manually through the management GUI or via scripting. 

VSS_DestroyedVolumesSnapshots.pngWith the PureProvider there are four different options for managing shadow copies between the Windows Server host and the FlashArray volumes and VSS snapshots. By updating these registry values you can customize how snapshots and volumes for shadow copies behave. At the start of this article vulnerabilities were mentioned with shadow copies around the ability of deleting shadow copies from the host system and potentially the SAN that is mapped to specific volumes or snapshots. There are three utilities that come with Windows Server that can be used to perform shadow copy deletes. Each one of the examples illustrates how to delete local shadow copies without requiring user assistance.  

Wmic shadowcopy delete /nointeractive

diskshadow delete shadows /all

vssadmin delete shadows /all /quiet

Running the above commands makes several assumptions:

  • The command was manually run by mistake.
  • The command was included in a diskshadow script for automation.
  • The host has been compromised and a diskshadow script or manual command has been executed.

With the FlashArray we provide the ability to control what happens with a specific shadow copy through the use of PureProvider registry settings. Using these settings it is possible to control the behavior of how a shadow copy deletion on the FlashArray is handled. 

DeleteSnaps

All settings are DWORD. This setting controls how snapshots and volumes are managed using the PureProvider and diskshadow. These registry values only affect the “deletion” phase, and not the subsequent eradication. 

  • DeleteAll [0] (default) -- Deletes the exposed volume create using purevol copy and the snapshot.
  • DeleteVolumeCopy [1] -- Deletes the exposed volume. 
  • DeleteSnapshot [2] -- Deletes the shadow copy (VSS-###) snapshot from the parent volume. 
  • DeleteNone [3] -- Does not delete the exposed volume or snapshot. 
EradicateVolumes

Setting is DWORD. This setting controls whether or not a volume is eradicated from the FlashArray or not. These registry values only affects immediate eradication vs the FlashArray's 24hr eradication processing.

  • EradicateVolumes [0] -- Does not eradicate volumes and snapshots immediately from the system. 
  • EradicateVolumes [1] (default) -- Immediately eradicates the volumes and snapshots associated to a select or all shadow copies.

Best Practice

It is not recommended to change the default values for DeleteSnaps and EradicateVolumes. The default values provide volume and snapshot management to not have problems later.

To modify these settings the below Windows PowerShell can be used to Get and Set the values.

Get-ItemProperty -Path 'HKLM:\Software\PureStorage\VSS\Provider' -Name 'DeleteSnaps'
Get-ItemProperty -Path 'HKLM:\Software\PureStorage\VSS\Provider' -Name 'EradicateVolumes'

Set-ItemProperty -Path 'HKLM:\Software\PureStorage\VSS\Provider' -Name 'DeleteSnaps' -Value 0 #DeleteAll
Set-ItemProperty -Path 'HKLM:\Software\PureStorage\VSS\Provider' -Name 'DeleteSnaps' -Value 1 #DeleteVolumeCopy
Set-ItemProperty -Path 'HKLM:\Software\PureStorage\VSS\Provider' -Name 'DeleteSnaps' -Value 2 #DeleteSnapshot
Set-ItemProperty -Path 'HKLM:\Software\PureStorage\VSS\Provider' -Name 'DeleteSnaps' -Value 3 #DeleteNone

The PureProvider caches the registry key values the first time it reads them during a volume/snapshot operation. If you change these values after the shadow copy operation the PureProvider service will still use the cached values and ignore the setting changes. A restart of the PureProvider is required for the new settings to take affect. Below is an example of how to restart the PureProvider service using Windows PowerShell. The service can be restarted using the Services utility included with Windows Server. 

Get-Service -Name PureProvider
Restart-Service -Name PureProvider

PureProvider Log Management

In addition to providing snapshot and volume management settings the PureProvider has different log settings for troubleshooting. Higher log level numbers implicitly includes all lower log levels.

  • LogLevel = 4 (default)
    • LOG_ERROR = 0
    • LOG_WARN = 1
    • LOG_INFO = 2
    • LOG_DEBUG = 3
    • LOG_TRACE = 4

The PureProvider log files are stored under C:\ProgramData\PureStorage. There are two log files:

1. PureProviderConfig.log -- Contains logging information for the PureProvider configuration.

Jan 10 12:31:47.343 20AC I Pure.Logger >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ==== ( LOG SESSION STARTED ) ==== <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Jan 10 12:31:47.358 20AC I Pure.Logger LogLevel: 4, LogFileName: C:\ProgramData\PureStorage\PureProviderConfig.log.
Jan 10 12:31:47.374 20AC I Pure.Logger Version: 1.6.0. BUILD: .16.16 (64-bit Release). OS: 6.2
Jan 10 12:31:47.374 20AC I main() Command-line: C:\Program Files\Pure Storage\VSS\Provider\PureProviderConfig.exe add --url https://10.0.0.1 --user pureuser TBL2-M20 
Jan 10 12:31:47.390 20AC D main() Starting Application - CONSOLE mode
Jan 10 12:31:47.405 20AC D PurityRegCli.Start Purity Reg Tool Log initialized
Jan 10 12:31:47.405 20AC D Pure.RegTool Cli ENTERING: AddPureArrayConfiguration()
Jan 10 12:31:47.421 20AC D Pure.RegTool Implementation ENTERING: GetPassword()
Jan 10 12:31:49.811 20AC D Pure.RegTool Implementation Password Entered: hidden, Exiting: GetPassword()
Jan 10 12:31:49.826 20AC D Pure.RegTool Implementation ENTERING: AddPureArray() with Parameters: Name=TBL2-M20, URL=https://10.0.0.1, Username=pureuser, Password hidden

2. PureVSSHardwareProvider.log -- Contains logging information for VSS operations. When troubleshooting VSS operations this is the log file that has the pertinent information.

Jan 10 12:28:35.804 1DF4 I Pure.Logger >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ==== ( LOG SESSION STARTED ) ==== <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Jan 10 12:28:35.820 1DF4 I Pure.Logger LogLevel: 4, LogFileName: C:\ProgramData\PureStorage\PureVSSHardwareProvider.log.
Jan 10 12:28:35.820 1DF4 I Pure.Logger Version: 1.6.0. BUILD: .16.16 (64-bit Release). OS: 6.2
Jan 10 12:28:35.835 1DF4 I Pure.PureProvider Service Starting : ProcessID=00002370. Running As: SYSTEM on Machine:SERVER05
Jan 10 12:28:35.851 1DF4 I Pure.VssProvider ENTERING : ParseCommandLine()
Jan 10 12:28:35.866 1DF4 I Pure.VssProvider ENTERING: RegisterPureProvider()
Jan 10 12:28:35.945 1DF4 I Pure.VssProvider Provider registration succeeded
Jan 10 12:28:35.960 1DF4 I Pure.VssProvider EXITING: RegisterPureProvider()
Jan 10 12:28:35.960 1DF4 I Pure.VssProvider ENTERING : AddEventSource()
Jan 10 12:28:35.976 1DF4 I Pure.VssProvider EXITING: AddEventSource()
Jan 10 12:28:35.991 1DF4 I Pure.Logger <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ==== ( LOG SESSION ENDED ) ==== >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Jan 10 13:21:14.249 1D98 I Pure.Logger >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ==== ( LOG SESSION STARTED ) ==== <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Jan 10 13:21:14.264 1D98 I Pure.Logger LogLevel: 4, LogFileName: C:\ProgramData\PureStorage\PureVSSHardwareProvider.log.
Jan 10 13:21:14.280 1D98 I Pure.Logger Version: 1.6.0. BUILD: .16.16 (64-bit Release). OS: 6.2
Jan 10 13:21:14.280 1D98 I Pure.PureProvider Service Starting : ProcessID=00001d34. Running As: SYSTEM on Machine:SERVER05
Jan 10 13:21:14.295 1D98 I Pure.VssProvider ENTERING : ParseCommandLine()
Jan 10 13:21:14.295 1D98 I Pure.VssProvider EXITING : ParseCommandLine()
Jan 10 13:21:14.311 1DE0 D Pure.RestLibrary ENTERING : PureLibrary::PureLibrary()

Additional PureProvider Registry Values

There are three registry keys and values used by the PureProvider service that have not been discussed. The below example shows all of the settings with a focus on RestTimeout, RestVersion and Build. 

Windows Registry Editor Version 5.00 
[HKEY_LOCAL_MACHINE\SOFTWARE\PureStorage\VSS\Provider]
"Version"="1.6.0"
"Build"="N/A"
"RestVersion"="1.1"
"RestTimeout"="10"
"Logging"="Enabled"
"LogLevel"="4"
"EradicateVolumes"=dword:00000000
"DeleteSnaps"=dword:00000001
 [HKEY_LOCAL_MACHINE\SOFTWARE\PureStorage\VSS\Provider\TBL2-M20]
"APIKey"=hex:<APIKey>
"URL"="https://10.0.0.1"
  • RestTimeout = 10 (default) -- Serves as the HTTP request timeout that the PureProvider will wait after issuing an HTTP call to the REST services on the FlashArray before declaring a timeout error. 
  • RestVersion = "1.1" -- This is the version of the FlashArray REST API that the PureProvider uses to communicate. If a FlashArray only communicated using REST 1.4 for example the PureProvider would not work. 
  • Build = N/A -- This represents the build number and date of the PureProvider. Note -- There is a bug causing this registry value to show N/A. This registry value is not used for any other purpose so does not cause any issues with the PureProvider service. 

Video Demonstration

The video below shows how the registry settings can be used to customize how shadow copies are managed on the FlashArray.