How-to Reclaim Space After Deleting Files on Host Systems
Deleting a file on a host file system does not automatically free up block space on storage systems. To do this, you need to trigger SCSI_UNMAP in order to notify the storage vendor that space should be freed up. This is required for many storage solutions, including Pure Storage FlashArrays. An UNMAP allows an operating system to inform an SSD which blocks of data are no longer considered in use and so can be wiped. This process informs the FlashArray what space can be reclaimed and ensures that you are not consuming unnecessary space. For example, if a FlashArray consistently reports more data on a volume than host file-system utilities report for the file system, UNMAP is probably not being run regularly enough.
Host Operating System
File System Support
ESX 5.0 U1, 5.1, 5.5, 60, 6.5
|VMFS-5, VMFS-3(1) support||Yes|
(1) The VAAI primitive UNMAP only works where the partition offset is a multiple of 1 MB. Operation will fail on misaligned VMFS-3 datastores or misaligned VMFS-3 datastores that have been converted to VMFS-5.
VMware Guest OS Space Reclamation
Many file systems cannot trigger the SCSI_UNMAP primitive like UFS, JFS, JFS2, NTFS-w2k8, etc. Additionally, some of the recent ones that can perform SCSI_UNAMP natively (Windows 2008 R2, Windows 2012, EXT4, etc) still have trouble doing so when operating within a guest VM, unless you are using Raw Device Mapping (RDM).
VMware VMFS - UNMAP Outside of the Guest
To reclaim unused storage blocks on a VMFS datastore, use the following VMware processes:
Space Reclamation in VMWare ESXi 5.5
NOTE: As noted in the VMware Best Practices page, make sure the UNMAP block count is set to 1% of the VMFS free space.
Space Reclamation in VMWare ESXi 5.0/5.1
NOTE: When running
vmkfstoolsin 5.0/5.1 the process creates a balloon file that consumes space. We recommend that you reclaim 20% at a time using this command and do not process more than 2TB of reclaimed space at a time.
Reclaiming Space within the Guest
vSphere 5.0, 5.1, and 5.5
In order to reclaim space inside of a virtual disk, we recommend that you use of a zeroing tool inside of the guest. See the following KB articles for reclaiming space:
Since ESXi virtualized the SCSI layer, even if operating systems attempted to send UNMAP down to the virtual disk, it would not perform any function, nor would it make it to the FlashArray. So any native OS attempts to send an UNMAP (such as discards in Linux) will not work.
With the introduction of vSphere 6.0, the
EnableBlockDelete option was changed so that it enables VMFS block delete when UNMAP is issued from a guest operating system. That is, if you enable this option, ESXi will now permits guest operating systems to issue UNMAP commands to a virtual disk to be translated down to the FlashArray so that the space can be reclaimed. Beyond that, if the virtual disk is thin, it will be reduced in size by the amount of space that was reclaimed.
Note: Apparently only thin virtual disks support this guest-UNMAP functionality.
For in-depth information, we strongly recommend reading Direct Guest OS UNMAP in vSphere 6.0.
The release of vSphere 6.5 supports Linux-based In-Guest UNMAP.
Executing In-Guest UNMAP/TRIM with Linux
There are a three options for reclaiming space as guest in vSphere 6.5 using Linux:
- Mount the file system with the discard option:
mount -t ext4 -o discard /dev/sdc /mnt/UNMAP.This directs the system to automatically issue UNMAP when files are deleted from the file system.
- Run the command
sg_unmap. It allows you to run UNMAP on specific Logical Block Addresses (LBAs).
- Run the command
fstrim. It reclaims dead space across a directory or entire file system on demand. It does not require the
discardoption to be set, but is compatible with file systems that do have it enabled.
Trusted commentary on this subject recommends the
discard option is the best option. The command
sg_unmap requires extensive manual work and familiarity with logical block address placement and the command
fstrim can run into alignment issues.
For in-depth information, we strongly recommend reading What's New in ESXI 6.5 Storage Part I: UNMAP.