Skip to main content
Pure Technical Services

VASA and vVols Related Fixes by ESXi Release

One of the tricky problems observed with using VMware Virtual Volumes (vVols) has been knowing what issues with vVols or VASA (vSphere API for Storage Awareness) are fixed in which releases of ESXi.  VMware provides fairly detailed release notes for each ESXi patch and update releases.  This KB's goal is to compile a list of all vVols or VASA related fixes by ESXi Release.

ESXi 6.7

VMware ESXi 6.7 - Patch Release ESXi670-202004002 - Build 16075168

  • PR 2458201: Some vSphere Virtual Volumes snapshot objects might not get a virtual machine UUID metadata tag

    During snapshot operations, especially a fast sequence of creating and deleting snapshots, a refresh of the virtual machine configuration might start prematurely. This might cause incorrect updates of the vSphere Virtual Volumes metadata. As a result, some vSphere Virtual Volumes objects, which are part of a newly created snapshot, might remain untagged or get tagged and then untagged with the virtual machine UUID.

    This issue is resolved in this release.
     
  • PR 2424969: If the first attempt of an ESXi host to contact a VASA provider fails, vSphere Virtual Volumes datastores might remain inaccessible

    If a VASA provider is not reachable or not responding at the time an ESXi host boots up and tries to mount vSphere Virtual Volumes datastores, the mount operation fails. However, if after some time a VASA provider is available, the ESXi host does not attempt to reconnect to a provider and datastores remain inaccessible.

    This issue is resolved in this release.
     
  • PR 2424363: During rebind operations, I/Os might fail with NOT_BOUND error

    During rebind operations, the source protocol endpoint of a virtual volume might start failing I/Os with a NOT_BOUND error even when the target protocol endpoint is busy. If the target protocol endpoint is in WAIT_RBZ state and returns a status PE_NOT_READY, the source protocol endpoint must retry the I/Os instead of failing them.

    This issue is resolved in this release. With the fix, the upstream relays a BUSY status to the virtual SCSI disk (vSCSI) and the ESXi host operating system to ensure a retry of the I/O.
     
  • PR 2449462: You might not be able to mount a Virtual Volumes storage container due to a stale mount point

    If the mount point was busy and a previous unmount operation has failed silently, attempts to mount a Virtual Volumes storage container might fail with an error that the container already exists.

    This issue is resolved in this release.
     
  • PR 2467765: Upon failure to bind volumes to protocol endpoint LUNs on an ESXi host, virtual machines on vSphere Virtual Volumes might become inaccessible

    If a VASA provider fails to register protocol endpoint IDs discovered on an ESXi host, virtual machines on vSphere Virtual Volumes datastores on this host might become inaccessible. You might see an error similar to vim.fault.CannotCreateFile. A possible reason for failing to register protocol endpoint IDs from an ESXi host is that the SetPEContext() request to the VASA provider fails for some reason. This results in failing any subsequent request for binding virtual volumes, and losing accessibility to data and virtual machines on vSphere Virtual Volumes datastores.

    This issue is resolved in this release. The fix is to reschedule SetPEContext calls to the VASA provider if a SetPEContext() request on a VASA provider fails. This fix allows the ESXi host eventually to register discovered protocol endpoint IDs and ensures that volumes on vSphere Virtual Volumes datastores remain accessible.
     
  • PR 2429068: Virtual machines might become inaccessible due to wrongly assigned second level LUN IDs (SLLID)

    The nfnic driver might intermittently assign wrong SLLID to virtual machines and as a result, Windows and Linux virtual machines might become inaccessible.

    This issue is resolved in this release. Make sure that you upgrade the nfnic driver to version 4.0.0.44.

VMware ESXi 6.7 - Patch Release ESXi670-201912001 - Build 15160138

  • PR 2423301: After you revert a virtual machine to a snapshot, change block tracking (CBT) data might be corrupted

    When reverting a virtual machine that has CBT enabled to a snapshot which is not a memory snapshot, and if you use the QueryChangedDiskAreas() API call, you might see an InvalidArgument error.

    This issue is resolved in this release. With ESXi670-201912001, the output of the QuerychangedDiskAreas () call changes to FileFault and adds the message Change tracking is not active on the disk <disk_path> to provide more details on the issue.

    With the fix, you must power on or reconfigure the virtual machine to enable CBT after reverting it to a snapshot and then take a snapshot to make a full backup.

    To reconfigure the virtual machine, you must complete the following steps:
    1. In the Managed Object Browser graphical interface, run ReconfigVM_Task by using an url such as https://<vc or host ip>/mob/?moid=<the virtual machine Managed Object ID>&method=reconfigure.
    2. In the <spec> tag, add <ChangeTrackingEnabled>true</ChangeTrackingEnabled>.
    3. Click Invoke Method.
  • PR 2419339: ESXi hosts might fail with a purple diagnostic screen during power off or deletion of multiple virtual machines on a vSphere Virtual Volumes datastore

    ESXi hosts might fail with a purple diagnostic screen during power off or deletion of multiple virtual machines on a vSphere Virtual Volumes datastore. You see a message indicating a PF Exception 14 on the screen. This issue might affect multiple hosts.

    This issue is resolved in this release.
     
  • PR 2432530: You cannot use a batch mode to unbind VMware vSphere Virtual Volumes

    ESXi670-201912001 implements the UnbindVirtualVolumes () method in batch mode to unbind VMware vSphere Virtual Volumes. Previously, unbinding took one connection per vSphere Virtual Volume. This sometimes led to consuming all available connections to a vStorage APIs for Storage Awareness (VASA) provider and delayed response from or completely failed other API calls.

    This issue is resolved in this release.
     
  • PR 2385716: Virtual machines with Changed Block Tracking (CBT) enabled might report long waiting times during snapshot creation

    Virtual machines with CBT enabled might report long waiting times during snapshot creation due to the 8 KB buffer used for CBT file copying. With this fix, the buffer size is increased to 1 MB to overcome multiple reads and writes of a large CBT file copy, and reduce waiting time.

    This issue is resolved in this release.

VMware ESXi 6.7 Update 3 - Build 14320388

  • PR 2312215: A virtual machine with VMDK files backed by vSphere Virtual Volumes might fail to power on when you revert it to a snapshot

    This issue is specific to vSphere Virtual Volumes datastores when a VMDK file is assigned to different SCSI targets across snapshots. The lock file of the VMDK is reassigned across different snapshots and might be incorrectly deleted when you revert the virtual machine to a snapshot. Due to the missing lock file, the disk does not open, and the virtual machine fails to power on.

    This issue is resolved in this release.

  • PR 2363202: The monitoring services show that the virtual machines on a vSphere Virtual Volumes datastore are in a critical state

    In the vSphere Web Client, incorrect Read or Write latency is displayed for the performance graphs of the vSphere Virtual Volumes datastores at a virtual machine level. As a result, the monitoring service shows that the virtual machines are in a critical state.

    This issue is resolved in this release.

  • PR 2402409: Virtual machines with enabled Changed Block Tracking (CBT) might fail while a snapshot is created due to lack of allocated memory for the CBT bit map

    While a snapshot is being created, a virtual machine might power off and fail with an error similar to:
    2019-01-01T01:23:40.047Z| vcpu-0| I125: DISKLIB-CTK : Failed to mmap change bitmap of size 167936: Cannot allocate memory.
    2019-01-01T01:23:40.217Z| vcpu-0| I125: DISKLIB-LIB_BLOCKTRACK : Could not open change tracker /vmfs/volumes/DATASTORE_UUID/VM_NAME/VM_NAME_1-ctk.vmdk: Not enough memory for change tracking.

    The error is a result of lack of allocated memory for the CBT bit map.

    This issue is resolved in this release.

VMware ESXi 6.7 Update 2 - Build 13006603

  • PR 2250697: Windows Server Failover Cluster validation might fail if you configure Virtual Volumes with a Round Robin path policy

    If during the Windows Server Failover Cluster setup you change the default path policy from Fixed or Most Recently Used to Round Robin, the I/O of the cluster might fail and the cluster might stop responding.

    This issue is resolved in this release.

  • PR 2279897: Creating a snapshot of a virtual machine might fail due to a null VvolId parameter

    If a vSphere API for Storage Awareness provider modifies the vSphere Virtual Volumes policy unattended, a null VvolID parameter might update the vSphere Virtual Volumes metadata. This results in a VASA call with a null VvoId parameter and a failure when creating a virtual machine snapshot.

    This issue is resolved in this release. The fix handles the policy modification failure and prevents the null VvolId parameter.

  • PR 2227623: Parallel cloning of multiple virtual machines on a vSphere Virtual Volumes datastore might fail with an error message for failed file creation

    If a call from a vSphere API for Storage Awareness provider fails due to all connections to the virtual provider being busy, operations for parallel cloning of multiple virtual machines on a vSphere Virtual Volumes datastore might become unresponsive or fail with an error message similar to Cannot complete file creation operation.

    This issue is resolved in this release.

  • PR 2268826: An ESXi host might fail with a purple diagnostic screen when the VMware APIs for Storage Awareness (VASA) provider sends a rebind request to switch the protocol endpoint for a vSphere Virtual Volume

    When the VASA provider sends a rebind request to an ESXi host to switch the binding for a particular vSphere Virtual Volume driver, the ESXi host might switch the protocol endpoint and other resources to change the binding without any I/O disturbance. As a result, the ESXi host might fail with a purple diagnostic screen.

    This issue is resolved in this release.

VMware ESXi 6.7 Update 1 - Build 10302608

  • PR 2039186: VMware vSphere Virtual Volumes metadata might not be updated with associated virtual machines and make virtual disk containers untraceable

    vSphere Virtual Volumes set with VMW_VVolType metadata key Other and VMW_VVolTypeHint metadata key Sidecar might not get VMW_VmID metadata key to the associated virtual machines and cannot be tracked by using IDs.

    This issue is resolved in this release.

  • PR 2119610: Migration of a virtual machine with a Filesystem Device Switch (FDS) on a vSphere Virtual Volumes datastore by using VMware vSphere vMotion might cause multiple issues

    If you use vSphere vMotion to migrate a virtual machine with file device filters from a vSphere Virtual Volumes datastore to another host, and the virtual machine has either of the Changed Block Tracking (CBT), VMware vSphere Flash Read Cache (VFRC) or I/O filters enabled, the migration might cause issues with any of the features. During the migration, the file device filters might not be correctly transferred to the host. As a result, you might see corrupted incremental backups in CBT, performance degradation of VFRC and cache I/O filters, corrupted replication I/O filters, and disk corruption, when cache I/O filters are configured in write-back mode. You might also see issues with the virtual machine encryption.

    Тhis issue is resolved in this release.

  • PR 2145089: vSphere Virtual Volumes might become unresponsive if an API for Storage Awareness (VASA) provider loses binding information from the database

    vSphere Virtual Volumes might become unresponsive due to an infinite loop that loads CPUs at 100% if a VASA provider loses binding information from the database. Hostd might also stop responding. You might see a fatal error message.

    This issue is resolved in this release. This fix prevents infinite loops in case of database binding failures.

  • PR 2146206: vSphere Virtual Volumes metadata might not be available to storage array vendor software

    vSphere Virtual Volumes metadata might be available only when a virtual machine starts running. As result, storage array vendor software might fail to apply policies that impact the optimal layout of volumes during regular use and after a failover.

    This issue is resolved in this release. This fix makes vSphere Virtual Volumes metadata available at the time vSphere Virtual Volumes are configured, not when a virtual machine starts running.

ESXi 6.5

VMware ESXi 6.5 - Patch Release ESXi650-201912002 - Build 15256549

  • PR 2271176: ESXi hosts might fail with a purple diagnostic screen during power off or deletion of multiple virtual machines on a vSphere Virtual Volumes datastore

    ESXi hosts might fail with a purple diagnostic screen during power off or deletion of multiple virtual machines on a vSphere Virtual Volumes datastore. You see a message indicating a PF Exception 14 on the screen. This issue might affect multiple hosts.

    This issue is resolved in this release.
     
  • PR 2423302: After you revert a virtual machine to a snapshot, change block tracking (CBT) data might be corrupted

    When reverting a virtual machine on which CBT is enabled to a snapshot which is not a memory snapshot, and if you use the QueryChangedDiskAreas() API call, you might see an InvalidArgument error.

    This issue is resolved in this release. With ESXi650-201912002, the output of the QuerychangedDiskAreas () call changes to FileFault and adds the message Change tracking is not active on the disk <disk_path> to provide more details on the issue.

    With the fix, you must power on or reconfigure the virtual machine to enable CBT after reverting it to a snapshot and then take a snapshot to make a full backup.
     
  • PR 2385716: Virtual machines with Changed Block Tracking (CBT) enabled might report long waiting times during snapshot creation

    Virtual machines with CBT enabled might report long waiting times during snapshot creation due to the 8 KB buffer used for CBT file copying. With this fix, the buffer size is increased to 1 MB to overcome multiple reads and writes of a large CBT file copy, and reduce waiting time.

    This issue is resolved in this release.

VMware ESXi 6.5 Update 3 - Build 13932383

  • PR 2282080: Creating a snapshot of a virtual machine from a virtual volume datastore might fail due to a null VVolId parameter

    If a vSphere API for Storage Awareness provider modifies the vSphere Virtual Volumes policy unattended, a null VVolId parameter might update the vSphere Virtual Volumes metadata. This results in a VASA call with a null VvolId parameter and a failure when creating a virtual machine snapshot.

    This issue is resolved in this release. The fix handles the policy modification failure and prevents the null VVolId parameter.

  • PR 2265828: A virtual machine with VMDK files backed by vSphere Virtual Volumes might fail to power on when you revert it to a snapshot

    This issue is specific to vSphere Virtual Volumes datastores when a VMDK file is assigned to different SCSI targets across snapshots. The lock file of the VMDK is reassigned across different snapshots and might be incorrectly deleted when you revert the virtual machine to a snapshot. Due to the missing lock file, the disk does not open, and the virtual machine fails to power on.

    This issue is resolved in this release.

  • PR 2113782: vSphere Virtual Volumes datastore might become inaccessible if you change the vCenter Server instance or refresh the CA certificate

    vSphere Virtual Volume datastore uses VMware CA signed certificate to communicate with VASA providers. When the vCenter Server instance or the CA certificate changes, vCenter Server imports the new vCenter Server CA signed certificate and the vSphere Virtual Volume datastore gets SSL reset signal which might not be triggered. As a result, the communication between vSphere Virtual Volume datastore and VASA providers might fail and the vSphere Virtual Volume datastore might become inaccessible.

    This issue is resolved in this release.

  • PR 2278591: Cloning multiple virtual machines simultaneously on vSphere Virtual Volumes might stop responding

    When you clone multiple virtual machines simultaneously from vSphere on a vSphere Virtual Volumes datastore, a setPEContext VASA API call is issued. If all connections to the VASA Providers are busy or unavailable at the time of issuing the setPEContext API call, the call might fail and the cloning process stops responding.

    This issue is resolved in this release.

VMware ESXi 6.5, Patch Release ESXi650-201811002 - Build 10884925

  • PR 2119609: Migration of a virtual machine with a Filesystem Device Switch (FDS) on a VMware vSphere Virtual Volumes datastore by using VMware vSphere vMotion might cause multiple issues

    If you use vSphere vMotion to migrate a virtual machine with file device filters from a vSphere Virtual Volumes datastore to another host, and the virtual machine has either of the Changed Block Tracking (CBT), VMware vSphere Flash Read Cache (VFRC), or IO filters enabled, the migration might cause issues with any of the features. During the migration, the file device filters might not be correctly transferred to the host. As a result, you might see corrupted incremental backups in CBT, performance degradation of VFRC and cache IO filters, corrupted replication IO filters and disk corruption, when cache IO filters are configured in write-back mode. You might also see issues with the virtual machine encryption.

    Тhis issue is resolved in this release.

  • PR 2142767: VMware vSphere Virtual Volumes might become unresponsive if a vSphere API for Storage Awareness provider loses binding information from its database

    vSphere Virtual Volumes might become unresponsive due to an infinite loop that loads CPUs at 100% if a vSphere API for Storage Awareness provider loses binding information from its database. Hostd might also stop responding. You might see a fatal error message. This fix prevents infinite loops in case of database binding failures.

    This issue is resolved in this release.