January vCenter and ESXi 5.5 Patch Releases: Critical CBT Bug Fixed

VMware released new builds for vCenter and ESXi 5.5 today. The builds contain mostly bug fixes, but I wanted to point out one fix that had affected those who use CBT as part of their VM backup strategy. Veeam users where initially affected by the bug…though Veeam released a work around in subsequent Veeam 8 builds this ESXi patch officially fixes the issue.

When you use backup software that uses the Virtual Disk Development Kit (VDDK) API call QueryChangedDiskAreas(), the list of allocated disk sectors returned might be incorrect and incremental backups might appear to be corrupt or missing. A message similar to the following is written to vmware.log:

DISKLIB-CTK: Resized change tracking block size from XXX to YYY

For more information, see KB 2090639.

There are also a couple of other interesting fixes in the build:

  • Storage vMotion of thin provisioned virtual machine disk (VMDK) takes longer than the Lazy Zeroed Thick (LZT) disk.
  • When a quiesced snapshot is created on a Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, Windows Server 2008 R2, or a Windows Server 2012 virtual machine, duplicate disks might be created in the virtual machine.
  • On an ESXi 5.5 host, the NFS volumes might not restore after the host reboots. This issue occurs if there is a delay in resolving the NFS server host name to IP address.

For those who backup VMs with disks larger that 128GB I would be looking to deploy this patch ASAP.

4 comments

  • What I cannot figure out – if your hosts are patched (in my case, ESXi 5.5) with the latest and greatest, do you still have to shut down the VMs that are displaying that issue during backups? The VMware KB is kind of cryptic…it says the workaround requires a power down of the VM. Not sure if this is required if you patch your hosts with the fix.

  • Here’s a response from our support manager, Ben:

    I think there is some minor confusion here by way of the poster. We are likely speaking on two separate CBT problems. The issue that was fixed by VMware’s patch was not one that would be evident as part of any error message. You would just have bad data in the backup(s), with successful job passes in Veeam. This does not require a reboot of the VM(s) post patch.

    If ytsejamer1 is still getting error messages about CBT in his job(s) then that is likely pointing to an issue with the CBT files themselves, which requires a CBT ‘reset’ (http://www.veeam.com/kb1113). Note that the primary method for resetting CBT does request a VM to be powered off, configured, then powered back on. There is another method to do this that uses Powershell but does not require a reboot. The former method is what VMware is referring to in its KB in regards to a VM reboot.

  • Thanks to both of you for your comments! We’re no longer receiving the error notification on the backup job (we’re backing up with CommVault), but according to VMware Support, the workaround should still be implemented on the VM to ensure that the CBT settings are reset – even after the hosts are patched.

    It is possible that CV was resetting the CBT on the VM during its incremental backup run and hence why it’s not giving me an error now that my hosts are patched. To be safe, I’m going to go through with powering them down and back up. They’re not major critical systems that can’t go down early in the morning or whatever.