There is another NFS bug hidden in the latest ESXi 5.x releases…while not as severe as the 5.5 Update 1 NFS Bug it’s been the cause of increased Virtual Disk Latency and Overall Poor VM performance across a couple of the environments I manage.

The VMwareKB Article referencing the bug can be found here:

Symptoms

  • When virtual machines run on the same host and use the same NFS datastore and the IOPS limit is set on at least one virtual machine, you experience high virtual disks latency and low virtual disk performance.
  • Even when a different IOPS limit is specified for different virtual machines, IOPS limit on all virtual machines is set to the lowest value assigned to a virtual machine in the NFS datastore.
  • You do not experience the virtual disks latency and low virtual disk performance issues when virtual machines:
    • Reside on the VMFS datastore.
    • Run on the same ESXi host, but are placed on the NFS datastore in which none of the virtual machines have an IOPS limit set.
    • Run on the same ESXi host, but are placed on different NFS datastores that do not share the client connection
    • Run on different ESXi hosts, but are placed on the same NFS datastore

So in a nutshell if you have a large NFS Datastore with many VMs with IOPS Limits to prevent against Noisy Neighbours you may experience, what looks like unexplained VM Latency and overall bad performance.

Some proof of this bug can be seen in the screenshot below, where a VM residing on an NFS datastore with IOPS limits applied was exhibiting high Disk Command Latency. It had an IOPS limit of 1000 and wasn’t constrained by setting…yet it had Disk Command Latency in the 100s. The red arrow represents the point at which we migrated the VM to another host. Straight away the latency dropped and the VM returned to expected performance levels. This resembles a couple of the symptoms above…

NFS_IOPS_BUG_1

We also experimented by removing all IOPS Disk Limits on a subset of NFS datastores and looked at the affect on overall latency that had. The results where almost instant as you can see below. Both peaks represent us removing and then adding back in the IOPS Limits.

NFS_IOPS_BUG_2

As we are running ESXi 5.1 hosts we applied the latest patch release (ESXi510-201406001) which includes ESXi510-201404401-BG that addresses the bug in 5.1. After applying that we did see a noticeable drop in overall latency on the previously affected NFS datastores.

Annoyingly there is no available patch for ESXi 5.5, but I have been told by VMware Support that it’s due as of Update 2 for 5.5..no time frame on that release though.

One thing I’m interested in comments on is around VM Virtual Disk IOPS limits… Designed to lesson the impact of noisy neighbours…but what overall affect can it have…or does it have to LUN based latency? Or does it self contain latency to the VM thats restricted? I assume it works differently to SIOC and doesn’t choke disk queue depth? The IOPS limit simply puts the breaks on any Virtual Disk IO?