Troubleshooting disk latency

I was asked to troubleshoot a VM. SCOM was reporting the following message: "Logical disk transfer (reads and writes) latency is too high - The threshold for the Logical Disk\Avg.". Also, jobs (this is a SQL server) took longer to run (almost twice a long). Now, this seems like an issue with disk I/O: the path to storage endpoint (SAN), incorrect HBA configuration or perhaps congestion on the storage array - none of them are true. Actually, it doesn't have anything to do with disk latency!

CPU|Contention (%)

So I opened up VROPS. Looked at the CPU | Contention. These are high numbers. The contention would suggest this VM is fighting for resources, cannot get it, and would be able to perform better. So I checked the host and the VM's configuration. It's a large VM, with 14 vCPU's, on a 16 core host (32 logical CPU's, hyperthreading is enabled). BUT IT'S THE ONLY VM ON THIS HOST!!!

CPU|Demand (%)

Next, I took a look at the CPU | Demand. Demand was high, but while demand was high, the contention was low! So this confirmed the conflict with other VMs issue - there is none. Something else is going on here! I remembered an issue with C-State throttling on Intel CPUs. If a VM is low on CPU Demand, the physical CPU doesn't require to run to full clock speeds. However, once a VM demands more CPU resources, the CPU needs to switch to a higher rate, and the duration to switch to that faster state introduces latency. It's called %C_Lat and can be viewed in ESXTOP. %C_Lat is also taken into account when calculating CPU|Contention (%), so we need to check the performance setting in the BIOS on the host.

​So I opened up OneView, checked the profile and by default it's configured to 'Balanced Power and Performance'. I've switched it to 'Maximum Performance'. Have a look at the screenshot below, and see what changes:

Awesome... I've moved the VM back to it's original server. And we should see improvements...