One thing that should always be done when using 1Gb iSCSI is setting up Multipathing, and in all version of vSphere they have included NMP or Native Multi Pathing. With NMP comes a path selection method called Round Robin which as with anything else with that name means that IO is sent down each path, one at a time. By default, VMware has set a limit of 1000 IO’s to travel down each path before switching to the next path. However, by changing that default down to 1 IO before switching paths we can sometimes achieve much greater throughput because we can more effectively utilize our links.
There has been much debate over whether this is really worth it or not, so I thought why not try it out and see what effects it would have. The test environment that I used included the following hardware:
- Hp P4300 (Lefthand) SAN with 2 Storage Nodes
- Hp DL380 G7 Server with 2 – 6 core Intel 5675 CPUs (3.07Ghz ;)) and 72Gb of RAM
- Cisco 3560G switch
I setup the HP P4300 SAN with 802.3ad bonded links to the switch and turned on channel groups in passive mode on those links. As for the server, I loaded ESXi 5.0 and used two VMKernel interfaces with two NICs… I also did not use the software VMware software iSCSI initiator, but instead used the BroadCom iSCSI offload initiators because of the ease of setting them up in ESXi 5.0.
After mounting my LUN and letting it format VMFS5 I set the path selection policy to Round Robin. I then Installed a Windows 2008 R2 VM and followed the instructions from VMKtree.org for running IOmeter (just like I have for the other SAN tests).
Now lets see what turning the IO limit down to 1 IO per path per turn gets us. Below you can see that we can now achieve 207MBps on Read throughput and our RealLife IOps is now at 2091 IOps. Plus every other number went up as well!
This is an increase of 60% on the Maximum Read throughput and a 30% increase on the number of IOps!
So how do you change the limit ?
Setting the limit to a ower value is described in the HP EVA best practices guide, but I could not get their command to work.
The command for ESXi5 is:
esxcli storage nmp psp roundrobin deviceconfig set -d naa.devicename --iops 1 --type iops
and to get the device name you could use:
esxcli storage nmp device list | grep naa.600
If you want to check out their guide it can be found here: