After reading Duncan's Blog posts about Resource Pools misconfigurations, I couldn't resist writing a new article about it to emphasize the problem a bit further. I've done this in Prezi to make it more interactive. Be sure to watch it fullscreen - it shows the pictures a bit better.

Today not really a tech-post, but more or less a discussion starter... And the question is: "Are physical limits insignificant?"

The reason I've come up with this was my action last night at one of my customers. We needed to re-patch some ESX servers and since we didn't want any downtime, we've put the hosts in maintenance mode. Hardware was Blades with 16 ESX servers each. While putting all those hosts in maintenance mode, waiting 'till every VM was VMotionned, I kept time. To fully migrate the whole enclosure, it took us almost: 45 minutes! That's 45 minutes for migrating all those VMs! And the environment isn't that special, having about 20:1 / 30:1 ratios.

Anyway, I see a lot of trends where you have BIGGER, FASTER, LARGER hardware. Especially looking at Cisco UCS, which allows for a stunning 384GB of memory. Can you imagine how long it would take to VMotion that? Besides of that, how high is the impact you loose 1 piece of hardware, with all those VMs (or better yet, loose an enclosure, with 4, 8 or 16 blades)?

On the other hand, how high is the risk that a hardware piece is failing, using redundant power supplies, raid solutions, etc.? To be honest - I've seen large environments and different brands of hardware, and the only thing that breaks once in a while, is a management controller, HDD (which is in raid set anyway), or a small fan which can be easily replaced. So while the impact is very high, the risk is very low, so is it OK for us to get those gigantic machines and have incredible virtualization ratios?

Well, I think other things come into play when we do. One of the things is 'Human error'. Ever shutdown or put a host in standby without putting that host in maintenance mode first? Ever started an firmware update and see all hosts shutdown one-by-one. Ever pulled out the 'active' cable while you where thinking it was the standby one? Well, I can't say I did those things - but I've seen it happen. And an other thing - perhaps there is an SLA where you need to be able to empty a whole enclosure within a specific time range? OK, OK, that's a bit out of line, or is it? How long would it take to remediate ESX servers using Update Manager?

Anyway, lucky for us tweaking helps us a bit. While the maximum VMotions at a time is 2, you are able to tweak this to 12, so it would somewhat go faster. Check it out here on Boche his blog.

Cheers.

Today we tried to do a storage VMotion. For some reason it failed with the message: "A general system error occurred: Failed waiting for data. Error bad0007. Bad Parameter."

When looking at the '/var/log/vmkernel' log we see:

cpu0:1092)Migrate: vm 1093: 7383: Setting migration info ts = 1266331952655810, src ip = <hidden-ip> dest ip = <hidden-ip> Dest wid = 1812 using SHARED swap
cpu0:1092)World: vm 1587: 901: Starting world migSendHelper-1093 with flags 1
cpu0:1092)World: vm 1588: 901: Starting world migRecvHelper-1093 with flags 1
cpu1:1587)WARNING: MigrateNet: 309: 1266331952655810: 5-0x8021d80:Sent only 4088 of 4096 bytes of message data: Broken pipe
cpu1:1587)WARNING: Migrate: 6820: 1266331952655810: Couldn't send data for 7: Broken pipe
cpu1:1587)WARNING: Migrate: 1243: 1266331952655810: Failed: Broken pipe (0xbad0052) @0x9d5d9f

vmkping worked fine (to it's own ip, but also to another hosts). ping worked fine. Enough free space (vdf -h), no strange things in esxtop. Also enough memory free in Service Console. Even checked the '.vmx' file - no strange things. Then I did something which is easy to do: disable VMotion option on the VMkernel interface, and then enabled it again. Whoohoo - it worked.

I don't know what could cause this, but at least now you now how to solve it.

Today I've updated the popular removevmhba script to version 5.0. This version now includes the removal of the drivers in vSphere ESX 4.0 update 1 isos. Thanks to Dinny Davies who did excellent work again on finding a solution for removing them on vSphere ESX4 (he just beat me to it ). Check the original ESX 3.x.x version here, and the new ESX 4.x.x document here.

Go ahead, grab removevmhba from the downloads section and give it a try. It removes the drivers only during installation, so you don't need to bother disconnecting your SAN or zone out anything during installation (both Emulex and Qlogic - and also hardware initiated iSCSI adapters). It's much safer for a scripted installation of ESX using the UDA or EDA. After the installation you will have the drivers (since it is installed as a package) - so you will get connection back to your SAN.

Instructions

You can use an ESX server, but you need to have mkisofs installed and mkfs. So I just take a default Ubuntu installation or like Dinny, using a default CentOS installation. The script checks the requirements and quits with a message if they are not met.

Make sure you are logged in as root, and start the script with:

# removevmhba-updatenewbuildsv5.sh <ESX 3 or 4 iso>

The results should be:

Download

Download is for free, from here (I hope you enjoy it):

Download removevmhba

File Title: removevmhba (Details) File Type: sh File Version: 5.0 File Size: 18.34 Kb License: Downloads: 3986

I noticed a some tweets on Twitter. Eric Siebert has updated his Top 25 Blogs. I have to say congrats Duncan! You're at the top again, and it is by far the best resource on the web. Anyway, keeping up with the top 25 is a day's work - but it's worth it, so check it out here: http://vlp.vsphere-land.com/

Anyway, I did end up on place 50. Which is not bad, I've even found that someone voted #1 for me (and trust me, it wasn't me). So thank you very much for the guy (or girl) who voted #1 for me . Check out the results here.