PSOD Warning: IBM HS23 Blades, Emulex 10GB Network Adaptors and ESXi 5.1

This is a quick informational post to warn anyone running an Emulex based 10GbE Converged Ethernet adapter in an IBM Blade Center with HS23 Series Blade servers with ESXi 5.x … Fix is below.

You are at risk of simultaneous host failure via theÂ VMware PSOD under unspecified conditions if you do not update to the latest combination FW from IBM (Emulex) and be2net driver from VMware.

While the trigger is still unclear (some suggestion linking guest based operations in combination with DVS environments) …searching online for PSOD + ESX + Emulex returns many results…it seems to be an issue that’s been around from release. And while it’s not limited to IBM Blades…there are reports of this happening onÂ HP/EmulexÂ platforms, this post is specific to the IBM HS23 Blades.

The below KB links suggest issues in FC or FCoE SAN management stacks, but in our situation we where running iSCSI software initiators with the following revision of Emulex FW and be2net driver. It must be noted that this combination had been stable for 7 months.

Emulex Corporation OneConnect 10Gb NIC (be3)

1	Emulex Corporation OneConnect 10Gb NIC (be3)

Vendor Id.......................................0x19a2

1	Vendor Id.......................................0x19a2

Device Id.......................................0x0710

1	Device Id.......................................0x0710

Sub-Vendor Id...................................0x10df

1	Sub-Vendor Id...................................0x10df

Sub-Device Id...................................0xe70a

1	Sub-Device Id...................................0xe70a

Current driver version: be2net 4.2.327.0 Current firmware version: 4.1.455.6

1	Current driver version: be2net 4.2.327.0 Current firmware version: 4.1.455.6

This Emulex KB suggests that the condition is triggered whenÂ frames sourced from the SAN management are aborted. Once the abort occurs the conditions for a PSOD exist.Â One particular example of an aborted frame is if the target does not respond to a request. For example the management application sends a read to a controller LUN but the controller LUN does not respond. The driver will then send out an abort for this particular read command.

VMware KB2031192:Â As of the 6th of May, VMware is suggesting that you update the Emulex FW to at least version 4.6.146.62. We struggled with IBM support to get access to an updated FWÂ 4.6.166.9 (direct download to FW Deployment ISO here) and VMware support also suggested upgrading the be2net driver to at least 4.6.142.10 which was released on the 6th of June (direct downloadÂ here). We later confirmed with VMware that running the driver version slightly behind the FW version was ok.

At this point in time our systems have been stable since applying the updates, but the is the second occurrence of the PSOD since the IBM Blade System has gone into production (within the last 7-8 months) and confidence in the platform has taken a hit. We, along with other users of this Emulex based platform will be hoping this is the last round of issues with this card…I struggle to understand how such a serious issue can be allowed to manifest. But this combination seems problematic!

VIRTUALIZATION IS LIFE!