Monthly Archives: March 2015

Quick Post: E1000 vs VMXNET3

There are countless posts out there comparing E1000s and VMXNET3 and why the VMXNET3 should (where possible) always be used for Windows VMs.

Last week I was provisioning a new Windows 2012 R2 VM to act as a Veeam Repository. For mass storage we have MD3200i’s presenting block storage over iSCSI. Going through the motions of a build that I’ve done countless times…I deployed the OS Template and then added to additional NICs to complete the VM build.

After mounting the iSCSI Volume I went to do some basic benchmarking and throughput testing… Crystal Disk Mark is great for basic Performance Testing. The initial results where underwhelming to say the least.

Fairly poor read results …with OK sequential writes. The MD3200i Disk Groups are capable of doing 100-130MB/s so I knew something wasn’t right. I initially was looking at the physical network to blame…but everything checked out. I then looked at the iSCSI MPIO setup and again…everything checked out. Looking back though the VM hardware I found that I had mistakenly added E1000 NICs for the two additional iSCSI networks. After removing those and reconfiguring them as VMXNET3 I reran the tests and got the results I expected (though writes where low possibly due to concurrent operations) and all was well

This was a quick public service announcement post to ensure VMXNET3 is used where possible. If you want to search through your environment for Windows VMs with E1000s…have a look at this post using a CloudPhysics Card…and remember!

NSX Bytes: 6.1.3 Upgrade and vSphere 6.0

Earlier today NSX-v 6.1.3 was released. This update brings vSphere 6.0 Support as well as bug fixes and a couple minor feature enhancements. 

There is a warning in the release notes:

NSX vSphere 6.1.3 is compatible with vSphere 6.0. However, the new vSphere features introduced in vSphere 6.0 have not been tested with NSX vSphere. These new vSphere features should not be used in environments where NSX vSphere is installed as they are unsupported. For a list of specific NSX vSphere limitations with respect to vSphere 6.0, see the VMware Knowledge Base article 2110197.

Read through the above KB and make sure you understand what can and can’t be done when running NSX-v on vSphere 6.0…there are some significant constraints in terms of using some of the new shiny bits of vSphere 6.0 …some of which I see as being potential matches made in heaven for NSX-v Hybrid Solutions with vSphere based Platforms.

  • While vSphere 6.0 adds support for vMotion over larger network distances (where the RTT is up to 150ms), this has not been tested in deployments that use NSX-v
  • Multi-vCenter vMotion
  • Deploying NSX-v components from a vSphere Multisite Content Library has not been tested and is not supported.
  • The following improvements introduced in vSphere 6.0 have not been tested with NSX-v: vSphere Fault Tolerance improvements and vSphere Replication improvements

Hopefully the next Major Release of NSX-v will take advantage of the new features above, but for now at least we have the ability to have NSX-v running with vCenter and ESXi 6.0.

With regards to the upgrade process, nothing has changed in regards to the steps listed in my previous NSX Upgrade Posts:

NSX components must be upgraded in the following order:

  1. NSX Manager
  2. NSX controller
  3. Clusters and Logical Switches
  4. NSX Edge and Guest Introspection

Build Numbers shown above and Below for Manager and Controllers

After NSX has been upgraded as above, you can Upgrade vCenter Server to 6.0 (if running the VCSA 5.5 Appliance read here for Upgrade Details) from there you can upgrade your ESXi hosts to 6.0…upon reboot the NSX Manager pushes the ESXi 6.0 VIBs to the hosts. When the hosts show reboot required on the Hosts and Clusters tab in the left hand side of the vSphere Web Client, reboot the hosts a second time and the NSX VIBs for ESXi 6.0 are now enabled.

vSphere 6.0 vCenter Server Appliance: Upgrading from 5.x

Like most VMware Junkies over the past 24 hours I’ve downloaded the vSphere 6.0 bits and had them ready and primed to deploy and discover all the new features. Given the nature of a .0 release (not withstanding Change Control) upgrading production systems won’t be happening any time soon…however I had an opportunity right away to upgrade the Zettagrid Lab VCSA 5.5 Appliance to the 6.0 Appliance.

No word if this upgrade is possible via the old school method shown above, however the basic run through of the upgrade process is that the new VCSA is deployed and then imports data and settings from the existing VCSA while living on temporary network settings. It then shuts down the old VSCA and assumes it’s IP Address.

Working through the Upgrade Process, you need to download VMware-VCSA-all-6.0.0-2562643.iso and mount the VCSA ISO (Thanks @grantorchard) and from there double click on vcsa-setup.htm.

If you haven’t installed the 6.0 Client Integration Plugin located in the \vcsa folder of the ISO close down the browser and run that.

Rerun the vcsa-setup.htm and you should get the Install/Upgrade page…for me in Chrome I have to allow an External Protocol Request.

From there you have the option to Install or Upgrade to VCSA 6.0. This is an Upgrade that was chosen at which point the installed kicks off and gives you some details on what version of the VCSA you can upgrade from.

Clicking on Ok we are now presented with the VCSA Deployment Menu. First step is to enter in a target ESXi Host…not a vCenter like I first did to get the error below!

Moving through the Steps, we enter in a Virtual Machine Name and then specify the Source Appliance details which again is all host based.

Set the Appliance Size

Select the Datastore and then move onto the Temporary Network Settings…the explanation of what’s happening can be seen in the description below

Once all the settings are reviewed click on finish and the installer does all the work. For me this took about 20-30 minutes…bearing in mind that this vCenter was not large so import times will vary. You do have the option to bring across all existing Statistics…checking that option will vary the install time depending on how much data you have.

NOTE: My install progress stalled on Start the vCenter Web Client but that was more likely due to me not following instructions. When you select your target ESXi Host it states that DRS should be switched to Manual during the install process…my assumption is that the the VCSA was brought up on a different host other than the one specified and the installer lost track of the VM and wasn’t able to confirm the completion.

Regardless of that, I was able to log into the Web Client and ensure all Services where in tact and that the upgrade was successful…however I did start to see some strange behaviours and login issues over the weekend. I rolled back and went through the upgrade again, this time ensuring DRS was set to Manual…and had a 100% successful outcome.

I now can enjoy the benefits of the Web Client and start to get a feel for vSphere 6.0.

NSX Host Preparation: Cluster Not Ready – Legacy vCNS VXLAN VIBs Installed

I came across a situation today while going through an NSX Setup and Configuration where I came across a Cluster under the Host Preparation Tab that was reporting a status of Not Ready – Resolve and listed some hosts as Ready and Not Ready…What made this strange was that this was a fresh deployment of NSX over the top of an existing vCNS Environment.

I decided to investigate at the host level and check to see why the NSX Manager thought these hosts where already prepared with the NSX Host Extensions. I SSH’ed to each host and checked to see if the NSX VIBs where present.

The Hosts marked Ready each had the esx-vxlan and esx-dvfilter-switch-security VIBs installed but not the esx-vsip VIB. The version number got me thinking about the vCNS Implementation of VXLAN and sure enough I found a preconfigured VXLAN Multicast config present by discovering an associated VXLAN, MTU and Teaming Policy set next to the Cluster on the Logical Network Preparation Tab.

My first point of call was to try to uninstall the VIBs on the Hosts and then upon reboot have NSX Manager reinstall the NSX extensions…however all that happened was the previous two VIBs got reinstalled, leaving the esx-vsip VIB still missing…Kind of unexpected…but it appears the ghosts of vCNS remained in play.

With that action failing I decided to move onto uninstalling the old VXLAN config (ensuring no Transport Zones where tied to the Cluster) which removed the references to the old vCNS VXLAN configuration…from there I hit Resolve against the Cluster in the Host Preparation Tab. The Host in the Cluster that was Not Ready had the vCNS Host Extenions installed and took the Installation Status of the Cluster to Legacy with an Option to Update or Uninstall.

Confirming that the previously Not Ready Host only had the vCNS VIBs installed I hit the Update Link and confirmed the Update. The NSX Manager then worked with vCenter DRS to uninstall and then install the updated NSX Host Extensions…along the way putting each host into Maintenance Mode and rebooting for the upgrade to take effect.

One thing I did find is that I had to repeat the process a couple of times for the Updates to go through each host in the Cluster. The process would often do one Host and then require a Retry…not ideal but it got there in the end. I SSH’ed into a couple of the hosts to ensure the right VIBs where now installed.

Biggest take away here is to double check any existing vCNS Environment for Legacy configurations of VXLAN using the old Multicast/vCD method…if not required, make sure the config is removed before the NSX Manager Upgrade.

NSX vCloud Retrofit: Controller Deployment Internal Server Error Has Occurred

During my initial work with NSX-v I was running various 6.0.x Builds together with vCD 5.5.2 and vCenter/ESXi 5.5 without issue. When NSX 6.1 was released I decided to clone off my base Production Environment to test a fresh deployment of 6.1.2 into a mature vCloud Director 5.5.2 instance that had vCNS running a VSM version of 5.5.3. When it came time to deploy the first Controller I received the following error from the Networking & Security section of the Web Client:

Looking at the vCenter Tasks and Events the NSX Manager did not even try to kick off the OFV Deployment of the Controller VM…the error it’s self wasn’t giving to much away and suspecting a GUI bug I attempted to deploy the Controller directly via the RestAPIs…this failed as well however the error returned was a little more detailed.

Looking through the NSX Manager Logs the corresponding System Log Message looked like this:

The Issue:

While the error it’s self is fairly straight forward to understand in that the a value being entered into the database table was not of the right type…the reasons behind why it had shown up in the 6.1.1 and 6.1.2 releases after having no such issue working with the 6.0.x releases stumped everyone involved in the ensuing Support Case. In fact it seemed like this was the only/first instance (at the time) of this error in all global NSX-v installs.

The Fix:

NOTE: This can only be performed by VMware Support via an SR.

The fix is a simple SQL Query to alter the KEY_VALUE_STORE referenced in the error…however this can only done done by VMware Support as it requires special access to the NSX Manager Operating System to commit the changes. A word of warning…If you happen to have the secret password to get into the back door of the NSX Manager and apply these changes…your support for NSX could become null and void!

Once that’s been committed and the NSX Manager Service restarted Controllers can be successfully deployed…again, the fix needs to be applied by VMware Support.

The RCA:

In regards to the RCA of this issue,  the customers having a long upgrade history (5.0 onwards) will hit the issue, since there was a db migration happening from 5.0 to 5.1.x upgrade the alter table script for KEY_VALUE_STORE was missing. As per VMware engineering a new upgrade on NSX Manager is not going to override the DB schema change, since there is no such script to alter table on the upgrade path.

There was no indication of this being fixed in subsequent NSX Releases and no real explanation as to why it didn’t happen in 6.0.x but that aside the fix works and can be actioned in 5 minutes.

This was a fairly unique situation that contributed to this bug being exposed…my environment was a a vCNS 5.1 -> 5.5 -> 5.5.2 -> -> NSX 6.1 -> 6.1.1 -> 6.1.2 replica of one of our Production vCloud Zone that sits in our lab. Previously I’d been able to fully deploy NSX end to end using the same base systems which sat side by side in working order in a separate #NestedESXi Lab…but that was vCNS 5.5.2 upgraded to NSX 6.0.5 which was upgraded to 6.1.2.

So due to not too many deployments of NSX-v the issue only manifested in mature vCNS environments that where upgraded to 6.1.1 or 6.1.2. Something to look out for if you are looking at doing an NSX vCloud Retrofit. If you have a green fields site you will not come across the same issue.

Further Reading:

This blog series extends my NSX Bytes Blog Posts to include a more detailed look at how to deploy NSX 6.1.x into an existing vCloud Director Environment. Initially we will be working with vCD 5.5.x which is the non SP Fork of vCD, but as soon as an upgrade path for 5.5.2 -> 5.6.x is released I’ll be including the NSX related improvements in that release.

VCAP Exam Retirement – Not Well Received…

VMware Education dropped somewhat of a bombshell today when they announced the almost immediate retirement of the VCAP CIA/CID/DTA/DTD

Effective March 2, 2015 the following certification exams will be retired and no new registrations accepted:

  • VCID510 – VMware Certified Advanced Professional 5 – Cloud Infrastructure Design
  • VCIA510 – VMware Certified Advanced Professional 5 – Cloud Infrastructure Administration
  • VDTA510 – VMware Certified Advanced Professional 5 – Desktop Administration
  • VDTD510 – VMware Certified Advanced Professional 5 – Desktop Design

There has already been a fairly big outcry in the VMware Community around the decision to scrap the exams…and even though this move had been expected with the announcements of the new VCIX Advanced Exam Roadmaps, what I think people didn’t expect was the chopping off at the head of these VCAPs. This has left some people’s plans of VCDX defense attempts in limbo…specially for those who where looking to defend for Cloud based on vCloud Director…vCloud Director has been pulled from the vCloud Suite for Enterprise but is still very much alive for Service Providers…I know of a few guys (myself possibly, though not 100% committed) who where already building documentation around vCloud for a defense.

To me I this shows a clear disconnect from the VMware Education and Certification team from what is actually happening in customer and partner land…the way this has been done is a mistake and VMware needs to retract the immediate retirement and put a 3-6 month period of these exams to EOL. There also needs to be some direction as to a roadmap for Service Providers who will continue to use vCD+NSX+VIO who would not be looking at vRealize Automation.

Feel free to comment below on your thoughts around the retirements…hopefully VMware Edu Services will read the communities thoughts and respond with something positive.


Top Blog 2015 Voting

The Top vBlog Voting for 2015 at which is headed up and run by Eric Sebert (@ericsebert) is now open.

There are so many excellent Blogs out there and it’s a credit to the community that so much content is being created. It’s a hard gig to get a vote with almost 400+ sites listed, but if you are reading this hopefully you have found my content to be helpful over the past 12 or so months.

Search for Virtualization is Life! or Anthony Spiteri…here is to more vCloud Director/NSX/ESXi posts…as well as maybe finding a new bug or two!

Thanks for your support!


MVMUG – Pivoting and The New IT

Last week I attended my second Melbourne VMUG User Conference. The UserCon was again well attended and the Melbourne VMUG Team did a great job putting together a VMWorld Standard Lineup filled with excellent content …highlighted by Keynotes from John Troyer (@JTroyer) and Chad Sakac (@sakacc) as well as Vaughn Stewart (@vStewed).

I flew in early to catch one of my favourite Bands, Papa Roach play a show at The Forum Melbourne. We talk about passion often in the vExpert Community and Papa Roach are certainly not short of passion and energy and I had a great time during a high octane one and half hour set. I had waited 15+ years to see this band and I wasn’t going to miss out on grabbing the opportunity to see them…I’ll tie that in later in the post!

Back to the UserCon, I wanted to focus on the Keynote from John…His presentation was based around a change of expectation around what it is to be part of the IT Industry…and even though there are some IT Professionals that will not embrace the shift that’s currently happening…the VMware/vExpert/Virtualization/Cloud Community is at the forefront of driving that change and best positioned to harness the pivot that’s currently on offer.

So what is this pivot?

Technologies like Containers (Docker) and processes like System and Service Automation (APIs, Orchestration Management) are challenging how we have done things for the past 10+ years since the Virtualisation revolution began last decade and there are a number of movements (DevOps -sic) that are trying to gain traction in the industry with a commonly held thought that if change isn’t embraced those that choose to stay still will be left behind…Pivoting is being used to describe this change of direction…it’s popular in the Startup world as described by Silicon Valley.

Pivoting was referenced a lot during the day and featured heavily in the closing Keynote from Vaughn and Chad around what the Storage Industry is doing to keep up with disruptive change being introduced by new Vendors…However pivoting as it relates to an IT Professionals jobs is probably not representative of this New IT spoken by John. The message from John’s keynote that resonated with me the most was in regards to the work done outside a persons job description…its the side projects that drive change.

For me this is what New IT refers to…Passion driving innovation that drives learning and discovery of new technologies such as those mentioned above. Those lacking in passion in my opinion are likely to be left behind because the industry is shifting quicker than ever and most of what’s new can only be discovered by those who want to learn. John knows first hand the value of being interested outside the square…more than anyone in Tech community he has been responsible for a lot of IT Pro’s following their passion…diving into side projects and in that creating content that’s shared in the form of Blog Posts and other mechanisms are then hopefully used by others passionate enough to dive into their own new side projects…and so the cycle goes.

I talked about Grabbing the Opportunity earlier in the post and for those interested enough to look to better themselves the time is now to grab on and ride your on wave…have a listen at John’s full talk below and ask yourself if taking part in the New IT is worth it?

…the answer should be “YES!”