Monthly Archives: May 2016

Quick Post: ESXi 6.0 Patch Breaks Veeam Instant VM Recovery

This is a quick post to alert Veeam users to an issue that was raised in the Veeam Community Forums yesterday…firstly if you are a Veeam customer and are not registered for the Veeam Community Forum Digest that Anton Gostev releases every Sunday night then stop reading this and go register here! There is some awesome content that Anton covers and its not just limited to backups but general industry news and trends as well.

Once you have done that I thought I would bring to everyone’s attention an important note that Gostev mentioned in his last update relating to an issue with Veeam Instant Recovery and all dependent features when ESXi 6.0 Patch 6 (Build  3825889) is installed.

This patch was released on the 12th of May so chances are some people have deployed it and are being impacted if they use or rely on Instant Recovery. As Gostev mentions, Veeam have an ongoing support case with VMware but as is usual with Veeam they have gone ahead and got a workaround in place in the form of a hotfix which is applicable to Veeam 9.0 Update 1.

If you have deployed this ESXi 6.0 build and run Veeam contact their support to grab the hotfix. Again well done to the Veeam development teams for working around issues so efficiently.


Released – vCloud Director SP 8.10 with Full NSX Support and UI Additions!

Earlier this morning VMware released vCloud Director SP 8.10 Build 3879706. This is a significant release for vCloud Director as its the first release that contains the long overdue UI features that where previously only accessible via API calls. The vCD Service Provider community had been asking for this since the first release of the SP code stream and as had been well documented in this blog, VMware have gone back and forth in finally committing to vCDs future which meant a commitment back in December to start introducing the new API features into the UI and commit to further development and UI improvements in future builds.

I’ll dedicate a couple of blogs over the next few weeks to the new features in this release but I’ve summarized the main features and improvements below. There is also some very important news around support for vCNS (below) and also around NSX interoprability as well as a requirement to upgrade to an 8.0.x build before being able to upgrade to 8.10.

New Feature Highlights:

  • Expanded UI Access to Features Many operations that were previously accessible only from the vCloud API are now accessible in the vCloud Director Web Console. I’ll cover most of these in the next few weeks, but this is obviously the standout new feature.
  • Object Extensibility A new Object Extensibility feature has been introduced. vCloud Director object extensions are external applications that can participate in, influence, or override the logic that vCloud Director applies to complex operations like vApp instantiation and placement.
  • Expanded Hardware Version and Guest OS support The system now supports Hardware Version 11 and Windows 10. This allows full support of those running ESXi 6.0 and above.
  • Browser Support It looks as though most…if not all browsers are supported in this release except Microsoft Edge. The release notes state that all browsers on the MAC are officially supported as well and we still have to put up with Flash support…for the medium term anyways 😉
  • Guest OS Support  Still no Windows Server 2016 support, but hopefully that will come not long after it’s GA’ed. Check the release notes for a full list.


There is no support for upgrades from any 5.5.x or 5.6.x versions meaning that you needs to upgrade to at least 8.0.1 before upgrading to the 8.10 binaries. If you try to upgrade you will get the warning message below so continue at your own risk.

There is also a recommendation in the release notes to increase the resources of your vCD Cells to a minimum of 6GB of RAM…though in the installer is quotes 5GB of RAM.

vCNS and NSX:

No Support for vShield Manager
This release of vCloud Director does not support integration with VMware vShield Manager. Existing installations that are using vShield Manager must migrate their vShield Manager installations to VMware NSX Manager before upgrading to vCloud Director 8.10. Legacy vShield Manager Edge appliances are still supported in this release, but we recommend upgrading those Edges to NSX Edges as soon as possible.

As people on the BETA for 8.10 found out this release officially ends support for vCloud Networking and Security which is no surprise given its being EOL’ed later in the year but what it will do is hasten any vCD SP that’s still not upgraded to NSX who want to deploy this and any future build of vCloud Director. I also assume that if all VSE’s are to be upgraded to installed NSX Edge versions the Edge Service Configuration is 100% compatible with the NSX Edges and you only configure whats presented to you from the vCD UI even through other services are available on the Edge…this obvious paves the way for the Advanced Networking Services which is set for release later in the year.

Below is the official supportability matrix for all vCD SP release and NSX-v…as shown below, 8.10 is good with NSX 6.2.2, 6.1.5 and 6.1.6.

For those with the correct entitlements…download here.



Quick Post – VSAN and Log Insight Custom Alerting Example

Log Insight is one of those great VMware products that needs to get more airplay as it has quiet a few applications other than a run of the mill log parser…in this post I’ll go through configuring a basic VSAN alert to detect disk failures. Once VSAN has been configured and deployed there is a new set of alerting parameters that VMware Admins need to be aware of that would usually be part of a traditional storage platforms feature set. Like all storage we need to be made aware of any issues with the supporting hardware such as Storage Controllers and Physical Disks. VSAN 6.2 comes with an excellent Health Monitor that allows you to get a quick overview of a VSAN instances state and will alert through vCenter if any issues arise.

While vCenter Triggered Alerting is fine we had a situation recently where a failed disk was missed for a couple of days due to the default vCenter Alarming not configured correctly. The only way we found out about the failed disk was by visually seeing the alert against the vCenter and then taking a look at the VSAN Health Analyzer. While vCenter monitoring is ok, I don’t believe it should be your only/primary source of monitoring and alerting.

Having done a few alerts in Log Insight before, I looked at what Log Insight could provide by way of logging through the recently released VSAN Content Pack.

Using the Diskgroup Failures menu on the VSAN Content Pack Dashboard I searched through to try and locate the previous disk failure. As shown below a Disk Permanent Error had been registered.

Clicking through to the Interactive Analysis on that event you get a more detailed view of the error and the search parameters of the specific log entry.

To create a custom alert that emails when a Permanent Disk Failure occurs I removed the search fields that related directly to the disk and host and clicked on the Create Alert Icon (Red Bell top left of the image)

As shown below configuring the alert is simple and there are a number of different hooks to use as methods of notification. One of the great things about using Log Insight to trigger Alert notification is the suppression mechanisms to stop alert floods.

Apart from creating custom alerts the VSAN Content pack comes with a number of pre-canned alerts that are disabled by default. To view and enable these click on the Manage Alerts button and filter for VSAN.

If you haven’t had a chance to look at Log Insight, take a look at the features page and if you own a vCenter license you already own 25 OSI Pack of Log Insight.



VMworld 2016 – #vGolf Las Vegas

Last year at VMworld 2015 in San Francisco a couple of us headed out on the Sunday morning for a walk up start of 9 holes of Golf at the Golden Gate Golf Club. #vGolf at VMworld is something that I’ve been wanting to do since I first attended in 2012 and being able to play last year was a great way to kick off the week and take in some of the scenery outside of the business district.

This year as we know, VMworld is in Las Vegas which is renowned for it’s local Golf Courses and I’ve decided to formalize and extend #vGolf to see if we can make it a bigger event for this years VMworld. At this stage i’m collecting numbers of those interested as well as preferred days and price range. On the suggestion of @rytalws I thought I would reach out to see if any vendor wanted to sponsor part of the costs…anything would be a bonus but at the end of the day it’s about experiencing some great golf in Las Vegas with members of the VMware community.

If you are interested in playing 9 holes on the Friday or Saturday please fill out the embedded form below:

Thanks and…

Top vBlog 2016 – Still Time to Vote

While I have resisted temptation to post a blog on this years Top vBlog voting I thought with a couple of days to go it was worth giving it a shout just in case there where some of you who hadn’t had the chance to vote or didn’t know about the Top vBlog vLaunchPad list organised and maintained by Eric Siebert of vShere-Land.

As Eric mentions the vBlog voting should be based on blog content based around longevity, length, frequency and quality of the posts. There is such great content creators out there in the VMware community and the 300+ lists of active bloggers is testament to the effort and passion shown by members of the community.

As has been the case in previous years, there has been talk of this being a popularity contest and there has even been some other comments around gender participation this year which is disappointing to have around what should be a legitimate way to help recognize the vBlog community. In my opinion the best way to vote was described by @Virten shown below.

Export Browser History…
Filer URLs…
Calculate Sums..
Filter out non VMware related sites..

How to vote for Top vBlog 2016

However you do it, or have done it good luck to all those who are listed and for those who haven’t voted yet click on the link below to cast your vote. If you feel inclined and enjoy my content around vCloud Director, NSX, VSAN and Cloud and Hosting in general…It would be an honor to have you consider in your Top 12 and also in the Independent Blogger category. 

Thanks again to Eric Siebert.

NSX Bytes: Friends Don’t Let Friends Delete The VTEP PortGroup

Last week I posted a tweet saying “Friends don’t let friends delete the NSX-v VTEP PortGroup” and as most of us do in our industry we learn by doing and I found out the hard way that you shouldn’t mess with the PortGroup created during the Host Preparation of the NSX setup and configuration stage. This PortGroup is used by the Hosts in an NSX Enabled Cluster for the VMKernel Interfaces that are the VTEPs or VXLAN Tunnel End Points.

In a production environment this action is actually near on impossible to do because you can’t delete a PortGroup when it’s in use. Where I found myself in this situation was in trying to clone off a lab environment and restore components of the existing lab into new lab with new hosts. With that the following is something that could be handy in lab environments.

Once the new hosts have been prepared I went to configure the VXLAN against the cluster which creates a new VMKernel Interface on each host and assigns it a VTEP address from DHCP or from a pre-configured IP Pool but got an error. When I looked at the event logs in vCenter I saw the following error.

DVPortGroup dvportgroup-148806 couldnot be found
 The object or item referred to could not be found

Instantly I remembered that I had “cleaned up” the cloned vCenter configuration and removed any surplus PortGroups…in doing so I deleted the PortGroup NSX was referencing. I tried to recreate the PortGroup with the same name but it was clear that the configuration was referencing the MOID of the PortGroup and asking vCenter to use that to complete the job. Even an export/import of the Distributed Switch configuration from the original vCenter didn’t do the trick as the import increments the MOID already contained in the vCenter Database.

GSS Support Fix:

Thinking back to previous NSX related cases I’ve raised with VMware support I knew that the NSX Manager Database kept a very simple structure of vCenter objects and I guessed that some backend SQL search and replace could do the trick. After raising a case I had the guys in GSS enter into the NSX Manager backend, that can only be access with a secret VMware password and search for the table that referenced the MOID of the PortGroup. As can be seen below the fix is simple if you know the MOID of the old and the new PortGroup.

Note: Only VMware Support can action this fix.

With that modification committed I was able configure the VTEPs for the new hosts and continue to rebuild up the cloned instance. So if you ever get yourself in a situation where you have managed to do as I have done…there is a fix that can be done to avoid a complete start from scratch scenario.

VSAN 6.2 ESXi Patch Updates + DELL PERC Firmware Updates

I wanted to cover off a couple of important updates in this post relating to the DELL PERC storage controller Firmware and software drivers as well as an important new release of ESXi 6.0 that addresses a couple of issues with VSAN and also fixes to more VMXNET3 problems which seem to keep popping up. Read further below for the ESXi fixes but firstly a couple of weeks ago I posted about the new certified driver updates for the DELL PERC based storage controllers that VMware released for VSAN 6.2. This driver was only half of the fix as DELL also released new Firmware for most of the PERC based controllers listed below.

It’s important to match the PERC Firmware with the updated driver from VMware as together they protect against the LSI issues mentioned here. The workaround after the driver has been installed is just that and it requires the FW upgrade to be fully protected. As shown below you want to be on at least version

Side note: While you are at it looking at the DELL Drivers and Download site you should also consider upgrading to the latest iDRAC Firmware and any other component that contains fixes to issues that could impact you.

Just on that new VMware driver…even if you are running earlier versions of VSAN with the Health Checker if you update the HCL database and run a health check you will see a warning against PERC FW Controller Driver versions prior to lsi_mr3 (6.903.85.00-1OEM.600.0.0.2768847) as shown below.

New ESXi 6.0 Update 2 Build VSAN Fixes:

Last week VMware released ESXi 6.0 Build 3825889 that addressed a couple of big issues relating to VSAN datastore updates and also a bad VMXNET3 PSOD issue. Of most importance to me looking to upgrade existing VSAN 6.1 clusters to VSAN 6.2 there was an issue with CBT enabled VMs when upgrading the VSAN filesystem from 2.0 to 3.0.

Attempts to upgrade a Virtual SAN cluster On-Disk format version from version 2.0 to 3.0 fails when you Power On CBT-enabled VMs. Also, CBT-enabled VMs from a non-owning host might fail due to on-disk lock contention on the ctk files and you might experience the following issues:

  • Deployment of multiple VMs from same CBT enabled template fail.
  • VMs are powered off as snapshot consolidation fails.
  • VM does not Power On if the hardware version is upgraded (for example, from 8 or 9 to 10) before registering the VM on a different host

So that’s not too cool specially if you are using Veeam or some other VDP based backup solution but glad there is a fix for that. Again I don’t get why or how these things slip through…but it seems like things haven’t improved too much when it comes to the QA of ESXi releases. But again, the relative turn around time to have these issues fixed seems to be somewhat acceptable.

As mentioned there are a few more significant fixes so when the time is right this update should be applied to existing ESXi 6.0 Update 2 installations.


VMworld 2016 : Session Voting #8808

Well this has crept up on us quickly this year! It’s time to vote for the VMworld Sessions that will be part of the US and Europe VMworld’s held later in the year. The Session Voting is more Session liking as you have the ability to mark multiple sessions as ones that you would like to see. There are 1574 sessions that where submitted and are listed in the Online Catalog.

As with the last couple of years I have submitted a session. This year I have decided to focus on a Real World Look at how we here at Zettagrid implemented NSX into our Service Offering. Have a read of the description below and if you want to hear more about NSX (with a little vCloud Director thrown in) in the real world and the lessons I’ve learnt over the past couple of years please consider Session #8808 for a vote.

NSX in Service Provider Land – Lessons Learnt from the Real World [8808]

VMware NSX has been out for two years and in that time has already gone through some significant updates and feature addons. The power of the NSX platform and what that offers Service Providers meant that getting NSX into Zettagrid become priority number 1 leading up to the end of 2015. Come and hear how Zettagrid successfully retrofitted NSX into an existing vCloud Platform that spans three Hosting Zones across Australia. In this session I will take you through how we brought NSX from Lab to limited BETA release to full production productization and deployment. You will hear about how working on the bleeding edge of network virtualization had its challenges and how NSX as an overlay network fit into our existing physical layer as well as the operational and architecture challenges we ran into before being able to offer our NSX Advanced Networking Product as part of our vCloud Director Virtual DC portfolio extending our Hybridity capabilities.

Track:  Software-Defined Data Center
Product and Topic:  Customer Story
Session Type:  Breakout Session
Audience:  IT – All, IT – Network, IT – Operations, IT – Telecom, Research and Development, Technical Support
Sub track:  Networking and Security

There is a lot of competition around the NSX space with 302 sessions listed that contain NSX in the title or the description but I’m hoping my experiences are interesting enough to get some votes and then make it through the internal voting process. On a separate note it was also pleasing to see an increase in the number of vCloud Director sessions this year.

As per the VMworld Session Voting FAQ Session Voting is open May 12 – May 26 at 11:59 pm PDT.


VMworld 2016 : VMTurbo Pass Giveaway

It’s that time of year again! VMworld is about 100 days away and VMTurbo are giving away three full passes to VMworld being held in Las Vegas. There are three sweepstakes draws happening over the next three months.

When will the drawings take place?
  • May 27, 2016
  • June 17, 2016
  • July 15, 2016

Head to the link below and Enter for Free!

Once entered your name will be in the pool for all three draws.


New NSX License Tier Thoughts and Transformers

Overnight there was a couple of NSX related events that took place…one was expected and one not so much. I have known about the next release of NSX code named Transformers going back to VMworld last year where I attended a couple of NDA sessions…What I wasn’t expecting was the announcement of separate licensing tiers for NSX-V which is detailed in this KB and this landing page.

Transformers – NSX-MH:

Before going into my thoughts around the features options that come with each licensing tier the release of NSX Transformers 1.0 represents a fundamental shift in the way that NSX is released in that it now is the one code base that unifies NSX-MH and NSX-V. For the moment though the 1.0 release which was available earlier in the day is only for current MH customers and we probably shouldn’t expect any release for V customers for a while…and when that drops it will be interesting to see the upgrade path and product interoprabilites.

NSX-V Edition Thoughts:

Going back to the new pricing its clear to me that this is aimed at trying to keep the momentum going for the NSBU and try to entice the market to a lower entry point, however going through the edition feature matrix I’m left a little confused at some of the choices especially for those who are looking to replace vCloud Networking and Security editions that’s set to be end of lifed later this year. (click here for details on that)

  • NSX Standard Edition – For organizations needing agility and automation of the network.
  • NSX Advanced Edition – For organizations needing Standard, plus a fundamentally more secure data center with micro-segmentation.
  • NSX Enterprise Edition – For organizations needing Advanced, plus networking and security across multiple domains.

VMware have increased the cost of the full featured Enterprise edition which existing customers have while creating the Standard and Advanced editions. Where previously the list (USD) for NSX was $5,996 per Socket the new editions come in at $1,995, $4,995 and $6,995 per Socket. The Standard edition is well priced but taking a look through the Matrix in the official KB you are getting an extremely slimmed down version of NSX…short of the bells and whistles that make it the awesome SDN platform that it is, however I’m sure the feature set will be attractive for some.

For existing customers, as stated in the FAQ

Customers with active support contracts who have purchased NSX prior to the new licensing model goes into effect in May, 2016 will be entitled to the same functionality in the Enterprise offering.

vCNS Edition Match:

Looking at the current functionality of vCNS and then trying to match it to the available functions in each edition as per the KB there doesn’t seem to me to be a perfect fit except for the Enterprise Edition…which isn’t what those looking to upgrade from vCNS want to hear. The new NSX-V Standard Edition doesn’t include VPN, SSLVPN or Load Balancing functionality, though it does include the third party integration which I suspect would represent a large part of the current vCNS users that are not Service Providers.

Feature Standard Advanced Enterprise
Hypervisors Supported      
ESXi 5.5 Yes Yes Yes
ESXi 6.0 Yes Yes Yes
vCenter 5.5 Yes Yes Yes
vCenter 6.0 Yes Yes Yes
Switching Encapsulation Format      
VXLAN Yes Yes Yes
Replication Mode for VXLAN      
Multicast Yes Yes Yes
Edge Routing (N-S)      
Edge Routing Static – IPv4 Yes Yes Yes
Edge Routing Static – IPv6 Yes Yes Yes
DHCP Relay Yes Yes Yes
VLAN Trunk (sub-interface) support Yes Yes Yes
NAT Support for NSX Edge      
NAT Support for NSX Edge Yes Yes Yes
Source NAT Yes Yes Yes
Destination NAT Yes Yes Yes
DHCP Server Yes Yes Yes
DHCP Relay Yes Yes Yes
DNS Relay Yes Yes Yes
Edge Firewall      
Edge Firewall Yes Yes Yes
Edge High-Availability Yes Yes Yes
Third Party Integration      
Endpoint Service Insertion – Guest Introspection Yes Yes Yes 
Public API based Integration Yes Yes  Yes
Edge Load-Balancing      
TCP (L4 – L7) No Yes Yes
HTTP No Yes Yes
HTTPS (Pass-through) No Yes Yes
LB Methods No Yes Yes
Round Robin No Yes Yes
Src IP Hash No Yes Yes
Least Connection No Yes Yes
Health Checks
TCP No Yes Yes
HTTP No Yes Yes
HTTPS No Yes Yes

What about the vCAN and Service Provider Pricing?:

At this point in time I haven’t heard through any official channels of changes to the NSX Bundle options that come with the vCAN Program and I don’t really expect any changes to what currently exists for SP consumption of NSX as anything but the full Enterprise feature list would be required for SPs to take full advantage of NSX.

Final Thoughts:

Time will tell if these changes lead to greater uptake of non Enterprise or Service provider customers. One thing is forsure…VMware are without question needing NSX to be a success and I’ve got no doubt that with what currently exists in addition to what’s coming in future releases NSX will continue to gain market penetration and be the defacto SDN platform going round.


« Older Entries