Monthly Archives: November 2015

NSX Bytes: No NSX Managers Listed in Web Client After vCenter Certificate Upgrade

Certificates and VMware don’t have a great history and there are a lot of posts out there centered around people’s struggles with vCenter, Lookup Service or Web Client Certificate management. I’ve recently had a little fun with a revoked vCenter certificate (thanks Symantec) that required replacement. Without going into the details of the pain I went through to successfully get the certificate updated and working with vCenter and the Web Client, when I did eventually get things in working order with the new publicly signed certificate I logged back into the Web Client and saw that I had no NSX Managers listed in the Web Client.

I’ve blogged before about how to deal with that particular error as it relates to user permissions, but as nothing had changed from a permissions point of view this was surly due to the certificate changes on the vCenter. Logging into the NSX Manager and going to the Manage Tab and NSM Management Service the vCenter Server Status was listed as Disconnected.

I also found corresponding errors in the Manager Logs as shown below.

The reason for this happening is the NSX Manager trusted the previous certificate and needs to be reconnected so that the new certificate can be trusted and accepted.

Once that’s been done you should have a green light and the NSX Manager will resync up with the vCenter Inventory and all operations will be back to normal…an easy fix to a logical issue!

vCloud Air and Virtustream – Just kill vCloud Air Already?!?

I’ve been wanting to write some commentary around the vCloud Air and Virtustream merger since rumours of it took place just before VMworld in Auguest and I’ve certainly been more interested in the whole state of play since news of the EMC/VMware Cloud Services spin off was announced in late October…the basis of this new entity is to try and get a strangle hold in the Hybrid Cloud market which is widely known to make up the biggest chunk of the Cloud market for the foreseeable future topping $90 billion by 2020.

Below are some of the key points lifted from the Press Release:

  • EMC and VMware  plan to form new cloud services business creating the industry’s most comprehensive hybrid cloud portfolio
  • Will incorporate and align cloud capabilities of EMC Information Infrastructure, Virtustream and VMware to provide the complete spectrum of on- and off-premises cloud offerings
  • The new cloud services business will be jointly owned 50:50 by VMware and EMC and will operate under the Virtustream brand led by CEO Rodney Rogers
  • Virtustream’s financial results to be consolidated into VMware financial statements beginning in Q1 2016
  • Virtustream is expected to generate multiple hundreds of millions of dollars in recurring revenue in 2016, focused on enterprise-centric cloud services, with an outlook to grow to a multi-billion business over the next several years
  • VMware will establish a Cloud Provider Software business unit incorporating existing VMware cloud management offerings and Virtustream’s software assets — including the xStream cloud management platform and others.

I’ve got a vested interest in the success or otherwise of vCloud Air as it directly impacts Zettagrid and the rest of the vCloud Air Network as well as my current professional area of focus however I feel I am still able to provide leveled feedback when it comes to vCloud Air and the time was finally right to comment after yesterday evening comming across the following LinkedIn Post from Nitin Bahdur

It grabbed my attention not only because of my participation in the vCloud Air Network but also because the knives have been out for vCloud Air almost before the service was launched as vCloud Hybrid Services. The post its self from Nitin though brief, was suggesting that VMware should further embrace it’s partnership with Google Cloud and just look to direct VMware Cloud customers onto the Google Cloud. The suggestion was based on letting VMware Orchestrate workloads on Google while letting Google do what it’s best at…which was surprisingly Infrastructure.

With that in mind I want to point out that vCloud Air is nowhere near the equal of AWS, Azure or Google in terms of total service offerings but in my opinion it’s never been about trying to match those public cloud players platform services end to end. Where VMware (and by extension it’s Service Provider Partners) does have an advantage is in the fact that in reality, VMware does do Infrastructure brilliantly and has the undisputed market share among other hypervisor platforms therefore giving it a clear advantage when talking about the total addressable market for Hybrid Cloud services.

As businesses look to go through their natural hardware refresh cycles the current options are:

  • Acquire new compute and storage hardware for existing workloads (Private – CapEx)
  • Migrate VM workloads to a cloud based service (IaaS – OpEx)
  • Move some application workloads into modern Cloud Services (SaaS)
  • Move all workloads to cloud and have third parties provide all core business services (SaaS, PaaS)

Without going into too much detail around each option…at a higher level where vCloud Air and the vCloud Air Network has the advantage in that most businesses I come across are not ready to move into the cloud holistically and for the next three to five years existing VM workloads will need a home as businesses work out a way to come to terms with an eventual move towards the next phase of cloud adoption which is all about platform and software delivered in a cloud native way.

Another reason why vCloud Air and the Air Network is attractive is because migration and conversion of VMs is still problematic and a massive pain (in the you know what) for most businesses to contemplate undertaking…let alone spend the additional capital on. A platform that offers the same underlying infrastructure as what’s out there as what vCloud Air, the vCloud Air Network partners and what Virtustream offers should continue to do well and there is enough ESXi based VMs out there to keep VMware based cloud providers busy for a while yet.

vCloud Air isn’t even close to being perfect and has a long way to go to even begin to catch up with the bigger players and VMware/EMC/DELL might well choose to wrap it up but my feel is that that would be a mistake…certainly it needs to evolve but the platform has a great advantage and it, along with the vCloud Air Network should be able to cash in.

In the next part I will look at what Virtustream brings to the table and how VMware can combine the best of both entities into a service that can and should do well over the next 3-5 years as the Cloud Market starts to mature and move into different territory leading into the next shift in cloud delivery.


Quick Fix: VCSA Web Client 6.0 Throws Monitoring Errors

This quick fix post is for those out there who are still using vCenter Operations Manager 5.8.x and are or thinking about deploying or upgrading to vCenter 6.0…I came across this annoying situation all of a sudden while working on a new vCenter instance when the Web Client started to report the error shown below.

This can be ignored by clicking no and you will still be able to operate most areas of the Web Client but you will find that Monitoring and Health pages fail to load and give you a generic Error #2036 as shown below.

It took me a while to realize that the error was related specifically to the monitoring modules and it finally clicked in my head that the error started happening when I Registered the vCenter against my lab vCOPs instance. I was still running vCOPs (not vRA) and the instance hadn’t been upgraded to the latest build. Having a look through the VMwareKBs I came across KB 2111224 which explained the cause.

This issue occurs because vRealize Operations Manager versions prior to 5.8.5 are not supported in the vSphere 6.0 environment.

Upgrading the vCOPs Appliances to build 5.8.5-2532416 sorted the issue and I was able to browse through the Web Client without the error and have the integrated Health Monitoring work without issue.


Dell PowerEdge FX2: VSAN Disk Configuration Steps

When you get your new DELL FX2s out of the box and powered on for the first time you will notice that the disk configuration has not been setup with VSAN in mind…If you where to log into ESXi on the blades in SLOT1a and 1c you would see that each host will have each SAS disk configured as a datastore. There is a little pre-configuration you need to do in order to get the drives presented correctly to the blades servers as well as remove and reconfigure the datastores and disks from within ESXi.

With my build I had four FC430 Blades with two FD332 Storage Sleds that contained 4x200GB SSDs and 8x600GB SAS drives in each sled.  By default the storage mode is configured in Split Single Host mode which results in all the disks being assigned to the hosts in SLOT1a and SLOT1c and both controllers as also assigned to the single host.

You can configure individual storage sleds containing two RAID controllers to operate in the following modes:

  • Split-single – Two RAID controllers are mapped to a single compute sled. Both the controllers are enabled and each controller is connected to eight disk drives
  • Split-dual – Both RAID controllers in a storage sled are connected to two compute sleds.
  • Joined – The RAID controllers are mapped to a single compute sled. However, only one controller is enabled and all the disk drives are connected to it.

To take advantage of the FD332-PERC (Dual ROC) controller you need to configure Split-Dual mode. All hosts need to be powered off to change the default configuration and change it to Split Dual Hosts for the VSAN configuration.

Head to Server Overview -> Power and from here Gracefully Shutdown all four servers

Once the servers have been powered down, click on the Storage Sleds in SLOT-03 and SLOT-04 and go to the Setup Tab. Change the Storage Mode to Split Dual Host and Click Apply.

To check the distribution of the disks you can Launch the iDRAC to each blade and go to Storage -> Enclosures and check to see that each Blade now has 2xSSDs and 4xHDD drives assigned. With the FD332 there are 16 total slots with 0-7 belonging to the first blade and 8-16 belonging to the seconds blade. As shown below we are looking at the config of SLOT1a.

The next step is to reconfigure the disks within ESXi to make sure VSAN can claim them when configuring the Disk Groups. Part of the process below is to delete any datastores that exist and clear the partition table…by far the easiest way to achieve this is via the new Embedded Host Client.

Install the Embedded Host Client on each Host

Log into the Hosts via the Embedded Client from https://HOST_IP/ui and go to the Storage Menu and delete any datastores that where preconfigured by DELL.

Click on Devices Tab in the Storage Menu and Clear the Partition Table so the VSAN can claim the disks that have been just deleted.

From here all disks should be available to be claimed by VSAN to create your disk groups.

As a side note it’s important to update to the latest driver for the PERC.


The Power of Network Automation: How a Huge Low Turned into a Great High!

A few weeks back at Zettagrid we released our NSX Advanced Networking product that we have been working on for the best part of 12 months. I’m particularly  proud of this release as it represents a significant realisation of a vision myself and others have had in trying to integrate NSX into the Zettagrid IaaS platform. Furthermore the release held a deeper meaning as it showed off what can be achieved when faced with disappointment and failure.

Taking myself back to February of 2014 I was presenting to a government panel for a Cloud Computing tender which ended up going horribly wrong…Notwithstanding the fact that the tender had specified IaaS as the basis of the tender the presentation actually ended up being a practical test on deploying a three tier application into a Virtual Datacenter in an allotted time period which was more akin to an Managed Services Provider than an Infrastructure Provider. Cutting a long story short I was able to get vCloud Director configured in such a way to get the vShield to do basic load balancing but failed to produce a working IIS Default page externally which would have meant passing the test and us making it through to the next stage of the process.

I came out of that presentation as deflated as I have been in my career…I don’t usually fail and up until that point every presentation and demo I had given had resulted in success…as I sunk down a couple of whiskey’s in the pub next to the government agency building I was trying to think to myself what went wrong? Surly there had to be a more efficient way to deploy, configure and manage networks in a cloud environment…it was decided there and then that Zettagrid would look at NSX as a way to improve network efficiency via automation.

Looking back at the tender process the Government agency got it all wrong…they expected the tenderer to deploy and configure the full environment themselves…they expected a Managed Service instead of a pure IaaS. In fact it should have been that the roles were reversed and that instead of us being handed the practical example to work through the design configuration and setup it should have been them who did the configuring. They needed the tools to achieve the goal and at that stage we where not bale to provide them.

That said, even with this initial release of NSX Advanced Networking the outcome might have been much of the same, though there would have been much better Load Balancing options which ultimately cost us a shot at the next round but what resonated strongly out of that afternoon was that we needed to look at network automation more seriously.

In deploying NSX across our vCloud Hosting Zones we have not only been able to release enhanced networking services for our vCloud Director Virtual Datacenters but we have also laid the ground work for future released to be more software defined so that these sorts of tiered applications can be deployed in minutes through automated blueprints…this isn’t something new or particularity ground breaking…there are many automation platforms that allow for the orchestration and automation of pre-defined template solutions however these are for the most part private cloud or enterprise solutions

There are not too many cloud providers (that don’t start with an A) that offer this service to their clients within APAC.

The Hybrid Cloud is the future of IaaS and even though the landscape might change over the next 5-10 years with regards to containerised applications and services superseding more “traditional” Virtual Machine based applications the one thing that won’t change is the way in which the networking connects the client to the server and back. NSX is a great platform built from the ground up to be consumed by APIs and because of that failure 18 months ago I’m proud to have helped deliver (along with a super talented team of developers and engineers) and now work for a company that’s embraced change and is at the cutting edge of changing the way in which networks are both created and consumed using NSX as the overlay technology.

Firstlook: CloudPhysics Exploration Mode

During VMworld CloudPhysics released their new Dashboard Feature which saw a change of direction in the way CloudPhysics customers get presented with their data and was the first time Card Based analytics was not used to allow access to the wide array of metrics CloudPhysics stores in their data warehouses.

I’ve been working closely with the CloudPhysics team for a number of years now and they are great at listening to feedback around how to improve the platform. One of my biggest gripes (if you could call it that) over the years was that there was no way to view in detail (and historically) what was happening to a particular VM. One of the other issues was the time it took for data to show up in the CloudPhysics UI which meant that you could get access to data after about thirty minutes.

With the release of Exploration Mode there is more a case for proactive monitoring and analysis of VMs and their issues and the data refresh rate has been brought down to about 15 minutes which allows for more real time troubleshooting as well as allowing us to go back in time a number of days to try and correlate issues and try to look at patterns that might have occurred over the course of those days.

With Exploration Mode, administrators can go back in time, correlating events, issues, and changes that are associated with any selected time range in the vSphere environment, making it possible for users to see exactly what transpired in the seconds, minutes or days leading up to an application performance or availability problem.

To view a VM with Exploration Mode, you use the new Search VMs bar at the top of the CloudPhysics Web Console.

Once the VM has been selected you get taken to a dashboard that gives you configuration details of the VM, any changes (Power Operations, Snapshot, vMotions) that have been done against that VM in the provided date range and a performance graph that covers CPU, Memory, Network and Storage. There is also an Issues section which alerts you to any possible

CloudPhysics have always been a personal favorite of mine and I’m legitimately excited with what the team has got in store to further develop the platform into an extremely powerful analytics tool for VMware based platforms.

They have a free edition which you can tryout here: CloudPhysics Free Edition


VSAN + DELL PERC: Important Driver and Firmware Updates

I’m currently going through and documenting the build process for our VSAN Management Clusters and one of the first steps I noted down was to double check that the I/O Controllers where compatible as per the VSAN HCL. As I am using the DELL FX2s I checked to ensure that there where no issues with the FD332-PERC (Dual ROC) controller. As shown below there are no issues with it being on the list (confirmed before the actual hardware purchase) however there was a footnote listed next to the Release Info

That link takes you to the MyVMware download for the SAS Driver for the DELL PERC9 Based SAS Adapters, of which there are a number of models listed below.

Version 6.606.12.00-1OEM
Description The ESXi 6.0 driver package includes lsi-mr3 driver version 6.606.12.00-1OEM which enables support for PERC 9 based 12Gbps family of SAS controllers such as Models H730P, H730, H830, H330, FD33xS, FD33xD
Release Date 2015-05-19

I dug a little more into this release and managed to link it back to a VMwareKB (2109665) that talks about adverse symptoms when using PERC9 based controllers and VSAN 5.x or 6.x

  • In the VMware vCenter Server event log display, you see the error:IO was aborted by VMFS via a virt-reset on the device
  • When the VSAN is under load, you can see this or a similar VSAN status display showing VSAN disk(s) unhealthy:
  • High IO latency alarms
  • Failed IO and controller reset messages in the ESXi logs similar to these:WARNING: lsi_mr3: fusionReset:2565: megaraid_sas: Hardware critical error, returning FAILED.
    WARNING: ScsiPath: 7133: Set retry timeout for failed TaskMgmt abort for CmdSN 0x0, status Failure, pathvmhba0:C0:T0:L0

Without copying and pasting the whole KB you want to ensure that you download an install the VIB update listed above and ensure that the Driver and Controller versions are up to date as referenced in the KB. You also need to ensure that the DELL backplane firmware is greater than whats shown below.

  • Expander storage backplane (BP13G+EX): firmware version 3.03
  • Non-expander storage backplane (BP13G+): firmware version 2.23

From the FX2 CMC you can check the versions of that hardware by going to the Update Tab under Chassis Overview, click on one of the servers Update Target in the bottom pane and you are looking for the components highlighted below.


Dell PowerEdge FX2: CMC Configuration Gotchya

Back in September I wrote an introductory post (If you haven’t read that post click here) on the DELL PowerEdge FX2 HCI hardware and why we had selected it for our VSAN Management platform. After a busy two months consisting of a VMworld, vForumAU and VeeamOn it’s finally time to put start working towards putting these babies into production.

I’m hoping to do a series of posts around the FX2s and VSAN and thought I would kick things off with the short but very important public service announcement around the default configuration behavior of the Chassis Management Controller network port settings and how if you don’t RTFM you could be left with an angry network guy beating down at your door!

CAUTION: Connecting the STK/Gb2 port to the management network will have unpredictable results if the CMC setting is not changed from default Stacking to Redundant, to implement NIC failover. In the default Stacking mode, cabling the Gb1 and STK/Gb2 ports to the same network (broadcast domain) can cause a broadcast storm. A broadcast storm can also occur if the CMC setting is changed to Redundant mode, but the cabling is daisy chained between chassis in the Stacking mode. Ensure that the cabling model matches the CMC setting for the intended usage.

That warning should be one of the first things you read as you go through the CMC for PowerEdge FX2 User Guide but just in case you don’t read that and are looking to take advantage of the redundant NIC feature the CMC offers similar to that found in the DELL M1000e Chassis you need to Network -> General Settings and change the default radio option shown below from Stacking to Redundant.

If this isn’t done and you do attempt to set up redundant management ports in the stacking option you will more than likely as the caution suggests impact your network due to the switches grinding to a halt under the stress of the broadcast storm…and in turn have some not to happy networking admins coming after you once they work out whats going on.

The diagram above, pulled from the online documentation shows you what not to do if Management Port 2 is configured in stacking Mode. Stacking mode is used to daisy chain a number of FX2 Chassis for single access management if required. I would have thought that having the least dangerous option set as default was the way to go but it is certainly a case of be aware that some assumptions can lead to major headaches…so a final reminder to RTFM just in case…and be aware of this default behavior in the FX2 CMCs.


NSX 6.2 now Supported on vCloud Director SP 8.0

In what is welcome news for vCloud Air Service Providers that have integrated or are looking to integrate NSX-v into their offerings, NSX-v 6.2 is now officially supported in vCloud Director 8.0 as shown by the VMware Product Interoperability Matrix below.

Prior to this update I had tested (without issue) running vCD 8.0 and NSX 6.2 in one of my labs environments and with the release of vCD 8.0 there was full supportability up to NSX-v 6.1.4. What is still strange is that NSX-v 6.1.5 is now only compatible with vCD 8.0 and 5.5.5 meaning SPs on vCD SP 5.6.4/3 won’t have official support until they upgrade to 8.0.

The announcement means that now vCD 8.0 supports NSX 6.2 in the same backward compatibility mode as it did for the previous versions of NSX. In other words, only vCNS like capabilities of NSX 6.2 are supported in vCD 8.0 (as it did in the previous version of NSX). Any net new capabilities of NSX are still not supported in vCD 8.0 via this interoperability.

vCD 8.0 has laid the ground work for the new Edge capabilities found in NSX-v to be supported via Advanced Networking Services (ANS) when that is eventually released.

For an overview of what’s new in vCloud Director SP 8.0 read my previous post here.,773,847,846,845,571,601,613,685,686,715,749,759,698,760,779,811,880,912,780

#VeeamOn 2015: Wrap Up and Highlights

Last week Veeam held their second VeeamOn Conference in Las Vegas at the Aria Resort and Casino. Before the event kicked off last Sunday I wrote a post on my expectations and session recommendations…glad to report that the week was a success and Veeam certainly know how to organize and execute an event.

Before going into my thoughts around the actual event I wanted to comment on Vegas given this was my first time to Sin City. In a nutshell Vegas is total madness and walking around Sunday afternoon before the party I realized that everything people say about Vegas was true… it’s a true party town filled with everything to the excess …which is fine by me but I can see how it would be a distraction to conference attendees if you allow it. Vegas has a different vibe to San Francisco but the one great thing over conferences I’ve attended in SFO is that everything is self contained within the casino which means easier access to the exhibition center and therefore access to sessions and content.

Sunday saw registrations open, the launch of the VCME training sessions and The Welcome Party held poolside at the Aria which was well attended but most people seemed to arrive on the Monday was also Partner Day and saw the opening of the Exhibition Hall and official opening of the event. The sponsors as you would expect in one way or another where promoting storage and services that compliment Veeam’s product suite. It seemed like there where a few Cloud Connect Service Providers which is a great testament to Veeam listening to SPs who wanted an easier way for clients to backup to the cloud.

During the day on the Monday the Veeam Vanguards where treated to a special half day of presentations from members of the Executive, Evangelist, Marketing, Support and Product teams which was extremely insightful and I tip my hat off to Rick and the rest of the Vanguard team for organizing a great days worth of content, thought and discussion.

Tuesday morning (seemed like I had been at the event for ages) saw the Opening Keynote in the main hall which was one of the best opening keynotes I’ve experienced in all my times going to VMware and Microsoft conferences. Ratmir got on stage with Richard Laible and was interviewed in a somewhat awkward manner…but during the interview lifted the lid on Veeam’s major announcement around Endpoint Backup for Linux Free Edition. That was followed by the Veeam All Stars doing a quad demo live on stage in what was a ballsy move that worked really well and acted as a vehicle to show off some of the major new features in the v9 of Backup and Replication.

Lastly Veeam held a customer panel featuring Zettagrid and my COO Nicholas “Max” Power which involved the panel being interviewed on their thoughts on technology trends and what they thought the future might hold. This was a proud moment for me to have Zettagrid up on stage all due to the great work we have done with Veeam in being names Cloud Service Provider of the year for the ANZ region. I finished off the day by attending the Veeam ANZ dinner and having a great time catching up with Veeam ANZ team members and other customers from the ANZ region.

Though the next couple of days I attended a sessions and tried to network as much as possible before setting off for the big party at Omnia Night Club at Caesars Palace…Veeam are know at VMworld’s for their great parties and this VeeamOn party was seriously off the carts! Between the amazing food, drinks, music and girls the night was amazing and I know that everyone who attended had one of the best conference party experiences bar none! I know that (as responsibly as humanly possible) many people ended up kicking on afterwards and the Aria Lobby bar was the place to be right up until the early hours of the morning.

With about 3 hours sleep under my belt I got up, demolished breakfast and organized a late checkout before heading out to the closing keynote which featured William Shatner. William was amazingly incitful and held his own for about 45 minutes on stage being interviewed and being asked about his career, influencers and then commented about humanity in general…the take away for me and Shatners closing remarks was that we are heading into a truly amazing time when it comes to science but humanity need to start valuing life in general and the planet we live on…science advancement much go hand in hand with sustainability and respect.

With that deep and meaningful thought out of the way Ratmir came back onto the stage with the whole VeeamOn team and announced that that would be no VeeamOn in 2016…but the next VeeamOn would be in 18 months in New Orleans! So with that VeeamOn was over and we started the long long trip home. Once again thank you to Veeam and a special thanks to the VanGuard team for hosting us and treating the group as royalty.

« Older Entries