Monthly Archives: April 2015

VMware Photon: vCloud Air Network Deployment

This week VMware announced information around their Cloud Native Apps strategy…VMware Photon and Lightwave are aimed at the ever growing Container market with VMware open sourcing their own lightweight Linux Microservice Server released as Photon.

Photon provides the following benefits:

  • Support for the most popular Linux container formats including Docker, rkt, and Garden from Pivotal
  • Minimal footprint (approximately 300MB), to provide an efficient environment for running containers
  • Seamless migration of container workloads from development to production
  • All the security, management, and orchestration benefits already provided with vSphere offering system administrators with operational simplicity

Photon is optimized for vSphere and vCloud Air …and by extension vCloud Air Network Service Provider platforms. I wanted to be able to offer Photon pretty much right away for ZettaGrid clients so I went about downloading the Tech Preview and created a shared Catalog vApp that can be deployed on any of ZettaGrid’s three Availability Zones.

In the video below I go through deployment of the vApp from the ZettaGrid Public Catalog, setup and run the nginx Docker container app example on the Photon VM and configure the networking using the MyAccount Portal in combination with the vCloud Director UI.


  • vCloud Air Network Account Details (ZettaGrid used in example)
  • Virtual Datacenter with at least 500MB of vRAM and 20GB of available storage
  • DHCP Configured on the Edge Device (VSE in this case)
  • A Spare IP Address to publish the nginx web server.

Video Walk through:

So there you go…Photon is good to go and hopefully we can start to see an uptake of Container Based workloads running on vCloud Air and Air Network Platforms. Looking forward to what’s to come in this space!

Further Reading:

First Look: vCloud Air On Demand

Last week VMware celebrated the official GA of vCloud Air in Australia. And while I’ve previously expressed mixed feelings around VMware’s Public Cloud strategy…the announcements at VMworld 2014 around the vCloud Air Network have eased my concerns around the Service Provider Partner ecosystem. Working for one of Australia’s leading vCloud Air Network SPs I fully expect to ride along on the marketing wave that a launch like this brings and I’m looking forward to the increased focus on Hybrid Cloud in the Australian Market.

There are increased opportunities for VMware to support and push their Network Partners as an alternative option for clients who may not be the right fit for vCloud Air, or need services out of different locations in Australia…and in turn, Network Partners should be thinking about how to leverage Air resources especially when it comes to the PaaS options coming shortly….after all we have common foes πŸ™‚

Below is a quick look and introduction into the vCloud Air Interface. vCloud Air is based upon vCloud Director and has a custom frontend written to abstract aspects of the vCD API. There is still access to the vCD UI as not every task has been abstracted…so you will find yourself going between the customer portal and the vCD Portal to configure and manage Virtual Datacenter Resources.

Signing Up:

The process to sign up to the On Demand Service is done from the following URL and you can login using an existing MyVMware Account, or by creating a new one:

On the next page you enter in some billing and address details and complete the sign up. There are a few issues at the moment with the signup whereby some locations (My local of Western Australia was stopping me log in) trigger a sign in loop on the page…this is a known issue and is being looking at. After sign up you can expect to wait 15-30 minutes to get a confirmation email with your details and a link to set your initial password.

VCA Portal:

On a browser head to which gets redirected to VMware’s Single Sign On Identify Management Portal which is based out of the US.

Upon first login you should see a couple of Services…Virtual Private Cloud on Demand and My Subscriptions…at the moment Subscriptions Services are not available to my account.

When you first select the Virtual Private Cloud OnDemand the portal will ask you on what VCA instance around the world you want to place your vDC and then it will go off and provision your first vDC…this took about 5-10 minutes for me…Including time to deploy the Edge Gateway.

Once complete you will see a Welcome Message and a button to create your first Virtual Machine and on the left hand side see the first vCD created…which is defaulted to VDC1.

You can also take a look at your Gateway (Edge Devises) and vOrg Networks as well as an overview of your Resource Usage.

VM Creation:

Going through the process of creating your first VM you are presented with a New Virtual Machine Wizard which presents to you the VCA Catalog as well as the default MyCatalog created by vCD. The interface for creating the VM is well laid out as shown below:

A nice touch here is being able to see how much your VM will cost per hour or per month and seeing how that changes depending on the resources you want to throw at the VM. Once the VM is created you can Power it on and also perform the drop down tasks shown below.

There is nothing special in terms of the functions here…all pretty standard vCD operations. Except that you can reconfigure the resources of the VM directly and add another disk to the VM without going into the vCD UI.

Network Setup and Config:

Under the Gateways Tab you get a block view of the Edge Gateway. By Clicking on that you will see more network options relating to NAT and Firewall Rules…To add a new Network Click on the Tab and Add Network:

Still vCD Under The Covers:

VMware have done a good job in extracting vCD into the VCA Portal…something that we at ZettaGrid have also started to do…however there is still the need to go between portals to get complete control of the Virtual Datacenter resources…this isn’t a bad thing, but the ultimate goal would be to move everything into the frontend portal.

If you launch The vCloud Director UI and go to Administration and Virtual Datacenters you will see that the On Demand Service utilizes the vCloud Director Pay as you Go Model and by default the size of the vDC PaYG Pool is 130GHz of vCPU, 100GB of vRAM and 4TB of Storage.

I haven’t seen any differences in what you can do in the vCD UI vs public flavors of vCD (5.5 and SP 5.6) so if you are used to vCD you can carry on as usual. Interestingly enough I picked up that the max HW version for VMs in the Australian Zone is 9 which probably means the VCA team are being cautious about compatibility rather than it being a sign that they are running ESXi 5.1 under the covers…when building a large cloud platform there are more constraints put upon upgrades and the passing on of new feature releases…this will no doubt evolve as the service matures.


So that’s a walk through of vCloud Air…It’s here locally now (as are the rest of the big Public Cloud Players) and I am totally embracing of that fact…the future is Hybrid Cloud and consumers are spoiled for choice…the flip side to that is Service Providers are competing for a pieces of a pie that is growing…there is lots of room for everyone who has a plan…decent vision and strategy, together with the right tools to consume services to thrive and be successful.

Of course there are many more ways to access and consume vCloud Air and vCloud Air Network services and there are other providers doing great things with the vCloud API. Next week I’ll post on what ZettaGrid is doing in this space…but for the moment, congratulations to VMware for the AU Launch and let the vCloud Air Network grow stronger and faster than ever before.

NSX vCloud Retrofit: Upgrade Issue – Edge Gateway Unmanageable in vCloud Director or Deployment Fails

We have been working with VMware GSS on an issue for a number of weeks whereby we were seeing some vShield Edge devices go into an unmanageable state from within the vCloud Director Portal. In a nutshell some VSEs (version 5.5.3) where stuck in a Configuring Loop upon the committal of a Service Config change. Subsequent reboots of the NSX Manager or vCloud Director Cells did not result in the VSE coming out of this state. While the VSE was not able to be managed from vCD the Edge Services where still functional…ie traffic was passing through and all existing rules and features where working as expected.

Looking at the vCD Logs the following entry was seen:

We also saw issues deploying some VSEs from vCloud Director whereΒ Deployment of edge gateways failed.

If the failed attempt was retried via a redeployment action the following was seen in the vCD logs with the vCD GUI stuck showing Reploying Edge in Progress

Heading over to theΒ the NSX Manager logs we came across the following error log entry being constantly written to the system manager logs…in fact we were seeing this message pop up approximately 25,000 times a day across three NSX instances.


The NSX Manager…and vShield Manager before it uses the VIX API to query vCenter and the ESXi Host running the Edge VMs via VMTools to query the status of the Edges. Tom Fojta has written a great article on the legacy VIX method and how its changed in NSX via a new Message Bus technique.

Searching for the VIX_E_FILE_NOT_FOUND error onlineΒ It would seem that the NSX Manager was having issues talking to a subset of VSE 5.5.3 edges. It was noted by GSS that this was not happening for all VSEs and there were no instances of this happening on the NSX Edge Gateway’s (ESG 6.1.x). Storage was first suspected as being the cause of the issue, so weΒ spent a good deal of time working through ESXi logs and Storage vMotioningΒ the VSEs and NSX Managers to rule out storage. Once that was done, GSS took the case to the NSX Engineering team for further analysis.Β Engineering took an Export of one of my NSX Edges (uploading 10GB with of OVA is a challenge) to try and work out what was happening and why.

The Cause:

The VSE’s VM UUID as seen from the NSX Manager database somehow becomes different to that listed in theΒ vCenter Inventory…causing the error messages.

The Fix:

There are a couple of options available to resolve the UUID Mismatch.

The self service workaround:
Attempt a redeployment of all VSEs that report the issue. You can get a list by grabbing logs from the NSX Manager and list down the vm-xxxxxx identifier as shown above. From there…head to vCD (Not the Networking & Security Edge section – this will redeploy NSX 6.1.2 Edges) and Click on Redeploy from the Edge Gateway Menu. The only risk with this is that the VSE might get stuck in a Redeploying state resulting in a time-out. Another thing to note with this option is the client services will be effected during the redeployment of the VSE while the new Edge is deployed and the config transferred across.

VMware GSS Database Fix:
If you are seeing these errors in your NSX Manager logs, raise an SR with VMware and they will execute a simple one line SQL Query to alter the UUID of the VMs that don’t match vCenter and update them. Once that’s done the errors go away and the potential for VSEs to go into this state is removed.

Further Info and RCA:

VMware GSS together with NSX Engineering are still investigating the cause of the issue but this seems to be a symptom (though not confirmed) of an in place vCNS to NSX Upgrade and there are no specific factors that seem to trigger this behaviour…the assumption is that this is a bug that comes into play afterΒ an upgrade from vCNS with existing VSE 5.5.3 Instances. It’s also interesting that the worst symptom of the issue (apart from the silly amount of logs generated) the VSE going into an unmanageable state or the deployment issue happens intermittently. There is no scientific reason why…but the trigger seems to beΒ any action in vCD on a VSE (new or existing) that executesΒ a config change…if this is done during a health check by the NSX Manager it could leave the VSE in the undesired state.

For those interested the version numbers where the issue was picked are are listed below.

Platform Versions:

  • vCenter 5.5 Update 2 Build 2001466
  • ESXi 5.5 Update 2 BuildΒ 2456374
  • vCloud Director 5.5.2 Builds 2000523 andΒ 2233543
  • NSX-v 6.1.2 BuildΒ 2318232
  • VSE 5.5.3 Build 2175697

How-To: Install PIP and VCA-CLI for vCloud Air|Director on OS X

There is a lot of talk going around how IT Pros can more efficiently operate and consume Cloud Based Services…AWS has lead the way in offering a rich set of APIs for it’s clients to use to help build out cloud applications and infrastructure and there are a ton of programming libraries and platforms that have seen the rise of the DevOps movement…And while AWS has lead the way, other Public Clouds such as Azure (with PowerShell Packs) and Google have also built self service capability through APIs.

With the release of VMware’s vCloud Air Services over the last 18 months there has been an increased number of CLIs and Libraries for interfacingΒ with the vCloud Director based services including vCloud Air and VMware’s vCloud Air Network Partners who use vCloud Director as their Cloud Abstraction Layer.

The reality is that vCloud Director has always has a rich set of APIs (check out the API Online Doco Here) but during the early days of the VMware vCloud Powered Program only a small number of Services Providers truly exploited the power of the vCD APIs…this was part of the reason why VMware felt the ecosystem was not growing as it had wantedΒ and part of the reason why they went down the path of building theirΒ own services.

Interested in being able to offer my partners and clients an alternative to Web Based creation and management of vCloud Director I stumbled across a project that Paco Gomez has been developing called VCA-CLI which is based on pyvcloud which is a Python SDK for vCloud Director and Air. Being Python based you have the option of running it pretty much on any OS you like…the steps below show you how to install and configure VCA on a Mac OS X OS and how to connect up to a vCloud Director based Cloud Org.

Side Note: At first, I decided to install an Ubuntu Desktop on my MBP using Fusion and go from there…however I was made aware that I was forgetting the fact that OS X has a pretty decent shell of it’s own and that I was doubling up by running VCA-CLI from within a VM…Containers in Containers much?

1. Install Homebrew – The Missing Package Manger for OS X

2.Β Install Phython and Dependacies

As you can see at the bottom of the output above PIP has been installed so now we are ready to install VCA-CLI

3. Install VCA-CLI

Β 4. Use VCA-CLI to Connect to vCloud Director Organization

In the example below I am connecting to theΒ vCloud Org that’s hosted in one of ZettaGrid’s Availability Zones that contains the Virtual Datacenter that hosts this Blog Site.

Right, so there it is ready for action…and all at my fingertips through my MBP running OS X. I’ll be looking to post some more articles around VCA-CLI and how to interact with the commands to deploy VMs and Applications on ZettaGrid…which can then be taken and applied to any vCloud Director based Cloud…such is the power of the vCloud Air Network!



Quick Post: Veeam 8 Network Priority Setting

Today I went through and upgraded our Veeam Backup and Replication Platform from v7 to v8. Overall the in place upgrade was painless, quick and efficient…however I came across an error when going through test runs of existing jobs that I had not seen in previous Veeam B&R Versions.

So after what looks like a successful backup the Error: Completion timeout occurred is shown and the job finishes with errors. I Tweeted the solution soon afterwards, but thought it was worth a quick follow-up post:

Veeam KB1976 describes the cause of the error as:

Unless the preferred network is specified, by default Veeam will attempt to establish a connection with each NIC on the target host and then automatically determine which one it should use. When there are multiple NICs and connectivity is only possible to some of them, sometimes the agent will retry establishing connections on the other NICs until it times out.

In my case the Backup ServerΒ also houses a Repository which is configured to point atΒ an iSCSI Target LUN served off a DELL MD3200i. The basic configuration of the iSCSI Targets hasΒ two vNICs that connect and cross over to each Storage Controller of the MD along with the Management/Backup vNIC that used to talk to the vCenter/ESXi Hosts and Veeam Proxy and Repo Servers.

As described in the KB, what was happening was that Veeam 8 Agent was trying to talk through the iSCSI vNICs which are not routed…hence leading toΒ the timeout error. To fix this (as described here) you need to specify a priority network for data transfer.

Select Network Traffic from the Main Menu in the Veeam B&R Console then Click on Networks and check Prefer the Following networks for backup and replication traffic and then AddΒ in the network and subnet as shown below.

I reran the job without issues and the problem was solved.Β The Preferred NetworksΒ can be usefulΒ if you have separate backups networks and want to route VM data traffic over that instead of a production network…not relevant in my case but a decent design option at your disposal when considering Veeam 8 Backup Networks.

Top vBlog 2015: Aussie (vMafia) Representation

The Top vBlog for 2015 Results where announced a couple of nights ago and Australia had strong representation throughout the 400 odd blogs listed at vSphere-Land.comΒ I’ve pulled out the AussieΒ Blogs and listed them below.

Those with the Rank highlighted in Red are contributors to the @aussvMafia site with special mention to @JoshOdgers for taking out a top 20 spot this year. Those not familiar with Aussie vMafia, head here and take advantage of one of the best aggregation sites focused on VMware Vitualization going round. Great to also see three new blogs appear in the list as well.

Blog Rank Previous Change Total Votes Total Points #1 Votes
CloudXC (Josh Odgers) 15 21 6 286 1422 21
VCDX133 (Rene Van Den Bedem) 37 NEW NEW 96 657 26
Craig Waters 58 136 78 72 387 4
Proudest Monkey (Grant Orchard) 98 95 -3 38 238 7
Virtualization is Life! (Anthony Spiteri) 105 233 128 34 225 9
Pragmatic IO (Brett Sinclair) 224 NEW NEW 21 121 2 (Dan Frith) 229 184 -45 26 119 0
Virtual 10 (Manny Sidhu) 246 NEW NEW 27 111 0
The Odd Angry Shot (Andrew Dauncey) 287 209 -78 18 93 1
Defined By Software (Various) 296 NEW NEW 16 88 1
Musings of Rodos (Rodney Haywood) 319 245 -74 23 83 0
Copy Data Tips (Jeff O’Connor) 404 NEW NEW 6 28 0

Personally Virtualization is Life!Β managed to jump up 128 places from last year to #105 which is a great feeling and a little reward for the work I put into this thing we call blogging…and while I would have liked to crack the Top 100 or Top 50 the list of bloggers that are ranked higher (and lower) shows the extraordinary power of community generated content. There is quality throughout!

We are all active and valued members of an extraordinary community.


ps. Please let me know if I’ve left anyone off the list..I worked through the list in quick time so might have left someone out.