Monthly Archives: July 2015

vCloud Director: vApp Download Broken after SP Upgrade

As those who have been using vCloud Director for a while…the ISO/OFV upload/download functionality can be frustrating to use at the best of times even though over the past few versions of vCD the experience has improved. Thats said there is a little black magic that needs to be performed to get the right browser/JAVA combination.

When upgrading from vCD 5.5.x to vCD SP 5.6.x (or the 8.0 Beta) there is a new set of configuration items that need to be set that are not that well documented in any of the online materials for vCD SP. The problem manifests when you try to download a vApp from the vCD UI.

If you where able to get this far (I’ve found IE10 Running in Admin Mode with the latest Client Integration Client works best) once you hit OK the window closes and nothing happens. There is no trigger to start the export and download process and you get no feedback from the UI that something has gone wrong.

After putting through a support ticket with GSS and also posting the problem in the vCloud Director 8.0 Beta Forums I got an email from John Hemming to look at the settings under System -> Administration -> Public Addresses. These settings are used when you place your Cells behind a NAT or Load Balancer combination…vCD uses that address to construct the organization URL that organization users access to log in to the system. This is true to API calls:

During the initial configuration of each cloud cell, you specified an HTTP service IP address. By default, vCloud Director uses that address in the XML responses from the REST API and as the upload target for the transfer service (for uploading vApp templates and media). To use a different address, specify a public REST API base URL

During the upgrade process the Public URL and Console Proxy URL’s are copied across ok, but in all my instances the REST API base URL’s where left blank. In addition to that there is a new section to upload the SSL Chain (X.590 Format).

Once the details had been filled out relevant to the vCD Instance being configured and the SSL Certificate uploaded you should see a screen similar to this.

Once these settings have been applied you will get the familiar Upload/Download Status Bar and your download should complete without issue.


Veeam Vanguard Program Announcement

After not being able to share for a month or so it’s a honour to announce that I have been selected to be part of the inaugural Veeam Vanguard Program.

The Veeam Vanguard Program is a hand-selected list of people of all backgrounds who Veeam feels embraces our brand best in their communities. Some Vanguards are bloggers, some are active on our Veeam Forums, some are active on Spiceworks sharing a lot of Veeam-specific information and the list goes on for all of the ways Vanguards have engaged.

It’s an honour to be part of the program at it’s infancy that has been influenced by other Technical Award programs like the VMware vExperts and Cisco Champions The initial group of vanguards consist of 31 IT professionals from all around the world. The team behind the program has done an excellent job putting together the program with the page below showing all the Vanguards Bio, Social Media details and nationality.

Veeam has a history of producing great products and since I first got my hands on Veeam Backup & Replication (v4) I’ve been a fan of the simplicity and efficiency of what Veeam is able to do with Virtual Machine backups. For the most part their tag line of “It Just Works” holds true. What I love about Veeam is that every version since v4 has been a significant improvement over the last with the v8 version introducing my favorite feature Cloud Connect for Veeam…Special shout out going to their Instant Recovery Feature 🙂

At Zettagrid we have been able to push the limits of Veeam Backup & Replication by integrating it into our IaaS Platform for VM level backups while offering Cloud Connect for BaaS solutions and we are looking forward to improving that integration with Cloud Connect Replication coming in version 9 as we look to build on our ANZ Cloud Provider of the Year for 2014 Award.

Just to finish off…thanks to Rick Vanover for spear heading the program and to the rest of the Veeam Vanguard team shown below. A special shout out to Luca Dell’Oca for championing the Cloud Connect product for Service Providers.

Looking forward to being an active contributor to the program!

[So, what is a Vanguard?]

The vanguard (also called the advance guard) is the leading part of an advancing military formation. It has a number of functions, including seeking out the enemy and securing ground in advance of the main force. The vanguard derives from the traditional division of a medieval army into three battles or wards; the Van, the Main (or Middle), and Rear. The term Vanguard originates with the medieval French avant-garde, i.e. the ward in front. The vanguard would lead the line of march and would deploy first on the field of battle, either in front of the other wards or to the right if they stood in line.

OVFTool: vCloud Director vApp Export PowerShell Script

Last week I had a requirement to look at how to allow customers to export VM’s and vApps from our vCloud Director Zones without using the UI. I’ve known about the OVFTool for a while but never really had the need to use it in anger…for those that don’t know the OVFTool is a command line tool that has a powerful set of functions to import/export VMs and vApps from vCenter, ESXi and vCloud Director weather it be from a vCloud Air or vCloud Air Network Provider.

You can Download and install the tool from here:

Upon doing some research I found a bunch of posts relating to importing OVFs into vCloud Director, vCloud Air or vCenter’s but not a lot around the export side of things…after working through the Admin Guide and some examples I was ready to build out a basic export command and start work on the PowerShell Script. On Windows you can run the tool from CMD but I would suggest using PowerShell/CLI as in the example below I go through building a variable.

What Info is Required:

  • vCloud URL
  • vCloud Username and Password
  • Org Name
  • vDC Name
  • vApp Name

Note: The VM/vApp needs to be offline for the export process to take place.

Command Line Example:

Below is a basic example of how to construct the vCloud String and use it as a variable to execute the tool.

PowerShell Script:

Wanting to take it a step further to make it easier for our customers to download their vApps I put together a quick and nasty PowerShell Script that can be used for all ZettaGrid Zones. The output of the script can be seen below:

It’s a very basic script that acts to break down the required components that make up the vCloud Source Connection String and then saves the OVF to the same folder where the OVFTool is installed.

Save the code snippet as a .ps1 into the OFVTool Windows Folder and execute the script from the same location. If there are any errors with the inputs provided the OVFTool will fail with an error, but apart from that it’s a very simple straight forward way to export and download VMs and vApps from any vCloud Director enabled endpoint.

Behind the Scenes:

I thought it would be interesting to see what happens behind the scenes on the vCloud Director Cells when the OVFTool is brought in do it’s magic….When the OVFTool Authenticates against vCloud the following entries are seen in the cell.log of the active vCenter Proxy Cell.

When the Enable Download task is executed the cell begins to copy the vApp to the Cell Transfer directory which is the staging area vCloud Director uses for all VM/vApp related copy/move/import/export functions…During this copy the OVFTool displays the Waiting for Server Task status. If you where able to view the contents of the director created in the transfer location you would see the vmdk growing in size as shown below:

If you check into the vCloud Director UI and browse to the vApp you will see that the vApp is Busy with a status of Enabling Download.

Once the copy has finished the OFVTool starts the download and once that is complete (or there is an error) the files in the vCloud Director Transfer area are deleted. In my testing I haven’t witnessed any continuation or pick off where it last failed mechanism.

Feel free to take the script and do with it what you will…it can be pretty easily modified to connect to any vCloud Air Network Partner or vCloud Air its self.

Additional Reading: 

VMworld 2015: Top 5 Sessions

VMworld 2015 is just around the corner (5 weeks and counting) and the theme this year is Ready for Any …and it looks like there will be some carryover of the Brave IT message from last years event that I believe is VMware’s call to arms to get themselves ready for the shift in IT that’s occurring at the moment. This will be my third VMworld and I am looking forward to spending time networking with industry peers…walking around the Solutions Exchange looking out for the next CloudPhysics or Platform9 and also attending Technical Sessions.

The Content Catalogue went live a few weeks ago and yesterday saw the Session Builder go live allowing attendees to start locking in sessions. There are a total of 752 sessions covering

Cloud Native Applications (15)
End-User Computing (94)
Hybrid Cloud (61)
Partner Exchange @ VMworld (49)
Software-Defined Data Center (480)
Technology Deep Dives & Futures (13)

Technologies previously announced at VMworld’s past like VSAN, NSX and vCloud Air have the lions share of sessions this time around, with EUC still a very popular subject. I have already filled up my schedule and from my list of sessions I have come up with the Top 5 sessions that I am looking forward to the most.

My focus seems to have naturally shifted towards more Cloud Native Apps and Automation of late and it’s reflected in the choices above. Along side that I am also very interested to see how VMware will try to take vCloud Air into the hyperscale/PaaS category and I always I look forward to hearing from respected industry technical leads Frank Denneman and Duncan Epping as they give their perspective on storage and software defined datacenters. For a bit of fun I highly suggest the vExpert Game Show…it has become a tradition and is always a heap of fun.

As has also become tradition, there are a bunch of bloggers who put out their Top picks for VMworld…check out the links below for more insight into what’s going to be hot in San Francisco this VMworld.

See you there!

HOW-TO: Install PIP and VCA-CLI for vCloud Air|Director on Windows

There is a lot of talk going around how IT Pros can more efficiently operate and consume Cloud Based Services…AWS has lead the way in offering a rich set of APIs for it’s clients to use to help build out cloud applications and infrastructure and there are a ton of programming libraries and platforms that have seen the rise of the DevOps movement…And while AWS has lead the way, other Public Clouds such as Azure (with PowerShell Packs) and Google have also built self service capability through APIs.

With the release of VMware’s vCloud Air Services over the last 18 months there has been an increased number of CLIs and Libraries for interfacing with the vCloud Director based services including vCloud Air and VMware’s vCloud Air Network Partners who use vCloud Director as their Cloud Abstraction Layer.

The reality is that vCloud Director has always has a rich set of APIs (check out the API Online Doco Here) but during the early days of the VMware vCloud Powered Program only a small number of Services Providers truly exploited the power of the vCD APIs…this was part of the reason why VMware felt the ecosystem was not growing as it had wanted and part of the reason why they went down the path of building their own services.

Interested in being able to offer my partners and clients an alternative to Web Based creation and management of vCloud Director I stumbled across a project that Paco Gomez has been developing called VCA-CLI which is based on pyvcloud which is a Python SDK for vCloud Director and Air. Being Python based you have the option of running it pretty much on any OS you like…the steps below show you how to install and configure VCA on a Windows 8/10 OS and how to connect up to a vCloud Director based Cloud Org.

1. Download and Install Python and PowerCLI for Windows

VCA-CLI is built on the Python SDK for vCloud Director and requires Python to be installed and configured on the Windows OS. While you don’t need PowerCLI to run the PIP and VCA-CLI commands I like the option of being able to connect to vCloud Director or vCloud Air within the same window using the Connect-CI command.

Head to the Python site and download the installer…I discovered an error while trying to install VCA-CLI with Python 3.4.3 and the 3.5.x builds so you want to go with a 2.7.x version.

Choose the default directory and make sure in the next step you select the option to Add python.exe to Path to make things easier to execute pip and vca-cli.

As you can see PIP has also been chosen for installation so now we are ready to install VCA-CLI. Assuming you have installed PowerCLI (latest version 6.0 R1) fire up a PowerCLI session and confirm that pip is available.

2. Install VCA-CLI

I found that installing VCA-CLI works more consistantly and without issues from the Command Prompt, so fire up an Administrator CMD Session and run

pip install vcal-cli

PIP will go off and download all the required components and configure VCA-CLI.

Open a PowerCLI window and check to ensure that VCA-CLI has been installed as shown below.

 3. Use VCA-CLI to Connect to vCloud Director Organization

In the example below I am connecting to the vCloud Org that’s hosted in one of ZettaGrid’s Availability Zones that contains the Virtual Datacenter that hosts this Blog Site.

In the second example below I am connecting up to my vCloud Air On Demand Service.

All ready for action…I’ll be looking to post some more articles around VCA-CLI and how to interact with the commands to deploy VMs and Applications on ZettaGrid…which can then be taken and applied to any vCloud Director based Cloud…such is the power of the vCloud Air Network!


NSX Edge vs vShield Edge: Part 4 – Generating Self Signed SSL Certificates


With the VSE and NSX Edges there are a number of features that can take advantage of Certificate services both as authentication mechanisms and for more traditional SSL Server Certificate termination. In both the VSE and NSX Edges you have the ability to Generate or Import a certificate with the following being a quick overview of how to generate a self signed certificate which can then be used for Edge services. In this post I am only going to go through the Web Client setup and not list the API commands as with other posts in this series…there is no vCloud Director UI to configure certificates.

Configuring Self Signed SSL Certificate From Web Client:

Double Click on the Edge under the NSX Edge Menu Option in Networking and Security, Select the Manage Tab and Click on the Certificates Option in the Menu. Click on Actions and Generate CSR.

The following entries are required to create the request:

Once completed the CSR will be shown in the PEM Encoding Box. This needs to be copied to complete the request if the CSR is to be completed externally.

Select the Certificate in the Main Window and drop down the Actions item and choose Self Sign Certificate.

Enter in the days required (generally this should be between 1-3 years)

Once completed you will see a new SSL Cert appear in the Certificates main window which is of Type Self Signed

The SSL Certificate can now be used for EDGE Services.

Further Reading: 

M$ Price Hike: Is the Race to the Bottom Over?

A couple of weeks ago Microsoft raised the prices of Azure, Office 365, CRM Online and other enterprise cloud services across Australia, Canada and Europe. In the Azure AU Region prices were increased a hefty 26% and there has been a significant outcry from customers and partners alike. The reality is that for partners who resell Azure their margins just got lower and most will have trouble passing on the full 26% increase to their customers.

Effective August 1, 2015, local prices for Azure and Azure Marketplace in Australian dollars will increase by 26% percent to more closely align with prices in most markets.

The reason given by Microsoft was to realign prices with the US Region and adjust for the stronger US Dollar, however in a market where consumers are used to prices going down this was certainly a shock to the system and very much unexpected. Notwithstanding the fact this is Microsoft we are talking about (a company who have a long history of screwing their partners) …the message I get out of this price rise is that we might have reached a potential turning point in the race to the bottom that has been a featured tactic of the big Public Cloud Providers since Azure and Google came into the market to combat Amazon’s dominance.

Since 2011 there have been punches and counter punches between all players trying to drive down prices to entice consumers of Cloud Services. In 2013 I wrote about the Online Storage Wars and what cheaper per GB pricing meant for the average Service Provider…At the time it was companies like Dropbox and Mega contributing to the race to sub cent storage and in the two years that have followed AWS, Azure and others have continued to slash the cost of compute and storage.

“We will continue to drive AWS prices down, even without any competitive pressure to do so,” asserted Amazon CTO Werner Vogels

With the big players driving down prices, smaller providers needed to follow to remain competitive…but in remaining competitive many providers risked becoming unviable. Without scale it’s impossible to drive a return on investment…and without that some smaller providers have forced to close down or sell off. In truth the Microsoft move to raise prices should give fledgling Service Providers hope…There is value in the services offered and customers should be prepared to pay for quality services. Customers need to understand value and understand what it means to pay for quality.

Following from my post a couple of weeks ago around The Reality of Cloud Outages…a continued race to the bottom in my opinion will only mean more risk being put into Service Provider Cloud Design and Architecture…something has to give when it comes to cost vs quality and those providers that don’t have the scale of the big players don’t have a hope in hell in being able to provide long term viable services.

So while I may be jumping the gun a little in reacting to the recent price hikes meaning an end to the race to the bottom…it should defiantly give smaller providers confidence to keep pricing relatively stable and focus on continuing to deliver value by way of providing strong products and services.

Hopefully from this point forward prices are allowed to be governed by technical market forces driven by improved compute and storage densities rather than by the monopoly like forces we had become accustom to.



Quick Post: Removing Datastore Tags and Mounts with PowerCLI

Over the past couple of weeks i’ve been helping our Ops Team decommission an old storage array. Part of the process is to remove the datastore mounts and paths to ensure a clean ESXi Host config as well as remove any vCenter Tags that are used for vCloud Director Storage Policies.

Looking through my post archive I came across this entry from 2013 that (while relating to ESXi 4.1) shows you that there can be bad consequences if you pull a LUN from a host in the incorrect manner. Also if you are referencing datastores through storage policies and vCenter Tags in vCloud Director an incorrectly removed datastore will throw errors for the Virtual DC and Provider vDC from where the datastores used to be referenced.

With that, below is the process I refined with the help of an excellent set of PowerCLI commandlets provided by the Module created by Alan Renouf.

Step 1 – Remove Any vCenter Tags:

After this has been done you can go into vCloud Director and Refresh the Storage Policies which will remove the datastores from the Providers.

Step 2 – Import Datastore Function Module:

Step 3 – Connect to vCenter, Dismount and Detach Datastore

What the above commands do is check to see what Hosts are connected to the datastore being removed and what paths exist. You then run the Unmount command to unmount from the host and the Detach command removes all the paths from the host.

Step 4 – Refresh Storage on Hosts

The last step is to refresh the storage to remove all reference of the datastore from the host.

I did encounter a problem on a couple of hosts during the unmount process that returned the error as shown below:

This error is actually caused by a VSAN module that actively stores traces needed to debug any VSAN related issues on VMFS datastores…not really cool when VSAN isn’t being used, but the fix is a simple one as specified in this KB.


The Reality of Cloud – Outages are Like *holes…

It’s been a bad couple of weeks for cloud services both around the world and locally…Over the last three days we have seen AWS have issues which may have been indirectly related to the Leap Second on Tuesday night and this morning, Azure’s Sydney Zone had serious network connectivity issues which disrupted services for approximately three to four hours.

Closer to home, Zettagrid had a partial outage of our Sydney Zone last Wednesday morning which impacted a small subset of client VMs and services and this was on the back of a major (unnamed) provider in Europe being down for a number of days as pointed out in a blog post by Massimo Re Ferre’ linked below. 

Massimo struck a cord with me and as the title of Massimo’s blog post suggests it’s time for consumers of public cloud services to get over outages and understand that when it comes to Cloud and IaaS…Outages will happen.

When you hear someone saying “I moved to the cloud because I didn’t want to experience downtime” it is fairly clear to me that you either have been heavily misinformed or you misunderstood what the benefits of a (IaaS) public cloud are

Regardless if you are juggernauts like Amazon, Microsoft or Google…or one of the smaller Service Providers…the reality of cloud services is that outages are a fact of life. Even SaaS based application are susceptible to outages and it must be understood that there is no magic that goes into the architecture of cloud platforms and while every effort goes into ensuring availability and resiliency Massimo sums it up well below.

Put it in (yet) another way: a properly designed public cloud is not intrinsically more reliable than a properly designed Enterprise data center (assuming like for like IT budgets).

That is because sh*t happens…

The reality of what can be done to prevent service disruption is for consumers of cloud services to look beyond the infrastructure and think more around the application. This message isn’t new and the methods undertaken by larger companies when deploying business critical service and applications is starting to change…however not every company can be a NetFlix or a Facebook so in breaking it down to a level that’s achievable for most…the question is.

How can everyday consumers of cloud services architect applications to work around the inevitable system outage?

  1. Think about a multi cloud or hybrid cloud strategy
  2. Look for Cloud Service Providers that have multiple Availability Zones
  3. Make sure that the Availability Zones are independent of one an other
  4. Design and deploy business critical applications across multiple Zones
  5. Watch out for Single Points of Failures within Availability Zones
  6. Employ solid backup and recovery strategies

They key to the points above is to not put all your eggs into one basket and then cry foul when that basket breaks…do not set an expectation whereby you become complacent in the fact that all Cloud Service Providers guarantee a certain level of system up time through SLA’s and then act surprised when an outage occurs. Most providers who are worth their salt do offer separate availability zones…but it’s very much up to the people designing and building services upon Services Provider Clouds to ensure that they are built to take advantage of this fact…you can’t come in stamping your feet and crying foul when the resources that are placed at your disposal to ensure application and service continuity are not taken advantage of.

Do not plan for 100% uptime…it does not exist! Anyone who tries to tell you otherwise is lying! You only have to search online to see that Outages are indeed like Assholes…everyone has them!


NSX Edge vs vShield Edge: Part 3 – IPsec and L2 VPN


NSX and vShield Edges support site to site IPSec VPN between Edge instances and remote sites. Behind each remote VPN router, you can configure multiple subnets to connect to the internal network behind an Edge through IPSec tunnels. These subnets and the internal network behind the Edges must have address ranges that do not overlap. You can have a maximum of 64 tunnels across a maximum of 10 sites.

NSX Edges are also capable of L2 VPNs where you can stretch both VXLAN and VLAN across geographical sites…This allows VMs to remain on the same subnet when they are moved between sites with the IP addresses not changing. L2 VPN allows seamless migration of workloads backed by VXLAN or VLAN between physically separated locations. Specifically for Service Providers L2 VPN provides a mechanism to on-board tenants without modifying IP addresses for VM workloads.

In this post I am only going to go through IPsec VPN configuration…feel there is a whole separate post required to do L2 VPN justice. The biggest difference between an NSX and vShield Edge when looking to configure VPNs is that when you are managing a vShield Edge you will not see the options to configure L2 VPN as shown in the configuration example below.

Configuring IPsec VPN From Web Client:

Configuration Items Required:

  • Local Endpoint
  • Local Subnets
  • Peer Endpoint
  • Peer Subnets
  • Encryption Algorithm and Authentication mechanism
  • Pre Shared Key
  • Diffie-Hellman Group

Double Click on the Edge under the NSX Edge Menu Option in Networking and Security, In the VPN Tab under Configuration click on Enable next to IPsec VPN Service Status and then hit Publish Changes

To create a new Tunnel, click on + and enter in the details collected as per the items listed above.

Click ok and then Publish the Changes…from there the Status should show a green tick. Once the other side has been configured check to see that the Tunnel(s) are up by clicking on Show IPsec Statistics.

If both sides are happy you should be able to talk between the configured subnets. Shown below you see an example of a Site to Site with One Tunnel configured up…and one down.

Configuring IPsec VPN From vCloud Director UI:

For vShield Edges managed via vCloud Director, head to the vCD UI and under Administration and the Edge Gateways. Right Click on the Edge and Configure Services. Under the VPN Tab you first want to Enable VPN and Configure the Public IPs.

Enter in the Public IP as shown above and click ok.

Click on Add and enter in the details collected. For Site to Site VPNs drop down the Establish VPN to: dropdown to a remote network and configure the rest of the settings.

Once done, you should see the Enabled and Status Column with green ticks.

A nice addition to the vCD UI (sometimes the UI team gets things right) is the Peer Settings Button which shows you the bits required to configure the other end of the connection.

Enabling/Disabling/Viewing IPsec With REST API:

Below are the key API commands to configure and manage IPsec VPN.