Category Archives: Service Providers

Released: vCloud Director 9.1 – New HTML5 Features, vCD-CLI and more!

Overnight VMware released vCloud Director 9.1 (build 7905680) which builds on the 9.0 release that came out last September. This continues to deliver on VMware’s promise to release major vCD updates every six months or so. This update, on the surface contains fewer big ticket items than the 9.0 release however the enhancements included are actually significant and continue to build on where 9.0 left off.

New Features and Enhancements:
  • Enhanced Tenant Portal
  • HTML Provider Portal
  • User Interface Extensibility
  • Service Integration
  • Standalone VMRC
  • Multi-Site Management View
  • SR-IOV
  • FIPS Mode
  • Python SDK
  • vCD-CLI
  • vRealize Orchestrator Integration
Enhanced Tenant Portal:

The new Tenant UI features include vApp and Catalog enhancements while delivering on probably the biggest pain point with the Flex UI tenant portal…that is OFV/OVA management. We now have native upload and download integration without the need for the client integration plugin.

You now also get an overview of resources consumed in your Virtual Datacenters and also get a view of the multiple organisation feature introduced into 9.0.

A new Provider Portal has been seeded in this release and at the moment can only be used for the new vRealise Orchestrator extensibility functionality. The administrator can import workflows from vRO through the import option. An administrator clicks the import workflow button, selects the vRO instance, and then chooses all the workflows they would like to import. On that note, there is an updated vRO Plug-In that allows both providers and tenants to automate tasks from the portal which is an excellent feature.

There is also a new workflow for the provision of standalone VMs and vApps.

Standalone VMRC:

If the management of OVAs/OVFs wasn’t the number one pain point with the FlexUI then the next one would have had to be the pain caused by the lack of functionality in the Console window. A HTML VM console is supported in version 9.0, but 9.1 now adds support for standalone VMware Remote Console. The VMRC provides more functions such for the tenant and significantly improves access to the VM consoles and gives greater flexibility accessing the VMs.


I’ve blogged about the old VCA-CLI on a number of occasions and it’s great to see the project officially brought back into the vCD world. Development on this stopped for a while with the demise of vCloud Air, however I’m glad to see it picked up on as it’s a great tool for managing vCloud Director tenant Organisations and objects from a command line without having to get stuck into the APIs directly. It’s also used for the new Container Services Extension that has also been released side by side with this release of vCD.

Compatibility with Veeam, vSphere 6.5 and NSX-v 6.4.x:

vCloud Director 9.1 is compatible with vSphere 6.5 Update 1 and NSX-v 6.4 and supports full interoperability with other versions as shown in the VMware Product Interoperability Matrix. With regards to Veeam support, I am sure that our QA department will be testing the 9.1 release against our integration pieces at the first opportunity they get, but as of now, there is no ETA on offical support.

A list of known issues can be found in the release notes.


Overall this is a very strong release with a lot of emphasis on extensibility behind the visual enhancements and functionality of the ever evolving HTML Tenant UI. As usual, I’ll look to write a few more blog posts on specific 9.1 features over the next couple of weeks.

There is a White Paper where you can find more details about what’s contained in the 9.1 release. Tom Fojta and Daniel Paluszek VMware have a what’s new blog posts as well.



VMware vCloud Director 9.1 is out!

NSX Bytes: Updated – NSX Edge Feature and Performance Matrix

For a few years now i’ve been compiling features and throughput numbers for NSX Edge Services Gateways. This started off comparing features and performance metrics between vShield Edges and NSX Edges. As the product evolves, so does it’s capabilities and given the last time I updated this was around the time of NSX-v 6.2 I thought it was time for an update.

A reminder that VMware announced the End of Availability (“EOA”) of the VMware vCloud Networking and Security 5.5.x that kicked in on the September of 19, 2016 and that from vCloud Director 8.10 and above vShield Edges are no longer supported…hence why I don’t have the VSE listed in the tables. For those still running VSEs for what ever reason, you can reference my original post here.

As a refresher…what is an Edge device?

The Edge Services Gateway (NSX-v) connects isolated, stub networks to shared (uplink) networks by providing common gateway services such as DHCP, VPN, NAT, dynamic routing, and Load Balancing. Common deployments of Edges include in the DMZ, VPN Extranets, and multi-tenant Cloud environments where the Edge creates virtual boundaries for each tenant.

The following relates to ESG maximums per NSX and ESXi maximums.

Item Maximums
ESGs per NSX Manager 2,000
ESGs per ESXi Host 250
ESG Interfaces 10 (Including Internal, Uplink and Trunk)
ESG Subinterfaces 200
The function of an ESG is as follows:

The ESG gives you access to all NSX Edge services such as firewall, NAT, DHCP, VPN, load balancing, and high availability. You can install multiple ESG virtual appliances in a datacenter. Each ESG virtual appliance can have a total of ten uplink and internal network interfaces. With a trunk, an ESG can have up to 200 subinterfaces. The internal interfaces connect to secured port groups and act as the gateway for all protected virtual machines in the port group. The subnet assigned to the internal interface can be a publicly routed IP space or a NATed/routed RFC 1918 private space. Firewall rules and other NSX Edge services are enforced on traffic between network interfaces.

Below is a list of services provided by the NSX Edge.

Service Description
Firewall Supported rules include IP 5-tuple configuration with IP and port ranges for stateful inspection for all protocols
NAT Separate controls for Source and Destination IP addresses, as well as port translation
DHCP Configuration of IP pools, gateways, DNS servers, and search domains
Site to Site VPN Uses standardized IPsec protocol settings to interoperate with all major VPN vendors
SSL VPN SSL VPN-Plus enables remote users to connect securely to private networks behind a NSX Edge gateway
Load Balancing Simple and dynamically configurable virtual IP addresses and server groups
High Availability High availability ensures an active NSX Edge on the network in case the primary NSX Edge virtual machine is unavailable
Syslog Syslog export for all services to remote servers
L2 VPN Provides the ability to stretch your L2 network.
Dynamic Routing Provides the necessary forwarding information between layer 2 broadcast domains, thereby allowing you to decrease layer 2 broadcast domains and improve network efficiency and scale. Provides North-South connectivity, thereby enabling tenants to access public networks.

Below is a table that shows the different sizes of each edge appliance and what (if any) impact that has to the performance of each service. As a disclaimer the below numbers have been cherry picked from different sources and are subject to change.

NSX Edge (Compact) NSX Edge (Large) NSX Edge (Quad-Large) NSX Edge (X-Large)
vCPU 1 2 4 6
Memory 512MB 1GB 1GB 8GB
Disk 512MB 512MB 512MB 4.5GB + 4GB
Interfaces 10 10 10 10
Sub Interfaces (Trunk) 200 200 200 200
NAT Rules 2,048 4,096 4,096 8,192
ARP Entries
Until Overwrite
1,024 2,048 2,048 2,048
FW Rules 2000 2000 2000 2000
FW Performance 3Gbps 9.7Gbps 9.7Gbps 9.7Gbps
DHCP Pools 20,000  20,000  20,000  20,000
ECMP Paths 8 8 8 8
Static Routes 2,048 2,048 2,048 2,048
LB Pools 64 64 64 1,024
LB Virtual Servers 64 64 64 1,024
LB Server / Pool 32 32 32 32
LB Health Checks 320 320 320 3,072
LB Application Rules 4,096 4,096 4,096 4,096
L2VPN Clients Hub to Spoke 5 5 5 5
L2VPN Networks per Client/Server 200 200 200 200
IPSec Tunnels 512 1,600 4,096 6,000
SSLVPN Tunnels 50 100 100 1,000
SSLVPN Private Networks 16 16 16 16
Concurrent Sessions 64,000 1,000,000 1,000,000 1,000,000
Sessions/Second 8,000 50,000 50,000 50,000
LB Throughput L7 Proxy) 2.2Gbps 2.2Gbps 3Gbps
LB Throughput L4 Mode) 6Gbps 6Gbps 6Gbps
LB Connections/s (L7 Proxy) 46,000 50,000 50,000
LB Concurrent Connections (L7 Proxy) 8,000 60,000 60,000
LB Connections/s (L4 Mode) 50,000 50,000 50,000
LB Concurrent Connections (L4 Mode) 600,000 1,000,000 1,000,000
BGP Routes 20,000 50,000 250,000 250,000
BGP Neighbors 10 20 100 100
BGP Routes Redistributed No Limit No Limit No Limit No Limit
OSPF Routes 20,000 50,000 100,000 100,000
OSPF LSA Entries Max 750 Type-1 20,000 50,000 100,000 100,000
OSPF Adjacencies 10 20 40 40
OSPF Routes Redistributed 2000 5000 20,000 20,000
Total Routes 20,000 50,000 250,000 250,000

Of interest from the above table it doesn’t list any Load Balancing performance number for the NSX Compact Edge…take that to mean that if you want to do any sort of load balancing you will need NSX Large and above. To finish up, below is a table describing each NSX Edge size use case.

Use Case
NSX Edge (Compact) Small Deployment, POCs and single service use
NSX Edge (Large) Small/Medium DC or mult-tenant
NSX Edge (Quad-Large) High Throughput ECMP or High Performance Firewall
NSX Edge (X-Large) L7 Load Balancing, Dedicated Core

The Quad Large model is suitable for high performance firewall abilities and the X-Large is suitable for both high performance load balancing and routing. You can convert between NSX Edge service gateway sizes upon demand using a non-disruptive upgrade process, so the recommendation is to begin with the Large model and scale up if necessary. A Large NSX Edge service gateway is suitable for medium firewall performance but as detailed later, the NSX Edge service gateway does not perform the majority of firewall functions.


Quick Look: Installing Veeam Powered Network Direct from a Linux Repo

Last week, Veeam Powered Network (Veeam PN) was released to GA. As a quick reminder Veeam PN allows administrators to create, configure and connect site-to-site or point-to-site VPN tunnels easily through an intuitive and simple UI all within a couple of clicks. Previously during the RC period there where two options for deployment…The appliance was available through the Azure Marketplace or downloadable from the website and deployable on-premises from an OVA.

With the release of the GA a third option is available which is installation direct from the Veeam Linux Repositories. This gives users the option to deploy their own Ubuntu Linux server and install the packages required through the Advanced Package Tool (APT). This is also the mechanism that works in the background to update Veeam PN through the UI via the Check for Updates button under Settings.

The requirements for installation are as follows:

  • Ubuntu 16.04 and above
  • 1 vCPU (Minimum)
  • 1 GB vRAM (Minimum)
  • 16 GB of Hard Drive space
  • External Network Connectivity

The Azure Marketplace Image and the OVA Appliance have been updated to GA build

Installation Steps:

To install Veeam PN and it’s supporting modules you need to first add the Veeam Linux Repository to you system and configure APT to be on the lookout for the Veeam PN packages. To do this you need to download and add the Veeam Software Repository Key, add Veeam PN to the list of sources in APT and run an APT update.

Once done you need to install two packages via the apt-get install command. As shown below there is the Server and UI component installed. This will pick up a significant list of dependancies that need to be installed as well.

There is a lot that is deployed and configured as it goes through the package installs and you may be prompted along the way to ask to overwrite the existing iptables rules if any existing on the system prior to install. Once completed you should be able to go to the Veeam PN web portal and perform the initial configuration.

The username to use at login will be the root user of your system.

So that’s it…an extremely easy and quick way to deploy Veeam Power Network without having to download the OVA or deploy through the Azure Marketplace.

As a reminder, i’ve blogged about the three different use cases for Veeam PN:

Clink on the links to visit the blog posts that go through each scenario and download or deploy the GA from the website or Azure Marketplace and now directly from the Veeam Linux Repos and give it a try. Again, it’s free, simple, powerful and a great way to connect or extend networks securely with minimal fuss.

AWS re:Invent – Expectations from a VM Hugger…

Today is the first day offical day of AWS re:Invent 2017 and things are kicking off with the global partner summit. Today also is my first day of AWS re:Invent and I am looking forward to experiencing a different type of big IT conference with all previous experiences being at VMworld or the old Microsoft Tech Eds. Just buy looking at the agenda, schedule and content catalog I can already tell re:Invent is a very very different type of IT conference.

As you may or may not know I started this blog as Hosting is Life! and the first half of my career was spent around hosting applications and web services…in that I gravitated towards looking at AWS solutions to help compliment the hosting platforms I looked after and I was actively using a few AWS services in 2011 and 2012 and attended a couple of AWS courses. After joining Zettagrid my use of AWS decreased and it wasn’t until Veeam announced supportability for AWS storage as part of our v10 announcements that I decided to get back into the swing of things.

Subsequently we announced Veeam Availability for AWS which leverages EBS snapshots to perform agentless backups of AWS instances and more recently we where announced as a launch partner for VMware Cloud on AWS data availability solutions. For me, the fact that VMware have jumped into bed with AWS has obviously raised AWS’s profile in the VMware community and it’s certainly being seen as the cool thing to know (or claim to know) within the ecosystem.

Veeam isn’t the only backup vendor looking to leverage what AWS has to offer by way of extending availability into the hyper-scale cloud and every leading vendor is rushing to claim features that offload backups to AWS cloud storage as well as offering services to protect native AWS workloads…as with IT Pros this is also the in thing!

Apart from backup and availability, my sessions are focused on storage, compute, scalability and scale as well as some sessions on home automation with Alexa and alike. This years re:Invent is 100% a learning experience and I am looking forward to attending a lot of sessions and taking a lot of notes. I might even come out taking the whole serverless thing a little more seriously!

Moving away from the tech the AWS world is one that I am currently removed from…unlike the VMware ecosystem and VMworld I wouldn’t know 95% of the people delivering sessions and I certainly don’t know much about the AWS community. While I can’t fix that by just being here this week, I can certainly use this week as a launching pad to get myself more entrenched with the technology, the ecosystem and the community.

Looking forward to the week and please reach out if you are around.

Released: vCloud Director 9.0 – The Most Significant Update To Date!

Today is a good day! VMware have released to GA vCloud Director 9.0 (build 6681978) and with it come the most significant feature and enhancements of any previous vCD release. This is the 9th major release of vCloud Director, now spanning nearly six and half years since v1.0 was released in Feburary of 2011 and as mentioned from my point of view it’s the most significant update of vCloud Director to date.

Having been part of the BETA program I’ve been able to test some of the new features and enhancements over the past couple of months and even though from a Service Provider perspective there is a heap to like about what is functionally under the covers, but the biggest new feature is without doubt the HTML5 Tenant Portal however as you can see below there is a decent list of top enhancements.

Top Enhancements:


  • Multi-Site vCD – Single Access point URL for all vCD instances within same SP federated via SSO
  • On-premises to Cloud Migration – Plugin that enables L2 connectivity, warm and cold migration
  • Expanded NSX Integration – Security Groups, Logical Routing for east-west traffic and audit logging
  • HTML5 Tenant UI – Streamlined workflows for VM deployment, UI Extensibility for 3rd party services/functionality
  • HTML5 Metrics UI – Basic Metrics for VMs shown through tenant portal
  • Extensible Service Framework – Service enablement, SSO Ready
  • Application Extensibility – Plugin Framework
  • PostGres 9.5 Support – In addition to MSSQL and Oracle, Postgres is now supported.
  • …and more under the hood bits

I’m sure there will be a number of other blog posts focusing on the list above, and i’ll look to go through a few myself over the next few weeks but for this GA post I wanted to touch on the new HTML5 Tenant UI.

There is a What’s New in vCloud Director 9.0 PDF here.

New HTML5 Tenant UI:

The vCD team laid the foundation for this new Tenant UI in the last release of vCD in bringing the NSX Advanced HTML5 UI to version 8.20. While most things have been ported across there may still be a case for tenants to go back to the old Flex UI to do some tasks, however from what I have seen there is close to 100% full functionality.

To get to the new HTML5 Tenant UI you go to: https://<vcd>/tenant/orgname

Once logged in you are greeted with a now familiar looking VMware portal based on the Clarity UI. It’s pretty, it’s functional and it doesn’t need Flash…so haters of the existing flex based vCD portal will have to bite their tongues now 🙂

The Networking menu is inbuilt into this same Tenant portal and you you can access it directly from the new UI, or in the same way as was the case with vCD 8.20 from the flex UI. Below is a YouTube video posted by the vCD team that walks through the new UI.

There is also VM Metrics in the UI now, where previously they where only accessible after configuring the vCD Cells to route metric data to a Cassandra database. The metrics where only accessible via the API and some providers managed to tap into that and bring vCD Metrics into their own portals. With the 9.0 release this is now part of the new HTML5 Tenant UI and can be seen in the video below.

As per previous releases this only shows up to two weeks worth of basic metrics but it’s still a step in the right direction and gives vCD tenant’s enough info to do basic monitoring before hitting up a service desk for VM related help.


vCloud Director 9.0 has delivered on the what most members of the VMware Cloud Provider Program had wanted for some time…that is, a continuation of the commitment to the the HTML5 UI as well as continuing to add features that help service providers extend their reach across multiple zones and over to hybrid cloud setups . As mentioned over the next few weeks, I am going to expand on the key new features and walk through how to configure elements through the UI and API.

Compatibility with Veeam, vSphere 6.5 and NSX-v 6.3.x:

vCloud Director 9.0 is compatible with vSphere 6.5 Update 1 and NSX 6.3.3 and supports full interoperability with other versions as shown in the VMware Product Interoperability Matrix. With regards to Veeam support, I am sure that our QA department will be testing the 9.0 release against our integration pieces at the first opportunity they get, but as of now, there is no ETA on offical support.

A list of known issues can be found in the release notes.



VMware Announces New vCloud Director 9.0

VMware Cloud on AWS: Thoughts One Year On

Last week at VMworld 2017 in the US, VMware announced the initial availability of VMware Cloud on AWS. It was the focal point for VMware at the event and probably the most important strategic play that VMware has undertaken in it’s history. This partnership was officially announced at last year’s VMworld and at the time I wrote a couple of blog posts commenting on the potential impact to the then, vCloud Air Network (now VCPP) and what needed to be done to empower the network.

As you can imagine at the time, I was a little skeptical about the announcement, but since that time we have seen the fall of vCloud Air to OVH and a doubling down of the efforts around enhancing vCloud Director and general support for the VMware Cloud Provider Program. Put this together with me stepping out of my role within the VCPP to one that is on the outside supporting it I feel that VMware Cloud on AWS is good for VMware and also good for service providers.

What It Looks Like:

This time last year we didn’t know exactly what VMC would look like apart from using vSphere, NSX and vSAN as it’s compute, networking and storage platforms or how exactly it would work on top of AWS’s infrastructure. For a detailed look under the hood, Frank Denneman has published a Technical Overview which is worth a read. A lot of credit needs to go to the engineering teams at both ends for achieving what they have achieved within a relatively small period of time.

The key thing to point out is the default compute and storage that’s included as part of the service. Four ESXi hosts will have dual E5-2686 v4 CPUs @2.3GHz with 18 Cores and 512GB of RAM. Storage wise there will be 10TB raw of All Flash vSAN per host, meaning depending on the FTT of vSAN a usable minimum of 20TB. The scale-out model enables expansion to up to 16 hosts, resulting in 576 CPU cores and 8TB of memory which is insane!

What does is Cost:

Here is where is starts to get interesting for me. Pricing wasn’t discussed during the Keynotes or in the announcements but looking at the pricing page here you can see what this base cluster will cost you. It’s going to cost $8.37 USD per host per hour for the on-demand option, which is the only option until VMware launches one year and three year reserved instances in the future where there looks to be a thirty and fifty percent saving respectively.

Upon first glance this seems expensive…however it’s only expensive in relative terms because there is the default resources that come the service. You can’t get anything less than the four hosts with all the trimmings at the moment which, when taken into consideration might lock out non enterprise companies from taking the service up.

Unless pricing changes by way of offering a smaller resource footprint I can see this not being attractive in other regions like ANZ or EMEA where small to medium size enterprises are more common. This is where VCPP service providers can still remain competitive and continue to offer services around the same building blocks as VMC on their own platforms.

CloudPhysics have an interesting blog post here, on some cost analytics that they ran.

How Can it be Leveraged:

With Veeam being a launch partner with VMware Cloud on AWS offering availability services it got me thinking as to how the service could be leveraged by service providers. A few things need to fall into place from a technology point of view but I believe that one of the best potential use cases for VMC is for service providers to leverage it for failover, replication and disaster recovery scenarios.

The fact that there this service posses auto-scaling of hosts means that it has the potential to be used as a resource cluster for disaster recovery services. If I think about Cloud Connect Replication, one of the hardest things to get right as a provider is sizing the failover resources and the procurement of the compute and storage to deal with customer requirements. As long as the base resources are covered the auto scaling capabilities mean that service providers only need to cover the base resources and pay any additional costs if a failover event happens and exceed the default cluster resources.

It must be pointed out that Cloud Connect can’t use a VMC cluster as a target at the moment due to the networking used…that is VXLAN on top of AWS VPN networking.

As I wrote last year, I feel like there is a great opportunity for service providers to leverage VMC as vCloud Director provider clusters however I know that this currently isn’t being supported by VMware. I honestly feel that service providers would love the ability to have cloud based Provider vDCs available across the world and I’m hoping that VMware realise the potential and allow vCloud Director to connect and consume VMC.

VMworld End of Show Report on VMware Cloud on AWS:


Cloud to Cloud to Cloud Networking with Veeam Powered Network

I’ve written a couple of posts on how Veeam Powered Network can make accessing your homelab easy with it’s straight forward approach to creating and connection site-to-site and point-to-site VPN connections. For a refresh on the use cases that I’ve gone through, I had a requirement where I needed access to my homelab/office machines while on the road and to to achieve this I went through two scenarios on how you can deploy and configure Veeam PN.

In this blog post I’m going to run through a very real world solution with Veeam PN where it will be used to easily connect geographically disparate cloud hosting zones. One of the most common questions I used to receive from sales and customers in my previous roles with service providers is how do we easily connect up two sites so that some form of application high availability could be achieved or even just allowing access to applications or services cross site.

Taking that further…how is this achieved in the most cost effective and operationally efficient way? There are obviously solutions available today that achieve connectivity between multiple sites, weather that be via some sort of MPLS, IPSec, L2VPN or stretched network solution. What Veeam PN achieves is a simple to configure, cost effective (remember it’s free) way to connect up one to one or one to many cloud zones with little to no overheads.

Cloud to Cloud to Cloud Veeam PN Appliance Deployment Model

In this scenario I want each vCloud Director zone to have access to the other zones and be always connected. I also want to be able to connect in via the OpenVPN endpoint client and have access to all zones remotely. All zones will be routed through the Veeam PN Hub Server deployed into Azure via the Azure Marketplace. To go over the Veeam PN deployment process read my first post and also visit this VeeamKB that describes where to get the OVA and how to deploy and configure the appliance for first use.


  • Veeam PN Hub Appliance x 1 (Azure)
  • Veeam PN Site Gateway x 3 (One Per Zettagrid vCD Zone)
  • OpenVPN Client (For remote connectivity)

Networking Overview and Requirements

  • Veeam PN Hub Appliance – Incoming Ports TCP/UDP 1194, 6179 and TCP 443
    • Azure VNET
    • Azure Veeam PN Endpoint IP and DNS Record
  • Veeam PN Site Gateways – Outgoing access to at least TCP/UDP 1194
    • Perth vCD Zone
    • Sydney vCD Zone
    • Melbourne vCD Zone
  • OpenVPN Client – Outgoing access to at least TCP/UDP 6179

In my setup the Veeam PN Hub Appliance has been deployed into Azure mainly because that’s where I was able to test out the product initially, but also because in theory it provides a centralised, highly available location for all the site-to-site connections to terminate into. This central Hub can be deployed anywhere and as long as it’s got HTTPS connectivity configured correctly to access the web interface and start to configure your site and standalone clients.

Configuring Site Clients for Cloud Zones (site-to-site):

To configuration the Veeam PN Site Gateway you need to register the sites from the Veeam PN Hub Appliance. When you register a client, Veeam PN generates a configuration file that contains VPN connection settings for the client. You must use the configuration file (downloadable as an XML) to set up the Site Gateway’s. Referencing the digram at the beginning of the post I needed to register three seperate client configurations as shown below.

Once this has been completed you need deploy a Veeam PN Site Gateway in each vCloud Hosting Zone…because we are dealing with an OVA the OVFTool will need to be used to upload the Veeam PN Site Gateway appliances. I’ve previously created and blogged about an OVFTool upload script using Powershell which can be viewed here. Each Site Gateway needs to be deployed and attached to the vCloud vORG Network that you want to extend…in my case it’s the, and vORG Networks.

Once each vCloud zone has has the Site Gateway deployed and the corresponding XML configuration file added you should see all sites connected in the Veeam PN Dashboard.

At this stage we have connected each vCloud Zone to the central Hub Appliance which is configured now to route to each subnet. If I was to connect up an OpenVPN Client to the HUB Appliance I could access all subnets and be able to connect to systems or services in each location. Shown below is the Tunnelblick OpenVPN Client connected to the HUB Appliance showing the injected routes into the network settings.

You can see above that the, and static routes have been added and set to use the tunnel interfaces default gateway which is on the central Hub Appliance.

Adding Static Routes to Cloud Zones (Cloud to Cloud to Cloud):

To complete the setup and have each vCloud zone talking to each other we need to configure static routes on each zone network gateway/router so that traffic destined for the other subnets knows to be routed through to the Site Gateway IP, through to the central Hub Appliance onto the destination and then back. To achieve this you just need to add static routes to the router. In my example I have added the static route to the vCloud Edge Gateway through the vCD Portal as shown below in the Melbourne Zone.


Summerizing the steps that where taken in order to setup and configure the configuration of a cloud to cloud to cloud network using Veeam PN through its site-to-site connectivity feature to allow cross site connectivity while allowing access to systems and services via the point-to-site VPN:

  • Deploy and configure Veeam PN Hub Appliance
  • Register Cloud Sites
  • Register Endpoints
  • Deploy and configure Veeam PN Site Gateway in each vCloud Zone
  • Configure static routes in each vCloud Zone

Those five steps took me less than 30 minutes which also took into consideration the OVA deployments as well. At the end of the day I’ve connected three disparate cloud zones at Zettagrid which all access each other through a Veeam PN Hub Appliance deployed in Azure. From here there is nothing stopping me from adding more cloud zones that could be situated in AWS, IBM, Google or any other public cloud. I could even connect up my home office or a remote site to the central Hub to give full coverage.

The key here is that Veeam Power Network offers a simple solution to what is traditionally a complex and costly one. Again, this will not suit all use cases but at it’s most basic functional level, it would have been the answer to the cross cloud connectivity questions I used to get that I mentioned at the start of the article.

Go give it a try!

Attack from the Inside – Protecting Against Rogue Admins

In July of 2011, Distribute.IT, a domain registration and web hosting services provider in Australia was was hit with a targeted, malicious attack that resulted in the company going under and their customers left without their hosting or VPS data. The attack was calculated, targeted and vicious in it’s execution… I remember the incident well as I was working for Anittel at the time and we where offering similar services…everyone in the hosting organization was concerned when starting to think about the impact a similar attack would have within our systems.

“Hackers got into our network and were able to destroy a lot of data. It was all done in a logical order – knowing exactly where the critical stuff was and deleting that first,”

While it was reported at the time that a hacker got into the network, the way in which the attack was executed pointed to an inside job and all though it was never proved to be so it almost 100% certain that the attacker was a disgruntled ex-employee. The very real issue of an inside attack has popped up again…this time Verelox, a hosting company out of the Netherlands has effectively been taken out of business with a confirmed attack from within by an ex-employee.

My heart sinks when I read of situations like this and for me, it was the only thing that truely kept me up at night as someone who was ultimately responsible for similar hosting platforms. I could deal and probably reconcile with myself if I found myself in a situation where a piece of hardware failed causing data loss…but if an attacker had caused the data loss then all bets would have been off and I might have found myself scrambling to save face and along with others in the organization, may well have been searching for a new company…or worse a new career!

What Can Be Done at an Technical Level?

Knowing a lot about how hosting and cloud service providers operate my feeling is that 90% of organizations out there are not prepared for such attacks and are at the mercy of an attack from the inside…either by a current or ex-employee. Taking that a step further there are plenty that are at risk of an attack from the inside perpetrated by external malicious individuals. This is where the principal of least privileged access needs to be taken to the nth degree. Clear separation of operational and physical layers needs to be considered as well to ensure that if systems are attacked, not everything can be taken down at once.

Implementing some form of certification or compliancy such as ISO 27001, SOC and iRAP will force companies to become more vigilant through the stringent processes and controls that are forced upon companies once they meet compliancy. This in turn naturally leads to better and more complete disaster and business continuity scenarios that are written down and require testing and validation in order to pass certification.

From a backup point of view, these days with most systems being virtual it’s important to consider a backup strategy that not only looks to make use of the 3-2-1 rule of backups, but also look to implement some form of air-gapped backups that in theory are completely seperate and unaccessible from production networks, meaning that only a few very trusted employees have access to the backup and restore media. In practice implementing a complete air-gapped solution is complex and potentially costly and this is where service providers are chancing their futures on scenarios that have a small percentage chance of happening however the likelihood of that scenario playing out is greater than it’s ever been.

In a situation like Verelox, I wonder if, like most IaaS providers they didn’t backup all client workloads by default, meaning that backup services was an additional service charge that some customers didn’t know about…that said, if backup systems are wiped clean is there any use of having those services anyway? That is to say…is there a backup of the backup? This being the case I also believe that businesses need to start looking at cross cloud backups and not rely solely on their providers backup systems. Something like the Veeam Agent’s or Cloud Connect can help here.

So What Can Be Done at an Employee Level?

The more I think about the possible answer to this question, the more I believe that service providers can’t fully protect themselves from such internal attacks. At some point trust supersedes all else and no amount of vetting or process can stop someone with the right sort of access doing damage. To that end making sure that you are looking after your employee’s is probably the best defence against someone feeling aggrieved enough to carry out an malicious attack such as the one Verelox has just gone through. In addition to looking after employee’s well being it’s also a good idea to…within reason, keep tabs on an employee’s state in life in general. Are they going through any personal issues that might make them unstable, or have they been done wrong by someone else within the company? Generally social issues should be picked up during the hiring process, but complete vetting of employee stability is always going to be a lottery.


As mentioned above, this type of attack is a worst case scenario for every service provider that operates today…there are steps that can be taken to minimize the impact and protect against an employee getting to the point where they choose to do damage but my feeling is we haven’t seen the last of these attacks and unfortunately more will suffer…so where you can, try to implement policy and procedure to protect and then recover when or if they do happen.

Vote for your favorite blogs at vSphere-land!

Top vBlog Voting 2017


Verelox (Netherlands hosting company) servers wiped by ex-admin from sysadmin