Monthly Archives: March 2017

World Backup Day …We should be thinking Availability #WAD

Yesterday (30th of March) was Veeam’s World Availability Day. This is a day that Veeam has declared to make people aware about how availability plays a part in all organizations as an extension of Backup and Replication. In it’s self, WAD is a marketing initiative from us here at Veeam that backs onto World Backup Day…which is happening today (31st of March).

Veeam celebrates World Availability Day in recognition of the fact that for modern businesses and service providers, it’s not only about having backups of data anymore, it’s about being Available. Veeam helps organizations of any size and shape to achieve Availability for both their virtual and physical infrastructures, as well as provide data protection solutions in the cloud, whether private, public or hybrid.

World Backup Day focuses around Backup but you can’t forget that replication plays a critical roll in organizations ensuring they are covered for disaster with low recovery times via RaaS and DraaS service offerings. Cloud Connect Replication offered by our Veeam Cloud and Service Providers offers industry leading replication platform and all Veeam customers can take advantage of Cloud Connect as it’s baked right into the core Backup & Replication product.

For those not aware, Veeam has a Find a Veeam Cloud Provider Directory that lets you search for any of our listed VCSPs based on criteria that is relevant to your backup or replication needs.

https://www.veeam.com/find-a-veeam-cloud-provider.html

More on Veeam Cloud Connect Replication:

Advanced image-based VM replication through Veeam Cloud Connect is simple to set up and easy to use through a VCSP and is as easy to setup as going to the Service Providers menu in the Veeam Backup & Replication console and choosing your service provider of choice. Once set up you have access to hardware plans that provide compute, storage and networking resources at the service provider end to which you can configure cloud replication jobs and manage failover scenarios by way of failover plans.

When looking at disaster recovery testing and failover, one of the biggest challenges is in the networking. Generally speaking, there is complexity that surrounds ensuring VM replicas that are brought up at a disaster recovery site have the right networking in order for their applications and services to work. With the Network Extension Appliance, the tenant has the ability to map the internal VM networks as well as configure and publish external services to ensures seamless transition to the VM replicas during a failover.

Data protection and disaster recovery tasks targeted at the cloud host are performed by tenants. Tenants can set up necessary replication jobs and perform failover operations on Veeam backup servers deployed on their side. Tenants can perform the following operations:

  • Replicate VMs to the cloud host
  • Perform failover tasks with VM replicas on the cloud host
  • Full site failover, when all critical production VMs fail over to their replicas on the cloud host in case the whole production site becomes unavailable
  • Partial site failover, when one or several VMs become corrupted and fail over to their replicas on the cloud host
  • Perform failback tasks with VM replicas on the cloud host.

Tasks associated with full site failover can be performed either by the tenant or by the service provider or through a Cloud Connect Portal hosted by the service provider that offers failover/failback operations. This flexibility allows the service provider to test the full site failover process and switch tenant’s production site to the cloud hosts upon request in the case that the tenant has no access to the backup infrastructure after a disaster. Ensuring that failover from the replica VM is successful is critically important, but there should also be a way back to access the on-premises infrastructure. Cloud Connect Replication features an option to easily failback to the primary infrastructure, by copying only data blocks changed while replica VMs were running on the service provider infrastructure.

Reminder: Veeam and its partners are giving away $200 MILLION in Cloud services ($1,000 USD per customer)!

Veeam and its VCSP partners are giving away $1,000 in FREE Cloud Services to each and every one of our +230K customers using Veeam Availability Suite, Veeam Backup & Replication and Veeam Backup Essentials.

Because of this unprecedented offer, right now is the BEST TIME to try backup and DRaaS in the cloud. You can avoid the cost and complexity of building and maintaining your off-site infrastructure while meeting business and regulatory requirements for off-site backup and DR.

Visit the FREE Cloud services promo page to learn more about this offer.

VMworld 2017 : Session Analysis and Voting Open…Already!

Well this has crept up on us quickly this year! It’s time to vote for the VMworld Sessions that will be part of the US and Europe VMworld’s held later in the year. The Session Voting is more Session liking as you have the ability to mark multiple sessions as ones that you would like to see. Apparently there are 1900 odd sessions, however the voting site lists 1500 that where submitted and are listed in the Online Catalog.

[EDIT]: I totally forgot that David Hill submitted a joint session

David Hill, VMware
Anthony Spiteri, Veeam
Service Providers globally are enthusiastically embracing hybrid cloud as both a way of reducing costs and improving the quality of service they provide to end customers. To achieve this, Service Providers are looking to VMware vCloud Air Network and Veeam to help them build a scalable cost effect cloud solution. In this session we will get into the details of the technology. We’ll focus on how these solutions are architected and what that implies in real-life implementations. A participant in this session will leave with a technical understanding of how to leverage technology to provide a successful cloud based storage service.
Session Type:  Breakout Session
Track :  Integrate Public Clouds
Integrate Public Clouds Subtrack:  Leverage Hybrid Clouds
Market Segment:  No Specific Segment
Session Audience:  IT – All, IT – Operations, Technical Support
Product and Topics:  vCloud
Technical Level:  Technical – Advanced
Event Submitted For:  Both

Having gone through the session catalog it’s pleasing to see the number of cloud and service provider content feature prominently again. NSX, vSAN and vCloud Director all have a number of decent sessions. Click below on the links to view the related sessions.

I was surprised with the number of sessions submitted for NSX-T, but there is obviously a lot of interest in the hypervisor agnostic NSX…along side the huge number of NSX-v sessions listed. AWS is mentioned in 51 sessions showing you another change of direction at VMworld and there is a decent amount of automation and container based sessions. It will be interesting to see what the final cut is and how those numbers look when the offical catalog becomes available.

As per the VMworld Session Voting FAQ Session Voting is open March 28 to April 16…which is a lot earlier than last years May 12 to May 26 voting period. So it’s not just me that thinks this has come around sooner than previous years.

VMworld 2017 registration goes live on April 4!

It’s ok to steal… VMUG UserCon Key Take Aways

Last week I attended the Sydney and Melbourne VMUG UserCons and apart from sitting in on some great sessions I came away from both events with a renewed sense of community spirit and enjoyed catching up with industry peers and good friends that I don’t see often enough. While the VMUG is generally struggling a little around the world at this point in time, kudos goes to both Sydney and Melbourne chapter leaders and steering committee in being able to bring out a superstar bunch of presenters (see panel below)…there might not be a better VMUG lineup anywhere in the world this year!

There was a heavy automation focus this year…which in truth was the same as last years events however last years messaging was more around the theory of _change or die_ this year there was more around the practical. This was a welcome change because, while it’s all well and good to beat the change messaging into people…actually taking them through real world examples and demo’s tends to get people more excited and keen to dive into automation as they get a sense of how to apply it to their every day jobs.

In the VMware community, there are not better examples of automation excellence than Alan Renouf and William Lam and their closing keynote sessions where they went through and deployed a fully functional SDDC vSphere environment on a single ESXi host from a USB Key was brilliant and hopefully will be repeated at other VMUGs and VMworld. This project was born out of last years VMworld Hackerthon’s and ended up being a really fun and informative presentation that showed off the power of automation along with the benefits of what undertaking an automation project can deliver.

“Its not stealing, its sharing” 

During the presentation Alan Renouf shared this slide which got many laughs and resonated well with myself in that apart from my very early failed uni days, I don’t think I have ever created a bit of code or written a script from scratch. There is somewhat of a stigma attached with “borrowing” or “stealing” code used to modify or create scripts within the IT community. There might also be some shame associated in admitting that a bit of code wasn’t 100% created by someone from scratch…I’ve seen this before and I’ve personally been taken to task when presenting some of the scripts that I’ve modified for purpose during my last few roles.

What Alan is pointing out there is that it’s totally ok to stand on the shoulders of giants and borrow from what’s out there in the public domain…if code is published online via someones personal blog or put up on GitHub then it’s fair game. There is no shame in being efficient…no shame in not having to start from scratch and certainly no shame in claiming success after any mods have been done… Own it!

Conclusion and Event Wrap Up:

Overall the 2017 Sydney and Melbourne UserCons where an excellent event and on a personal note I enjoyed being able to attend with Veeam as the Platinum Sponsor and present session on our vSAN/VVOL/SPBM support and introduce our Windows and Linux Agents to the crowd. The Melbourne crowd was especially engaged and asked lots of great questions around our agent story and where looking forward to the release of Veeam Agent for Windows.

Again the networking with industry peers and customers is invaluable and there was a great sense of community once again. The UserCon events are of a high quality and my thanks goes out to the leaders of both Sydney and Melbourne for working hard to organise these events. And which one was better? …I won’t go there but those that listened to my comment during our Sponsor giveaways at the end of the event knows how I really feel.

Until next year UserCon!

Cloud Connect Replication Partial Failover – Example

Veeam Cloud Connect Replication has been part of Veeam’s Backup & Replication product since version 9 was released in early 2016 and like Cloud Connect Backup before it, Replication is starting to get traction in the market due to it’s ease of use, intuitive interface and best of breed disaster recovery networking technologies that are all baked into the core Backup & Replication product.

Without doubt one of the best/little known features of VCCR is the partial failover. Most disaster recovery scenarios focus on the total failure of all systems due to natural events or localised datacenter issues however the most common outage occurs at an virtual machine or instance level…this is generally an issue with the application or the operating system. With that, the ability to failover a single VM at a time is an often overlooked feature when looking into replication and disaster recovery platforms.

Veeam provides partial failovers within Cloud Connect Replication and once a VM has a replica copied up to the cloud provider you have the ability to perform a partial failover. I’ve created a video showing the process involved to initiate a partial failover which starts the VM replica up in the cloud providers hosting platform and then creates a L2 Tunnel via Network Extension Appliances that are deployed at the production and cloud ends. For an explanation of the Network Extension Appliance click here. Without diving into the specifics of what’s happening underneath the surface the NEAs talk to each other via the Cloud Connect Gateway and bridge the layer 2 network providing layer 3 access so that the replica VM that’s been partially failed over can communicate with VMs on the production network and vice versa.

This effectively means services and applications will be available over the internet without the need to employ expensive WAN connectivity mechanisms…the NEAs do all the work for you. In the demo video I am simulating the failure of a VM that hosts a WordPress site. That VM is brought up at the cloud providers end and, as can be seen in the video is shown to be running in a failed over state. From there I trigger a failback which replicates any changes made at the cloud end during the failover state back to the production site on premises. Once I am happy I commit the failback and the VM resumes normal operations on-premises.

The uptake of Cloud Connect Replication through Veeam’s VCSPs has been steady and we are seeing the number of replication VM licenses reported gather pace and grow month on month. As the Backup and Disaster Recovery markets mature I fully expect Cloud Connect to be a central part of our customers 3-2-1 rule of backup and availability with Cloud Connect Replication becoming the best of breed Replication/Disaster Recovery as a Service offering in the market.

References:

https://helpcenter.veeam.com/docs/backup/cloud/cloud_connect_partial_site_failover.html?ver=95

ESXi 6.5 Storage Performance Issues and Fix

[NOTE] : I decided to republish this post with a new heading and skip right to the meat of the issue as I’ve had a lot of people reach out saying that the post helped them with their performance issues on ESXi 6.5. Hopefully people can find the content easier and have a fix in place sooner.

The issue that I came across was to do with storage performance and the native driver that comes bundled with ESXi 6.5. With the release of vSphere 6.5 yesterday, the timing was perfect to install ESXI 6.5 and start to build my management VMs. I first noticed some issues when uploading the Windows 2016 ISO to the datastore with the ISO taking about 30 minutes to upload. From there I created a new VM and installed Windows…this took about two hours to complete which I knew was not as I had expected…especially with the datastore being a decent class SSD.

I created a new VM and kicked off a new install, but this time I opened ESXTOP to see what was going on, and as you can see from the screen shots below, the Kernel and disk write latencies where off the charts topping 2000ms and 700-1000ms respectively…In throuput terms I was getting about 10-20MB/s when I should have been getting 400-500MB/s. 

ESXTOP was showing the VM with even worse write latency.

I thought to myself if I had bought a lemon of a storage controller and checked the Queue Depth of the card. It’s listed with a QD of 31 which isn’t horrible for a homelab so my attention turned to the driver. Again referencing the VMware Compatibility Guide the listed driver for the controller the device driver is listed as ahci version 3.0.22vmw.

I searched for the installed device driver modules and found that the one listed above was present, however there was also a native VMware device drive as well.

I confirmed that the storage controller was using the native VMware driver and went about disabling it as per this VMwareKB (thanks to @fbuechsel who pointed me in the right direction in the vExpert Slack Homelab Channel) as shown below.

After the host rebooted I checked to see if the storage controller was using the device driver listed in the compatibility guide. As you can see below not only was it using that driver, but it was now showing the six HBA ports as opposed to just the one seen in the first snippet above.

I once again created a new VM and installed Windows and this time the install completed in a little under five minutes! Quiet a difference! Upon running a crystal disk mark I was now getting the expected speeds from the SSDs and things are moving along quiet nicely.

Hopefully this post saves anyone else who might by this, or other SuperMicro SuperServers some time and not get caught out by poor storage performance caused by the native VMware driver packaged with ESXi 6.5.


References
:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2044993

VMUG UserCon – Sydney and Melbourne Events are Huge This Year!

Last year I claimed that the Melbourne VMUG Usercon was the “Best Virtualisation Event Outside of VMworld!” …that was a big statement if ever there was one however, fast forward a year and credit goes to the VMUG steering committee’s of both Sydney and Melbourne as it seems they have bettered themselves this time around for the 2017 editions. Both events happen a couple of days apart from each other on the 21st and 23rd of March and both are filled with quality content, quality presenters and a great community feel.

This will be my fourth Melbourne UserCon and my second Sydney UserCon…The last couple of years I have attended the event in Melbourne I have taken away a lot of great technical and non-technical knowledge back home with me and with keynote speakers the likes of no less than Duncan Epping and Amy Lewis together with other industry superstars William LamAlan Renouf, Emad Younis, Josh Atwell and other great local presenters I expect the same for the 2017 editions.

With what is the strongest lineup in the Usercon’s history, it promises to be a worthwhile event to attend…if you haven’t already registered head to the registration pages below and sign up.

Both the Sydney and Melbourne Agenda’s are jam packed with virtualisation goodness and it’s actually hard to attend everything of interest with schedule conflicts happening throughout the day…the agenda’s are not yet 100% completed on the sites but make you check back later in the week to get details on who is presenting what and when.

Veeam is sponsoring both events as Platinum level sponsors and I’ll be presenting a session on Availability made easy for your vSphere infrastructure where I’ll go through some tips and tricks about getting the most out of Veeam and vSphere as well as talk about how we extend availability into the cloud.

If you are in Sydney or Melbourne next week try and get down to The Westin and The Crown Casino respectively to participate, learn and contribute and hopefully we can catch up for a drink.

Quick Fix: vCloud Director SP None of the Cells have a vCenter Proxy Service Running. SSL Protocol Fix

vCloud Director SP 8.20 was released a few weeks ago and I wanted to highlight an issue I ran into while testing of the BETA. I hadn’t come across this issue in previous versions of vCD and even though it relates to the fact I had a vCenter 5.5 I thought it worth a post now that 8.20 has GA’ed.

After I upgraded my cells I got the fairly common error message under the Cloud Cells section of the Manage & Monitor menu telling me that I didn’t have a vCenter Proxy service running. It’s something all vCD administrators would have seen over the years, so I did the usual troubleshooting step of going of reconnecting the vCenter under vSphere Resources. This didn’t work, so I did what comes naturally and cleared the Quartz Tables in the vCD database without any success.

Failed to connect to the vCenter. Please check if this is a valid vCenter server and the credentials are correct.

The NestedESXi lab was running vCenter 5.5 U3b and after a bit of searching I came across a post in the vCloud BETA forums relating to this issue:

Starting with VDC 8.20, the SSL protocol ‘TLSv1’ is no longer supported by default in the product for security reasons (as a server to serve the REST API request, but also as a client when talking to vCenter).
The version of vCenter you are running (please confirm which version), is older and probably only supports TLSv1.

Which explains the errors I also had been observing. Note that from 5.5 Update 3e and 6.0 Update 3 and later TLS v1.0 has been disabled and should be disabled.

Due to security concerns in the TLSv1.0 protocol, both Payment Card Industry (PCI) and BSI organizations have suggested to implement and enable TLSv1.1 or TLSv1.2, and move away from the use of TLSv1.0 as soon as possible

Even though it’s not suggested I needed to enable TLS v1 so that vCD SP 8.20 could connect to the vCenter. The following steps where done to enable TLSv1 which was based off this VMwareKB outlining why cells no longer enable SSL v3 by default and talks about a cell management tool command that configures the allowed SSL Protocols vCD uses during the handshake process with vCenter.

The SSL V3 protocol has serious vulnerability, described in CVE-2014-3566. As of vCloud Director 5.5.3, cells no longer enable SSL V3 by default for internal and external HTTPS connections. The vCloud Director cell management tool has been updated with a new subcommand that enables the system administrator to configure the set of SSL protocols that the cell offers to use during the SSL handshake process. This new subcommand has been made available in vCloud Director 5.5.3

Run the following command on the vCD cell in /opt/vmware/vcloud/bin/

./cell-management-tool ssl-protocols -d SSLv3,SSLv2Hello

After that is done restart the cell and check to make sure you have a listener and that vCenter is connected. If you run the ssl-protocols command with a -l flag it will show you what ssl-protocols are allowed. By default you should now only have TLS v1.1 and 1.2 enabled, but in my case I also needed v1.

Finally, it’s worth repeating that TLS v1 shouldn’t be used in production, but if you are still running older versions of 5.5 and 6.0 in your labs then this will help.

References:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2112282

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2145796

Looking Beyond the Hyper-Scaler Clouds – Don’t Forget the Little Guys!

I’ve been on the road over the past couple of weeks presenting to Veeam’s VCSP partners and prospective partners here in Australia and New Zealand on Veeam’s Cloud Business. Apart from the great feedback in response to what Veeam is doing by way of our cloud story I’ve had good conversations around public cloud and infrastructure providers verses the likes of Azure or AWS. Coming from my background working for smaller, but very successful service providers I found it almost astonishing that smaller resellers and MSPs seem to be leveraging the hyper-scale clouds without giving the smaller providers a look in.

On the one hand, I understand why people would choose to look to Azure, AWS and alike to run their client services…while on the other hand I believe that the marketing power of the hyper-scalers has left the capabilities and reputation of smaller providers short changed. You only need to look at last week’s AWS outage and previous Azure outages to understand that no cloud is immune to outages and it’s misjudged to assume that the hyper-scalers offer any better reliability or uptime than the likes of providers in the vCloud Air Network or other IaaS providers out there.

That said, there is no doubt that the scale and brain power that sits behind the hyper-scalers ensures a level of service and reliability that some smaller providers will struggle to match, but as was the case last week…the bigger they are, the harder they fall. The other things that comes with scale is the ability to drive down prices and again, there seems to be a misconception that the hyper-scalers are cheaper than smaller service providers. In fact most of the conversations I had last week as to why Azure or AWS was chosen was down to pricing and kickbacks. Certainly in Azure’s case, Microsoft has thrown a lot into ensuring customers on EAs have enough free service credits to ensure uptake and there are apparently nice sign-up bonuses that they offer to partners.

During that conversation, I asked the reseller why they hadn’t looked at some of the local VCSP/vCAN providers as options for hosting their Veeam infrastructure for clients to backup workloads to. Their response was, that it was never a consideration due to Microsoft being…well…Microsoft. The marketing juggernaut was too strong…the kickbacks too attractive. After talking to him for a few minutes I convinced him to take a look at the local providers who offer, in my opinion more flexible and more diverse service offerings for the use case.

Not surprisingly, in most cases money is the number one factor in a lot of these decisions with service uptime and reliability coming in as an important afterthought…but an afterthought non-the less. I’ve already written about service uptime and reliability in regards to cloud outages before but the main point of this post is to highlight that resellers and MSP’s can make as much money…if not more, with smaller service providers. It’s common now for service providers to offer partner reseller or channel programs that ensure the partner gets decent recurring revenue streams from the services consumed and the more consumed the more you make by way of program level incentives.

I’m not going to do the sums, because there is so much variation in the different programs but those reading who have not considered using smaller providers over the likes of Azure or AWS I would encourage to look through the VCSP Service Provider directory and the vCloud Air Network directory and locate local providers. From there, enquire about their partner reseller or channel programs…there is money to be made. Veeam (and VMware with the vCAN) put a lot of trust and effort into our VCSPs and having worked for some of the best and know of a lot of other service provider offerings I can tell you that if you are not looking at them as a viable option for your cloud services then you are not doing yourself justice.

The cloud hyper-scalers are far from the panacea they claim to be…if anything, it’s worthwhile spreading your workloads across multiple clouds to ensure the best availability experience for your clients…however, don’t forget the little guys!