Author Archives: Anthony Spiteri

v10 Enhancements – Mounting Object Storage Repository for DR

Version 10 of Veeam Backup & Replication isn’t too far away and we are currently at the end of a second private BETA for our customers and partners. There has been a fair bit of content released around v10 functionality and features from our Veeam Vanguard’s over the past couple of weeks and as we move closer to GA, as part of the lead up, I am doing a series on some of the cool new enhancements that are coming as part of the release. These will be quick short takes that give a glimpse into what’s coming as part of the v10 release.

Mounting Object Storage Repository for Streaming Disaster Recovery

The Cloud Tier was introduced in Veeam Backup & Replication 9.5 Update 4 and focused on the offloading of data from local repositories to Object Storage repositories. Essentially looking to reduce the cost and overheads of ever growing local primary repositories. Due to the smarts we built into the feature, the use cases for Cloud Tier started to expand beyond the offloading of data and looked at recovery options.

Because we hold a replicated copy of the VBK metadata as well the actual backup data that is indexed as blocks in Object Storage we have the ability to leverage the data sitting there for recovery purposes. I’ve already shown this a number of times this year, and presented on the recovery and resiliency of the Cloud Tier at Cloud Field Day 5.

With v10, we have made this process even easier by introducing a Mount function that will enable users to import backup restore points for recovery purposes in the case of disaster. This can even be done with the Community Edition which means that the Cloud Tier now becomes a mechanism for recovery from any device to almost any platform.

Quickly going over how this works, the first step is to recreate the Object Storage Repository with the same settings as the one existed in the original location.

At this point we can leverage the new v10 feature that allows you to Import the backup data contained on the Object Storage Repository by right clicking on the repository and selecting Import Backups.

This will store the available restore points in the Backup & Replication database and have them appear under Imported Backups in the console.

It’s important and cool to note, that as this stage we haven’t downloaded the metadata shells that constitute the de-hydrated VBK. One of the extra smart things we have built into this feature is that the metadata and VBK shells are only downloaded once a restore operation has been started, meaning quicker setup and more specific re-syncing of the metadata shells.

On that note, all existing restore operations are available at this point.

Once a restore operation is triggered, only then is the required metadata downloaded and reconstructed into the required shell chain to a temp directory. The example below shows the shells of a full and an incremental triggered by an Instant VM Recovery (IVMR) Operation.

The data required to perform the IVMR is streamed from the Object Storage Repository (Capacity Tier Extent).

Once restore operations have been completed you can go back to the Object Storage Repository, right click and select Detach

This unmounts the Object Storage and removed the restore points from the Imported Backup view and deletes the downloaded contents of the temp folder where the metadata shells where staged.

Wrap Up:

That was a quick look at one of my personal favourite new enhancements in v10. We have improved an operation that was being leveraged in 9.5 Update 4 due to the smarts built into the Cloud Tier for recovery operations and made it quicker and more efficient. This also allows users to effectively restore to any platform from any device that has Veeam Backup & Replication installed and has access to the Object Storage platform!

When put together with the new Copy mode being introduced into v10, we all of a sudden have a solution that can achieve very low RPO and RTO for disaster recovery… more to come on that aspect when v10 launches.

Stay tuned over the next few weeks as I go through some more hidden gems.

Disclaimer: The information and screen shots in this post is based on BETA code and may be subject to change come final GA.

Important ESXi 6.0 and 6.7 Patch Release – CBT Fixes and More!

Over the last few years the amount of CBT related issues has decreased significantly from VMware. I remember back in my previous roles of having to deal with multiple issues and wrote some pretty heavy posts around the topic. While those posts where not popular in some circles at the time, they certainly highlighted the importance of having stable CBT code in vSphere for backup vendors to leverage as part of their solutions. The threat of corruption in backup data was a worst case scenario back then… but even more so now, given how critical data has become.

Contained in the latest patch releases for ESXi 6.0 (November) and 6.7 (last week) is a fix for the an an issue that surfaced a few months ago in relation to a very specific condition upon reverting a SnapShot where CBT data might become corrupted. ESXi 6.5 patch has not been released yet.

The VMwareKB has the information here.

For both ESXi 6.0 patch release and the ESXi 6.7 patch release, the description of the issue and the detail of the fix is listed below.

  • When reverting a virtual machine that has CBT enabled to a snapshot which is not a memory snapshot, you might see InvalidArgument error if you use the QueryChangedDiskAreas() API call. After you revert a virtual machine to a snapshot, change block tracking (CBT) data might be corrupted. When reverting a virtual machine that has CBT enabled to a snapshot which is not a memory snapshot, and if you use the QueryChangedDiskAreas() API call, you might see an InvalidArgument error. This issue is resolved in this release. With ESXi670-201912001, the output of the QuerychangedDiskAreas () call changes to FileFault and adds the message Change tracking is not active on the disk <disk_path> to provide more details on the issue.

In regards to Veeam customers, our SVP of Product, Anton Gostev highlighted the patches in his Weekly Forum Digest (which everyone should be following) and there will be an update to confirm all is well in next weeks digest when it comes to Veeam Backup & Replication working transparently with the fix from VMware.

The patch for 6.7 contains a number of additional fixes.. a lot of them centered around storage so it’s worth looking through the Resolved Issues section of the


Released: Backup for AWS …Free for up to 10 EC2 Instances!

This week at AWS re:Invent, exciting news for a lot of us in Veeam was announced as we made available for GA Veeam Backup for AWS (Build Available through the AWS Marketplace the product can be deployed within five minutes and be ready to backup EC2 instances. From my point of view, apart from the technical aspects, probably the biggest news of this release is that the FREE edition will backup up to 10 EC2 instances out of the box and is fully featured.

Apart from being free there are a number of capabilities in this initial release which are innovative and will prove valuable for our customers consuming the product.

  • Automates Amazon EBS snapshots for frequent backup and fast restores
  • Backup to Amazon S3 Repository for long-term retention
  • Policy-based protection and job creation
  • On-Demand Worker Nodes that work as data movers
  • Web-based management UI
  • Built-in cost estimation
  • Support for IAM role separation as well as cross-region and cross-account configuration and Multi-factor authentication compatible
  • Restore to the original instance
  • Restore to a new instance
  • File-level recovery

The Veeam Product Strategy Team has already produced a few blog posts around the release which goes into some greater details around the core components and features of Backup for AWS.

As we look to quickly iterate on this initial v1 release we will look to support even more AWS services but for now, this is a significant moment in Veeam’s history as we look to broaden our own in-house capabilities and extend the Veeam Backup Platform to cover even more workloads and make them portable as they land in our repositories and in our portable data format.

Links and Downloads:

Quick Fix – Terraform Linux Remote-Exec Provisioner Errors without Exit Status

When deploying and configuring Linux as part of my Terraform plans, I have generally used a combination of script based or in-line Remote-Exec declarations to get the job done. A while back, while working on my Ansible CentOS Terraform deployment hack, I came across the following error when the executable shell script got to the point where I needed to either log-out or reboot to close off the existing session.

The reboot was required to set the correct Python version for the user session in Bash.

Effectively, because Terraform didn’t receive a correct exit status or signal it threw the error and stopped the plan. After digging around and discovering that Terraform can’t send a reboot command its self to a vSphere VM (Unless i’ve missed the command) I came across the Failure Behavior for Provisioners.

Failure Behavior

By default, provisioners that fail will also cause the Terraform apply itself to fail. The on_failure setting can be used to change this. The allowed values are:

  • “continue” – Ignore the error and continue with creation or destruction.
  • “fail” – Raise an error and stop applying (the default behavior). If this is a creation provisioner, taint the resource.

So the Quick Fix is after the remote-exec provisioner declaration, to simply add in:

on_failure = “continue”

This will allow Terraform to continue to the next declaration and continue to proceed with executing the plan. The default value is to fail, so there isn’t really any need to define a fail on the failure… as it will exit as per normal. Again, it would be nice is Terraform added a feature where the vSphere Provider can reboot/reset the VM as a declarable action.


Released: Veeam Powered Network 2.1 … Upgrade in Under a Minute!

A couple of days ago we snuck out a minor update to Veeam Powered Network (Veeam PN) bringing the version up to 2.1 (Build We released 2.0 back in May which was a significant release as we ripped out OpenVPN and replaced it with WireGuard for the site to site networking functionality. Version 2.1 bring with it some minor under the hood enhancements but it mainly focused on supporting Veeam PN being deployed from the AWS Marketplace.

For those that need a little refresher on what Veeam PN does… it presents a simple and intuitive Web Based User Interface for the setup and configuration of site-to-site and point-to-site VPNs. Veeam PN has become popular in the IT enthusiast and home lab worlds as a simple and reliable way to remain connected while on the road, or to mesh together with ease networks that where spread across disparate platforms.

I use Veeam PN almost all the time from home and more importantly, when I am away on work trips to connect back to my home office as well as remote into various platforms where I run lab workloads.

Upgrading to 2.1:

For those still running 1.0 there is no direct upgrade to 2.x due to the replacement of OpenVPN with WireGuard. For those on 2.0, the upgrade is simple and can be completed in under a minute.

When you login, you should see a message above the main dashboard

Clicking on that will take you to the Settings -> Updates Tab

Click on the Update Now button and the upgrade will begin. Note that there is a services restart so all existing connections will be disconnected

After about a minute or so, you can log back in, and all end points should automatically reconnect. There is no need to re-download the site-to-site or point-to-site configurations.

Note that you can also upgrade via the command line using apt-get.

New DNS Configuration:

In the Veeam PN hub portal, you can see the list of configured sites, DNS suffixes and DNS servers. If you want to disable DNS on a network hub you can toggle the DNS setting to off.  DNS forwarding and configuring was introduced in 2.0 to resolve FQDNs in connected sites.


Released: Backup for Office 365 4.0 …Now with Object Storage Support!

Yesterday, Veeam Backup for Microsoft Office 365 (VBO) version 4.0 ( went GA. This new version again builds on the 3.0 release that enhanced overall performance for Exchange, SharePoint and OneDrive as well as further enhancements around backup policy, reporting and APIs. Version 4.0 is heavily focused on the addition of Object Storage support for the storing VBO data which has been a top request for a long time by our customers and partners.

Version 3.0 was released just seven months ago and it’s a credit to the R&D team for pushing out a major update in such a small space of time. I talked a little bit about the Object Storage support here when we released the public beta but it’s an understatement to say that being able to leverage Public or Private cloud Object Storage to store the sheer volume of data produced by Office 365 Organizations is key to the future success of this product in terms of what it offers customers longer term.

What’s New in 4.0:

While understanding how best to deal with the backing up of SaaS based services where a lot of what happens is outside of the control of the backup vendor, there are challenges around the backing up and restoring of Exchange, SharePoint and OneDrive. With the release of version 4.0 one of the biggest pain points in terms of data growth has been solved with the leveraging of Object Storage. We have also added efficiencies around the backing up of SharePoint sites in better dealing with Office 365 throttling mechanisms by leveraging multiple backup accounts helps to distribute the load on Office 365 servers and significantly reducing the risk of backups throttling.

Apart from the headline new features and enhancements there are also a number of additional ones that have been implemented into Backup for Microsoft Office 365 4.0.

  •  Exclude retention for contacts and calendars allows you to protect all contacts and calendar items for as long as an associated mailbox is protected and skip these items from the retention cleanup.
  • Additional automation through non-mail enabled Office 365 security groups support allows for easier backup jobs management. Azure Active Directory security groups populated to Office 365 (or created in Office 365 directly) can now be used as a source for backup jobs.
  • Enhanced Mailbox Protection report now includes protection statistics for Office 365 Group, Public, Shared and Resource (Equipment/Room) mailboxes.
  • Added a Cloud Credential Manager and Password Manager for maintaining a list of account records to connect to object storage.
  • Console will now automatically disconnect from the backup server after 30 minutes of inactivity to reduce the load on the server.

For another look at what’s new, Niels Engelen goes through his top new features in detail here and for service providers out there, it’s worth looking at his Self Service Portal which has also been updated to support 4.0.

Architecture and Components:

Obviously the biggest change to the architecture is when you want to leverage Object Storage for a repository. This keeps only metadata and a cache with all backup data located in Object Storage. Because multiple Repositories can be created and mapped to one or more users, there is flexibility in being able to have a choice of local storage or one or more Object Storage platforms. When using object storage, data can be protected with optional encryption at-rest.

Proxies are the work horses of VBO and can be scaled out again depending on the size of the environment being backed up. Again, this could be Office 365 or on-premises Exchange or SharePoint instances.

Installation Notes:

You can download the the latest version of Veeam Backup for Microsoft Office 365 from this location. The download contains three installers that covers the VBO platform and two new versions of the Explorers. Explorer for Microsoft OneDrive for Business is contained within the Explorer for Microsoft SharePoint package and installed automatically.

  • for Veeam Backup for Microsoft Office 365
  • for Veeam Explorer for Microsoft Exchange
  • for Veeam Explorer for Microsoft SharePoint

To finish off…It’s important to read the release notes here as there are a number of known issues relating to specific situations and configurations.

Backup for Office 365 has been a huge success for Veeam with a growing realisation that SaaS based services require an availability strategy. The continuity of data on SaaS platforms like Office 365 is not guaranteed and it’s critical that a backup strategy is put into place.

Links and Downloads:

Kubernetes… Kubernetes… Kubernetes!

Kubernetes is getting centre stage at KubeCon and Cloud Native Con this week and it has attracted 12000 attendees which is amazing for an open source non vendor specific conference. Kubernetes has dominated the IT water-cooler talk this year and a lot of people talk about it… but are they also doing it? This post is more or less a little social experiment testing the lure of industry keywords and current trend topics.

I have written a legitimate opinon piece today, which can be found here around KubeCon and some thoughts on Kubernetes in relation to OpenStack, Docker and the rest of the Cloud Native landscape. Apart from this being honeypot(ish) post, I am legitimately interested in polling people on the following:

Have you installed, configured and used Kubernetes?

View Results

Loading ... Loading ...

Is your company actively deploying Cloud Native Applications today?

View Results

Loading ... Loading ...

Is your company actively deploying Containerised Applications today?

View Results

Loading ... Loading ...

#Kubcon 12000 people cant be wrong…right?

#Kubcon – 12,000 People Can’t Be Wrong… Right?

This week, KubeCon and Cloud Native Con is happening in San Diego. In the lead up to the event, there was talk about 12,000 registrations which puts it up there with one of the fastest growing industry events by numbers and apparently the biggest independent, non vendor specific event ever in our industry. When you look at the fact that VMworld US had approx. 20,000 attendees the attendance for KubeCon is impressive to say the least.

Most people are aware that Kubernetes is hot right now. I’ve written a couple of articles this year on the subject and been also tracking the rise of Kubernetes in the vernacular of the more traditional infrastructure IT community over the past few years.

There is no doubt there is a significant element of #FOMO associated with the rise of Kubernetes, but looking at the breath of the conference on a whole it’s more about the Cloud Native aspect of the ecosystem. Kubernetes as a “theme” is the draw, but I would put a bet on the fact that a large chunk of the attendees (Dev community and those directly associated with Cloud Native aside) have not grasped the cloud native movement that powers KubeCon and Cloud Native Con… I will admit that it is something that I am yet to comes to terms with as well.

Are there shades of OpenStack here?

Docker Enterprise was acquired last week my Mirantis which in it’s self started life as an OpenStack offering to rival the likes of more managed OpenStack platforms from VMware and a like. OpenStack is now just a block in the overall picture of what Mirantis offers. OpenStack was set to dominate the IT industry and change the world and back when it was ontop of its own hype curve I remember a lot of similar FOMO conversations happening.

Kubernetes appears to be more than a block at the moment. I’ve been talking to a lot of people and discovering the power of Kubernetes myself. While I will reserve judgement on the holy war battleground fight that appears to have been run and won. What some may find interesting is that Docker is still the containerisation platform that powers the orchestration and management layer that is Kubernetes. It obviously extends and is being extended to more use cases, but the parallels in terms of hype between Kubernetes and OpenStack are noted.

Large Complex Ecosystem

Orchestration engines for Docker was always the battleground in this cloud native space. That space only exists because there is a ground swell of developers creating applications on Cloud Native Platforms. While traditional/monolithic software development isn’t going away any time soon, it’s clear that the Cloud Native approach is well and truly mainstream.

That doesn’t make things easier, now that Cloud Native is more mainstream. In fact the CNCF confirms the existing complex ecosystem via its Cloud Native Trail Map that begins to point organizations down the right path as they start their Cloud Native journey.

To finish off, i’ll leave you with the image below. It’s the Cloud Native Landscape as it sits today. This isn’t your every day IT Infrastructure ecosystem. There are literally hundreds (thousands) of different permutations and choices consumers of IT need to think about when looking to go down their Cloud Native journeys. Kubernetes is a building block and one part of the puzzle… though an important one that does bring together a lot of the other elements you see below.

1,277 cards with a total market cap of $14.55T and funding of $63.28B cant be wrong… right?

This is also worth a watch from theCUBE guys.


KubeCon + CloudNativeCon North America 2019


Tech Field Day Recap #TFD20

Tech Field Day 20 has come and gone, and it was an honour to play a small part in the 10th year anniversary Tech Field Day event. This was my second TFD event for the year having attended Cloud Field Day 5 back in April. It’s always a privilege to present to the delegates and to those tuning in on the livestream. The significance is not lost on me, the impact that TFD has had on peoples careers. In an indirect way, it helped my land this role at Veeam as @RickVanover got his break having attended the first TFD. If Rick hadn’t gone to that, he wouldn’t have been hired by Veeam and further down the track I might not have had the opportunity to join…possibly.

In any case, well done to Stephen Foskett and GestaltIT on 10 years and the on the impact you have had on many peoples career in our extended tech community.

Veeam Recap:

We had the second slot on the Wednesday from 10am-12pm and presented around three main topics as well as a very quick re-introduction to Veeam and how we are doing in the market today.

Rick then took everyone through a Scale Out Backup Repository (SOBR) 101 and a quick recap of the Cloud Tier as it was released as part of Veeam Backup & Replication 9.5 Update 4. We actually could have level set a bit more at this point, but time was already short. With that I put together a quick post last week to further debunk some of the terminology we use when talking about SOBR and the Cloud Tier.

Veeam Cloud Tier Glossary

Following that, I went through two of the most anticipated features in our upcoming Veeam Backup & Replication v10 release. That is enhancements to the Cloud Tier by adding Copy Mode and Immutability for Amazon S3.

Michael Cade then took us through the v10 Enhanced NAS feature which is probably our most eagerly and long awaited/overdue feature in years. Michael does a great job of going through the differences between us and our competitors and also why we have waited this long to release backup for NAS… even though this is now much much more.

As an extra, Michael put out this video the next day further explaining how have implemented CRC into the feature for more efficient backup performance.

Finally, I had 15 minutes to race through a feature that is not coming as part of v10, but coming in 2020… CDP! It’s taken us a while, but as I said in the video, I believe we will have the most reliable and stable implementation of CDP. This isn’t something you want to mess around with, and I know all to well from experience the impact problematic CDP implementations can have.

#TFD20 Follow Up – Veeam Cloud Tier Glossary

Yesterday I presented at Tech Field Day 20. My first topic was on the enhancements we are bringing to Cloud Tier in our Backup & Replication v10 release. Rick Vanover setup the v10 enhancement session by doing some ground work on what a Scale Out Backup Repository is and briefly went over the initial features of Cloud Tier released in Backup & Replication Update 4.

We had a few questions around some of the terminology being used with regards to the Cloud Tier so I thought as a followup I would list out the glossary of terminology I’ve been building since the Update 4 release with the additions of the new v10 enhancements.

  • Cloud Tier – Cloud Tier is the name given to this feature in Veeam Backup & Replication 9.5 Update 4
  • Object Storage Repository – Object Storage Repository is the name given to a repository that is backed by Amazon S3, Azure Blob or IBM Cloud
  • Scale Out Backup Repository (SOBR) – Scale-Out Backup Repository is a Veeam feature first introduced in Veeam Backup & Replication v9. It consists of one or more Performance Tier extents and exactly one Capacity Tier extent.
  • Capacity Tier – Capacity Tier is the name given to extent on a SOBR using an Object Storage Repository.
  • Performance Tier – Name given to the one or more extents on a SOBR using a standard backup repository
  • Move Mode – Name given to a policy introduced in Update 4 that offloads data from sealed chains and has it in either Performance or Capacity Tier
  • Copy Mode – Name given to policy coming in v10 that immediately duplicates backup files from Performance to Capacity Tier once a backup job has completed
  • Offload Job – Name given to the process that moves data from Performance to Capacity Tier
  • Immutability Period – New feature coming in v10 that sets an Amazon S3 or S3 Compatible Object Lock on blocks copied or moved from the Performance or Capacity Tier protecting them against accidental or malicious deletion.

In addition to that, I have pasted a link to the offical Deep Dive Veeam Whitepaper for Cloud Tier that goes into the why the what and the how of the Cloud Tier and dives into the innovative technologies we have built into the feature.

White Paper Link:

If you want to catch the Cloud Field Day 5 presentation on Cloud Tier, as well as the most recent one yesterday at Tech Field Day 20, I have embedded them below.

« Older Entries