With the release of Backup & Replication 9.5 Update 3 we introduced the concept of a Recycle Bin for customers sending offsite cloud backups to VCSPs using Veeam Cloud Connect. This deleted backup protection…or Insider Protection allows the VCSP to enable the deleted backups protection option for specific tenants and looks to add another level of data security for cloud based backups in the case of a malicious user gaining access to the Backup & Replication Console or in the case of accidental deletion by an administrator.
As shown above, this is set by checking a box (Also via PowerShell) in the properties of the tenant account. Once checked the SP will choose the retention period by setting the Keep deleted Backup files for <N> days option. With this option enabled, when a backup or a specific restore point in the backup chain is deleted or aged out from the cloud repository. The actual backup files are not deleted immediately, instead, they are moved to a _RecycleBin folder on the repositories.
Once moved, backup files in the recycle bin do not consume tenant quota however they obviously consume general storage. With that in mind it should be considered by the SP to charge for that used storage. I will release a post shortly detailing some tips on how best to size and charge for the recycle bin storage per client.
At the tenant end those backup files that are moved into the recycle bin are not registered and will not show up in the job information window. They can’t access or do anything with the files in the recycle bin. For the moment if a tenant wants to restore data they must contact the SP to obtain the necessary backup files. Once the retention period has expired all files that fall out of that period are deleted.
Basic Mechanics:
When the option is checked for a tenant a new folder is created under the _RecycleBin\<tenant> folder of the repository. In the case of a Scale Out Backup Repository there is a recycle bin folder created per extent which ensured that any split tenant VM files are processed locally and not between extents.
Once files in the repository start to age out the tenant folder will start to populate with backup files. If there is an event that triggers a change of retention or a VM removed from a job or the deletion of a whole job, any remaining VBK or VIB files in the tenant repository are moved into the recycle bin.
The files remain in the _RecycleBin folder until the retention period has passed or if the service provider moves them out of the folder for recovery purposes.
Working Example:
I have a Cloud Connect Backup account that I am using to back up five VMs that reside on premises, using a standard Backup Job with Forward Incrementals and a Synthetic Full done once a week. I have configured this job to keep two restore points.
I then have configured a secondary destination for the job via a Backup Copy Job to the Cloud Repository and I have set a GFS to happen weekly so I have a full archive offsite. If I hadn’t enabled GFS retention (for those running Update 3) a warning would appear as shown below.
Tip: If the tenant plans to create off-site copies of backed-up data with a backup copy job, it should enable GFS retention settings in the job properties. This way, Veeam Backup & Replication will be able to protect backups created by the job against an attack when a hacker reduces the job’s retention policy and creates a few incremental backups to remove backed-up data from the backup chain.
The Cloud Connect Tenant account has a deleted backup protection setting of 2 days configured as shown in the first image of this post.
Below is the local jobs folder structure:
Looking at the Cloud Connect repository (split over two SOBR extents) you can see that the main repository holds the VM backup files as per the job configuration. Notice the GFS _W files there as well.
Taking a look at the _RecycleBin folder for the tenant after a few days the aged out incremental will start to appear in the folder. Notice that there are no full backup files in the recycle bin at this stage.
Tip: The retention period will look at all backup jobs completed in a 24 hours period and have any expiring or deleted backup files moved into the recycle bin directory. This means that if you are copying up VMs that have a local backup interval of every 4 hours you will have six lots of backup files ageing out daily.
In this example I’m simulating an malicious attack or accidental deletion the VM (TPM03-RMQ-01/VM-120) from the backup. For the sake of this example we are deleting the VM from the Backup & Replication Console under Backups and Cloud. If the Included Archived copies option was chosen then the GFS weekly full backup file is also moved into the recycle bin.
Once the deletion process has been completed the _RecycleBin folder for the tenant will now be populated with the deleted full, plus three incremental files. If the Included Archived copies option was chosen then the GFS weekly full backup file is also moved into the recycle bin.
These will stay in the recycle bin until the retention period is met. From here these files can be transported back to the tenant to be recovered (see here for full process) from within the on-premises Backup & Replication console.
Conclusion:
As shown above, deleted backup protection or Insider Protection is an excellent enhancement to Cloud Connect Backup. It goes some way to having an air gapped backup in the cloud and protects against malicious attacks and rogue or clumsy administrators. There is a lot happening behind the scenes to make it work, however the concept is simple and this features extends the 3-2-1 rule by protecting that offsite copy as part of the Cloud Connect solution. VCSP’s should be looking to offer this as a value add to their clients and Veeam customers should be looking to take advantage of Cloud Connect Backup and Replication for their offsite backup and replication needs.
References:
https://helpcenter.veeam.com/docs/backup/cloud/cloud_connect_bin.html?ver=95
https://helpcenter.veeam.com/docs/backup/cloud/cloud_connect_bin_restore.html?ver=95
