Category Archives: Azure

Cloud Tier Data Migration between AWS and Azure… or anywhere in between!

At the recent Cloud Field Day 5 (CFD#5) I presented a deep dive on the Veeam Cloud Tier which was released as a feature extension of our Scale Out Backup Repository (SOBR) in Update 4 of Veeam Backup & Replication. Since we went GA we have been able to track the success of this feature by looking at Public Cloud Object Storage consumption by Veeam customers using the feature. As of last week Veeam customers have been offloading petabytes of backup data into Azure Blob and Amazon S3…not counting the data being offloaded to other Object Storage repositories.

During the Cloud Field Day 5 presentation, Michael Cade talked about the Portability of Veeam’s data format, around how we do not lock our customers into any specific hardware or format that requires a specific underlying File System. We offer complete Flexibility and Agnosticity where your data is stored and the same is true when talking about what Object Storage platform to choose for the offloading of data with the Cloud Tier.

I had a need recently to setup a Capacity Tier extent that was backed by an Object Storage Repository on Azure Blob. I wanted to use the same backup data that I had in an existing Amazon S3 backed Capacity Tier while still keeping things clean in my Backup & Replication console…luckily we have built in a way to migrate to a new Object Storage Repository, taking advantage of the innovative tech we have built into the Cloud Tier.

Cloud Tier Data Migration:

During the offload process data is tiered from the Performance Tier to the Capacity Tier effectively Dehydrating the VBK files of all backup data only leaving the metadata with an Index that points to where the data blocks have been offloaded into the Object Storage.

This process can also be reversed and the VBK file can be rehydrated. The ability to bring the data back from Capacity Tier to the Performance Tier means that if there was ever a requirement to evacuate or migrate away from a particular Object Storage Provider, the ability to do so is built into Backup & Replication.

In this small example, as you can see below, the SOBR was configured with a Capacity Tier backed by Amazon S3 and using about 15GB of Object Storage.

The first step is to download the data back from the Object Storage and rehydrate the VBK files on the Performance Tier extents.

There are two ways to achieve the rehydration or download operation.

  1. Via the Backup & Replication Console
  2. Via a PowerShell Cmdlet
Rehydration via the Console:

From the Home Menu under Backups right click on the Job Name and select Backup Properties. From here there is a list of the Files contained within the job and also the objects that they contain. Depending on where the data is stored (remembering that the data blocks are only even in one location… the Performance Tier or the Capacity Tier) the icon against the File name will be slightly different with files offloaded represented with a Cloud.

Right Clicking on any of these files will give you the option to Copy the data back to the Performance Tier. You have the choice to copy back the backup file or the backup files and all its dependancies.

Once this is selected, a SOBR Download job is kicked off and the data is moved back to the Performance Tier. It’s important to note that our Intelligent Block Recovery will come into play here and look at the local data blocks to see if any match what is trying to be downloaded from the Object Storage… if so it will copy them from the Performance Tier, saving on egress charges and also speeding up the process.

In the image above you can see the Download Job working and only downloaded 95.5MB from Object Storage with 15.1GB copied from the Performance Tier… meaning the data blocks for the most that are local are able to be used for the rehydration.

The one caveat to this method is that you can’t select bulk files or multiple backup jobs so the process to rehydrate everything from the Capacity Tier can be tedious.

Rehydration via PowerShell:

To solve that problem we can use PowerShell to call the Start-VBRDownloadBackupFile cmdlet to do the bulk of the work for us. Below are the steps I used to get the backup job details, feed that through to variable that contains all the file names, and then kick off the Download Job.

The PowerShell window will then show the Download Job running

Completing the Migration:

No matter which way the Download job is initiated, we can see the progress form the Backup & Replication Console under the Jobs section.

And looking at the Disk and Network sections of Windows Resource Monitor we can see connections to Amazon S3 pulling the required blocks of data down.

Once the Download job has been completed and all VBKs have been rehydrated, the next step is to change the configuration of the SOBR Capacity Tier to point at the Object Storage Repository backed by Azure Blob.

The final step is to initiate an offload to the new Capacity Tier via an Offload Job…this can be triggered via the console or via Powershell (as shown in the last command of the PowerShell code above) and because we have already a set of data that satisfies the conditions for offload (sealed chains and backups outside the operational restore window) data will be dehydrated once again…but this time up to Azure Blob.

The used space shown below in the Azure Blob Object Storage matches the used space initially in Amazon S3 All recovery operations show Restore Points on the Performance Tier and on the Capacity Tier as dictated by the operational restore window policy.
Conclusion:

As mentioned in the intro, the ability for Veeam customers to have control of their data is an important principal revolving around data portability. With the Cloud Tier we have extended that by allowing you to choose the Object Storage Repository of your choice for cloud based storage or Veeam backup data…but also given you the option to pull that data out and shift when and where desired. Migrating data between AWS, Azure or any platform is easily achieved and can be done without too much hassle.

References:

https://helpcenter.veeam.com/docs/backup/powershell/object_storage_data_transfer.html?ver=95u4

Quick Look – New Cloud Credentials Manager in Update 4

With the release of Update 4 for Veeam Backup & Replication 9.5 we further enhanced our overall cloud capabilities by adding a number of new features and enhancements that focus on tenants being able to leverage Veeam Cloud and Service Providers as well as Public Cloud services. With the addition of Cloud Mobility, External Repository and Cloud Connect Replication supporting vCloud Director we decided to break out the existing credential manager and create a new manager dedicated to the configuration and management of Cloud specific credentials.

The manager can be accessed by clicking on the top left dropdown menu from the Backup & Replication Console and then choosing Manage Cloud Credentials.

You can use the Cloud Credentials Manager to create and manage all credentials that are planned to use to connect to cloud services.

The following types of credentials can be configured and managed:

  • Veeam Cloud Connect (Backup and Replication for both Hardware Plans and vCD)
  • Amazon AWS (Storage and Compute)
  • Microsoft Azure Storage (Azure Blob)
  • Microsoft Azure Compute (Azure and Azure Stack)

The Cloud Connect credentials are straight forward in terms of what they are used for. There is even a way for non vCloud Director Authenticated tenants to change their own default passwords directly.

When it comes to AWS and Azure credentials the manager will allow you to configure accounts that can be used with Object Storage Repositories, Restore to AWS (new in Update 4), Restore to Azure and Restore to Azure Stack (new in Update 4).

PowerShell is still an Option:

For those that would like to configure these accounts outside of the Backup & Replication Console, there is a full complement of PowerShell commands available via the Veeam PowerShell Snap-in.

As an example, as part of my Configure-Veeam GitHub Project I have a section that configures a new Scale Out Backup Repository with an Object Storage Repository Capacity Tier backed by Amazon S3. The initial part of that code is to create a new Amazon Storage Account.

For a full list of PowerShell capabilities related to this, click here.

So there you go…a very quick look at another new enhancement in Update 4 for Backup & Replication 9.5 that might have gone under the radar.

References:

https://helpcenter.veeam.com/docs/backup/vsphere/cloud_credentials.html?ver=95u4

Veeam Availability Console now available from Azure Marketplace

Last week the Veeam Availability Console Azure Marketplace appliance went live. This allows Veeam Cloud and Service Providers to easily deploy VAC into any Azure region. In it’s previous incarnation the Managed Backup Portal was only available as an Azure marketplace appliance and not available to install by a VCSP. Now that VAC 2.0 is out, VCSPs who don’t have the ability to host Cloud Connect or VAC on their infrastructure can deploy it in Azure and have the service up and running within fifteen minutes.

There are some limitations that come along with deploying VAC into Azure and it won’t be for everyone. The biggest caveat is that you can only have one Cloud Connect Server per VAC instance and as part of the deployment, Cloud Connect services is installed on the same Virtual Machine. You can’t offer Replication services from the Azure instance, and if offering Cloud Connect backup you need to understand it’s own scalability and performance bottlenecks. That said, as a remote management, monitoring, reporting, billing and self service platform there is a lot to like about having VAC in Azure.

Marketplace Deployment Steps:

You can start the deployment by searching for Veeam Availability Console in the Azure Marketplace or you can go direct to the product page here.

Click on Create to start the configuration steps.

The Basics includes VM name, hard disks type, username and password as well as selecting the subscription, the ability to use a new or existing resource group and finally the Azure location you want to deploy into.

In Step 2 you need to choose the Size of the Azure instance. The template provides the recommended configurations. The sizes are relative to the amount of agents and/or Backup & Replication instances you are going to be managing from this instance. You can find sizing guides here for larger environments.

I ended up going with an A2 standard for my instance which removes the load balancing functionality from the configuration and offers a little less IOPS. Step 3 contains some optional extra’s to ensure a higher level of availability for the VM instance and lets you configure the networking. Once that’s done you can review your configuration settings and start the deployment. It took just over 8 minutes for the deployment to succeed.

If you click on the Virtual Machine object in the Azure Portal you will see an overview of the VM and it’s configuration.

Addition Azure Configuration:

If you notice in the image above, a DNS name is listed in the overview. This was something that I had to set manually after the deployment. You set this by going into the Networking of the resource pool and click on IP Configuration. Here, you can enter in a DNS name relative to the Azure zone you are in. You can then use this to connect to the VAC Console, Cloud Connect Service and to RDP to the VM and helps in the event of having a dynamic, rather than a static Azure IP.

Speaking of networking and ports, below is a list of the default port rules created during the deployment. Note that WinRM is open as well.

Finalizing Deployment:

After deploying the Azure Marketplace appliance you can RDP into the VM and complete the setup that includes configuring Cloud Connect and VAC it’s self. A few things have been done for us as part of the deployment, however the first thing you need to do is get a license. This is a BYO license situation, so once you have deployed the Marketplace appliance you will need to source a VAC license from the Veeam Licensing Portal and apply.

Head to the VAC Web Portal and Install the License.

Once done the last step is to configure Cloud Connect from the Backup & Replication Console. Again, you will need a valid Cloud Connect license as you are greeted with the Free Edition when you connect to the console for the first time. As per normal with Cloud Connect, you need to configure the SSL Certificate first and then configure a new Cloud Gateway. Configure the Networking as shown below using the DNS name that was created in the steps above.

Once this is completed you can go into the VAC Console and work through the normal Configuration steps. The only thing you don’t need to do is add the Cloud Connect Server to the VAC instance as this has already been done during the initial deployment process.

It’s worth noting that the versions of Backup & Replication (9.5.0.1536) and Availability Console (2.0.1.1343) are up to date and include the latest Hot-Fixes for VAC. The intent is to have the templates as up to date as possible, however once deployed you can upgrade as per usual.

Conclusion:

So there you have it…within fifteen minutes you can have a fully working Veeam Availability Console instance running in Azure and ready to be used to offer all the goodness that VAC offers our Cloud and Service Provider partners. For an overview as to what VAC offers, click here and have a read of my GA post on What’s in It for Service Providers.

Links:

https://azuremarketplace.microsoft.com/en-us/marketplace/apps/veeam.veeam-availability-console?tab=Overview