Tag Archives: Cloud Connect

Veeam Vault #11 – VBR, Veeam ONE, VAC Releases plus Important Update for Service Providers

Posted on by

Welcome to the 11th edition of Veeam Vault and the first one for 2019! It’s been more than a year since the last edition, however in light of some important updates that have been released over the past couple of weeks and months, I thought it was time to open up the Vault again! Getting stuck into this edition, I’ll cover the releases of Veeam Backup & Replication 9.5 Update 4b, Veeam One Update 4a as well as an update for Veeam Availability Console and some supportability announcements.

Backup & Replication 9.5 Update 4b and Veeam ONE 4a:

In July we released Update 4b for Veeam Backup & Replication 9.5. It brought with it a number of fixes to common support issues as well as a number of important platform supportability milestones. If you haven’t moved onto 4b yet, it’s worth getting there as soon as possible. You will need to be on at least 9.0 Update 2 (build 9.0.0.1715) or later prior to installing this update. After the successful upgrade, your build number will be 9.5.4.2866.

Veeam ONE 9.5 Update 4a was released in early September and containers similar platform supportability to Backup & Replication as well as a number of fixes. Details can be found in this VeeamKB.

Backup & Replication Platform support

  • VMware vCloud Director 9.7 compatibility at the existing Update 4 feature levels.
  • VMware vSphere 6.5 and 6.7 U3 Supportability vSphere 6.5 and 6.7 U3 GA is officially supported with Update 4b.
  • Microsoft Windows 10 May 2019 Update and Microsoft Windows Server version 1903 support as guest VMs, and for the installation of Veeam Backup & Replication and its components and Veeam Agent for Windows 3.0.2 (included in the update).
  • Linux Kernel version 5.0 support by the updated Veeam Agent for Linux 3.0.2 (included in the update)

For a full list of updates and bug fixes, head to the offical VeeamKB. Update 4b is a cumulative update, meaning it includes all enhancements delivered as a part of Update 4a. There are also a number of fixes specifically for Veeam Cloud & Service Providers that offer Cloud Connect services. For the full change log, please see this topic on the private VCSP forum.

https://www.veeam.com/kb2970

VAC 3.0 Patch:

Update 3 for Veeam Availability Console v3 (build 2762) was released last week, and containers a number of important fixes and enhancements. The VeeamKB lists out all the resolved issues, but i’ve summerized the main ones below. It is suggested that all VAC installations are updated as soon as possible. As a reminder, don’t forget to ensure you have a backup of the VAC server before applying the update.

  • UI – Site administrators can select Public IP Addresses belonging to a different site when creating a company. Under certain conditions, “Used Storage” counter may display incorrect data on the “Overview” tab.
  • Server – Veeam.MBP.Service fails to start when managed backup agents have duplicate IDs (due to cloning operation) in the configuration database.
  • Usage Reporting – Under certain conditions, usage report for managed Veeam Backup & Replication servers may not be created within the first ten days of a month.
  • vCloud Director – Under certain conditions, the management agent may connect to a VAC server without authentication.
  • Reseller – Reseller can change his or her backup quota to “unlimited” when creating a managed company with “unlimited” quota.
  • RESTful APIs – Querying “v2/tenants/{id}” and “/v2/tenants/{id}/backupResources” information may take considerable amount of time.

https://www.veeam.com/kb3003

Veeam Cloud Connect Replication Patch:

Probably one of the more important patches we have released of late has to do with a bug found in the stored procedure that generates automated monthly license usage reports for Cloud Connect Replication VMs. This displays an unexpected number of replicated VMs and licensed instances which has been throwing off some VCSP license usage reporting. If VCSPs where using the PowerShell command Get-VBRCloudTenant -Name “TenantName”, the correct information is returned.

To fix this right now, VCSPs offering Cloud Connect Replication servers can visit this VeeamKB, download an SQL script and apply it to the MSSQL server as instructed. There will also be an automated patch released and the fix baked into future Updates for Backup & Replication.

https://www.veeam.com/kb3004

Quick Round Up:

Along with a number of platform supportability announcements at VMworld 2019, it’s probably important to reiterate that we now have a patch available that allows us to support restores into NSX-T for VMware Cloud on AWS SDDCs environments. This also means that NSX-T is supported on all vSphere environments. The patch will be baked into the next major release of Backup & Replication.

Finally, the Dell EMC SC storage plug-in is now available which I know will be popular among our VCSP community who leverage SCs in their Service Provider platforms. Being able to offload the data transfer of backup and replication jobs to the storage layer introduces a performance advantage. In this way, backups from storage array snapshots provide a fast and efficient way to allow the Veeam backup proxy to move data to a Veeam backup repository.

Orchestration of NSX by Terraform for Cloud Connect Replication with vCloud Director

Posted on by

That is probably the longest title i’ve ever had on this blog, however I wanted to highlight everything that is contained in this solution. Everything above works together to get the job done. The job in this case, is to configure an NSX Edge automatically using the vCloud Director Terraform provider to allow network connectivity for VMs that have been replicated into a vCloud Director tenant organization with Cloud Connect Replication.

With the release of Update 4 for Veeam Backup & Replication we enhanced Cloud Connect Replication to finally replicate into a Service Providers vCloud Director platform. In doing this we enabled tenants to take advantage of the advanced networking features of the NSX Edge Services Gateway. The only caveat to this was that unlike the existing Hardware Plan mechanism, where tenants where able to configure basic networking on the Network Extension Appliance (NEA), the configuration of the NSX Edge had to be done directly through the vCloud Director Tenant UI.

The Scenario:

When VMs are replicated into a vCD organisation with Cloud Connect Replication the expectation in a full failover is that if a disaster happened on-premises, workloads would be powered on in the service provider cloud and work exactly as if they where still on-premises. Access to services needs to be configured through the edge gateway. The edge gateway is then connected to the replica VMs via the vOrg Network in vCD.

In this example, we have a LAMP based web server that is publishing a WordPress site over HTTP and HTTPs.

The VM is being replicated to a Veeam Cloud Service Provider vCloud Director backed Cloud Connect Replication service.

During a disaster event at the on-premises end, we want to enact a failover of the replica living at in the vCloud Director Virtual Datacenter.

The VM replica will be fired up and the NSX Edge (the Network Extension Appliance pictured is used for partial failovers) associated to the vDC will allow the HTTP and HTTPS to be accessed from the outside world. The internal IP and Subnet of the VM is as it was on-premises. Cloud Connect Replication handles the mapping of the networks as part of the replication job.

Even during the early development days of this feature I was thinking about how this process could be automated somehow. With our previous Cloud Connect Replication networking, we would use the NEA as the edge device and allow basic configuration through the Failover Plan from the Backup & Replication console. That functionality still exists in Update 4, but only for non vCD backed replication.

The obvious way would be to tap into the vCloud Director APIs and configure the Edge directly. Taking that further, we could wrap that up in PowerShell and invoke the APIs from PowerShell, which would allow a simpler way to pass through variables and deal with payloads. However with the power that exists with the Terraform vCloud Director provider, it became a no brainer to leverage this to get the job done.

Configuring NSX Edge with Terraform:

In my previous post around Infrastructure as Code vs APIs I went through a specific example where I configured an NSX Edge using Terraform. I’m not going to go over that again, but what I have done is published that Terraform plan with all the code to GitHub.

The GitHub Project can be found here.

The end result after running the Terraform Plan is:

  • Allowed HTTP, HTTPS, SSH and ICMP access to a VM in a vDC
    • Defined as a variable as the External IP
    • Defined as a variable as the Internal IP
    • Defined as a variable as the vOrg Subnet
  • Configure DNAT rules to allow HTTP, HTTPS and SSH
  • Configure SNAT rule to allow outbound from the vOrg subnet

The variables that align with the VM and vORG network are defined in the terraform.tfvars file and need to be modified to match the on-premises network configuration. The variables are defined in the variables.tf file.

To add additional VMs and/or vOrg networks you will need to define additional variables in both files and add additional entires under the firewall_rules.tf and nat_fules.tf. I will look at ways to make this more elegant using Terraform arrays/lists and programatic constructs in future.

Creating PowerShell for Execution:

The Terraform plan can obviously be run standalone and the NSX Edge configuration can be actioned at any time, but the idea here is to take advantage of the script functionality that exists with Veeam backup and replication jobs and have the Terraform plan run upon completion of the Cloud Connect Replication job every time it is run.

To achieve this we need to create a PowerShell script:

GitHub – configure_vCD_VCCR_NSX_Edge.ps1

The PowerShell script initializes Terraform and downloads the Provider, ensures there is an upgrade in the future and then executes the Terraform plan. Remembering that that variables will change within the Terraform Plan its self, meaning these scripts remain unchanged.

Adding Post Script to Cloud Connect Replication Job:

The final step is to configure the PowerShell script to execute once the Cloud Connect Replication job has been run. This is done via a post script settings that can be found in Job Settings -> Advanced -> Scripts. Drop down to selected ps1 files and choose the location of the script.

That’s all that is required to have the PowerShell script executed once the replication job completes.

End Result:

Once the replication component of the job is complete, the post job script will be executed by the job.

This triggers the PowerShell, which runs the Terraform plan. It will check the existing state of the NSX Edge configuration and work out what configuration needs to be added. From the vCD Tenant UI, you should see the recent tasks list modifications to the NSX Edge Gateway by the user configured to access the vCD APIs via the Provider.

Taking a look at the NSX Edge Firewall and NAT configuration you should see that it has been configured as specified in the Terraform plan.

Which will match the current state of the Terraform plan

Conclusion:

At the end of the day, what we have done is achieved the orchestration of Veeam Cloud Connect Replication together with vCloud Director and NSX… facilitated by Terraform. This is something that Service Providers offering Cloud Connect Replication can provide to their clients as a way for them to define, control and manage the configuration of the NSX edge networking for their replicated infrastructure so that there is access to key services during a DR event.

While there might seem like a lot happening, this is a great example of leveraging Infrastructure as Code to automated as otherwise manual task. Once the Terraform is understood and the variables applied, the configuration of the NSX Edge will be consistent and in a desired state with the config checked and applied on every run of the replication job. The configuration will not fall out of line with what is required during a full failover and will ensure that services are available if a disaster occurs.

References:

https://github.com/anthonyspiteri/automation/tree/master/vccr_vcd_configure_nsx_edge

Update 4 for Service Providers – Targeting vCloud Director with Cloud Connect Replication

Posted on by

When Veeam Backup & Replication 9.5 Update 4 went Generally Available in late January I posted a What’s in it for Service Providers blog. In that post I briefly outlined all the new features and enhancements in Update 4 as it related to our Veeam Cloud and Service Providers. As mentioned each new major feature deserves it’s own seperate post. I’ve covered off the majority of the new feature so far, and for the final post in the series I am looking at something that is close to my heart…vCloud Director Support for Veeam Cloud Connect Replication.

As a reminder here are the top new features and enhancements in Update 4 for VCSPs.

Leveraging the Best of vCloud Director for Stronger DRaaS:

VMware vCloud Director is the de facto standard for service providers who offer Infrastructure as a Service based on VMware and Veeam has had a long history supporting vCloud Director, with the industry’s first support for vCloud Director aware backups released in Veeam Backup & Replication v7 following on with the first stand alone Self Service Backup Portal in v9.5.

With the release of Update 4, we have added support for Veeam Cloud Connect to replicate directly into vCloud Director virtual datacenters, allowing both our Cloud and Service Provider Partners (VCSP) and customers to take advantage of the enhancements VMware has built into the platform. While this has been a long time coming, this support represents a significant enhancement to the way in which our VCSPs offer DRaaS.

With tenants consuming vCloud Director resources, it allows them to take advantage of more powerful features when dealing with full disaster, or the failure of individual workloads. Full and partial failovers will be more transparent with the use of the vCloud Director HTML5 Tenant UI and the vCloud Director HTML5 UI will also allow tenants to see what is happening to workloads as they boot and interact with the guest OS directly. This takes the pressure of the VCSPs helpdesk and gives tenants more control of their replicas once failed over.

Enhanced Edge Networking with NSX:

From a networking point of view, being able to access the NSX Edge Gateway for replicated workloads means that tenants can leverage the advanced networking features available on the NSX Edge Gateway. The Network Extension Appliance did a great job in offering basic network functionality however the NSX Edge offers:

  • Advanced Firewalling and NAT
  • Advanced Dynamic Routing (BGP, OSPF and more)
  • Advanced Load Balancing
  • IPsec and L2VPN
  • SSL VPN
  • SSL Certificate Services

Once a failover has been triggered from the Veeam Backup & Replication Console or via the VCSPs own Portals, the ability to manage and configure everything through the vCloud Director HTML5 UI utilizing NSX via vCloud Director enhances Cloud Connect Replication for both service providers and tenants.

Network Automation During Partial Failovers with the NEA:

There are a number of options that can be used to extend the tenant network to the service provider cloud network when actioning a partial failover. Tenants and service providers can configure:

  • Custom IPsec VPN
  • IPsec or L2VPN via the NSX Edge Gateway
  • NEA to NEA L2 VPN

The Network Extension Appliance is still available for deployment in the same way as before Update 4 and can be used directly from within a vCloud Director virtual datacenter. The NEA’s automate the extension of a tenant network so that the failed over workload can be accessible from the tenant network, even though it resides in the service provider’s environment. The NEA to NEA option is the simplest and most effective way to extend the tenants network to the cloud network.

NOTE: I will be dedicating a seperate blog post to this feature as I believe this is one of the leading innovative features we have as part of Cloud Connect Replication.

vCloud Director 9.7 Compatibility:

Just a quick note to finish that at the time of writing this post, Veeam Backup & Replication 9.5 Update4a does not officially support vCloud Director 9.7. We currently support up to vCloud Director 9.5 but will be looking to add supportability for 9.7 within the next 90 days.

Wrap Up:

DRaaS is something that is only just becoming recognized as something that organizations require as part of their overall data protection strategy. Veeam has had a strong offering delivered through our VCSPs for a while now, with a strong focus on network automation which is typically the hardest part of any DRaaS offering. With Cloud Connect Replication now targeting vCloud Director we now have a very compelling DRaaS product that is simple, flexible and reliable…yet still delivers enterprise grade functionality.

Update 4 for Service Providers – Tenant Connectivity with Cloud Connect Gateway Pools

Posted on by

When Veeam Backup & Replication 9.5 Update 4 went Generally Available a couple of weeks ago I posted a What’s in it for Service Providers blog. In that post I briefly outlined all the new features and enhancements in Update 4 as it related to our Veeam Cloud and Service Providers. As mentioned each new major feature deserves it’s own seperate post. I’ve covered off Tape as a Service and RBAC Self Service, and today i’m focusing on a much requested feature…Cloud Connect Gateway Pools

As a reminder here are the top new features and enhancements in Update 4 for VCSPs.

Gateway Pools for Cloud Connect

Cloud Connect has become the central mechanism for connectivity and communication between multiple Veeam services. When first launched with Cloud Connect Backup in v8 of Backup & Replication, the Cloud Connect Gateways where used for all secure communications between tenant backup server instances and the Veeam Cloud and Service Provider (VCSP) Cloud Connect backup infrastructure. This expanded to support Cloud Connect Replication in v9 and from there we have added multiple products that rely on communications brokered by Cloud Connect Gateways.

As of today Cloud Connect Gateways facilitate:

  • Cloud Connect Backup
  • Cloud Connect Replication
  • Full and Partial Failovers for Cloud Connect Replication
  • Remote Console Access
  • Veeam Availability Console Tenant and Agent Management
  • Veeam Backup for Microsoft Office 365 Self Service

With regards to acting as the broker for Cloud Connect Backup or Replication, prior to Update 4 the only way in which a VCSP could design and deploy the Gateways was in an all or nothing approach when it came to configuring the IP address and DNS for the service endpoint. When considering VCSPs that also provide connectivity such as MPLS for their customers it meant that to leverage direct connections that might be private the options where to either use the public address or setup a whole new Cloud Connect environment for the customer.

Now with Update 4 and Gateway Pools a VCSP can configure one or many Gateway Pools and allocate one or more Cloud Connect Gateways to those pools. From there, tenants can be assigned to Gateway Pools.

Cloud Gateways in a Gateway Pool operate no differently to regular Cloud Gateways. As with previous Cloud Gateways, If the primary gateway is unavailable, the logic built into Veeam Backup & Replication will failover to another Cloud Gateway in the same pool.

If tenants are not assigned a Cloud Gateway Pool they can use only gateways that are not a part of any cloud gateway pool. That situation is warned in the UI when configuring the gateways.

Wrap Up:

The introduction of Cloud Connect Gateway Pools un Update 4 was undertaken due to direct feedback from our VCSPs who wanted more flexibility in the way in which the Cloud Gateways where deployed and configured for customers. Not only can they be used to seperate tenants connecting from public and private networks, but they can also be used for Quality of Service by assigning a Gateway Pool to specific tenants. They can also be used to control access into a VCSPs Cloud Connect infrastructure if located in different geographic locations.

For a great overview and design considerations of Cloud Connect Gateway Pools and Gateways themselves, check out Luca’s Cloud Connect Book here.

References:

https://helpcenter.veeam.com/docs/backup/cloud/cloud_gateway_pool.html?ver=95u4

VeeamOn 2018: Recognizing Innovation and what it means to be Innovative

Posted on by

True innovation is solving a real problem…and though for the most, it’s startups and tech giants that are seen to be the innovators, their customers and partners also have the ability to innovate. Innovation drives competitive advantages and allows companies to differentiate themselves compared to others. In my previous roles I was lucky to be involved with teams of talented people that did great things with great technologies. Like others around the world we where innovating with leading vendor technologies to create new service offerings that add value and compliment the underlying technology.

Innovation requires these teams of people to be experimental at heart and try to build or enhance upon already existing technologies. The Service Provider industry has always found a way to innovate ontop of vendor platforms and successful vendors are those that offer the right tools and guidance for providers to creative innovative solutions ontop of their platforms. The are problem solvers!

Orchestrations, automation, provisioning and billing are driving factors in how service providers can differentiate themselves and gain that competitive advantage in the marketplace. Without innovating ontop of these platforms, service offerings become generic, don’t stand out and are generally operationally expensive to manage and maintain.

Introducing the Veeam Innovation Awards for 2018:

When visiting and talking to different partners across the world it’s amazing to see some of the innovation that’s been built ontop of Veeam technologies and we at Veeam want to reward our customers and partners who have done great things with our technologies.

At VeeamON 2018, we’ll be celebrating some of these innovative solutions, so please let us know how you’ve built upon the Veeam Availability Platform. Nominations can be made from March 29 to April 30, with the winners being recognized during the VeeamON main stage keynote. Self nominations or those from partners, providers, or Veeam field-team members are encouraged — click here to nominate for a Veeam Innovation Award.

I can think of a number of VCSPs that have done great things with building upon Cloud Connect, Backup & Replication IaaS backups and working with Veeam’s API’s and PowerShell to solve customer problems and offer value added services. These guys have brought something new to the industry and we want to reward that.

Having previously come from a successfully innovate company within their own space, being innovative is now something I try to preach to all customers and partners I visit. It is an absolute requirement if you want to win business and stand out in the backup and availability industry…innovation is key and we want to hear about it from you!

References:

https://www.veeam.com/executive-blog/nominations-veeamon-2018-innovation-awards.html

Veeam Availability Console now available from Azure Marketplace

Posted on by

Last week the Veeam Availability Console Azure Marketplace appliance went live. This allows Veeam Cloud and Service Providers to easily deploy VAC into any Azure region. In it’s previous incarnation the Managed Backup Portal was only available as an Azure marketplace appliance and not available to install by a VCSP. Now that VAC 2.0 is out, VCSPs who don’t have the ability to host Cloud Connect or VAC on their infrastructure can deploy it in Azure and have the service up and running within fifteen minutes.

There are some limitations that come along with deploying VAC into Azure and it won’t be for everyone. The biggest caveat is that you can only have one Cloud Connect Server per VAC instance and as part of the deployment, Cloud Connect services is installed on the same Virtual Machine. You can’t offer Replication services from the Azure instance, and if offering Cloud Connect backup you need to understand it’s own scalability and performance bottlenecks. That said, as a remote management, monitoring, reporting, billing and self service platform there is a lot to like about having VAC in Azure.

Marketplace Deployment Steps:

You can start the deployment by searching for Veeam Availability Console in the Azure Marketplace or you can go direct to the product page here.

Click on Create to start the configuration steps.

The Basics includes VM name, hard disks type, username and password as well as selecting the subscription, the ability to use a new or existing resource group and finally the Azure location you want to deploy into.

In Step 2 you need to choose the Size of the Azure instance. The template provides the recommended configurations. The sizes are relative to the amount of agents and/or Backup & Replication instances you are going to be managing from this instance. You can find sizing guides here for larger environments.

I ended up going with an A2 standard for my instance which removes the load balancing functionality from the configuration and offers a little less IOPS.
Step 3 contains some optional extra’s to ensure a higher level of availability for the VM instance and lets you configure the networking.
Once that’s done you can review your configuration settings and start the deployment. It took just over 8 minutes for the deployment to succeed.

If you click on the Virtual Machine object in the Azure Portal you will see an overview of the VM and it’s configuration.

Addition Azure Configuration:

If you notice in the image above, a DNS name is listed in the overview. This was something that I had to set manually after the deployment. You set this by going into the Networking of the resource pool and click on IP Configuration. Here, you can enter in a DNS name relative to the Azure zone you are in. You can then use this to connect to the VAC Console, Cloud Connect Service and to RDP to the VM and helps in the event of having a dynamic, rather than a static Azure IP.

Speaking of networking and ports, below is a list of the default port rules created during the deployment. Note that WinRM is open as well.

Finalizing Deployment:

After deploying the Azure Marketplace appliance you can RDP into the VM and complete the setup that includes configuring Cloud Connect and VAC it’s self. A few things have been done for us as part of the deployment, however the first thing you need to do is get a license. This is a BYO license situation, so once you have deployed the Marketplace appliance you will need to source a VAC license from the Veeam Licensing Portal and apply.

Head to the VAC Web Portal and Install the License.

Once done the last step is to configure Cloud Connect from the Backup & Replication Console. Again, you will need a valid Cloud Connect license as you are greeted with the Free Edition when you connect to the console for the first time. As per normal with Cloud Connect, you need to configure the SSL Certificate first and then configure a new Cloud Gateway. Configure the Networking as shown below using the DNS name that was created in the steps above.

Once this is completed you can go into the VAC Console and work through the normal Configuration steps. The only thing you don’t need to do is add the Cloud Connect Server to the VAC instance as this has already been done during the initial deployment process.

It’s worth noting that the versions of Backup & Replication (9.5.0.1536) and Availability Console (2.0.1.1343) are up to date and include the latest Hot-Fixes for VAC. The intent is to have the templates as up to date as possible, however once deployed you can upgrade as per usual.

Conclusion:

So there you have it…within fifteen minutes you can have a fully working Veeam Availability Console instance running in Azure and ready to be used to offer all the goodness that VAC offers our Cloud and Service Provider partners. For an overview as to what VAC offers, click here and have a read of my GA post on What’s in It for Service Providers.

Links:

https://azuremarketplace.microsoft.com/en-us/marketplace/apps/veeam.veeam-availability-console?tab=Overview

 

Configuring Service Provider Self Service Recovery with Veeam Backup for Microsoft Office 365

Posted on by

For a while now I’ve talked about the increasing functionality of the the Cloud Connect Gateway and that it is central to a lot of features and services that exist within Veeam Backup & Replication. With the release of 9.5 Update 3 we added a feature that allows multi-tenant self service recoverability of a tenants Office365 mailbox backup hosted by Veeam Cloud and Service Providers utilising Veeam Backup for Microsoft Office 365 1.5 that was released late last year.

Overview:

Tenant admins communicate with the Service Provider via the Cloud Gateway component which handles flow of data. The Service Provider grants the ability to their tenants so that each tenant can perform self restore operations using Veeam Explorer for Microsoft Exchange. By default, tenants are not able to restore anything from the backup without a Service Provider assistance.

The steps above show the self restore scenarios performed by the Tenant:

  • Tenants use Veeam Explorer for Microsoft Exchange to send restore requests via Veeam Cloud Gateway directly to the Service Provider.
  • On the Service Provider side, Veeam Backup for Microsoft Office 365 management server detects a proxy server responsible for processing tenant data.
  • Veeam Backup for Microsoft Office 365 management server locates an associated repository that contains a backup file that belongs to the Tenant.
  • Corresponding backup data is then transferred back to the tenant via Veeam Cloud Gateway.

IMPORTANT!

When planning solution components deployment, remember that Veeam Backup for Microsoft Office 365 v1.5 and Veeam Backup & Replication 9.5 Update 3 must be installed on the same server.
Example:

These days I don’t have access to a local Exchange Server or to a corporate Exchange Online instance but I did migrate my personal domain over to Office365 just before Christmas. That account has only one mailbox, but that’s enough to demonstrate the Office365 Service Provider backup and tenant self service recovery use case.

Service Provider Side:

For Service Providers to backup tenants on-premises or Office 365 Exchange mailboxes they need to first configure a new organization in Veeam Backup for Office 365. I’m not going to go through the steps for that as it’s been covered in other posts and is very simple to configure, however to prepare for the self service capability the service provider needs to ensure that the Cloud Connect Gateways are setup and configured and accessible externally.

In Backup for Office 365 you have to enable and configure the RestAPI and Authentication Settings under their respective tabs in the Options menu. This includes selecting an SSL certificate for both services…I’m just using a self signed certificate but obviously service providers will want a correctly signed public certificate to productise this feature.

With the organization configured I created a new job and backed up the Exchange Organization. Again, for this example I just have the one mailbox but the theory is the same weather it’s one, five, fifty or five thousand mailboxes.

From here, without any self service configured the Service Provider can access the mailboxe(s) to perform whole or granular item level recovery using the Veeam Explorer for Exchange. As shown below I can access any mailbox from the service provider’s end and perform recovery to a number of different locations. 

For each tenant (not per Exchange User) there needs to be a Cloud Connect tenant account created on the Backup & Replication server. This will be used at the tenant end by the admin to configure a Service Provider in the Backup & Replication console which will then be detected and used by the Veeam Explorer for Exchange to use to connect into the service provider and authenticate with an applicable Exchange account.

Tenant End:

For the tenant admin to use Veeam Explorer for Exchange to perform mailbox recovery you first have to configure a Service Provider using Cloud Connect tenant credentials as provided by the Service Provider. It’s worth mentioning here that you can have no license installed in Backup & Replication and are still able to add a Service Provider to the Backup Infrastructure menu. Once connected, firing up the Explorer for Exchange you will use the Service Provider option in the Add Store dropdown.

In the drop down list, select the Service Provider account configured in the Backup Infrastructure menu. If multiple exist you will see each one in the drop down. You also configure the username and password that connects to the Exchange Organization. This can be an admin account that is allowed impersonation, or you can enter in an individual account.

Once connected (which can take some time with the GUI of the Explorer for Exchange) any mailbox that the account has authorization over will be seen and mailbox recovery can begin.

An interesting thing to do is to check what is happening from a network connectivity point of view during this process. While performing a restore you can see open connections from the tenant side to Cloud Connect gateway on port 6180 and also you can see a connection to Office365 on port 443 completing the loop.

Back at the Service Provider end in the Backup for Office365 console you can see active Explorer for Exchange sessions as running jobs. Below you can see the local one, plus a remote session.

Automation:

For Service Providers with the capability to automate the setup and provisioning of these services through PowerShell or the RestAPIs here is a great example of what can be achieved with Backup for Office365 and the creation of a self service portal web interface. You can use the built in Swagger UI to evaluate the capabilities of RestAPIs.

The Swagger UI can be accessed via the following URL:

https://<Backup-Office365>:<Port>/swagger/ui/index

From there you can authenticate and work through the live examples.

Conclusion:

The market for Office365 backups is significant and we have built in some pretty cool technology into Backup & Replication that works with Backup for Office365 that allows easy, self service capabilities that can be productized by Service Providers out of the box. Not only can Service Providers offer services to backup client Exchange Organisations but they can also extend that to offer self service which increases overall operational efficiencies at the provider end while also offering enhanced services to clients.

References:

https://helpcenter.veeam.com/docs/vbo365/guide/vbo_mail_baas.html?ver=15

https://helpcenter.veeam.com/docs/vbo365/rest/swaggerui.html?ver=15

Creating a Custom Cloud Connect Maintenance Mode Message

Posted on by

Last week I wrote an article on Maintenance Modes in Cloud Connect and also Veeam Availability Console. For Cloud Connect there is a default error message that get’s shown in the Job Status if any jobs are started if the Cloud Connect Maintenance Mode is turned on.

We have the ability to customize that message via a registry key addition as documented in the online Veeam Help Centre.

To create a custom Maintenance mode notification, on the SP Veeam backup server, create the new registry value HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\CloudMaintenanceModeMessage = <message> (String), where <message> is a Maintenance mode notification that you want to display on the tenant side.

Adding the key via Registry Editor is simple enough and this is what you are left with from within the Registry Editor.

And the error message at the tenant end now reflects the custom message.

To make this easier for Service Providers, i’ve written a quick PowerShell script that does a couple of things. The first thing is report on the current registry value for the Maintenance Mode and then give you the option to delete the key and return the message to it’s default state. The second thing it does is prompt you enter in the desired custom message and set that in the registry.

References:

https://helpcenter.veeam.com/docs/backup/cloud/cc_maintenance_message.html?ver=95

Cloud Connect and VAC Portal Maintenance Modes

Posted on by

Lately i’ve been digging deeper into the Veeam Availability Console and have been wrapping my head around it’s extended feature set. With that I thought it would be good to start a series of short blog posts pointing out examples of how certain parts are configured and what is happening under the covers. To kick things off I am going to talk about Maintenance Modes in VAC and also how it translates back to Cloud Connect Maintenance mode and also start off by covering that new Update 3 feature.

Maintenance Mode for Cloud Connect in Backup & Replication 9.5 Update 3

In Backup & Replication 9.5 Update 3 we introduced a Maintenance Mode feature for Cloud Connect. In a nutshell this makes the Service Provider cloud resources unavailable for tenants to perform backup or backup copy job operations. This is true for jobs running on Backup & Replication 9.5 Update 3, Agent for Windows 2.1 and Agent for Linux 2.0.

To enable Maintenance Mode from the VBR console Right Click on the Cloud Connect top level tree item and click on Maintenance Mode

Read the message and click Yes

Once completed you should see the following status in the Cloud Connect menu tree

You can also set and check this state in PowerShell

Once triggered, any running jobs are gracefully stopped. Within that the current task is allowed to complete but all subsequent jobs will fail. In the case of an agent the whole job is allowed to complete. Any new backup or backup copy job that tries to start after Maintenance Mode has been initialed will fail with an error which is shown below.

Tying this into the Veeam Availability Console you can also trigger Maintenance Mode from the VAC UI. To enable maintenance mode for Veeam Cloud Connect, log in to Veeam Availability Console as a Portal Administrator and at the top right corner click Configuration and under Portal Configuration click Cloud Connect Server and click Enable Maintenance Mode.

Click Yes to confirm the operation.

The message isn’t 100% correct based on what I talked about earlier. The current job task will be completed and not dropped as suggested here.

You can disable Maintenance Mode by clicking on the menu option if it’s enabled.

Maintenance Mode for Veeam Availability Portal UI

For those times when you may need to perform configuration changes or OS updates to the system hosting the VAC Portal you have the ability to put the portal its self into maintenance mode. When enabled, all users will not be able to login to the portal remotely and you will see a message on the welcome page as shown below.

To toggle this setting go to the top right of the VAC console and click Configuration and then under Server Settings click on Settings and go to the Maintenance Mode Tab. Set the toggle to on or off to enable or disable and click save.

Once in Maintenance Mode you can only log back into the portal from the local console of the server hosting the VAC UI role. Note that while under Maintenance Mode you can only modify the SQL Server Configuration or toggle Maintenance Mode off.

Conclusion:

I’ve gone through the Maintenance Mode options for both Veeam Availability Console and Cloud Connect and how each one is enabled and what their purpose is. For the moment, in Backup & Replication 9.5 Update 3 the Maintenance Mode is limited to Backup and Backup copy job operations. There are a other operations that are not currently impacted by this mode such as vCloud Director backups or Cloud Connect Replication operations however this will be looked at in upcoming releases.

To read more about Maintenance Mode head to the Veeam Help Documentation page here.

References:

https://helpcenter.veeam.com/docs/backup/cloud/cc_maintenance_mode.html?ver=95

https://helpcenter.veeam.com/docs/vac/enterprise_admin/enable_disable_vac_maintenance_mode.html?ver=20

A Deeper Look at Insider Protection in 9.5 Update 3

Posted on by

With the release of Backup & Replication 9.5 Update 3 we introduced the concept of a Recycle Bin for customers sending offsite cloud backups to VCSPs using Veeam Cloud Connect. This deleted backup protection…or Insider Protection allows the VCSP to enable the deleted backups protection option for specific tenants and looks to add another level of data security for cloud based backups in the case of a malicious user gaining access to the Backup & Replication Console or in the case of accidental deletion by an administrator.

As shown above, this is set by checking a box (Also via PowerShell) in the properties of the tenant account. Once checked the SP will choose the retention period by setting the Keep deleted Backup files for <N> days option. With this option enabled, when a backup or a specific restore point in the backup chain is deleted or aged out from the cloud repository. The actual backup files are not deleted immediately, instead, they are moved to a _RecycleBin folder on the repositories.

Once moved, backup files in the recycle bin do not consume tenant quota however they obviously consume general storage. With that in mind it should be considered by the SP to charge for that used storage. I will release a post shortly detailing some tips on how best to size and charge for the recycle bin storage per client.

At the tenant end those backup files that are moved into the recycle bin are not registered and will not show up in the job information window. They can’t access or do anything with the files in the recycle bin. For the moment if a tenant wants to restore data they must contact the SP to obtain the necessary backup files. Once the retention period has expired all files that fall out of that period are deleted.

Basic Mechanics:

When the option is checked for a tenant a new folder is created under the _RecycleBin\<tenant> folder of the repository. In the case of a Scale Out Backup Repository there is a recycle bin folder created per extent which ensured that any split tenant VM files are processed locally and not between extents.

Once files in the repository start to age out the tenant folder will start to populate with backup files. If there is an event that triggers a change of retention or a VM removed from a job or the deletion of a whole job, any remaining VBK or VIB files in the tenant repository are moved into the recycle bin.

The files remain in the _RecycleBin folder until the retention period has passed or if the service provider moves them out of the folder for recovery purposes.

Working Example:

I have a Cloud Connect Backup account that I am using to back up five VMs that reside on premises, using a standard Backup Job with Forward Incrementals and a Synthetic Full done once a week. I have configured this job to keep two restore points.

I then have configured a secondary destination for the job via a Backup Copy Job to the Cloud Repository and I have set a GFS to happen weekly so I have a full archive offsite. If I hadn’t enabled GFS retention (for those running Update 3) a warning would appear as shown below.

Tip: If the tenant plans to create off-site copies of backed-up data with a backup copy job, it should enable GFS retention settings in the job properties. This way, Veeam Backup & Replication will be able to protect backups created by the job against an attack when a hacker reduces the job’s retention policy and creates a few incremental backups to remove backed-up data from the backup chain.

The Cloud Connect Tenant account has a deleted backup protection setting of 2 days configured as shown in the first image of this post.

Below is the local jobs folder structure:

Looking at the Cloud Connect repository (split over two SOBR extents) you can see that the main repository holds the VM backup files as per the job configuration. Notice the GFS _W files there as well.

Taking a look at the _RecycleBin folder for the tenant after a few days the aged out incremental will start to appear in the folder. Notice that there are no full backup files in the recycle bin at this stage.

Tip: The retention period will look at all backup jobs completed in a 24 hours period and have any expiring or deleted backup files moved into the recycle bin directory. This means that if you are copying up VMs that have a local backup interval of every 4 hours you will have six lots of backup files ageing out daily.

In this example I’m simulating an malicious attack or accidental deletion the VM (TPM03-RMQ-01/VM-120) from the backup. For the sake of this example we are deleting the VM from the Backup & Replication Console under Backups and Cloud. If the Included Archived copies option was chosen then the GFS weekly full backup file is also moved into the recycle bin.

Once the deletion process has been completed the _RecycleBin folder for the tenant will now be populated with the deleted full, plus three incremental files. If the Included Archived copies option was chosen then the GFS weekly full backup file is also moved into the recycle bin.

These will stay in the recycle bin until the retention period is met. From here these files can be transported back to the tenant to be recovered (see here for full process) from within the on-premises Backup & Replication console.

Conclusion:

As shown above, deleted backup protection or Insider Protection is an excellent enhancement to Cloud Connect Backup. It goes some way to having an air gapped backup in the cloud and protects against malicious attacks and rogue or clumsy administrators. There is a lot happening behind the scenes to make it work, however the concept is simple and this features extends the 3-2-1 rule by protecting that offsite copy as part of the Cloud Connect solution. VCSP’s should be looking to offer this as a value add to their clients and Veeam customers should be looking to take advantage of Cloud Connect Backup and Replication for their offsite backup and replication needs.

References:

https://helpcenter.veeam.com/docs/backup/cloud/cloud_connect_bin.html?ver=95

https://helpcenter.veeam.com/docs/backup/cloud/cloud_connect_bin_restore.html?ver=95

« Older Entries