Author Archives: Anthony Spiteri

Backing up 6.7 Update 1 VCSA to Cloud Connect Fails

A few weeks ago I upgraded my NestedESXi homelab to vSphere 6.7 Update 1. Even though Veeam does not have offical supportability for this release until our Backup & Replication 9.5 Update 4 release there is a workaround that deals with the change of vSphere API version that out of the box, causes backup to fail. After the upgrade and the application of the workaround I started to get backup errors while trying to process the main lab VCSA VM which was now running vCenter 6.7 Update 1. All other VMs where being backed up without issue.

Processing LAB-VC-67 Error: Requested value ‘vmwarePhoton64Guest’ was not found.

The error was interesting and only impacted the VCSA VM that I had upgraded to 6.7 Update 1. I do have another VCSA VM in my lab which is on the GA of 6.7 which was backing up successfully. What was interesting is that it appears like the GuestOS type of the VM had changed or was being recognised as PhotonOS from within the upgraded vCenter on which it lived it’s self.

Looking at the VM Summary, it was being listed as VMware Photon OS (64-bit)

My first instinct was to change this back to what I saw the other VCSA to be, which was Other 3.x Linux (64-bit)

However, due to the chicken or the egg nature of having the management VCSA on the same vCenter when I logged into the ESXi host (also upgraded to 6.7 Update 1) I saw that it didn’t match what was being shown in vCenter.

Thinking it was due to a mismatch, I changed the Guest OS type here to Photon OS However the same issue occurred. Next I tried to get a little creative and change the Guest OS Type to Other Linux (64-bit) but even though I changed it to that from ESXi…from vCenter (its self) it was still reporting Photon OS and failed.

The Issue:

I submitted a support ticket and from the logs the Support team where able to ascertain that the issue actually lied at the Cloud Connect Providers end. I was sending these backups directly to the Cloud Connect Provider, so my next step to confirm this was to try a local backup test job and sure enough the VM processed without issues.

I then attempted a Backup Copy job from that successful test job to the Cloud Connect Provider and that resulted in the same error.

From the job logs it became clear what the issue was:

[07.11.2018 03:00:12] <01> Info [CloudGateSvc 119.252.79.147:6180]Request: [Service.Connect] SessionType:4, SessionName:Lab Management, JobId:54788e4d-7ba1-488a-8f80-df6014c58462, InstallationId:30ee4690-01c9-4368-94a6-cc7c1bad69d5, JobSessionId:b1dba231-18c2-4a28-9f74-f4fa5a8c463b, IsBackupEncrypted:False, ProductId:b1e61d9b-8d78-4419-8f63-d21279f71a56, ProductVersion:9.5.0.1922,
[07.11.2018 03:00:13] <01> Info [CloudGateSvc xx.xx.xx.xx:6180]Response: CIResult:b4aa56f4-fd02-4446-b893-2c39a16e535e, ServerTime:6/11/2018 7:00:13 PM, Version:9.5.0.1536,

At my end, I am running Backup & Replication 9.5 Update 3a, while at the provider end, they are running Backup & Replication 9.5 Update 3. Update 3a introduced supportability to vSphere 6.7 and other platform updates…this included the list at Veeam’s end of support Guest OS Types. In a nutshell the Veeam Cloud Connect Backup server still needs to understand what type of VM/Guest its backing up in its Cloud Repository. For this to be resolved the provider would need to upgrade their Cloud Connect infrastructure to Update3a…meanwhile, I’m backing up the VM locally for the time being.

Timely Message for VCSPs running Cloud Connect:

As we approach the release of another Update for Backup & Replication it’s important for Veeam Cloud and Service Providers to understand that they need to keep in step with the latest releases. This is why we typically have an RTM build given to providers at least two weeks before GA.

With vSphere 6.7 Update 1 starting to be deployed to more organisations it’s important to be aware of any issues that could stop tenant backups from completing successfully. This has generally been a consideration for providers offering Cloud Connect over the years…especially with Cloud Connect Replication, where the target platform needs to be somewhat in check with the latest platforms that are available.

References:

https://www.veeam.com/kb2443

https://www.veeam.com/kb2784

Hybrid World… Why IBM buying RedHat makes sense!

As Red October came to a close…at a time when US Tech stocks were taking their biggest battering in a long time the news came out over the weekend that IBM had acquired RedHat for 34 billion dollars! This seems to have taken the tech world by surprise…the all-cash deal represents a massive 63% premium on the previous close of RedHat’s stock price…all in all it seems ludicrous.

Most people that I’ve talked to about it and from reading comments on social media and blog sites suggests that the deal is horrible for the industry…but I’ve felt this is more a reaction to IBM than anything. IBM has a reputation as swallowing up companies whole and spitting them out the other side of the merger process a shell of what they once were. There has also been a lot of empathy for the employees of RedHat, especially from ex-IBM employees who have experience inside the Big Blue machine.

I’m no expert on M&A and I don’t pretend to understand the mechanics behind the deal and what is involved…but when I look at what RedHat has in its stable, I can see why IBM have made such an aggressive play for them. On the surface it seems like IBM are in trouble with their stock price and market capitalization falling nearly 20% this year and more than 30% in the last five years…they had to make a big move!

IBM’s previous 2013 acquisition of SoftLayer (for a measly 2 billion USD) helped them remain competitive in the Infrastructure as a Service space and if you believe the stories, have done very well out of integrating the SoftLayer platform into what was BlueMix, and is now IBM Cloud. This 2013 Forbes article on the acquisition sheds some light as to why this RedHat acquisition makes sense and is true to form for IBM.

IBM sees the shift of big companies moving to the cloud as a 20-year trend…

That was five years ago…and since then a lot has happened in the Cloud world. Hybrid cloud is now the accepted route to market with a mix of on-premises, IaaS and PaaS hosted and hyper-scale public cloud services being the norm. There is no one cloud to rule them all! And even though AWS and Azure continue to dominate and be front of mind there is still a lot of choice out there when it comes to how companies want to consume their cloud services.

Looking at RedHat’s stable and taking away the obvious Linux distro’s that are both enterprise and open sources the real sweet spot of the deal lies in RedHat’s products that contribute to hybrid cloud.

I’ve heard a lot more noise of late about RedHat OpenStack becoming the platform of choice as companies look to transform away from more traditional VMware/Hyper-V based platforms. RedHat OpenShift is also being considered as an enterprise ready platform for containerization of workloads. Some sectors of the industry (Government and Universities) have already decided on their move to platforms that are backed by RedHat…the one thing I would comment here is that there was an upside to that that might now be clouded by IBM being in the mix.

Rounding out the stable, RedHat have a Cloud Suite which encompasses most of the products listed above. CloudForms for Infrastructure as Code, with Ansible for orchestration…together with RedHat Virtualization together with OpenStack and OpenShift..it’s a decent preposition!

Put all that together with the current services of IBM Cloud and you start to have a compelling portfolio covering almost all desired aspects of hybrid and multi cloud service offerings. If the acquisition of SoftLayer was the start of a 20 year trend then IBM are trying to keep themselves positioned ahead of the curve and very much in step with the next evolution of that trend. That isn’t to say that they are not playing catchup with the likes of VMware, Microsoft, Amazon, Google and alike, but I truly believe that if they don’t butcher this deal they will come out a lot stronger and more importantly offer valid completion in the market…that can only be a good thing!

As for what it means for RedHat itself, their employees and culture…that I don’t know.

References:

https://www.redhat.com/en/about/press-releases/ibm-acquire-red-hat-completely-changing-cloud-landscape-and-becoming-world%E2%80%99s-1-hybrid-cloud-provider

IBM sees the shift of big companies moving to the cloud as a 20-year trend

vSphere 6.7 Update 1 – Top New Features and Platform Supportability

Last week VMware released vSphere 6.7 Update 1. While the buzz around this release was less than the previous release it still contains a ton of enhancements for vCenter, ESXi and vSAN. Like 6.7 before it, this is a lot more than a point release and represents a significant upgrade from vSphere 6.7.

Looking through the release notes, there appears to be less for service providers in this release though I still feel like it’s important to highlight the base hypervisor (ESXi) as well as the management platform (vCenter). vSAN has had another significant update and that will warrant a post on it’s on. I’ll also talk about current interoperability with vCloud Director and NSX as well as current Veeam supportability for vSphere 6.7 Update 1 as well as touch on Veeam’s current supportability.

  • New (almost 100%) Fully functional HTML5 client
  • Upgrade path from vSphere 6.5 U2 to vSphere 6.7 Update 1
  • Enhanced support for NVIDIA Quadro vDWS VMs and support for Intel FPGA
  • New vCenter Convergence Tool
  • Updated vSAN
  • Enhanced vSphere Content Library
Fully Functional HTML5 Client

Most functions have now been ported across to the HTML5 vSphere Client. This results in administrators not having to switch back and forth between the FLEX Web Client and the HTML5 client. Update 1 features:

  • vCenter High Availability (VCHA)
  • Auto Deploy
  • Host Profiles
  • vSphere Update Manager
  • Network Topology Diagrams
  • Performance Charts
  • Improved Searching
  • Dark Theme

Emad Younis has a detailed post here that goes through the new features.

Upgrade Path from vSphere 6.5 Update 2 to vSphere 6.7 Update 1

One of the issues with vSphere 6.7 was the fact that the vSphere 6.5 Update 2 release would not be able to be upgraded to vSphere 6.7.  With the release of vSphere 6.7 Update 1. vSphere 6.5 Update 2 to vSphere 6.7 Update 1 is now a fully supported.

Enhanced Content Library

New improvements to the content library in vSphere 6.7 Update 1 enables the importing of OVA templates from a HTTPS endpoint and also local storage.  Importing now verifies the certificate of the OVA bundle and also now natively supports VM templates (VMTX) and associated operations such as deploying a VM directly from Content Library.

vCenter Specific Enhancements

With vCenter Server 6.7 Update 1, you can move a vCenter Server with an Embedded Platform Services Controller from one vSphere domain to another vSphere domain. Services such as tagging and licensing are retained and migrated to the new domain.

There is a new Burst Filter to manage event bursts and prevent the database of vCenter Server from flooding with identical events over a short period of time.

vCenter Server 6.7 Update 1 supports VMware vSphere vMotion between on-premises vCenter’s and VMware Cloud on AWS. You can use either the vSphere Client or vSphere Web Client, or the API. Both sides need to be at 6.7 Update 1.

you can import Open Virtual Appliance (OVA) files in a Content Library. The OVA files are unzipped during the import, providing manifest and certificate validations, and create an OVF library item that enables deployment of virtual machines from a Content Library.

With vCenter Server 6.7 Update 1, you can use the Appliance Management User Interface to configure and edit the firewall settings of the vCenter Server Appliance.

ESXi Specific Enhancements

There are a few vendor/hardware related features and enhancements in Update 1 for ESXi 6.7. The release notes cover them in detail here. But as mentioned above, probably the biggest addition here is the ability to upgrade from ESXi 6.5 Update 2 which I know a few service providers where stuck on. In terms of known issues the release notes also contain a good list. There are some here that impact Service Providers so it’s worth reading through them.

vCD and NSX Supportability:

Shifting from new features and enhancements to an important subject to talk about when talking service provider platform…VMware product compatibility. For those VCPP Service Providers running a Hybrid Cloud you should be running a combination of vCloud Director SP or/and NSX-v of which the 6.4.3 and 6.4.2 versions are supported at release. Most providers should be on these releases so that’s good news.

Looking at vCloud Director, it looks like 9.5 is the only supported version at the moment

Veeam Backup & Replication Supportability: 

Veeam commits to supporting major version releases within 90 days or sooner of GA. There has been many discussions going round whether an Update is a major release these days…and general consensus now is that VMware is releasing these updates with enough changes to potentially impact backup supportability.

So with that, those Service Provider that are also VCSPs using Veeam to backup their infrastructure should not upgrade to vSphere 6.7 until Backup & Replication Update 4 is released. For those that are bleeding edge and have updated your only is to go with the workaround that is detailed here. It works…but again, it’s a work around.

Wrapping Up:

Rounding off this post, in the Known Issues section there is a fair bit to be aware of for 6.7 Update 1. it’s worth reading through all the known issues just in case there are any specific issues that might impact you.

Happy upgrading!

References:

https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-vcenter-server-671-release-notes.html

https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-esxi-671-release-notes.html

Quick Fix – VCSA 6.7.0.10000 Can’t Update via URL from Management Interface

I had an issue with my VCSA today trying to upgrade to vCenter 6.7 Update 1 whereby the Management Interface Upgrade option was not detecting the update to upgrade the appliance to 6.7 Update 1. It was a similar issue to this VMwareKB, however the URL that is mentioned in that instance was already in the VCSA Settings.

My first instinct was to check the disk space and see if there where any pressures in that area. I did find that the /dev/sda3 partition was low on space, so I expanded the disk following advice given by Mark Ukotic. After a reboot and resize I had plenty of storage left, but still couldn’t trigger an update from the URL. At this point I did download the Update patch ISO from the VMware Patch center and loaded it up manually…however the issue of it not popping up automatically was annoying me.

As mentioned, the settings of the VCSA Update window has the following URL listed:

https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.10000.latest/

Having asked around a little the quick fix was provided by Matt Allford who provided me with the URL that was present in his VCSA after he upgraded successfully via the CLI.

https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.20000.latest/

I added that as a custom repository as shown below…

I was then able to rescan and choose from the list of updates for the VCSA.

And perform the upgrade from the Management Interface as first desired.

Interestingly enough, after the upgrade the default Update Repository was set to the one Matt provided for me.

This is the first time i’ve seen this behavior from the VCSA but I had reports of people being able to upgrade without issue. I’m wondering if it might be the particular build I was on, though that in it’s self was not picking up any patches to update to either. If anyone has any ideas, feel free to comment below.

Quick Fix: Terraform Plan Fails on Guest Customizations and VMware Tools

Last week I was looking to add the deployment of a local CentOS virtual machine to the Deploy Veeam SDDC Toolkit project so that it included the option to deploy and configure a local Linux Repository. This could then can be added to the Backup & Replication server. As part of the deployment I call the Terraform vSphere Provider to clone and configure the virtual machine from a pre loaded CentOS template.

As shown below, I am using the Terraform customization commands to configure VM name, domain details as well as network configuration.

In configuring the CentOS template i did my usual install of Open VM Tools. When the Terraform plan executes we applied the VM was cloned without issue, but it failed at the Guest Customizations part.

The error is pretty clear and to test the error and fix, I tried applying the plan without any VMware Tools installed. In fact without VMware Tools the VM will not finish the initial deployment after the clone and be deleted by Terraform. I next installed open-vm-tools but ended up with the same scenario of the plan failing and the VM not being deployed. For some reason it does not like this version of the package being deployed.

Next test was to deploy the open-vm-tools-deploypkg as described in this VMwareKB. Now the Terraform plan executed to the point of cloning the VM and setting up the desired VM hardware and virtual network port group settings but still failed on the custom IP and hostname components of the customisation. This time with a slightly different error.

The final requirement is to pre-install the perl package onto the template. This allows for the in guest customizations to take place together with VMware Tools. Once I added that to the template the Terraform Plan succeeded without issue.

References:

https://kb.vmware.com/s/article/2075048

 

 

Released: vCloud Director 9.5 – Full HTML5 Tenant UI, NSX-T Thoughts and More!

Last week VMware released vCloud Director 9.5 (build 10266189) which builds on the 9.1 release that came out earlier this year. This continues to deliver on VMware’s promise to release major vCD updates every six months or so. This update completes the HTML5 Tenant Portal port as well as continuing to enhance the usability of the HTML5 interface by extending the Provider UI to be more functional. Under the hood there are a number of networking enhancements as well as the initial introduction of a vCD Cell Appliance.

New Features and Enhancements:

  • Fully Functional HTML5 Tenant Portal
  • Cross-OrgVDC and Multi-Site Cross-VDC Networking
  • Initial Support for NSX-T
  • Enhanced Role Base Access Control (RBAC)
  • vCloud Director Appliance
  • IPv6 Support for Guest VMs
  • Updated Plugin for vRealize Orchestrator
  • API and SDK Enhancements
  • Container Service Extension (CSE) 1.2

In this post, I am going to focus more on the HTML5 Tenant and Provider Portal as well as touch on some of the important changes to supportability this release brings. As you can see from the list above, there are a number of major features to talk about, and i’ll try to put together a few more posts over the next few weeks digging into them specifically.

Tenant UI Reaches Feature Parity:

Starting from this release the reliance on the old Flex based portal is no more. All tenant tasks have been ported over to the HTML5 portal along with a lot of additional enhancements. If I think back a couple years ago when vCloud Director was at a cross roads in terms of how VMware continued to develop it, it’s amazing to see this new UI fully complete.

Everything that Tenant’s could see in the Flex UI is present in the HTML5 UI. Some of the additions include a recent tasks pane, support for independent disks is not only an API only feature now and can be accessed via the UI as well as Affinity Rules being configurable from the HTML portal.

Provider UI Improvements:

Heading over to /provider will get you into the HTML5 Provider UI. This now lists all vCD Organizations and you can create a new Org and then click through to the Tenant UI as Administrator to perform configuration tasks

You can also manage Catalogs and as with vCD 9.1 you can manage the Content Library through the provider UI. What else is new in 9.5 is the ability to allow the management of users, groups, roles, global roles.

Depreciated APIs and Functionality:

vCD 9.5 brings with it the end of support for Oracle Database which brings full circle the requirement for Oracle. Many of you who started on vCD when it was in Beta or v1 remember that it needed an Oracle database and didn’t support MSSQL. With the support of PostgreSQL it’s now ironcially MSSQL’s days that are numbered with 9.5 being the last release to support MSSQL as the vCD Database. 

For those that use vCloud Network Isolation (VCDNI), that is now also no longer supported as well as a continued end of support for Older API Versions with version 19.0 and earlier no longer supported.

From a networking point of view vCD 9.5 is the last release to support the creation edge devices in the non-advanced mode which is effectively the old vShield mode. Only edge devices that have been created or converted to advanced will be supported by the HTML5 UI.

Compatibility with Veeam, vSphere 6.5, 6.7, NSX-v 6.4.x and NSX-T 2.2 Support:

On the NSX-T front…from the release notes:

vCloud Director 9.5 is the first version to support NSX-T, which can be combined with the existing support for NSX-V in the same vCloud Director installation. You can add a NSX-T Manager and the corresponding vCenter(s) as a resource in vCD (via API) and create a Provider VDC (PVDC) that is backed by NSX-T. All the vCenters in this PVDC should be backed the same NSX-T manager. All the hosts in these vCenters then will be installed with the DPDK switch. A VLAN backed network pool for each OrgVDC can be created,
from this the network configuration on tenant side is the same as with NSX-V.

NSX-T is something that VMware is pushing very hard now, and i’ll be honest in saying that i’ve not had a chance to tinker with it. I’m still very much in tune with NSX-v however it’s clear from the push of NSX-T into VMware Cloud on AWS and now into vCD that it is the network virtualization platform of choice moving forward…though I must check on the progress of the Edge devices. These are critical to tenant edge services that front a vDC and there is a lot of power in the current NSX-v edges.

Current NSX Platform? Future Direction?

View Results

Loading ... Loading ...

vCloud Director 9.5 is compatible with the latest vSphere 6.7, 6.5 Update 2 (but not 6.5 GA) and NSX-v 6.4.3 and supports full interoperability with other versions as shown in the VMware Product Interoperability Matrix. Interestingly enough, 9.5 has more supportability for NSX-v and obviously with NSX-T having initial limited support.

With regards to Veeam support, I am sure that our QA department will be testing the 9.5 release against our integration pieces at the first opportunity they get, but as of now, there is no ETA on offical support.

There are only two resolved issues in this build and there are a number of known issues that can be found here.

Conclusion:

Overall this is again, a very strong release and it’s clear to now see that vCD is 100% supported and backed by VMware. You can start to see a shift of the platform away from just being an abstraction layer to becoming what could be a brokerage engine expanding on the extensibility thats being built into the product under the hood. vCloud Director 9.5 continues to fulfil the promise of enabling SDDC functionality to VMware service providers.

There is a White Paper where you can find more details about what’s contained in the 9.5 release. Tom Fojta and Daniel Paluszek from VMware have a what’s new blog posts as well.

#LongLivevCD

References:

https://cloudsolutions.vmware.com/assets/blt4e4a9fe9b7954100/What’s%20New%20with%20vCloud%20Director%209.5.pdf

https://docs.vmware.com/en/VMware-vCloud-Director-for-Service-Providers/9.5/rn/vmware-vcloud-director-for-service-providers-95-release-notes.html

Enhanced Self Service Restore in Backup for Office 365 v2.0

Earlier in the year I gave an overview on the Self Service recovery capability of Veeam Backup for Office 365 which gave Veeam Cloud and Service Providers the ability to offer self service to their tenants for the recovery of Exchange data that’s been backed up on their platforms as a service.

As a bit of a refresher:

Tenant admins communicate with the Service Provider via the Cloud Gateway component which handles flow of data. The Service Provider grants the ability to their tenants so that each tenant can perform self restore operations using Veeam Explorer for Microsoft Exchange. By default, tenants are not able to restore anything from the backup without a Service Provider assistance.

The steps above show the self restore scenarios performed by the Tenant:

  • Tenants use Veeam Explorer for Microsoft Exchange to send restore requests via Veeam Cloud Gateway directly to the Service Provider.
  • On the Service Provider side, Veeam Backup for Microsoft Office 365 management server detects a proxy server responsible for processing tenant data.
  • Veeam Backup for Microsoft Office 365 management server locates an associated repository that contains a backup file that belongs to the Tenant.
  • Corresponding backup data is then transferred back to the tenant via Veeam Cloud Gateway.
What’s Changed in v2.0:

As mentioned, one of the big limitations in VBO v1.5 was the fact you could only restore the most recently backed up recovery point which limited it’s usefulness for most administrators looking to take advantage of the feature. That’s changed in VBO v2.0 with the ability to now choose a point in time from the Explorers. This is true for both Veeam Explorer for Exchange and Sharepoint (Which also does OneDrive).

Shown below is a Service Provider view of a restore operation for the Sliema organisation. As with the previous versions you have the ability to use latest or go back to a point in time.

As a reminder…the retention is set against the Backup Repository in VBO. Organisations are assigned to Repositories which dictates their own retention. At the tenant end, once the Veeam Explorer has been launched and the Connect to a Service Provider option has been chosen, you now see similar options to either do the latest, or go to a point in time.

If you go to choose a point in time that precedes the date of the first backup you will get the error below. Once a correct point in time has been selected the Self Service can begin. Shown below i’m able to go back to the 3rd of May 2018 restore point and perform actions on mail items. In this case, I was looking for a AWS Bill that I had deleted out of the mailbox and had gone way past my default Exchange retention settings. Back on the Service Provider end, you can see the active restore job session which is being facilitated through Cloud Connect. Conclusion:

To reiterate, the market for Office365 backups is significant and we have built in some pretty cool technology into Backup & Replication that works with Backup for Office365 that allows easy, self service capabilities that can be productized by Service Providers out of the box. Not only can Service Providers offer services to backup client Exchange, SharePoint or OneDrive Organisations but they can also extend that to offer self service which increases overall operational efficiencies at the provider end while also offering enhanced services to clients.

References:

https://helpcenter.veeam.com/docs/vbo365/guide/vex_sp_add.html?ver=20#pit

Configuring Service Provider Self Service Recovery with Veeam Backup for Microsoft Office 365

Quick Post – Veeam Backup for Office 365 v2 Important Patch plus Self Service Warning Fix

Last week we snuck out an important cumulative patch for Veeam Backup for Office 365 v2 bring the build number up to 2.0.0.567. The patch is actually fairly significant and I would recommend anyone running VBO to update as soon as possible. It covers Licensing, SharePoint and OneDrive, Group and Shared Mailbox fixes and enhancements as well as general server fixes.

To download and install the update, head to the VeeamKB here. There are some important notes about the upgrade process depending on your deployment configuration.

  • Execute VBO2.0-KB2765.msp as administrator on the Veeam Backup for Microsoft Office 365 server.
  • If there are any remote proxies in your environment please update those as described here 
  • If you use a remote VBO365 console and/or remote VBO365 PowerShell module installation, please contact technical support to assist you in upgrading those components.
Self Service Warning Fix:

Not related to the update, but something that I had happen to me on testing the upgraded VBO instance was that when I went to perform a Self Service through the Veeam Explorer for Exchange or Sharepoint I had the following pop up.

Once hitting ok, I didn’t have the ability to choose a Service Provider connection for the Self Service restore operation. This was the same for both Exchange the Sharepoint Explorer. Working with our support to ensure it wasn’t a regression in the latest patch we found an entry in the Explorer log files that pointed to the issue.

[28.09.2018 13:08:15] <37> Info [CloudCacheSync] Synchronizing provider 119.252.77.83

[28.09.2018 13:08:16] <37> Error Exception while connecting to endpoints [119.252.77.83]
[28.09.2018 13:08:16] <37> Error No connection could be made because the target machine actively refused it 119.252.77.83:6180 (System.Net.Sockets.SocketException)

[28.09.2018 13:08:17] <37> Error All cloud gateways are unavailable (Veeam.Backup.Core.CCloudGateSvc+CAllGatesUnavailableException)

[28.09.2018 13:08:17] <37> Error Credentials with id ‘a22f868a-a51a-473f-9f6d-cff9ff250fa3’ were not found (System.Exception)

Basically the issue was caused by the fact that I had an uncontactable Service Provider endpoint configured in the Backup & Replication Server. Once I removed the offending entry in the Service Provider section, I was able to reload the Explorers and have the ability to perform self service recoveries again. It’s probably something that won’t come up under normal tenant circumstances as I connect to multiple Service Providers from my NestedESXi Homelab instance…but something to take note of if the warning appears for you.

References:

https://www.veeam.com/kb2765

Quick Fix: Specified vCloud Director is not supported when trying to add vCD 9.1 to Veeam ONE

Back in May when VMware released vCloud Director 9.1 they also depreciated support for a number of older API versions:

End of Support for Older vCloud API Versions

  • vCloud Director 9.1 no longer supports vCloud API versions 1.5 and 5.1. These API versions were deprecated in a previous release.
  • vCloud Director 9.1 is the last release of vCloud Director to support any vCloud API versions earlier than 20.0. Those API versions are deprecated in this release and will not be supported in future releases.

Due to this, and being mid release cycle, Veeam ONE had issues connecting to vCD instances that where running version 9.1.

The error you would get if you tried to connect was:

Over the past few months i’ve had questions around this and if it was going to be fixed by way of a patch. While we are waiting for the next release of Veeam ONE that is due with Veeam Backup & Replication 9.5 Update 4 there is a way to get vCD 9.1 instances connected into the current build of Veeam ONE.

There is a HotFix available through Veeam Support to resolve the Known Issue. It involves stopping the Veeam ONE services, replacing a couple of DLL’s and then re-starting the services. Once implemented Veeam ONE is able to connect to vCD 9.1.

So if you have this problem, raise a support case, grab the HotFix and the issue will be sorted.

References:

https://docs.vmware.com/en/vCloud-Director/9.1/rn/rel_notes_vcloud_director_91.html#deprecated

Automated Configuration of Backup & Replication with PowerShel

As part of the Veeam Automation and Orchestration for vSphere project myself and Michael Cade worked on for VMworld 2018, we combined a number of seperate projects to showcase an end to end PowerShell script that called a number of individual modules. Split into three parts, we had a Chef/Terraform module that deployed a server with Veeam Backup & Replication installed. A Terraform module that deployed and configured an AWS VPC to host a Linux Repository with a Veeam PN Sitegateway. And finally a Powershell module that configured the Veeam server with a number of configuration items ready for first use.

The goal of the project was to release a PowerShell script that fully deployed and configured a Veeam platform on vSphere with backup repositories, vCenter server and default policy based jobs automatically configured and ready for use. This could then be adapted for customer installs, used on SDDC platforms such as VMware Cloud on AWS, or for POCs or lab use.

While we are close to releasing the final code on GitHub for the project, I thought I would branch out the last section of the code and release it separately. As I was creating this script, it became apparent to me that it would be useful for others to use as is or as an example from which to simplify manual and repetitive tasks that go along with configuring Backup & Replication after installation.

Script Overview:

The PowerShell script (found here on GitHub) performs a number of configuration actions against any Veeam Backup & Replication Server as per the included functions.

All of the variables are configured in a config.json file meaning nothing is required to be modified in the main PowerShell script. There are a number of parameters that can be called to trigger or exclude certain functions.

There are some pre-requisites that need to be in place before the script can be executed…most importantly the PowerShell needs to be executed on a system where the Backup & Replication Console is installed to allow access to the Veeam PowerShell Snap-in. From there you just need a new Veeam Backup & Replication server and a vCenter server plus their login credentials. If you want to add a Cloud Connect Provider offering Cloud Connect Backup or/and Replication you enter in all the details in the config.json file as well. Finally, if you want to add a Linux Repository you will need the details of that plus have it configured for key based authentication.

You can combine any of the parameters listed above. An example is shown above where -ClearVBRConfig has been used to reverse the -RunVBRConfigure parameter that was executed first to do an end to end configure. For Cloud Connect Replication, if you want to configure and deploy an NEA there is a specific parameter for that. If you didn’t want to configure Cloud Connect or the Linux Repository the parameters can be used individually, or together. If those two parameters are used, the Default Backup Repository will be used for the jobs that are created.

Automating Policy Based Backup Jobs:

Part of the automation that we where keen to include was the automatic creation of default backup jobs based on vSphere Tags. The idea was to have everything in place to ensure that once the script had been run, VMs could be backed up dependant on them being added to vSphere Tags. Once done the backup jobs would protect those VMs based on the policies set in the config.json.

The corresponding jobs are all using the vSphere Tags. From here the jobs don’t need to be modified when VMs are added…VMs assigned those Tags will be included in the job.

Conclusion:

Once the script has been run you are left with a fully configured Backup & Replication server that’s connected to vCenter and if desired (by default) has local and Cloud Connect repositories added with a set of default policy based jobs ready to go using vSphere Tags.

There are a number of improvements that I want to implement and I am looking out for Contributors on GitHub to help develop this further. At its base it is functional…but not perfect. However it highlights the power of the automation that is possible with Veeam’s PowerShell Snap-In and PowerCLI. One of the use-cases for this was for repeatable deployments of Veeam Backup & Replication into POCs or labs and for those looking to standup those environments, this is a perfect companion.

Look out for the full Veeam SDDC Deploy Toolkit being released to GitHub shortly.

References:

https://github.com/anthonyspiteri/powershell/tree/master/BR-Configure-Veeam

« Older Entries