A few weeks ago I upgraded my NestedESXi homelab to vSphere 6.7 Update 1. Even though Veeam does not have offical supportability for this release until our Backup & Replication 9.5 Update 4 release there is a workaround that deals with the change of vSphere API version that out of the box, causes backup to fail. After the upgrade and the application of the workaround I started to get backup errors while trying to process the main lab VCSA VM which was now running vCenter 6.7 Update 1. All other VMs where being backed up without issue.
Processing LAB-VC-67 Error: Requested value ‘vmwarePhoton64Guest’ was not found.
The error was interesting and only impacted the VCSA VM that I had upgraded to 6.7 Update 1. I do have another VCSA VM in my lab which is on the GA of 6.7 which was backing up successfully. What was interesting is that it appears like the GuestOS type of the VM had changed or was being recognised as PhotonOS from within the upgraded vCenter on which it lived it’s self.
Looking at the VM Summary, it was being listed as VMware Photon OS (64-bit)
My first instinct was to change this back to what I saw the other VCSA to be, which was Other 3.x Linux (64-bit)
However, due to the chicken or the egg nature of having the management VCSA on the same vCenter when I logged into the ESXi host (also upgraded to 6.7 Update 1) I saw that it didn’t match what was being shown in vCenter.
Thinking it was due to a mismatch, I changed the Guest OS type here to Photon OS
However the same issue occurred. Next I tried to get a little creative and change the Guest OS Type to Other Linux (64-bit) but even though I changed it to that from ESXi…from vCenter (its self) it was still reporting Photon OS and failed.
The Issue:
I submitted a support ticket and from the logs the Support team where able to ascertain that the issue actually lied at the Cloud Connect Providers end. I was sending these backups directly to the Cloud Connect Provider, so my next step to confirm this was to try a local backup test job and sure enough the VM processed without issues.
I then attempted a Backup Copy job from that successful test job to the Cloud Connect Provider and that resulted in the same error.
From the job logs it became clear what the issue was:
[07.11.2018 03:00:12] <01> Info [CloudGateSvc 119.252.79.147:6180]Request: [Service.Connect] SessionType:4, SessionName:Lab Management, JobId:54788e4d-7ba1-488a-8f80-df6014c58462, InstallationId:30ee4690-01c9-4368-94a6-cc7c1bad69d5, JobSessionId:b1dba231-18c2-4a28-9f74-f4fa5a8c463b, IsBackupEncrypted:False, ProductId:b1e61d9b-8d78-4419-8f63-d21279f71a56, ProductVersion:9.5.0.1922,
[07.11.2018 03:00:13] <01> Info [CloudGateSvc xx.xx.xx.xx:6180]Response: CIResult:b4aa56f4-fd02-4446-b893-2c39a16e535e, ServerTime:6/11/2018 7:00:13 PM, Version:9.5.0.1536,
At my end, I am running Backup & Replication 9.5 Update 3a, while at the provider end, they are running Backup & Replication 9.5 Update 3. Update 3a introduced supportability to vSphere 6.7 and other platform updates…this included the list at Veeam’s end of support Guest OS Types. In a nutshell the Veeam Cloud Connect Backup server still needs to understand what type of VM/Guest its backing up in its Cloud Repository. For this to be resolved the provider would need to upgrade their Cloud Connect infrastructure to Update3a…meanwhile, I’m backing up the VM locally for the time being.
Timely Message for VCSPs running Cloud Connect:
As we approach the release of another Update for Backup & Replication it’s important for Veeam Cloud and Service Providers to understand that they need to keep in step with the latest releases. This is why we typically have an RTM build given to providers at least two weeks before GA.
With vSphere 6.7 Update 1 starting to be deployed to more organisations it’s important to be aware of any issues that could stop tenant backups from completing successfully. This has generally been a consideration for providers offering Cloud Connect over the years…especially with Cloud Connect Replication, where the target platform needs to be somewhat in check with the latest platforms that are available.
References:
