Author Archives: Anthony Spiteri

AWS re:Invent 2018 – Veeam and N2WS Recap and Thoughts

There was so much to take away from AWS re:Invent last week. In my opinion, having attended a lot of industry events over the past ten or so years, this years re:Invent has left the industry with a lot to think about it! AWS vigorously defended their position as the number one Public Cloud destination (in their eyes) while trying to lay a path for future growth by expanding into the true enterprise space. Also, with the announcement of Outposts set a path to try and dominate the hybrid world with an on-premises offering.

Instead of writing down my extended thoughts it’s more consumable to hear Rick Vanover and myself talk about the event from a Veeam perspective in the short embedded video below. I’ve also embedded a video with David Hill and Sebastian Straub covering things from an N2WS perspective, as well as talk about the N2WS related announcements at re:Invent 2018.

I’ve also posted the Veeam session video here:

AWS re:Invent 2018 Recap – Times…they a̶r̶e̶ have a̶ Changi̶n̶g̶ed!

I wrote this sitting in the Qantas Lounge in Melbourne waiting for the last leg back to Perth after spending the week in Las Vegas at AWS re:Invent 2018. I had fifteen hours on the LAX to MEL leg and before that flight took off, I struck up a conversation (something I never usually do on flights) with a guy in the seat next to me. He noticed my 2017 AWS re:Invent jumper (which is 100x better than the 2018 version) and asked me if had attended re:Invent.

It ended up that he worked for a San Francisco based company that wrote middleware integration for Salesforce. After a little bit of small talk, we got into some deep technical discussions about the announcements and around what we did in our day to day roles. Though I shouldn’t have been surprised, just as I had never heard of his company, he had never heard of Veeam…ironically he was from Russia and now working in Melbourne.

The fact he hadn’t heard of Veeam in its self wasn’t the most surprising part…it was the fact that he claimed to be a DevOps engineer. But had never touched any piece of VMware software or virtualisation infrastructure. His day to day was exclusively working with AWS web technologies. He wasn’t young…maybe early 40s…this to me seemed strange in itself.

He worked exclusively around APIs using AWS API Gateway, CloudFormations and other technologies but also used Nginx for reverse proxy purposes. That got me thinking that the web application developers of today are far far different to those that I used to work with in the early 2000’s and 2010’s. I come from the world of LAMP and .NET applications platforms…I stopped working on web and hosting technologies around the time Nginx was becoming popular.

I can still hold a conversion (and we did have a great exchange around how he DevOp’ed his applications) around the base frameworks of applications and components that go into making a web application work…but they are very very different from the web applications I used to architect and support on Windows and Linux.

All In on AWS!

The other interesting thing from the conversation was that his Technical Director commands the exclusive use of AWS services. Nothing outside of the service catalog on the AWS Console. That to me was amazing in itself. I started to talk to him about automation and orchestration tools and I mentioned that i’d been using Terraform of late…he had never used it himself. He asked me about it and in this case I was the one telling him how it worked! That at least made me feel somewhat not totally dated and past it!

My takeaway from the conversation plus what I experienced at re:Invent was that there is a strong, established sector of the IT industry that AWS has created, nurtured and is now helping to flourish. This isn’t a change or die message…this is simply my own realisation that the times have changed and as a technologist in the the industry I owe it to myself to make sure I am aware of how AWS has shifted web and application development from what I (and from my assumption the majority of those reading this post) perceive to be mainstream.

That said, just like the fact that a hybrid approach to infrastructure has solidified as the accepted hosting model for applications, so to the fact that in the application world there will still be a combination of the old and new. The biggest difference is that more than ever…these worlds are colliding…and that is something that shouldn’t be ignored!

Veeam’s AWS re:Invent 2018 Session Posted

This week, myself and David Hill presented at AWS re:Invent 2018 around what at Veeam is offering by way of providing data protection and availability for native AWS workloads, VMware Cloud on AWS workloads and how we are leveraging AWS technologies to offer new features in the upcoming Update 4 release of Backup & Replication 9.5.

For those that where not at AWS re:Invent this week or for those who could not attend the session on Wednesday, the video recording has been posted on the offical AWS YouTube page.

We had some audio issues at the start which made for some interesting banter between David and myself…but once we got into it we talked about the following:

  • The N2WS 2.4 Release
  • Veeam VTL and AWS Storage Gateway
  • Update 4 Cloud Tier
  • Update 4 Cloud Mobility
  • Data Protection for VMware Cloud on AWS

I wanted to highlight the Cloud Tier section where I give an overview and quick deepdive into the smarts behind the new repository feature coming in Update 4. The live demo of me using our Patented Instant VM Recovery feature to bring up a VM with data residing in Amazon S3 is a great example of the power of this upcoming feature. Not only does it allow storage efficiencies locally but offloading old data to Object Storage for long term retention, but is also is intelligent enough to recover quickly and efficiently with its Intelligent Block Recovery.

Veeam at AWS re:Invent 2018

AWS re:Invent 2018 is happening next week and for the first time Veeam is at the event in a big way! Last year, we effectively tested the waters with a small booth, no main session and without the usual event presence that you would expect of Veeam at an VMworld or Microsoft Ignite. This year is a little different and we will be there as Diamond Sponsors of the event and with a lot to share in regards to how Veeam is leveraging AWS technologies to enhance our availability messaging.

We bolstered our native AWS capabilities earlier this year with the acquisition of N2SW who already where a leader in the protection of AWS workloads and with the upcoming release of Backup & Replication 9.5 Update 4 we will be further enhancing our ability to not only backup AWS workloads, but also leverage AWS technologies such as S3 to facilitate a change in mindset as to what it is to have a local backup repository. We will also be talking about migration into AWS and also how we are the best data protection choice for VMware Cloud on AWS.

Breakout Session:

At the event we will have a breakout session which myself and David Hill will be presenting. This will be on Wednesday at 5:30pm in the Aria Casino and we are looking forward to deep diving into what’s coming in Update 4 as well as showing off what’s coming in the next release of N2WS as we start to jointly develop solutions between the two companies.

STG206-S – A Deeper Look at How Veeam is Evolving Availability on AWS

Wednesday, Nov 28, 5:30 PM – 6:30 PM – Aria East, Level 1, Joshua 6

Veeam has made significant enhancements to its platform, focusing on the availability of AWS workloads over the past year. Join this technical deep dive where representatives from Veeam demonstrate how the company protects cloud-native workloads on AWS as well as how they back up to and from on-premises environments. They also discuss data protection for VMware Cloud on AWS. Finally, they review the enhancements to Veeam’s Backup and Replication feature set, which now includes cloud mobility to AWS and a cloud archive that leverages Amazon S3 for long-term data retention of backed-up workloads.

In terms of the technologies and solutions that we will be diving into and showing off via some live demos…we will be looking at:

  • The N2WS 2.4 Release
  • Veeam VTL and AWS Storage Gateway
  • Update 4 Cloud Tier
  • Update 4 Cloud Mobility
  • Data Protection for VMware Cloud on AWS

I will also be giving a Booth Presentation at the Cloudcheckr booth, Tuesday at 10am which will effectively be a slimmed down version of the main session happening on the Wednesday.

Booth and Show Floor:

As mentioned, this year we will have significant presence on the show floor with two areas to come and see Veeam technologies as well as chat to us about how we are protecting and leveraging AWS and AWS workloads. On the main show floor we will be at booth #1011 which is well positioned next to the GitHub booth and we will also have a second location at the Mirage called the Data Protection Lounge which will be a place to relax, enjoy a snack and engage in technical discussions with our experts…including myself!

Social Events:

This year we are jointly sponsoring a location for the re:Invent Pub Crawl which is happening on Tuesday night. Details are below

Pub Crawl – Veeam | N2WS and VMware
Date & Time: Tuesday, November 27, 6pm – 8pm
Location: Mercato della Pescheria – The Venetian Shoppes

Wrapping Up:

I’m looking forward to the event and being more than a spectator this year I’m expecting big things from it. Make sure you come visit us at our booth or at the lounge to check out what has been brewing from Veeam and N2WS R&D over the past twelve months…and also don’t forget to attend the session on Wednesday afternoon. I’m excited about some of the new features we will release as part of Update 4…and this session is a chance to see them working and get an understanding as to what they will be delivering.

If you would like to schedule a meeting with myself or any other member of the Veeam Product Strategy team attending, please reach out.

Veeam Availability Console 2.0 Update 1 – New Patch Release

Last week a patch was released for Veeam Availability Console 2.0 Update 1. There are a number of resolved issues around core VAC server functionality, discovery rules, monitoring and alarms, reporting and billing and also the ConnectWise Manage plug-in. The patch is advised to be deployed to all VCSPs running VAC as it will resolve a number of UI and under the hood issues.

To apply the patch, head to the VeeamKB here and follow the instructions. You need to have at least VAC 2.0 Update 1 Build 2.0.2.1750 or later as shown below.

From there, make sure you have a backup of the database, close down the Web UI and execute all three MSI packages as administrator on the server.

The first one updates the VAC server.

The second one updates the ConnectWise Manage Plugin.

The last one updates the Web UI.

Once completed the patches are applied and VAC 2.0 Update 1 is up to date running on version number Server Version 2.0.2.1850. Note that updated Windows for Agent Builds have been pushed out and can be upgraded as per my post a few months back.

References:

https://www.veeam.com/kb2694

Backing up 6.7 Update 1 VCSA to Cloud Connect Fails

A few weeks ago I upgraded my NestedESXi homelab to vSphere 6.7 Update 1. Even though Veeam does not have offical supportability for this release until our Backup & Replication 9.5 Update 4 release there is a workaround that deals with the change of vSphere API version that out of the box, causes backup to fail. After the upgrade and the application of the workaround I started to get backup errors while trying to process the main lab VCSA VM which was now running vCenter 6.7 Update 1. All other VMs where being backed up without issue.

Processing LAB-VC-67 Error: Requested value ‘vmwarePhoton64Guest’ was not found.

The error was interesting and only impacted the VCSA VM that I had upgraded to 6.7 Update 1. I do have another VCSA VM in my lab which is on the GA of 6.7 which was backing up successfully. What was interesting is that it appears like the GuestOS type of the VM had changed or was being recognised as PhotonOS from within the upgraded vCenter on which it lived it’s self.

Looking at the VM Summary, it was being listed as VMware Photon OS (64-bit)

My first instinct was to change this back to what I saw the other VCSA to be, which was Other 3.x Linux (64-bit)

However, due to the chicken or the egg nature of having the management VCSA on the same vCenter when I logged into the ESXi host (also upgraded to 6.7 Update 1) I saw that it didn’t match what was being shown in vCenter.

Thinking it was due to a mismatch, I changed the Guest OS type here to Photon OS However the same issue occurred. Next I tried to get a little creative and change the Guest OS Type to Other Linux (64-bit) but even though I changed it to that from ESXi…from vCenter (its self) it was still reporting Photon OS and failed.

The Issue:

I submitted a support ticket and from the logs the Support team where able to ascertain that the issue actually lied at the Cloud Connect Providers end. I was sending these backups directly to the Cloud Connect Provider, so my next step to confirm this was to try a local backup test job and sure enough the VM processed without issues.

I then attempted a Backup Copy job from that successful test job to the Cloud Connect Provider and that resulted in the same error.

From the job logs it became clear what the issue was:

[07.11.2018 03:00:12] <01> Info [CloudGateSvc 119.252.79.147:6180]Request: [Service.Connect] SessionType:4, SessionName:Lab Management, JobId:54788e4d-7ba1-488a-8f80-df6014c58462, InstallationId:30ee4690-01c9-4368-94a6-cc7c1bad69d5, JobSessionId:b1dba231-18c2-4a28-9f74-f4fa5a8c463b, IsBackupEncrypted:False, ProductId:b1e61d9b-8d78-4419-8f63-d21279f71a56, ProductVersion:9.5.0.1922,
[07.11.2018 03:00:13] <01> Info [CloudGateSvc xx.xx.xx.xx:6180]Response: CIResult:b4aa56f4-fd02-4446-b893-2c39a16e535e, ServerTime:6/11/2018 7:00:13 PM, Version:9.5.0.1536,

At my end, I am running Backup & Replication 9.5 Update 3a, while at the provider end, they are running Backup & Replication 9.5 Update 3. Update 3a introduced supportability to vSphere 6.7 and other platform updates…this included the list at Veeam’s end of support Guest OS Types. In a nutshell the Veeam Cloud Connect Backup server still needs to understand what type of VM/Guest its backing up in its Cloud Repository. For this to be resolved the provider would need to upgrade their Cloud Connect infrastructure to Update3a…meanwhile, I’m backing up the VM locally for the time being.

Timely Message for VCSPs running Cloud Connect:

As we approach the release of another Update for Backup & Replication it’s important for Veeam Cloud and Service Providers to understand that they need to keep in step with the latest releases. This is why we typically have an RTM build given to providers at least two weeks before GA.

With vSphere 6.7 Update 1 starting to be deployed to more organisations it’s important to be aware of any issues that could stop tenant backups from completing successfully. This has generally been a consideration for providers offering Cloud Connect over the years…especially with Cloud Connect Replication, where the target platform needs to be somewhat in check with the latest platforms that are available.

References:

https://www.veeam.com/kb2443

https://www.veeam.com/kb2784

Hybrid World… Why IBM buying RedHat makes sense!

As Red October came to a close…at a time when US Tech stocks were taking their biggest battering in a long time the news came out over the weekend that IBM had acquired RedHat for 34 billion dollars! This seems to have taken the tech world by surprise…the all-cash deal represents a massive 63% premium on the previous close of RedHat’s stock price…all in all it seems ludicrous.

Most people that I’ve talked to about it and from reading comments on social media and blog sites suggests that the deal is horrible for the industry…but I’ve felt this is more a reaction to IBM than anything. IBM has a reputation as swallowing up companies whole and spitting them out the other side of the merger process a shell of what they once were. There has also been a lot of empathy for the employees of RedHat, especially from ex-IBM employees who have experience inside the Big Blue machine.

I’m no expert on M&A and I don’t pretend to understand the mechanics behind the deal and what is involved…but when I look at what RedHat has in its stable, I can see why IBM have made such an aggressive play for them. On the surface it seems like IBM are in trouble with their stock price and market capitalization falling nearly 20% this year and more than 30% in the last five years…they had to make a big move!

IBM’s previous 2013 acquisition of SoftLayer (for a measly 2 billion USD) helped them remain competitive in the Infrastructure as a Service space and if you believe the stories, have done very well out of integrating the SoftLayer platform into what was BlueMix, and is now IBM Cloud. This 2013 Forbes article on the acquisition sheds some light as to why this RedHat acquisition makes sense and is true to form for IBM.

IBM sees the shift of big companies moving to the cloud as a 20-year trend…

That was five years ago…and since then a lot has happened in the Cloud world. Hybrid cloud is now the accepted route to market with a mix of on-premises, IaaS and PaaS hosted and hyper-scale public cloud services being the norm. There is no one cloud to rule them all! And even though AWS and Azure continue to dominate and be front of mind there is still a lot of choice out there when it comes to how companies want to consume their cloud services.

Looking at RedHat’s stable and taking away the obvious Linux distro’s that are both enterprise and open sources the real sweet spot of the deal lies in RedHat’s products that contribute to hybrid cloud.

I’ve heard a lot more noise of late about RedHat OpenStack becoming the platform of choice as companies look to transform away from more traditional VMware/Hyper-V based platforms. RedHat OpenShift is also being considered as an enterprise ready platform for containerization of workloads. Some sectors of the industry (Government and Universities) have already decided on their move to platforms that are backed by RedHat…the one thing I would comment here is that there was an upside to that that might now be clouded by IBM being in the mix.

Rounding out the stable, RedHat have a Cloud Suite which encompasses most of the products listed above. CloudForms for Infrastructure as Code, with Ansible for orchestration…together with RedHat Virtualization together with OpenStack and OpenShift..it’s a decent preposition!

Put all that together with the current services of IBM Cloud and you start to have a compelling portfolio covering almost all desired aspects of hybrid and multi cloud service offerings. If the acquisition of SoftLayer was the start of a 20 year trend then IBM are trying to keep themselves positioned ahead of the curve and very much in step with the next evolution of that trend. That isn’t to say that they are not playing catchup with the likes of VMware, Microsoft, Amazon, Google and alike, but I truly believe that if they don’t butcher this deal they will come out a lot stronger and more importantly offer valid completion in the market…that can only be a good thing!

As for what it means for RedHat itself, their employees and culture…that I don’t know.

References:

https://www.redhat.com/en/about/press-releases/ibm-acquire-red-hat-completely-changing-cloud-landscape-and-becoming-world%E2%80%99s-1-hybrid-cloud-provider

IBM sees the shift of big companies moving to the cloud as a 20-year trend

vSphere 6.7 Update 1 – Top New Features and Platform Supportability

Last week VMware released vSphere 6.7 Update 1. While the buzz around this release was less than the previous release it still contains a ton of enhancements for vCenter, ESXi and vSAN. Like 6.7 before it, this is a lot more than a point release and represents a significant upgrade from vSphere 6.7.

Looking through the release notes, there appears to be less for service providers in this release though I still feel like it’s important to highlight the base hypervisor (ESXi) as well as the management platform (vCenter). vSAN has had another significant update and that will warrant a post on it’s on. I’ll also talk about current interoperability with vCloud Director and NSX as well as current Veeam supportability for vSphere 6.7 Update 1 as well as touch on Veeam’s current supportability.

  • New (almost 100%) Fully functional HTML5 client
  • Upgrade path from vSphere 6.5 U2 to vSphere 6.7 Update 1
  • Enhanced support for NVIDIA Quadro vDWS VMs and support for Intel FPGA
  • New vCenter Convergence Tool
  • Updated vSAN
  • Enhanced vSphere Content Library
Fully Functional HTML5 Client

Most functions have now been ported across to the HTML5 vSphere Client. This results in administrators not having to switch back and forth between the FLEX Web Client and the HTML5 client. Update 1 features:

  • vCenter High Availability (VCHA)
  • Auto Deploy
  • Host Profiles
  • vSphere Update Manager
  • Network Topology Diagrams
  • Performance Charts
  • Improved Searching
  • Dark Theme

Emad Younis has a detailed post here that goes through the new features.

Upgrade Path from vSphere 6.5 Update 2 to vSphere 6.7 Update 1

One of the issues with vSphere 6.7 was the fact that the vSphere 6.5 Update 2 release would not be able to be upgraded to vSphere 6.7.  With the release of vSphere 6.7 Update 1. vSphere 6.5 Update 2 to vSphere 6.7 Update 1 is now a fully supported.

Enhanced Content Library

New improvements to the content library in vSphere 6.7 Update 1 enables the importing of OVA templates from a HTTPS endpoint and also local storage.  Importing now verifies the certificate of the OVA bundle and also now natively supports VM templates (VMTX) and associated operations such as deploying a VM directly from Content Library.

vCenter Specific Enhancements

With vCenter Server 6.7 Update 1, you can move a vCenter Server with an Embedded Platform Services Controller from one vSphere domain to another vSphere domain. Services such as tagging and licensing are retained and migrated to the new domain.

There is a new Burst Filter to manage event bursts and prevent the database of vCenter Server from flooding with identical events over a short period of time.

vCenter Server 6.7 Update 1 supports VMware vSphere vMotion between on-premises vCenter’s and VMware Cloud on AWS. You can use either the vSphere Client or vSphere Web Client, or the API. Both sides need to be at 6.7 Update 1.

you can import Open Virtual Appliance (OVA) files in a Content Library. The OVA files are unzipped during the import, providing manifest and certificate validations, and create an OVF library item that enables deployment of virtual machines from a Content Library.

With vCenter Server 6.7 Update 1, you can use the Appliance Management User Interface to configure and edit the firewall settings of the vCenter Server Appliance.

ESXi Specific Enhancements

There are a few vendor/hardware related features and enhancements in Update 1 for ESXi 6.7. The release notes cover them in detail here. But as mentioned above, probably the biggest addition here is the ability to upgrade from ESXi 6.5 Update 2 which I know a few service providers where stuck on. In terms of known issues the release notes also contain a good list. There are some here that impact Service Providers so it’s worth reading through them.

vCD and NSX Supportability:

Shifting from new features and enhancements to an important subject to talk about when talking service provider platform…VMware product compatibility. For those VCPP Service Providers running a Hybrid Cloud you should be running a combination of vCloud Director SP or/and NSX-v of which the 6.4.3 and 6.4.2 versions are supported at release. Most providers should be on these releases so that’s good news.

Looking at vCloud Director, it looks like 9.5 is the only supported version at the moment

Veeam Backup & Replication Supportability: 

Veeam commits to supporting major version releases within 90 days or sooner of GA. There has been many discussions going round whether an Update is a major release these days…and general consensus now is that VMware is releasing these updates with enough changes to potentially impact backup supportability.

So with that, those Service Provider that are also VCSPs using Veeam to backup their infrastructure should not upgrade to vSphere 6.7 until Backup & Replication Update 4 is released. For those that are bleeding edge and have updated your only is to go with the workaround that is detailed here. It works…but again, it’s a work around.

Wrapping Up:

Rounding off this post, in the Known Issues section there is a fair bit to be aware of for 6.7 Update 1. it’s worth reading through all the known issues just in case there are any specific issues that might impact you.

Happy upgrading!

References:

https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-vcenter-server-671-release-notes.html

https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-esxi-671-release-notes.html

Quick Fix – VCSA 6.7.0.10000 Can’t Update via URL from Management Interface

I had an issue with my VCSA today trying to upgrade to vCenter 6.7 Update 1 whereby the Management Interface Upgrade option was not detecting the update to upgrade the appliance to 6.7 Update 1. It was a similar issue to this VMwareKB, however the URL that is mentioned in that instance was already in the VCSA Settings.

My first instinct was to check the disk space and see if there where any pressures in that area. I did find that the /dev/sda3 partition was low on space, so I expanded the disk following advice given by Mark Ukotic. After a reboot and resize I had plenty of storage left, but still couldn’t trigger an update from the URL. At this point I did download the Update patch ISO from the VMware Patch center and loaded it up manually…however the issue of it not popping up automatically was annoying me.

As mentioned, the settings of the VCSA Update window has the following URL listed:

https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.10000.latest/

Having asked around a little the quick fix was provided by Matt Allford who provided me with the URL that was present in his VCSA after he upgraded successfully via the CLI.

https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.20000.latest/

I added that as a custom repository as shown below…

I was then able to rescan and choose from the list of updates for the VCSA.

And perform the upgrade from the Management Interface as first desired.

Interestingly enough, after the upgrade the default Update Repository was set to the one Matt provided for me.

This is the first time i’ve seen this behavior from the VCSA but I had reports of people being able to upgrade without issue. I’m wondering if it might be the particular build I was on, though that in it’s self was not picking up any patches to update to either. If anyone has any ideas, feel free to comment below.

Quick Fix: Terraform Plan Fails on Guest Customizations and VMware Tools

Last week I was looking to add the deployment of a local CentOS virtual machine to the Deploy Veeam SDDC Toolkit project so that it included the option to deploy and configure a local Linux Repository. This could then can be added to the Backup & Replication server. As part of the deployment I call the Terraform vSphere Provider to clone and configure the virtual machine from a pre loaded CentOS template.

As shown below, I am using the Terraform customization commands to configure VM name, domain details as well as network configuration.

In configuring the CentOS template i did my usual install of Open VM Tools. When the Terraform plan executes we applied the VM was cloned without issue, but it failed at the Guest Customizations part.

The error is pretty clear and to test the error and fix, I tried applying the plan without any VMware Tools installed. In fact without VMware Tools the VM will not finish the initial deployment after the clone and be deleted by Terraform. I next installed open-vm-tools but ended up with the same scenario of the plan failing and the VM not being deployed. For some reason it does not like this version of the package being deployed.

Next test was to deploy the open-vm-tools-deploypkg as described in this VMwareKB. Now the Terraform plan executed to the point of cloning the VM and setting up the desired VM hardware and virtual network port group settings but still failed on the custom IP and hostname components of the customisation. This time with a slightly different error.

The final requirement is to pre-install the perl package onto the template. This allows for the in guest customizations to take place together with VMware Tools. Once I added that to the template the Terraform Plan succeeded without issue.

References:

https://kb.vmware.com/s/article/2075048

 

 

« Older Entries