Category Archives: VMware

VMworld 2019 – Session Breakdown and Analysis

Everything to do with VMworld this year feels like it’s arrived at lightning speed. I actually thought the event was two weeks away as the start of the week… but here we are… only five days away from kicking off in San Francisco. The content catalog for the US event has been live for a while now and as is recently the case, a lot of sessions were full just hours after it went live! At the moment there is huge 1348 sessions listed which include the #vBrownBag Tech Talks hosted by the VMTN Community.

As I do every year I like to filter through the content catalog and work out what technologies are getting the airplay at the event. It’s interesting going back since I first started doing this to see the catalog evolve with the times… certain topics have faded away while others have grown and some dominate. This ebs and flows with VMware’s strategies and makes for interesting comparison.

What first struck me as being interesting was the track names compared to just two years ago at the 2017 event:

I see less buzz words and more tracks that are tech specific. Yes, within those sub categories we have the usual elements of “digital transformation” and “disruption”, however VMware’s focus looks to be focuses more around the application of technology and not the high level messaging that usually plagues tech conferences. VMworld has for the most and remains a technical conference for techs.

By digging into the sessions by searching on key words alone, the list below shows you where most of the sessions are being targeted this year. If, in 2015 you where to take a guess at what particular technology was having the most coverage at a VMworld at 2019…the list below would be much different than what we see this year.

From looking back over previous years, there is a clear rise in the Containers world which is now dominated by Kubernetes. Thinking back to previous VMworld’s, you would never get the big public cloud providers with airtime. If you look at how that has changed for this year we now have 231 sessions alone that mention AWS… not to mention the ones mentioning Azure or Google.

Strategy wise it’s clear that NSX, VMC and Kubernetes are front of mind for VMware and their ecosystem partners.

I take this as an indication as to where the industry is… and is heading. VMware are still the main touch point for those that work in and around IT Infrastructure support and services. They own the ecosystem still… and even with the rise of AWS, Azure, GCP and alike, they still are working out ways to hook those platforms into their own technology and are moving with industry trends as to where workloads are being provisioned. Kubernetes and VMware Cloud on AWS are a big part of that, but underpinning it is the network… and NSX is still heavily represented with NSX-T becoming even more prominent.

One area that continues to warm my heart is the continued growth and support shown to the VMware Cloud Providers and vCloud Director. The numbers are well up from the dark days of vCD around the 2013 and 2014 VMworld’s. For anyone working on cloud technologies this year promises to be a bumper year for content and i’m looking forward to catching as much vCD and VCPP related sessions as I can.

It promises to be an interesting VMworld, with VMware hinting at a massive shift in direction… I think we all know in a round about way where that is heading… let’s see if we are right come next week.

https://my.vmworld.com/widget/vmware/vmworld19us/us19catalog

Quick Fix – Issues Upgrading VCSA due to Password Expiration

It seems like an interesting “condition” has worked its self into recent VCSA builds where upon completing upgrades, the process seems to reset the root account expiration flag. This blocked my proceeding with an upgrade and only worked when I followed the steps listed below.

The error I got is shown below:

“Appliance (OS) root password is expired or is going to expire soon. Please change the root password before installing an update.”

When this happened on the first vCenter I went to upgrade, I thought that maybe there was a chance I had forgotten to set that to never expires… but usually by default I check that setting and set it to never expires… not the greatest security practice, but for my environments it’s something I set almost automatically during initial configuration. After reaching out on Twitter, I got some immediate feedback saying to reset the root password by going into single user mode… which did work.

When this happened a second time on a second VCSA, on which I without question set the never expires flag to true, I took a slightly different approach to the problem and decided to try reset the password from the VCSA Console, however that process fails as well.

After going back through the Tweet responses, I did come across this VMwareKB which lays down the issue and offers the reason behind the errors.

This issue occurs when VAMI is not able to change an expired root password.

Fair enough… but I don’t have a reason for the password never expires option not being honoured? Some feedback and conversations suggest that maybe this is a bug that’s worked its way into recent builds during upgrade procedures. In any case the way to fix it is simple and doesn’t need console access to access the command line… you just need to SSH into the VCSA and reset the root password as shown below.

Once done, the VCSA upgrade proceeds as expected. As you can see there we have also confirmed that the Password Expires is set to never. If anyone can confirm the behaviour regarding that flag being reset, feel free to comment below.

Apart from that, there is the quick fix!

References:

https://kb.vmware.com/s/article/67414

Orchestration of NSX by Terraform for Cloud Connect Replication with vCloud Director

That is probably the longest title i’ve ever had on this blog, however I wanted to highlight everything that is contained in this solution. Everything above works together to get the job done. The job in this case, is to configure an NSX Edge automatically using the vCloud Director Terraform provider to allow network connectivity for VMs that have been replicated into a vCloud Director tenant organization with Cloud Connect Replication.

With the release of Update 4 for Veeam Backup & Replication we enhanced Cloud Connect Replication to finally replicate into a Service Providers vCloud Director platform. In doing this we enabled tenants to take advantage of the advanced networking features of the NSX Edge Services Gateway. The only caveat to this was that unlike the existing Hardware Plan mechanism, where tenants where able to configure basic networking on the Network Extension Appliance (NEA), the configuration of the NSX Edge had to be done directly through the vCloud Director Tenant UI.

The Scenario:

When VMs are replicated into a vCD organisation with Cloud Connect Replication the expectation in a full failover is that if a disaster happened on-premises, workloads would be powered on in the service provider cloud and work exactly as if they where still on-premises. Access to services needs to be configured through the edge gateway. The edge gateway is then connected to the replica VMs via the vOrg Network in vCD.

In this example, we have a LAMP based web server that is publishing a WordPress site over HTTP and HTTPs.

The VM is being replicated to a Veeam Cloud Service Provider vCloud Director backed Cloud Connect Replication service.

During a disaster event at the on-premises end, we want to enact a failover of the replica living at in the vCloud Director Virtual Datacenter.

The VM replica will be fired up and the NSX Edge (the Network Extension Appliance pictured is used for partial failovers) associated to the vDC will allow the HTTP and HTTPS to be accessed from the outside world. The internal IP and Subnet of the VM is as it was on-premises. Cloud Connect Replication handles the mapping of the networks as part of the replication job.

Even during the early development days of this feature I was thinking about how this process could be automated somehow. With our previous Cloud Connect Replication networking, we would use the NEA as the edge device and allow basic configuration through the Failover Plan from the Backup & Replication console. That functionality still exists in Update 4, but only for non vCD backed replication.

The obvious way would be to tap into the vCloud Director APIs and configure the Edge directly. Taking that further, we could wrap that up in PowerShell and invoke the APIs from PowerShell, which would allow a simpler way to pass through variables and deal with payloads. However with the power that exists with the Terraform vCloud Director provider, it became a no brainer to leverage this to get the job done.

Configuring NSX Edge with Terraform:

In my previous post around Infrastructure as Code vs APIs I went through a specific example where I configured an NSX Edge using Terraform. I’m not going to go over that again, but what I have done is published that Terraform plan with all the code to GitHub.

The GitHub Project can be found here.

The end result after running the Terraform Plan is:

  • Allowed HTTP, HTTPS, SSH and ICMP access to a VM in a vDC
    • Defined as a variable as the External IP
    • Defined as a variable as the Internal IP
    • Defined as a variable as the vOrg Subnet
  • Configure DNAT rules to allow HTTP, HTTPS and SSH
  • Configure SNAT rule to allow outbound from the vOrg subnet

The variables that align with the VM and vORG network are defined in the terraform.tfvars file and need to be modified to match the on-premises network configuration. The variables are defined in the variables.tf file.

To add additional VMs and/or vOrg networks you will need to define additional variables in both files and add additional entires under the firewall_rules.tf and nat_fules.tf. I will look at ways to make this more elegant using Terraform arrays/lists and programatic constructs in future.

Creating PowerShell for Execution:

The Terraform plan can obviously be run standalone and the NSX Edge configuration can be actioned at any time, but the idea here is to take advantage of the script functionality that exists with Veeam backup and replication jobs and have the Terraform plan run upon completion of the Cloud Connect Replication job every time it is run.

To achieve this we need to create a PowerShell script:

GitHub – configure_vCD_VCCR_NSX_Edge.ps1

The PowerShell script initializes Terraform and downloads the Provider, ensures there is an upgrade in the future and then executes the Terraform plan. Remembering that that variables will change within the Terraform Plan its self, meaning these scripts remain unchanged.

Adding Post Script to Cloud Connect Replication Job:

The final step is to configure the PowerShell script to execute once the Cloud Connect Replication job has been run. This is done via a post script settings that can be found in Job Settings -> Advanced -> Scripts. Drop down to selected ps1 files and choose the location of the script.

That’s all that is required to have the PowerShell script executed once the replication job completes.

End Result:

Once the replication component of the job is complete, the post job script will be executed by the job.

This triggers the PowerShell, which runs the Terraform plan. It will check the existing state of the NSX Edge configuration and work out what configuration needs to be added. From the vCD Tenant UI, you should see the recent tasks list modifications to the NSX Edge Gateway by the user configured to access the vCD APIs via the Provider.

Taking a look at the NSX Edge Firewall and NAT configuration you should see that it has been configured as specified in the Terraform plan.

Which will match the current state of the Terraform plan

Conclusion:

At the end of the day, what we have done is achieved the orchestration of Veeam Cloud Connect Replication together with vCloud Director and NSX… facilitated by Terraform. This is something that Service Providers offering Cloud Connect Replication can provide to their clients as a way for them to define, control and manage the configuration of the NSX edge networking for their replicated infrastructure so that there is access to key services during a DR event.

While there might seem like a lot happening, this is a great example of leveraging Infrastructure as Code to automated as otherwise manual task. Once the Terraform is understood and the variables applied, the configuration of the NSX Edge will be consistent and in a desired state with the config checked and applied on every run of the replication job. The configuration will not fall out of line with what is required during a full failover and will ensure that services are available if a disaster occurs.

References:

https://github.com/anthonyspiteri/automation/tree/master/vccr_vcd_configure_nsx_edge

Quick Post – vCloud Director 9.5.0.3 Released as Critical Update

Late last week, on the same day as vCloud Director 9.7 was released to GA, an update was also released for vCloud Director 9.5.x which has been marked are critical. Specifically it relates to a vulnerability in previous vCloud Director 9.5.x with identifier CVE-2019-5523. Ironically this threat targets the new Tenant and Provider Portals.

VMware vCloud Director for Service Providers update resolves a Remote Session Hijack vulnerability in the Tenant and Provider Portals. Successful exploitation of this issue may allow a malicious actor to access the Tenant or Provider Portals by impersonating a currently logged in session.

Obviously given that vCloud Director 9.7 has just been release it’s unlikely that most Service Providers will upgrade right away, therefore the majority will be running vCloud Director 9.5.x for some time yet.

vCloud Director 9.0.x and 9.1.x are not affected.

References:

https://docs.vmware.com/en/vCloud-Director/9.5/rn/vCloud-Director-9503-for-Service-Providers-Release-Notes.html

https://www.vmware.com/security/advisories/VMSA-2019-0004.html

Quick Post – vCloud Director 9.5.0.2 Released

While we wait for the upcoming release of vCloud Director 9.7 next month (after the covers where torn off the next release in a blog post last week by the vCloud Team), VMware have released a new build (9.5.0.2 Build 12810511) of vCloud Director 9.5 that contains a number of resolved issues and is a recommended patch update.

Looking through the resolved issues it seems like the majority of fixes are around networking and to do with NSX Edge Gateway deployments as well as a few fixes around OVF template importing and API interactions.

While looking through the new layout of the VMware Docs page for vCloud Director I noticed that a few new builds for 9.1, 9.0 and 8.20 had shipped out over the past few months or so. I updated the vCloud Director Release History to reflect all the latest builds across all versions.

References:

https://docs.vmware.com/en/vCloud-Director/9.5/rn/vCloud-Director-9502-for-Service-Providers-Release-Notes.html

https://blogs.vmware.com/vcloud/2019/03/the-hybrid-cloud-gets-better-meet-vcloud-director-9-7.html

 

First Look – Runecast Adding Support for VMware HCL

Two years ago at the 2017 Sydney and Melbourne UserCons, I spent time with a couple of the founders of Runecast, Stanimir Markov and Ched Smokovic and got to know a little more about their real time analytics platform for VMware based infrastructure. Fast forward to today and Runecast have continued to build on the their initial release and have continued to add features and enhancements. The most recent of those, which is the ability to report on a ESXi Hosts VMware Hardware Compatibility List (HCL) is currently in beta and will be released shortly.

Currently, Runecast checks hardware versions, drivers and firmware against existing VMware KB articles and provides proactive findings for known issues that could impact your servers. With this addition Runecast will now show the compliance status of hardware against the VMware HCL.

This feature alone literally replaces hours of work to extract the needed data and match each server from your environment against the HCL. Critically, it can inform you if, where, and why your vSphere environment is not supported by VMware because of Hardware Compatibility issues.

In terms of what it looks like, as from the screen shot above you can see the new menu item that give you the Compatibly Overview. Your hosts are listed in the main window pane and are shows as green or red depending on their status against the HCL.

Clicking on the details you are shows the details of the host against the HCL data. If the host is out of whack with the HCL you will get an explanation similar to what is seen below. (note in the BETA I have installed this was not

With this feature you can identify which component is incompatible and unsupported. From there it will also indicate what the supportability options are for you.

Runecast keep adding great features to their platform… and most of their features are ones which any vSphere admin would find very helpful. That is the essence of what they are trying achieve.

For more information and to apply for the beta head here:

References:

https://www.runecast.com/blog/announcements/runecast-analyzer-support-for-vmware-hcl-beta

 

VMUG UserCon – Sydney and Melbourne Events!

A few years ago I claimed that the Melbourne VMUG Usercon was the “Best Virtualisation Event Outside of VMworld!” …that was a big statement if ever there was one however, over the past couple of years I still feel like that statement holds court even though there are much bigger UserCons around the world. In fairness, both Sydney and Melbourne UserCons are solid events and even with VMUG numbers generally struggling world wide, the events are still well attended and a must for anyone working around the VMware ecosystem.

Both events happen a couple of days apart from each other on the 19th and 21st of March and both are filled with quality content, quality presenters and a great community feel.

This will be my sixth straight Melbourne UserCon and my fourth Sydney UserCon…The last couple of years I have attended with Veeam and presented a couple of times. This year Veeam has UserCon Global Sponsorship which is exciting as the Global Product Strategy team will be presenting a lot of the UserCons around the world. Both the Sydney and Melbourne Agenda’s are jam packed with virtualisation and automation goodness and it’s actually hard to attend everything of interest with schedule conflicts happening throughout the day.

…the agenda’s are listed on the sites.

As mentioned, Veeam is sponsoring both events a the Global Elite level and I’ll be presenting a session on Automation and Orchestration of Veeam and VMware featuring VMware Cloud on AWS which is an updated followup to the VMworld Session I presented last year. The Veeam SDDC Deployment Toolkit has been evolving since then and i’ll talk about what it means to leverage APIs and PowerShell to achieve automation goodness with a live demo!

Other notable sessions include:

If you are in Sydney or Melbourne next week try and get down to Sydney ICC and The Crown Casino respectively to participate, learn and contribute and hopefully we can catch up for a drink.

NSX Bytes – What’s New in NSX-T 2.4

A little over two years ago in Feburary of 2017 VMware released NSX-T 2.0 and with it came a variety of updates that looked to continue to push NSX-T beyond that of NSX-v while catching up in some areas where the NSX-v was ahead. The NSBU has had big plans for NSX beyond vSphere for as long as I can remember, and during the NSX vExpert session we saw how this is becoming more of a reality with NSX-T 2.4. NSX-T is targeted at more cloud native workloads which also leads to a more devops focused marketing effort on VMware’s end.

NSX-T’s main drivers relate to new data centre and cloud architectures with more hetrogeneality driving a different set of requirements to that of vSphere that focuses around multi-domain environments leading to a multi-hypervisor NSX platform. NSX-T is highly extensible and will address more endpoint heterogeneity in future releases including containers, public clouds and other hypervisors.

What’s new in NSX-T 2.4:

[Update] – The Offical Release Notes for NSX-T 2.4 have been releases and can be found here. As mentioned by Anthony Burke

I only touch on the main features below…This is a huge release and I don’t think i’ve seen a larger set of release notes from VMware. There are also a lot of Resolved Issues in the release which are worth a look for those who have already deployed NSX-T in anger. [/Update]

While there are a heap of new features in NSX-T 2.4, for me one of the standout enhancements is the migration options that now exist to take NSX-v platforms and migrate them to NSX-T. While there will be ongoing support for both platforms, and in my opinion NSX-v still hold court in more traditional scenarios, there is clear direction on the migration options.

In terms of the full list of what’s new:

  • Policy Management
    • Simplified UI with rich visualisations
    • Declarative Policy API to configure networking, security and services
  • Advanced Network Services
    • IPv6 (L2, L3, BGP, FW)
    • ENS Support for Edge and DFW
    • VPN (L2, L3)
    • BGP Enhancements (allow-as in, multi-path-asn relax, iBGP support, Inter-SR routing)
  • Intrinsic Security
    • Identity Based FW
    • FQDN/URL whitelisting for DFW
    • L7 based application signatures for DFW
    • DFW operational enhancements
  • Cloud and Container Updates
    • NSX Containers (Scale, CentOS support, NCP 2.4 updates)
    • NSX Cloud (Shared NSX gateway placement in Transit VPC/VNET, VPN, N/S Service Insertion, Hybrid Overlay support, Horizon Cloud on Azure integration)
  • Platform Enhancements
    • Converged NSX Manager appliance with 3 node clustering support
    • Profile based installs, Reboot-less maintenance mode upgrades, in-place mode upgrades for vSphere Compute Clusters, n-VDS visualization, Traceflow support for centralized services like Edge Firewall, NAT, LB, VPN
    • v2T Migration: In-built UI wizards for “vDS to N-vDS” as well as “NSX-v to NSX-T” in-place migrations
    • Edge Platform: Proxy ARP support, Bare Metal: Multi-TEP support, In-band management, 25G Intel NIC support
Infrastructure as Code and NSX-T:

As mentioned in the introduction, VMware is targeting cloud native and devops with NSX-T and there is a big push for being able to deploy and consume networking services across multiple platforms with multiple tools via the NSX API. At it’s heart, we see here the core of what was Nicira back in the day. NSX (even NSX-v) has always been underpinned by APIs and as you can see below, the idea of consuming those APIs with IaC, no matter what the tool is central to NSX-T’s appeal.

Conclusion:

It’s time to get into NSX-T! Lots of people who work in and around the NSBU have been preaching this for the last three to four years, but it’s now apparent that this is the way of the future and that anyone working on virtualization and cloud platforms needs to get familiar with NSX-T. There has been no better time to set it up in the lab and get things rolling.

For a more in depth look at the 2.4 release, head to the official launch blog post here.

References:

vExpert NSX Briefing

https://blogs.vmware.com/networkvirtualization/2019/02/introducing-nsx-t-2-4-a-landmark-release-in-the-history-of-nsx.html/

Configuring Amazon S3 Access from VMware Cloud on AWS through an S3 Endpoint

When looking at how to configure networking for interactions between a VMware Cloud on AWS SDDC and an Amazon VPC there is a little bit to grasp in terms of what needs to be done to achieve traffic flow between the SDDC and the rest of the world.

As an example, by default if you want to connect to S3 the default configuration is to go through the Amazon ENI (Elastic Network Interface) which means that unless configured correctly, connectively to Amazon S3 will fail. Brian Gaff has a really good series of posts on Networking and Security Groups when working on VMware Cloud on AWS and are worth a read to get a deeper understanding of VMC to AWS networking.

There is a way to change this behaviour to make connectivity to Amazon S3 connect via the SDDCs Internet Gateway. This is done through the VMware Cloud Portal by going to the Networking section of the relevant SDDC.

Doing this, while easy enough means that you loose a lot of the benefits that passing traffic through the ENI provides. That is a high-bandwidth, low latency connection between the VPC and the SDDC which also provides free egress. In the case of S3 and the utilising the Veeam Cloud Tier it means more optimal connectivity between a Veeam Backup & Replication instance hosted in the SDDC and Amazon S3.

To allow communication between the SDDC and Amazon S3 over the ENI the following needs to be actioned.

Create Endpoint:

First step is to go into the AWS Console, go to the VPC thats connected to the VMC service and create a new Endpoint for S3 as shown below making sure you select the correct Route Table.

Configure Security Group:

Next is to configure the Security Group associated with your VPC to allow traffic to the logical network or networks. It’s a basic HTTPS Inbound rule where your source is the SDDN network or networks you want access from.

Create Compute Gateway Firewall Rule:

The final step is to configure a firewall rule on the SDDC Compute Gateway to allow HTTPS traffic to the Amazon VPC from the network or networks you want access to Amazon S3 from.

That’s pretty much it! After that, you should be able to access Amazon S3 over the ENI and get all the benefits that delivers.

References:

https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws-operations/GUID-B501FA3C-EAF9-4005-AC72-155C3F592281.html

Veeam for Service Providers…Ten Plus Years of Innovation!

I remember the day I first came across Veeam. It was mid 2010 and I was working for Anittel at the time. We had a large virtualisation platform that hosted a number of high profile sites including a well known e-commerce site. There had been a serious data breach on one of those site and we were required by the Australian Federal Police to restore the website logs from a couple weeks back when the breach had first taken place.

We were using a well known product at the time to backup our vSphere platform and from the outside everything seemed ok. All backup reports where green and we thought the backups where verified. To cut a long and painful story short, when we came to restore the website logs we found that the backups had not worked as expected and we couldn’t retrieve data off a secondary partition due to a huge unknown bug in the software.

That was the end for that backup application (and interestingly enough they went out of business a few years later) and that afternoon we downloaded Veeam Backup & Replication v4 and went to work pushing that out into production. We (and I have) never looked back from there. Veeam did in fact Just Work! At that stage there were enough features in the software to cover all of the requirements for a VMware based hosting platform, and over the years as v5 and v6 were released more and more features and enhancements were released that made Veeam even better service providers.

By the time I left Anittel and headed to Zettagrid, Veeam had introduced more innovative features like Instant VM Recovery, vCloud Director Support, Cloud Connect Backup, the Scale Out Backup Repository just to name a few. In fact Veeam impressed me so much with their Service Provider features that I joined the company where I now focus my time on working with Service Providers as part of the Veeam Product Strategy Team focusing on our cloud and service providers products and features.

While I could bang on about all the features that Veeam has released over the years to enable us to become a significant player in the Cloud and Service Provider space, a picture tells a thousand words…and an interactive timeline showing just how innovative and focused Veeam has been on enabling our Cloud and Service Provider partners to succeed is priceless!

No other vendor has this track record of producing specific Cloud and Service Provider features and enhancements over the years and as you can see over the last three to five years we have moved with the industry to continue innovating in the cloud space by accelerating feature development and bringing great technology to the market.

If you are a Cloud and Service Provider and not using Veeam…what are you waiting for?

https://anthonyspiteri.net/veeam-vcsp-reverse-roadmap/

« Older Entries Recent Entries »