Before anything else… if you haven’t patched your Cloud Director installations as per the below table… do it now!

With that done, lets continue…

This VMware Security advisory was released on the 19th of May and related to a series vulnerability in VMware Cloud Director. This was picked up by Citadelo, an Ethical Hacking company.

Citadelo discovered a vulnerability that could allow an attacker to gain access to sensitive data and take over control of private clouds within an entire infrastructure. The vulnerability would enable a user to gain control over all customers within the cloud. It also grants access to an attacker to modify the login section of the entire infrastructure to capture the username and password of another customer.

It was reported to VMware and in quick time, they released a serious of patches as listed above. The issue was that VMware Cloud Director does not properly handle input leading to a code injection vulnerability. VMware evaluated the severity of this issue to the severity range with a maximum CVSSv3 base score of 8.8.  An authenticated actor may be able to send malicious traffic to VMware Cloud Director which may lead to arbitrary remote code execution. This vulnerability can be exploited through the HTML5- and Flex-based UIs, the API Explorer interface and API access…basically all exposed entry points.

There is a quick workaround as described in this VMwareKB.

Citadelo have an amazingly detailed blog on the exploit here, and where able to perform the following actions using the Code injection:

• View content of the internal system database, including password hashes of any customers allocated to this infrastructure.
• Modify the system database to steal foreign virtual machines (VM) assigned to different organizations within Cloud Director.
• Escalate privileges from “Organization Administrator” (normally a customer account) to “System Administrator” with access to all cloud accounts (organization) as an attacker can change the hash for this account.
• Modify the login page to Cloud Director, which allows the attacker to capture passwords of another customer in plaintext, including System Administrator accounts.
• Read other sensitive data related to customers, like full names, email addresses or IP addresses.

Just to clarify at this stage, that this is not a negative post about VMware or Cloud Director in general, it’s clearly the best Cloud Management Platform in existence and has only gone from strength to strength over the past few years. VMware provided a quick fix, and then a patch in quick time. Exploits and vulnerabilities happen all the time on almost all software and hardware based platforms. It’s mainly just a case of them being discovered and exploited by the right people with the wrong intentions to cause pain.

This is EXACTLY what used to keep me up at night!

In my past life looking after a  number of vCloud Director based (and other IaaS) cloud platforms there was always one thing that kept me up at night. That was a scenario as listed above where a malicious user somehow gained access through the exposed endpoints wreaked havoc on the tenancies and underlying management infrastructure. It was a situation that I always thought, if bad enough, there would be no coming back from. No coming back for the business… and no coming back (for a long while at least) from the personal stigma and shame that would be attached to myself as the one with ultimate responsibility for the operations and management of the platform.

I can almost say, that this was not something I wanted hanging over my head, and to a certain extent, I was happy to walk away from the responsibility when I moved to Veeam. As much as I miss most elements of operating and managing a cloud hosting platform the threat of decimation was not one I wanted to hanging over my head…

If you search around the internet you will find a lot of examples of Hosting and/or Software as a Service business decimated after being exploited. Generally speaking, the malicious user isn’t so much doing it for financial gain, there is usually a personal vendetta in play. The threat of insider attack is real and was also something that I used to have to think about. To a certain extent, the insider threat is more common as these known exploits are generally complicated and require high levels of skill to explain… as shown by Citadelo in this specific instance.

Mitigation, Backup and Planning

So what happens if the worst case scenario comes into play? Ultimately, if a malicious user is successful they are going to systematically work through all systems and platforms and delete everything so that it is almost impossible to put things back together. The first thing that needs to be considered is to educate tenants and partners of the need to backup and protect their workloads and data. It’s amazing how many tenants did not have backups of VMs when I was working at my previous companies. Basically the assumption that the cloud is backed up is a dangerous one!

Secondly it’s important to think about protecting core infrastructure and have a plan to recover from some external, air-gapped backup. This is something that is difficult to achieve in reality and has an associated opportunity cost with it. How far do you go to backup the backup. Service Provider margins are tight at the best of times, so it is no surprise that cost pressures can dictate the level of backups undertaken. Loosing customer data is one thing, but not being able to recover core systems and data for yourself is virtual suicide.

Lastly the mitigation piece needs to be your first line of defence. Locking down systems, tracking access, reducing attack surfaces all while keeping track of exploits and vulnerability across all systems and platforms is not an easy task, and often does get overlooked. We all like to think we are better than the next provider, but the reality is that I would bet my bottom dollar that there are a lot of platforms running with known exploits that are not upgraded or going to be upgraded. That is systematic and probably won’t change in the service provider space… though I have been nearly four years out of the game, so maybe attitudes and procedures have changed since then.

Conclusion 

Regardless of the platform in question, this is an important reminder that no one is safe from potential attack. It’s so important to be aware of any exploits that are reported and put in place strategies to mitigate them. Even though I am not directly responsible anymore for a hosting platform, I still am heavily invested in the service provider industry… I don’t want to see any more examples of attacks that have the potential to impact lots and lots of people!

References:

https://kb.vmware.com/s/article/79091

https://www.vmware.com/security/advisories/VMSA-2020-0010.html

https://citadelo.com/en/blog/full-infrastructure-takeover-of-vmware-cloud-director-CVE-2020-3956/