Inhalt
In a recent case, we tried to reconstruct the attacker’s activities on an ESXi hypervisor. The logs available on the system were very limited, which made it difficult to analyze the attacker’s activities. The ESXi hypervisor in particular offers detailed logs that can be used for forensic analysis if configured appropriately. The topic of incident readiness in general was covered in a previous article, which is highly recommended reading. This article focuses on hypervisors, the risks they are exposed to and how to protect them.
What is a hypervisor?
Hypervisors are essential aspects of modern hosting environments. Fewer and fewer applications are hosted on “bare metal” servers ; instead, the server hosts several virtualized servers, which in turn host the applications. This virtualization offers advantages in terms of flexibility, scalability and reliability.
These virtualized servers are hosted in a hypervisor. It is obvious that the hypervisor is an essential building block in a modern hosting environment. Not only does the operation of each virtualized server and each application within it depend on the continuous functioning of the hypervisor, it also represents a single point of failure in restricting access to these servers. It is in the nature of things that a hypervisor grants access to every server hosted in it.
Hypervisors as targets
Hypervisors have therefore been targeted by various ransomware gangs looking to exploit their functionality to wreak havoc on a company’s network, sometimes by exploiting vulnerabilities in the hypervisor. Attacks that use access to a hypervisor to encrypt all of a company’s virtual machines are known as “big game hunting”. It is therefore essential to secure and monitor the hypervisor used.
VMWare ESXi
ESXi from VMware is a class 1 hypervisor that not only includes the core functions of a hypervisor, but also basic operating system functions. It is installed directly on the hardware without an underlying operating system. Its small footprint of 150 MB makes it one of the smallest hypervisors in the world. Across all products offered, VMware holds a market share of over 61% by 2023. This publication focuses on threat actors and vulnerabilities. However, general recommendations can also be applied to other specific technologies.
Known groups and ransomware
Unsurprisingly, several ransomware groups have already started targeting the ESXi hypervisor in 2021. Among them are the following:
- Storm-0506/Black Basta
- Storm-1175/Medusa
- Octo-Tempest/Scattered Spider
- SpriteSpider/ RansomEXX/Defray777
- CarbonSpider/FIN7
- Manatee Tempest/EvilCorp
- Cicada3301
- Qilin
- Black Cat/ALPHV – ALPHV Ransomware
- Lockbit
- Kuiper/RobinHood
- ShadowSyndicate/Infra Storm/Play
- Akira
- Mallox
- Cheerscrypt/Cheers/Babuk
- BlackSuit
- Hive
- AvosLocker
- Luna
- REvil/Sodinokibi
- ViceSociety/HelloKitty
- DarkSide/BlackMatter
- GwisinLocker
- RedAlert/N13V
- Conti
Most ransomware campaigns were carried out by gaining privileged access to the hypervisor and then simply encrypting all machines hosted within it. These include this year’s Play Group attacks, Cicada3301 and last year’s Scattered Spider attack on MGM.
Special attention should be paid to the fact that most ransomware programs shut down virtual machines before encryption. However, some examples such as BlackBasta, Luna Babuk and RedAlert (where the attacker has to activate this function manually) do not. This could result in the virtual machines being damaged, making recovery impossible, even with the appropriate keys. This could be seen as an additional incentive to avoid infection in the first place.
In our most recent case, the attackers used various means to gain access to a company’s network. These included credentials obtained from marketplaces on the dark web. The attackers used legitimate remote access to the system instead of specific malware. One of their actions on this network was to attempt to access the virtual machines hosted in the ESXi hypervisor. This shows that ransomware and whole-disk encryption are not the only threats to ESXi users. Hand-on-keyboard attacks, which were the main threat to hypervisors before the advent of targeted malware, are still a danger as an attacker will not miss an opportunity if it presents itself.
Prominent weak points
Complex software always has vulnerabilities, and the ESXi hypervisor is no exception. Two known vulnerabilities that have been exploited in the past are described below.
CVE-2021-21974
CVE-2021-21974 is a heap overflow vulnerability on port 427/tcp in OpenSLP as used in ESXi that can lead to remote code execution. In 2023, this vulnerability was used to deploy the ransomware “ESXiArgs” on a large scale on ESXi servers with direct internet connection. At the time this ransomware was distributed, the infection could have been avoided by properly updating the ESXi hypervisor. More information can be found in the excellent article.
CVE-2024-37085
On July 29, 2024, Microsoft announced the discovery of CVE-2024-37085. A vulnerability in domain-joined ESXi hypervisors that allowed access with full administrator rights. At that time, it had already been used by ransomware actors Storm-0506, Storm-1175, Octo Tempest, Akira and Manatee Tempest for system encryption.
The vulnerability was that any user who was a member of an AD group called “ESX Admins” was given full administrator rights on the ESXi hypervisor. The creation of this group or the addition of users to this group is usually not monitored in AD domains and is relatively easy to do. Further information. Broadcom classified this vulnerability as medium, which led to some controversy in security circles: https://www.theregister.com/2024/07/30/make_me_admin_esxi_flaw/
Preventive measures
System administrators of ESXi hypervisors can take numerous measures to limit the attack surface and speed up recovery in the event of a compromise.
The first priority is to apply patches and updates quickly and comprehensively. The vulnerabilities listed above show what attackers can gain from unpatched vulnerabilities. Of course, by definition, there is no way to patch against zero-day vulnerabilities, but CVE-2021-21974 has shown that serious vulnerabilities can remain unpatched for years and thus be exploited by attackers. There are many reasons why companies do not implement patches. Often a lack of manpower delays the patching process, or there is a lack of awareness of the number and type of systems that need to be maintained. While the latter can be remedied with a CMDB and a vulnerability scanning and management tool, the only real solution to the former is to hire additional staff.
Another concern is the potential impact on productivity. The prospect of bringing an entire business to a standstill because of a faulty patch is daunting, and the recent IT outages associated with CrowdStrike have reinforced this perception. When weighing up the risks and considering the different responsibilities for outages with different causes, a company should always reduce its attack surface by implementing vendor-approved patches.
If patching is not feasible or possible, e.g. because the attack uses a known zero-day vulnerability, virtual patching is an alternative. Virtual patching is a technique for preventing exploits by blocking them at network level with the help of an IPS or a next-generation firewall. This leads to the topic of protection mechanisms at network level. A simple firewall can even protect vulnerable systems from attackers by removing them from their reach. Of course, hypervisors should not be visible to the Internet. They should only be visible within the corporate network. If possible, in a highly privileged network segment. Remote access can take place via a VPN. An IPS solution offers additional protection against threats from within the company’s own network.
In addition, access to the hypervisor should be restricted to authorized personnel. Personalized accounts with strong, unique passwords should be used and, if possible, a second factor should be implemented. Access authorizations should be reviewed regularly and revoked if they no longer apply. The relevance of this basic principle must be emphasized with ESXi hypervisors as they do not have privilege management. Every user with access to the machine has full administrative rights.
The importance of regular backups cannot be overestimated. It is important that these backups are not only created regularly, but are also out of the reach of potential attackers. Most malware is capable of encrypting backups stored on the hypervisor itself. Air gapping or the immutability of the backups are ways of preventing the deletion or encryption of these backups.
Not all vulnerabilities lie in the software itself. Configuration errors can also lead to vulnerabilities. In addition to measures that apply to all software, such as deactivating functions that are not used and restricting access to the system and configuration to authorized users, there are measures that apply specifically to hypervisors in general and the ESXi hypervisor in particular.
Creating and hardening the AD group “ESX Admins” protects against CVE-37085. No user should be in this group or have the ability to change it. Even if all systems are already patched, it may be advisable to take this measure to guard against shadow IT, overlooked systems and systems that will be installed in the future. The best solution, of course, is to completely separate the hypervisor from the company’s Active Directory. This not only prevents attackers from gaining access to the hypervisor as a result of a successful attack on the company’s domain, but also increases availability as access to the hypervisor is not dependent on the operation of the domain controller, which is often hosted in a virtual machine itself.
UEFI Secure Boot on the physical server and TPM 2.0 help protect the system by ensuring the integrity of the boot files and code loaded at boot time through code signing, which protects the hypervisor from tampering with the boot files. In vSphere 7.0 U2 and newer, TPM 2.0 is also used to encrypt the configuration of the ESXi host to protect it from tampering.
Protecting the boot process of a system is always a good idea. With ESXi systems , this is of additional importance as the operating system is not located on a permanent file system. Instead, the file system is recreated from VIB files each time it boots. To achieve persistence on the system, the attacker must therefore attack the boot process.
The option execInstalledOnly=true is an option that prevents foreign code from being executed on the hypervisor. It is strongly recommended to use it. However, one must be aware that using this option as a runtime setting has little to no consequences. Given the aforementioned lack of privilege management in ESXi, any user with access to the machine automatically has the ability to disable this setting. On the other hand, configuring the execInstalledOnly option in the kernel means that it will remain active until the next reboot, even if it is changed.
Attackers who can access the hypervisor routinely do so via ssh. By disabling ssh, these attacks are nipped in the bud. Since the ESXi can be managed via the web interface, there is no good reason to leave ssh permanently enabled.
Finally, hypervisors should be closely monitored. Fortunately, most hypervisor solutions offer extensive logging capabilities. Of course, these logs need to be enabled and configured to be useful. For Windows-based Hyper-V solutions, please refer again to our article on Incident Readiness. Hypervisors based on Linux and ESXi can use Syslog. For several reasons, it is advisable to forward these logs to a SIEM. This removes the logs from the immediate access of an attacker who has already gained access to the hypervisor. In addition, a SIEM enables correlation with events from other log sources as well as event monitoring and alerting in the event of abnormal or malicious events. On ESXi systems, monitoring is even more important than on other systems, as there is no EDR and every user automatically has full access to the system.
Alarms could be triggered by events such as logins, (specific) configuration changes or restarts. Of course, 24/7 monitoring and pre-planned response plans must be in place to prevent encryption in the event of a compromise.
Following these recommendations will significantly improve the security of your organization’s hypervisor and protect it from encryption. For more tips and advice on protecting an ESXi hypervisor, we also recommend this presentation from this year’s Security Fest.
