NSX-T is a Software-Defined-Networking (SDN) solution of VMware which, as its
basic functionality, supports spanning logical networks across VMs on
distributed ESXi and KVM hypervisors. The central controller of the SDN is the
NSX-T Manager Cluster which is responsible for deploying the network
configurations to the hypervisor hosts.
This summer, I looked into the mechanism which is used to add new KVM hypervisor
nodes to the SDN via the NSX-T Manager. By tracing what happens on the KVM host,
I discovered that the KVM hypervisor got instructed to download the NSX-T
software packages from the NSX-T Manager via unencrypted HTTP and install them
without any verification. This enables a Man-in-the-Middle (MITM) attacker on
the network path to replace the downloaded packages with malicious ones and
compromise the KVM hosts.
We recently came across an issue when playing around with VMware NSX-T which not
anyone might be aware of when getting started with it. Because many of our
customers start with transitioning to NSX-T, we want to share this with you. In
short, the Distributed Firewall (DFW) of NSX-T can be easily bypassed in the
default configuration because it only works effectively if at the same time, the
SpoofGuard feature is enabled on all logical switch ports which is not the
case by default.
Matthias and I
had the pleasure to give a talk at the H2HC2018 in
São Paulo, Brazil about attacking VMware NSX. The talk is an introduction to
VMware NSX for security researchers, and it discusses possible attack vectors
including the management, controlling, and data exchange planes. We demonstrated
how to prepare a fuzzing and debugging setup for the ESXi kernel and the kernel
modules. It should be noted that Olli was also
supporting the research.
The slides can be found
here.
During a recent customer project we identified several vulnerabilities in the
VMware vRealize Automation Center such as a DOM-based cross-site scripting and a
missing renewal of session tokens during the login. The vulnerabilities have
been disclosed to VMware on November 20th, 2017. A security advisory for the
vulnerabilities has been made available
here on April
12th, 2018.
Just a few words regarding the cross-site scripting vulnerability. This
vulnerability is present within a GET request to the URL /vcac/gadgets/ifr
because of certain URL parameters whose values are directly passed to an eval
function call. The vulnerable parameters are gwt:onLoadErrorFn and
gwt:onPropertyErrorFn. It seems that these parameters are actually never used
by the application and we only found them by looking at the source code.
Yesterday 7Elements released
the description
of a Remote Code Execution vulnerability in VMware vCenter. The information came
in at a good point as I’m at the moment drafting a follow-up blogpost for
this one
which will summarize some of our approaches to virtualization security. The
vCenter vulnerability is both quite critical and particularly interesting in
several ways:
Once there is proper network isolation & restriction, the vulnerability should
not be exploitable from the overall corporate network (or maybe even the
Internet — a quick inaccurate shodan search for “vcenter” returned about 1800
results and at random checks actually revealed vCenter systems). It should
also not be exploitable from ESXi hosts managed through the vCenter: ESXi
hosts need to be able to connect to the vCenter for heartbeat messages,
however “only” on ports 443 and 902 — the vulnerability exploits a service
running on TCP
ports 9875 – 9877.
It is questionable whether the exploited Java RMI functionality is really
required for the operation of VMware infrastructures. This
blogpost provides further
detail on the known type of vulnerability in Java applications. VMware had a
similar issue back in 2010,
where their workaround
to fix a vulnerability was to just disable the affected component, resulting
in the impression that it wasn’t even required in the first place. Let’s see
whether the future will bring up more vulnerabilities which could have been
prevented by implementing more thorough hardening of all components (e.g.
following the minimal machine principle). Furthermore in 2011 there was a
similar 3^(rd) party component vulnerability in vCenter which we covered
in this blogpost.
The totality of our posts on VMware security can be found
here.
For high-security environments we have been recommending for some time to use
a dedicated vCenter per hypervisor cluster (i.e. if you have two hypervisor
clusters, one for internal and one for DMZ systems, you should use two
separate vCenter systems). Vulnerabilities like these illustrate the need for
that, given that the ESXi hosts need to be able to access the vCenter on the
network level.
While fairytales often start with “Once upon a time…”, our blogposts often start
with “During a recent security assessment…” — and so does this one. This time we
were able to spend some time on VMware’s
vCenter Operations Manager
(herein short: VCOPS). VCOPS is a monitoring solution for load and health of
your vSphere environment. In order to provide this service, two virtual machines
(analytics engine and Web-based UI) must be deployed (as a so-called vApp) that
are configured on startup by various scripts (mainly
/usr/lib/vmware-vcops/user/conf/install/firstbootcommon.sh) to match the actual
environment and communicate via an OpenVPN tunnel that is established directly
between the two machines. To gather the monitoring data, read-only access to the
vCenter is required.
In the course of our virtualization research, we came across a certain technical
issue we couldn’t find an easy solution on knowledge bases and the like.
However, as we found the question several times on the web, the following post
gives just a short hint on a technical detail.
If you want to connect two virtual machines in VMware Fusion using a serial port
(e.g. for debugging purposes), Fusion doesn’t provide you an GUI option to
configure that. However, if you just add the following config to the debugger
system’s VMX file:
In our last series of posts regarding the VMDK file inclusion attack, we focused
on read access
and
prerequisites
for the attack, but avoided stating too much about potential write access. But
as we promised to cover write access in the course of our future research, the
following post will describe our latest research results.
First of all, the same
prerequisites
(which will be refined a little bit more later on) as for read access must be
fulfilled and the same
steps have to
be performed in order to carry out the attack successfully. If that is the case,
there are several POIs (Partitions Of Interest) on a ESXi hypervisor that are
interesting to include:
As announced at last week’s
#HITB2012AMS, I’ll describe
the fuzzing steps which were performed during our initial research. The very
first step was the definition of the interfaces we wanted to test. We decided to
go with the plain text VMDK file, as this is the main virtual disk description
file and in most deployment scenarios user controlled, and the data part of a
special kind of VMDK files, the Host Sparse Extends.