NSX-T is a Software-Defined-Networking (SDN) solution of VMware which, as its basic functionality, supports spanning logical networks across VMs on distributed ESXi and KVM hypervisors. The central controller of the SDN is the NSX-T Manager Cluster which is responsible for deploying the network configurations to the hypervisor hosts.
This summer, I looked into the mechanism which is used to add new KVM hypervisor nodes to the SDN via the NSX-T Manager. By tracing what happens on the KVM host, I discovered that the KVM hypervisor got instructed to download the NSX-T software packages from the NSX-T Manager via unencrypted HTTP and install them without any verification. This enables a Man-in-the-Middle (MITM) attacker on the network path to replace the downloaded packages with malicious ones and compromise the KVM hosts.
After disclosing this issue to VMware, they developed fixes and published the vulnerability in VMSA-2020-0023 assigning a CVSSv3 base score of 7.5.
Adding a KVM Hypervisor to NSX-T
When adding a new hypervisor as a transport node in NSX-T Manager, you simply put in the IP address of the hypervisor and provide some credentials. The NSX-T Manager will then use them to login to the hypervisor via SSH and perform the required software installations and configurations.
As can be seen from the screenshot above, providing a thumbprint/fingerprint of the hypervisor is not required. However, you should definitely do this because otherwise, a MITM attacker could impersonate the hypervisor node. This is not related to the MITM vulnerability of this post (CVE-2020-3993) but still should be considered when adding new transport nodes over potentially untrusted networks.
Tracing Setup
For tracing all steps performed by the NSX-T Manager on a KVM hypervisor via SSH, I used a very simple setup consisting of the following:
- Adding a dedicated user to the system which was provided to the NSX-T Manager exclusively
- Logging all session activity of the user via sudosh which can be analyzed with sudoreplay
- Making and comparing snapshots of the filesystem before and after installation using snapper
- Capturing the network traffic on all interfaces with tcpdump
MITM Vulnerability
After adding the KVM hypervisor as a transport node and providing SSH credentials, the commands below are executed by the supplied user to download the NSX-T binaries from the Manager and install them. First, the installation files are downloaded as a tar archive via unencrypted HTTP:
/bin/bash -c wget http://<manager-ip>;:8080/repository/3.0.0.0.0.15945873/HostComponents/linux64-bionic/nsx-lcp-3.0.0.0.0.15945874-linux64-bionic_amd64.tar.gz -O /tmp/nsx-lcp-3.0.0.0.0.15945874-linux64-bionic_amd64.tar.gz
Then, the contents of the tar file are extracted:
/bin/bash -c tar xvC /tmp/ -f /tmp/nsx-lcp-3.0.0.0.0.15945874-linux64-bionic_amd64.tar.gz
Finally, the packages are installed via dpkg -i which does not perform any signature validation:
/usr/bin/sudo dpkg -i /tmp/nsx-lcp-linux64-bionic/openvswitch-datapath-dkms_2.12.1.15861219-1_all.deb /usr/bin/dpkg -i -B /tmp/nsx-lcp-linux64-bionic/libopenvswitch_2.12.1.15861219-1_amd64.deb /tmp/nsx-lcp-linux64-bionic/openvswitch-common_2.12.1.15861219-1_amd64.deb /tmp/nsx-lcp-linux64-bionic/openvswitch-pki_2.12.1.15861219-1_all.deb /tmp/nsx-lcp-linux64-bionic/openvswitch-switch_2.12.1.15861219-1_amd64.deb /tmp/nsx-lcp-linux64-bionic/python-openvswitch_2.12.1.15861219-1_all.deb /usr/bin/dpkg -i /tmp/nsx-lcp-linux64-bionic/cvm-snmp_1.1.3_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-agent_3.0.0.0.0.15945874_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-cli_3.0.0.0.0.15945874-bionic_all.deb /tmp/nsx-lcp-linux64-bionic/nsx-exporter_3.0.0.0.0.15945874-bionic_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-host_3.0.0.0.0.15945874_all.deb /tmp/nsx-lcp-linux64-bionic/nsx-monitoring_3.0.0.0.0.15945874_all.deb /tmp/nsx-lcp-linux64-bionic/nsx-mpa_3.0.0.0.0.15945874-bionic_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-nestdb_3.0.0.0.0.15945874-bionic_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-netopa_3.0.0.0.0.15945874-bionic_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-opsagent_3.0.0.0.0.15945874-bionic_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-platform-client_3.0.0.0.0.15945874-bionic_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-proxy_3.0.0.0.0.15945874-bionic_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-python-gevent_1.3.5.15670903-2_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-python-greenlet_0.4.14.15670904-2_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-python-logging_3.0.0.0.0.15945874_all.deb /tmp/nsx-lcp-linux64-bionic/nsx-python-protobuf_2.6.1.15670901-2_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-python-utils_3.0.0.0.0.15945874_all.deb /tmp/nsx-lcp-linux64-bionic/nsx-sfhc_3.0.0.0.0.15945874-bionic_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-shared-libs_3.0.0.0.0.15945874-bionic_amd64.deb /tmp/nsx-lcp-linux64-bionic/nsx-vdpi_3.0.0.0.0.15945874-bionic_amd64.deb
No other verification of the packages is performed before installation. Therefore, when on the same network or on the network path to the Manager, a MITM attack can be performed replacing the contents of the downloaded binaries and effectively compromising the transport node.
Ideally, all communication between the transport nodes and the NSX-T manager happens on a dedicated, trusted management network anyway where the presence of a MITM attacker is rather unlikely. However, there might be situations where this is not the case and the traffic flows over potentially untrusted networks or where the management network is shared with other components. In such situations, the vulnerability might have a severe impact.
Further Notes
As I also took a look at what happens after the NSX-T packages are installed, it should be noted that the following communication to the NSX-T Manager is rather well secured against MITM attacks. More specifically, there is client and server authentication for all other communication we looked at which effectively mitigates MITM attacks.
Disclosure Timeline
2020-07-08 Vulnerability reported tothe VMware Security Response Center, start of 90 day disclosure period of the vulnerability by ERNW to VMware.
2020-09-24 VMware asks ERNW for an extension of the disclosure deadline of about 12-14 days since the fix for NSX-T version 2.5 will be deployed in the second week of October 2020.
2020-09-25 Extension of the disclosure deadline granted by ERNW.
2020-10-20 Disclosure of the vulnerability by VMware in VMSA-2020-0023.
2020-11-26 Disclosure of the vulnerability details by ERNW.