We recently came across an issue when playing around with VMware NSX-T which not anyone might be aware of when getting started with it. Because many of our customers start with transitioning to NSX-T, we want to share this with you. In short, the Distributed Firewall (DFW) of NSX-T can be easily bypassed in the default configuration because it only works effectively if at the same time, the SpoofGuard feature is enabled on all logical switch ports which is not the case by default.
VMware NSX-T
In NSX-T, you can interconnect different hypervisors (currently: ESXi and KVM) as so-called Transport Nodes and then configure logical switches and routers across the VMs running on different hypervisors. The hypervisors then use an overlay network to forward the VM traffic between them and configure the local virtual switches on the hypervisors to forward the network traffic accordingly. Within logical switches and routers, all network traffic is allowed by default. This can be restricted by configuring the DFW based on tags you assign to the virtual machines. So lets assume we have 2 VMs on 2 different KVM hypervisors both connected to the same logical switch. Within the VMs, we manually configure the IP addresses 192.168.42.10 and 192.168.42.11 respectively. NSX-T will use ARP snooping to discover these IP addresses and add them to the Realized Bindings of the corresponding logical switch port as shown below for the first VM:
As stated in the documentation, this is done based on trust on first use:
By default, the discovery methods ARP snooping and ND snooping operate in a mode called trust on first use (TOFU). In TOFU mode, when an address is discovered and added to the realized bindings list, that binding remains in the realized list forever.
NSX-T Distributed Firewall (DFW)
For the DFW functionality, we assign the tag KVM-VM-1 to the first VM with the IP address 192.168.42.10 and the tag KVM-VM-2 to the second VM with the IP address 192.168.42.11. We then create two groups with the same names and make all VMs with the respective tag member of the corresponding group as shown below for the KVM-VM-1 group.
Now lets create a firewall rule which drops all traffic from KVM-VM-1 to KVM-VM-2:
Now, we are not able to ping from KVM-VM-1 to KVM-VM-2 anymore:
Bypassing the NSX-T Distributed Firewall (DFW)
However, in the default configuration of NSX-T using the default Switching Profiles, we can easily bypass the DFW restrictions by changing the IP address within the VM. KVM-VM-1 had the IP address 192.168.42.10 so lets change it to 192.168.42.42 by editing the /etc/network/interfaces on our Ubuntu-Server VM:
At this point we only need to reload the network configuration, for example by rebooting the VM, and all the restrictions by the DFW will be gone. We can now ping our KVM-VM-2 again:
This means that in the default configuration, anyone who controls a VM, either an admin of a VM or an attacker gaining access to one, can bypass the network restrictions of the DFW just by changing the IP address of the VM. So if you currently use the default switching profiles of NSX-T and think you apply DFW rules as a security mechanism, actually you are not and it’s time to do something about this.
SpoofGuard
By enabling SpoofGuard on the ports of the logical switches, you can prevent VMs from sending network traffic from different IP addresses by configuring an explicit IP binding or using the implicit IP binding discovered based on trust on first use. As stated in the documentation, SpoofGuard is required for the correct functioning of the DFW:
There are several reasons SpoofGuard might be used in your environment:
[…]
Guaranteeing that distributed firewall (DFW) rules will not be inadvertently (or deliberately) bypassed
[…]
However, if you just go with the defaults and configure your DFW as shown above, you will not be warned by the NSX-T Manager in any way that the rules you configure can easily be bypassed because you have not enabled SpoofGuard on the involved switch ports.
To enable it, create a new SpoofGuard Switching Profile which has the Address Binding Whitelist set to Enabled and assing this profile to the ports on the logical switches:
Instantly, all communication from KVM-VM-1 will be blocked again by default as long as its coming from the wrong IP address. Even if we add a DFW rule allowing ICMP traffic,
it will be dropped:
Only by changing the IP address back to the one which was initially learned by NSX-T or by configuring a manual binding in the NSX-T-Manager, the traffic will even be considered to be matched against the DFW rules. In environments where VMs should be able to change their IP address frequently, maintaining appropriate manual address bindings can be a challenging task.
Conclusion
If you want to use the NSX-T DFW as a security feature for restricting network traffic between virtual machines, you also need to enable the SpoofGoard feature on the logical switches. Otherwise, DFW restrictions can be easily bypassed.
that’s valid only if you use blacklisting approach (default rule any-any-allow), however if you adopt the whitelisting approach (default rule any-any-deny), then you explicitly allow (whitelist) certain traffic and nothing else will work, as it will hit the default deny rule.
That’s the so called Zero Trust – you allow only what you need.
This is really no mystery since even physical switches using ACLs also rely on IP addresses to filter traffic. Spoof guard is absolutely the recommended way to prevent this so that MAC and IP addresses will be locked down to the Segment port, as you discovered. Nice thing about the NSX DFW is that IP filtering can be combined with L4-L7 policies as well as IDS. Any nefarious behavior would then be identified affter the DFW blocked only traffic that is desired. Thing about Spoofguard is if it’s on by default, it can cause other problems when you WANT to change IP’s. So, if you want a secure environment, simple enable it.
If you set a specific block rule for two specific IP addresses, and leave the default “any any any ALLOW” fw rule at the bottom, then of course it will not block if you change the IPs to something else.
This is by design.
I don’t know how the web interface looks right now in more recent versions, but at least in the NSX-T version back then, as far as I remember, it was not obvious in the web interface that you configure a firewall rule between IP addresses but it suggested that you configure a firewall rule between VMs (based on the tag) as the screenshots show. The intention of this post was to raise awareness of the fact that you actually do not configure rules between VMs (which would also be enforced for a malicious VM) if you do not use SpoofGuard too.