This post is a short wrap-up of our Troopers talk about the research we did on IBM’s General Parallel File System. If you are interested in all the technical details take a look at our slides or the video recording. We will also give an updated version of this talk at the PHDays conference in Moscow next month.
The IBM General Parallel File System is a distributed file system used in large scale enterprise environments, high performance clusters as well as some of the worlds largest super computers. It is considered by many in the industry to be the most feature rich and production hardened distributed file system currently available. GPFS has a long and really interesting history, going back to the Tiger Shark file system created by IBM 1993.
Of course, this makes it an interesting target for security research. When looking at GPFS from an implementation point of view, the Linux version is made up of three different components: User space utilities and helper scripts, the mmfsd network daemon and multiple Linux kernel modules. We (Florian Grunow and me) spent some time analyzing the internals of these components and discovered critical vulnerabilities in all of them.
Userspace Utilities
GPFS includes a wealth of user space applications and helper scripts that can be used by administrators to setup and manage the file system. A significant number of these binaries have the setuid flag set, meaning they are potential targets for a local privilege escalation. By targeting the setuid tsstatus binary and exploiting a insecure TCP based communication channel between this binary and the file system daemon mmfsd, we were able to trigger a format string vulnerability and execute a shell with root privileges. This issue was assigned CVE-2015-0197.
Kernel Module
In order to enable communication between user space libraries and the GPFS kernel module, the module generates a device named /dev/ss0 that can be accessed by unprivileged accounts. Communication is implemented using IOCTL calls and each IOCTL handler performs its own privilege checks. When looking for functionality exposed to unprivileged users, we were able to identify the IOCL call kWaitforFlock. kWaitForFlock contains a trivial arbitrary memory write that can be triggered by an unprivileged attacker. This issue was assigned CVE-2015-0199.
Even though IBM disagrees with our assessment and only considered this vulnerability a DoS, privilege escalation is possible:
Both the tsstatus bug and the kernel issue require a local user account on a single cluster node. However thanks to an interesting property of GPFS having root privileges on one node means one is able to execute commands with root privileges on all nodes in the cluster:
“Currently, a prerequisite of any IBM General Parallel File System (IBM GPFS™) installation is that all the nodes on the cluster must be able to communicate with each other using the root user ID and that this is done without a password in order to facilitate automated tasks” (IBM Developerworks)
While this makes the first two vulnerabilities quite interesting for an attacker, a bug in the network communication daemon allowing a direct compromise without the need for a local user account would be preferable of course.
Communication Daemon
By default, mmfsd listens on TCP port 1191 on all interfaces of a host. This means that a vulnerability could be exploited even when an attacker only has access to a network interface not used for GPFS communication. GPFS communicates using a stateful protocol. Nodes establish multiple persistent TCP connections between themselves. Interestingly if network shared disks are used, file access to this disk will be transferred completely unencrypted by default:
In order to start a communication with the daemon, a client has to send a valid hello packet. This includes a unique cluster id, timestamps and the IP addresses of source and destination node inside the TCP payload. As we noticed during our research the only authentication mechanism in the default installation is a IP address check to make sure the source node is part of the cluster. However, this check uses the IP address stored inside the TCP payload, which of course can easily be manipulated by an attacker.
By faking an IP address we can start communicating with the cluster as if we were a member node. By design this already allows us to manipulate files stored on network shared disks and to disrupt the normal synchronization methods used by GPFS. However we were mainly interested in a direct command execution vulnerability.
Authentication is only based on these IP addresses, which allows an attacker to talk with the daemon as if she was a legitimate part of the cluster. This opens an extremely large attack surface and by abusing a command injection vulnerability in a certain message type, we were able to execute arbitrary commands with root privileges from a remote system. Due to the GPFS architecture, this bug cannot be fixed easily and can at this point in time only be fixed by enabling SSL with client certificates for the cluster communication. After spending some time mapping the different message types we discovered the MMRemote message that included a command injection vulnerability allowing us the direct execution of arbitrary commands with root privileges. This vulnerability was assigned CVE-2015-0198.
Because the missing authentication support between cluster nodes is a design issue of the GPFS protocol the released patch does not fix this underlying issue. Instead IBM recommends all users of GPFS to enable SSL with cert based authentication for their cluster communication.
IBM’s security advisory and information as for the patches can be found here. If you any questions or feedback feel free to comment on this post or contact me or Florian directly.
– Felix