Some notes on VMware vCenter Operations Manager

While fairytales often start with “Once upon a time…”, our blogposts often start with “During a recent security assessment…” — and so does this one. This time we were able to spend some time on VMware’s vCenter Operations Manager (herein short: VCOPS). VCOPS is a monitoring solution for load and health of your vSphere environment. In order to provide this service, two virtual machines (analytics engine and Web-based UI) must be deployed (as a so-called vApp) that are configured on startup by various scripts (mainly /usr/lib/vmware-vcops/user/conf/install/firstbootcommon.sh) to match the actual environment and communicate via an OpenVPN tunnel that is established directly between the two machines. To gather the monitoring data, read-only access to the vCenter is required.

While we were not able to find vulnerabilities which could be exploited without system access, we found several vulnerabilities within the UI system (we were only able to spend a certain amount on vulnerability research, however we’re planning to take a closer look soon). These vulnerabilities significantly increase the impact of externally exploitable vulnerabilities and lower the difficulty of privilege escalation attacks.

The UI virtual machine is basically a bunch of Java web applications running in (a bunch of) tomcat server(s) behind an Apache httpd in mod_proxy mode. There are several relevant aspects about this system:
1) The administrative user for the web interface is also an OS user (see following items).
2) All tomcat servers run as the one administrative user (in our test lab, “admin”).
3) The admin user is allowed to sudo without a password.
4) User passwords are stored encrypted, not hashed, in a PostgreSQL database.

Taking the first three issues into account, it is comparable to running your tomcat server and the Java web app as root — and I guess we don’t have to explain why this is a bad idea (especially given the state of web application security).

Issue 4) however is more interesting. If an attacker has access to the system, s/he can connect to the PostgreSQL server and extract the stored passwords:

postgres@secondvm-external:~> /opt/vmware/vpostgres/1.0/bin/psql -d alivevm
 psql.bin (9.0.13)
 Type "help" for help.

No entry for terminal type "xterm-256color";
 using dumb terminal settings.
 alivevm=# select username,password from useraccount;
 username | password
 ---------------+-----------------------------------------------
 hostname\user | ENCRYPTED_AND_BASE64ENCODED_DATA=@

The web application uses this data to decrypt the user passwords for the connection to the vCenter. The corresponding class com.integrien.alive.common.security.Encrypt.class can be found
in /usr/lib/vmware-vcops/common/lib/alive_common.jar. The first interesting aspect is the use of two different ways of encryption/decryption: oldDCipher and newDCipher. Passwords encrypted with newDCipher can be identified by the “@” at the very end of the encrypted data, like it is shown in the listing above. newDCipher also uses a password for decryption that is stored by default at /usr/lib/vmware-vcops/user/conf/key.txt:

String homeLocation = System.getProperty("ALIVE_BASE");
 if(homeLocation == null)
 homeLocation = "/usr/lib/vmware-vcops";
 String keyFileLocation = (new StringBuilder()).append(homeLocation).append(File.separator).append("user").append(File.separator).append("conf").append(File.separator).append("key.txt").toString();

In contrast, oldDCipher uses a hard-coded key:

private static String getOldKey() {
 int minlen = 7;
 String key = "sadklbvaiuewufhjgf";
 int len = key.length();
 if(len < minlen) {
 StringBuilder sb = new StringBuilder(key);
 for(int i = len; i <= minlen; i++)
 sb.append("M");

Even though storing the password is an inevitable problem when needing it to connect to another service, the hard-coded key is an interesting finding that can be used to exploit older VCOPS installations. We modified the original class a bit to support the decryption of encrypted passwords with both algorithms when the password is known. The tool can be found here (Java) and here (Python port).

Considering  the short analysis time and the kind of findings (basically running web applications as root, hard-coded crypto keys) into account, VCOPS should be additionally hardened (which is difficult as it is a VMware soft appliance) or network-wise restricted/isolated.

Stay tuned,
Niklaus & Matthias