As Kai and I will be holding a TROOPERS workshop on automation with ansible, we needed a setup for the attendees to use ansible against virtual machines we set up with the necessary environment. The idea was, that every attendee has their own VMs to run ansible against, ideally including one to run ansible from, as we want to avoid setup or version incompatibilities if they set up their own ansible environment on their laptop. Also they should only be able to talk to their own machines, thus avoiding conflicts because of accidental usage of wrong IPs or host names but also simplify the setup for the users.
This left us with the following requirements:
- Configurable number of attendees
- Configurable number of VMs for each attendee
- Easy automatic generation of VMs
- VMs in separated subnets
- Internet access for both VMs and attendees
Since we are doing the workshop in ansible, it was pretty clear that we wanted to do the underlying automation in ansible as well. On the one hand to freshen up our ansible (there have been plenty of changes in recent versions) and to be able to show which cool things were possible.
What we did
The first question was which virtualization technology to use. As ansible is an open source tool and we wanted to use an open source stack to keep it re-usable for eveyone. Vmware and Hyper-V would have been options for there are existing ansible modules as well, but we wanted to keep the setup completely open source. For lightweight setups like this one containers would have been a good choice, too but we decided to opt for libvirt with qemu/kvm on a Linux system. I’m skipping the argumentative to and fro about distributions, saying that we used a Debian 9 server as hypervisor system.
The base installation of the hypervisor was easy, the general configuration also was done via ansible to be able to re-use as much of the setup for other workshops without having to do much by hand again. Basic package installations and the creation of users for each of us and we were good to go. Installing the libvirt daemon and enabling ip_forward and we could test-spawn our first VM. Works like a charm <3
The default libvirt virtual networks do a great job and work really well, but as we wanted to automate a complex setup, we opted to do the whole network creation and management ourselves. It would probably also work with only libvirt networking, but we just wanted complete control over the configuration. This setup is IPv4 only in the first approach, but the addition of IPv6 capabilities will be added. If not until TROOPERS 18 then until next year at the latest.
How we did it
Now what did we need to have configurable? Of course the number of attendees and VMs, but it would be good to have the network parameters configurable as well. So the variables we feed the ansible hypervisor setup role with are:
- Number of users
- VMs per user
- Nameserver and search domain
- Names of external and internal network interfaces
- Names of management and default bridge interfaces
- VLAN offset
These are the facts that influence the base setup without which VM creation is impossible. So the prerequisites for creating the VMs were not only a working libvirt instance, but most important the network setup. As the VMs would be spawned in parallel they needed to be configured correctly from the start, avoiding additional overhead of moving or changing things afterwards.
Each user would be assigned an ID, as would be each VM, creating a number of hosts named after the scheme “u01vm01, u01vm02, u02vm02, …” so it would be possible to uniquely identify the machines and have an easy numeric scheme to address and count by during the automation process.
These hostnames, together with their corresponding IDs as variables, are put into the ansible hosts file so we can access the information and talk to the machines after creation. I will not go into detail on ansible mechanics here, since this post is already rather long – if you want to know how ansible works I know a great opportunity to learn about it at TROOPERS 18 *wink*
Network Setup
The setup we had in mind was something like using a /16 and assigning a /24 to each user where they could have their laptop and VMs in the same network and everyone uses the hypervisor itself as their router. This way we would need only a singe IP in the TROOPERS network for our workshop and don’t need to overload the network buildup crew with complicated additional setup requests. Also, this would be much easier to use for future workshops in unknown environments.
What we did to achieve this, was create empty bridge interfaces to later on feed to the libvirt and kvm, using the 10.10.0.0/16 IPs from the RFC1918 ranges and then simply apply the user ID as /24 – numbering the users from 1 to #attendees and reserving 0 for the trainer(s). This bridge would then be assigned to a VLAN that can later on be passed to the workshop switch, so each attendee has their own VLAN on their preconfigured access ports.
This is the template we used to create the corresponding interfaces in /etc/network/interfaces.d
# VLAN Definition
auto {{ eth_internal }}.{{ vlan_offset|int + br_id|int }}
iface {{ eth_internal }}.{{ vlan_offset|int + br_id|int }} inet manual
vlan-raw-device {{ eth_internal }}
# Bridge Definition
auto {{ br_name }}
iface {{ br_name }} inet static
address 10.10.{{ br_id }}.1
netmask 255.255.255.0
bridge_ports {{ eth_internal }}.{{ vlan_offset|int + br_id|int }}
bridge_maxwait 0
To get something up and running on these interfaces we obviously needed to actually assign corresponding IP addresses to the VMs. The easiest way to do this seemed to be using DHCP and utilizing static leases to be sure each VM would reliably be reachable where we wanted it to be. The attendee laptops could be assigned dynamic leases, since we don’t care where they are as long as they are in the same subnet.
An excerpt of the template for isc-dhcp-server dhcpd.conf
### Global Options ###
option domain-name "{{ domain }}";
option domain-name-servers {{ nameserver }};
### VM Subnets ###
{% for id in range(num_users) %}
subnet 10.10.{{ id + 1 }}.0 netmask 255.255.255.0 {
range 10.10.{{ id + 1}}.10 10.10.{{ id + 1}}.20;
option routers 10.10.{{ id + 1}}.1;
}
{% endfor %}
### Hosts ###
{% for host in groups['vms'] %}
host {{ host }} {
hardware ethernet 52:54:00:1e:{{ "%.02d" | format(hostvars[host]['user_id']|int) }}:{{ "%.02d" | format(hostvars[host]['vm_id']|int) }};
server-name "{{ host }}.{{ domain }}";
fixed-address 10.10.{{ "%.01d" | format(hostvars[host]['user_id']|int) }}.1{{ "%.02d" | format(hostvars[host]['vm_id']|int) }};
option host-name "{{ host }}";
}
{% endfor %}
As you can see, here we fill in the global options concerning nameserver and domain, then we create the subnets with the lease ranges for attendee laptops and for each host in the “vms” group of the inventory generate a static lease and add the option to provide it with its host name. We are going to use that information during the actual setup of the VM so we don’t have to do any templating there. The formatting is actually a little ugly, because we always use the ID in two-digit form. To fill in the MAC address, we need them like that and to fill in the IP address we need to shorten the numbers below ten to their one-digit representation. So however you do it, one of these times formatting gets a little messed up.
There is also a br00 and mgmt0 device. The br00 serves for the purpose of trainer VMs and laptops, allowing machines to be added for demonstration purposes – either with the other VMs or statically deploy an image for specific demonstrations. It also allows the trainers to connect to all trainee interfaces and machines, thus being able to properly debug any problems that may arise. The management interface is reachable from each VM to allow services on the hypervisor to be configured globally for all VMs, in our case mainly a caching proxy for the base system installation of the training machines.
The only thing left now was restricting or allowing the corresponding networks via iptables and creating a NAT rule to allow internet access for the attendees and their machines. As this is only a workshop setup and doesn’t need to be absurdly tight, we went with a minimal set of iptables rules deployed via ansible template once again.
These are only an excerpts showing the most basic rules for our setup:
# User Bridges
{% for id in range(num_users) %}
-A INPUT -i br{{ "%.02d" | format((id|int)+1) }} -j ACCEPT
{% endfor %}
# Allow bridges to themselves and external
{% for id in range(num_users) %}
-A FORWARD -i br{{ "%.02d" | format((id|int)+1) }} -o br{{ "%.02d" | format((id|int)+1) }} -j ACCEPT
-A FORWARD -i br{{ "%.02d" | format((id|int)+1) }} -o {{ eth_external }} -j ACCEPT
{% endfor %}
# Basic NAT rule
-A POSTROUTING -o {{ eth_external }} -j MASQUERADE
Virtual Machine Creation
To actually create the virtual machines – now that all the requirements are met – we use a pre-configured Ubuntu installer. This is a really easy process working for all Debian and Ubuntu versions. You just have to write a preseed file filling in the answers to every question you don’t want the installer to ask interactively (so in our case every single one of them) and re-build the installer’s initrd to contain the preseed.cfg in the root directory. Some time ago I built a little wrapper together with a very good friend of mine, extending the process to automatically build the initrd and include an addon folder and additional files as needed. This way we can also copy some basic configuration like ssh keys, authorized_keys files and even things like bashrc and similar. This additional configuration is optional, but deploying authorized_keys file is extremely helpful in our (and nearly every) case here.
An excerpt of the preseed configuration:
### Localization
d-i debian-installer/locale string en_US
### Network configuration
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string ansible
d-i netcfg/get_domain string troopers
### Mirror settings
d-i mirror/http/hostname string 10.0.0.1:9999
d-i mirror/http/directory string /ubuntu
### User Configuration
d-i passwd/username string default
d-i passwd/user-password password insecure
d-i passwd/user-password-again password insecure
### Clock and time zone setup
d-i clock-setup/utc boolean true
d-i time/zone string Europe/Berlin
### Partitioning
d-i partman-auto/disk string /dev/vda
d-i partman-auto/method string regular
d-i partman-auto/expert_recipe string \
myroot :: \
10240 4096 20480 ext4 \
$primary{ } \
$bootable{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext4 } \
mountpoint{ / } \
.
Now the only thing missing is actually telling kvm and the libvirtd about our VM and feeding it with the installation kernel and initrd. To do that we utilize the libvirt ansible module calles “virt” and provide a standardized xml description of our machine to kvm. In our setup, we use logical volumes as raw storage for the VMs, but the same setup could be achieved using e.g. qcow2 image files without making any difference to the resulting setup.
Excerpts from the ansible call and the virtual machine XML definition:
### From ansible create_vm.yml ###
- name: Virtual Machine definition
virt:
command: define
name: "{{ host }}"
xml: '{{ lookup("template", "firstboot.xml.j2") }}'
### From firstboot.xml ###
<domain type='kvm'>
<name>{{ host }}</name>
<memory unit='KiB'>1048576</memory>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='x86_64'>hvm</type>
<kernel>/var/lib/libvirt/images/linux</kernel>
<initrd>/var/lib/libvirt/images/initrd.gz</initrd>
<cmdline>auto=true noprompt</cmdline>
<boot dev='hd'/>
</os>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/guest/{{ host }}'/>
<target dev='hda' bus='virtio'/>
</disk>
<interface type='network'>
<mac address='52:54:00:1e:{{ "%.02d" | format(hostvars[host]['user_id']|int) }}:{{ "%.02d" | format(hostvars[host]['vm_id']|int) }}'/>
<source network='br{{ "%.02d" | format(hostvars[host]['user_id']|int) }}'/>
<model type='virtio'/>
</interface>
</devices>
</domain>
As you can see, the above configuration is named “firstboot”. To activate (boot) the VMs after the installation we need to re-define them with a final configuration without the direct kernel boot parameters pointing to the installer kernel and initrd or else we end up in an installer boot loop.
Optimization
To optimize the setup for better RAM and load distribution, we used ksmtuned to utilize kernel same-page merging. This is a mechanism allowing the kernel to merge identic pages in the RAM and only copy them for a virtual machine as they are changed. As we are spawning a lot of completely identical virtual machines, this saves a lot of space and has a reasonable impact on performance. We set the threshold for activation of KSM to 50% RAM usage.
Furthermore, we changed the IO scheduler to deadline, which had a massive impact on the performance of the installation process, as the write options were orchestrated better. As shown above in the XML excerpt, we also disabled caching within the VMs completely and set the IO mode to “native”, so the VMs would pass through the write actions directly to the hypervisor. Together with the underlying RAID0 for the VM volumes this was another reasonable impact.
To further optimize the installation process, we installed approx, a specialized caching proxy for Debian and Ubuntu package installation caching. This can be seen above in the preseed configuration, where the mirror is set to 10.0.0.1:9999. The IP is the management interface address of the hypervisor and the main reason for the existence of said interface, 9999 is the default port for approx. This way, the packages were not downloaded from the internet by every installed host but could be passed on directly from the hypervisor, minimizing network traffic. They are downloaded or updated when requested for the first time and then the cached version is delivered from the next request on. Only the first installed machine experiences some latency, the others can be installed a lot faster.
After all this optimization 50 virtual machines would be completely ready and configured for use after about 2:30 hours. And we have not yet touched optimization on the hardware RAID controller, where there currently is no write cache enabled, assuming this will give another boost when added.
Delivering Exercises
The only thing left to do now is delivering the exercises to the trainees. For that purpose we set up a git repository on the hypervisor, which can then simply be cloned by the attendees on their VMs using the ssh key deployed during the installation. During the workshop, we just push the current exercise to the repository thus making it instantly available for the attendees.
This is – apart from an ansible update necessary because Ubuntu 17.10 does not yet serve ansible version 2.4 – the only thing we need to roll out on the VMs with ansible after the installation process is complete. Even then, this is a mere convenience as the attendees could also issue the git clone command themselves, but we wanted to save time and thought this to be a good initial demonstration of ansible capabilities.
Learn more …
If you want to learn more about how to do cool stuff like this with ansible, we look forward to seeing you at TROOPERS 18 – there are still some spots availabe for the workshop!