umlmon

umlmon − Concepts and Architecture

UMLMON is an advanced monitor for User Mode Linux that controls the virtual machines (VMs). In this page we will give an overview over the UMLMON concept and explain its architecture.

If you read this text for the first time, it is recommended to read the section GETTING STARTED first (near the end).

UMLMON is a complete run−time environment for User Mode Linux. This environment implies a certain directory structure where disk files and other data file are stored. As the VMs are run in chroot jails it is quite important to understand this structure, because the VMs cannot access files outside the jail.

There is a monitor daemon for every VM. This daemon creates the run−time environment and starts the VM by executing the UML kernel. The daemon has an RPC interface over which commands can be sent (e.g. the command to start the VM). The daemon also determines certain kernel arguments that are passed to the UML kernel. Although one can simply pass any argument, the daemon has special support to set up arguments for memory size, virtual disks, virtual network interfaces, and console channels in a convenient way.

The monitor daemon includes also routines to do certain administration tasks like the creation of disks.

One of the superb parts of the monitor daemon is the ability to establish bidirectional connections to virtual consoles and virtual serial lines. This allows the user to have interactive sessions on the VMs.

The UMLMON directory service is another daemon that can be used to figure out which VMs are available on the system (or better, which monitor daemons are running).

UMLMON defines a certain directory structure, so the VM can run in a chroot jail. Unlike other such jails, UML does not need many support files in it, so the jail has a very simple layout. The few needed files are automatically created by the monitor.

The basic structure is:

/var/lib/umlmon/vmname:

This directory contains all data for the VM called vmname.

/var/lib/umlmon/vmname/jail:

This is the chroot jail, i.e. the new root directory.

/var/lib/umlmon/shared:

This is a directory for shared files. These files are accessible from the jails of all VMs.

/var/lib/umlmon/vmname/jail/shared:

This is where the shared directory is mounted inside the jail.

The VM cannot escape the jail (unless it runs as privileged user). When the VM accesses files, this is done relative to the root directory of the jail.

Some directories have defined purposes, and should be used in this way. The following paths are relative to the root of the jail, because they must usually be given in this form, and not in the fully absolute form where the jail root is prepended.

/disks:

This directory may contain disk images and block devices. The suffix of the file name determines the file type: .dsk files are flat images, .cow files are copy−on−write images, and .dev files are block devices.

/tmp:

This directory is actually a tmpfs filesystem used to represent virtual memory.

/shared/disks:

These are disks shared by several VMs. For example, one can place here backing files for copy−on−write disks that are shared by several VMs. To some extent, the monitor tracks whether the files are accessed sanely, and not corrupted by uncoordinated parallel accesses.

/shared/kernels:

This directory should contain the UML kernel executables.

The monitor daemon is a program with the name "umlmon". For every VM, a separate instance of umlmon is started. (This is done by the rc script.) Once started, umlmon puts itself into the background, and wait for commands on the RPC interface (see below). The umlmon daemon runs with full root privileges, so it can prepare the environment for the VMs (among other things, the daemon must be able to mount file systems, to create block devices, and to perform network setups).

When the VM is started, umlmon forks once, and the child process establishes the chroot environment and drops all the privileges. The parent and the child processes communicate over a private pipe. The child process starts the UML kernel, and watches its activity. Usually, the child process is also the controlling terminal for the UML kernel, so it is possible to kill UML by killing the child.

By some trickery with file descriptors, the UML kernel gets also access to the pseudo terminals it needs for the console and serial channels.

The umlmon daemon has an RPC interface that can be used to control it. This is classic SunRPC. Note, however, that the portmapper is not used, and that a private directory service ("umldir") is established.

There is a Unix Domain socket over which one can send RPC requests:

/var/lib/umlmon/vmname/ctrl

By default, any user can connect to this socket. The protocol itself does not enforce authentication when used over this socket. The only way to restrict access is to set the privileges of the parent directory.

It is also possible to enable TCP for the RPC interface. An anonymous port is used. Unlike the Unix Domain socket, the TCP socket has built−in authentication. As the standard SunRPC mechanisms are either insecure or overly complex, a simple HMAC−based authentication scheme has been added. The first RPC call must perform authentication, or the connection is immediately closed.

One can set a separate password for every VM, and a master password that enables access to all VMs. (The latter may be useful for the Web interface.) These passwords have nothing to do with system accounts, and can be simply set in the configuration file.

In order to find out the TCP port of the RPC interface of a certain VM, one can query the directory service. This service is provided by the daemon umldir. Actually, umldir provides a second type of RPC interface (with very few procedures). The key procedure is get_port allowing to find out the TCP port by VM name. Unlike umlmon, umldir listens on a defined TCP port. The port must be defined in /etc/services for the name "umldir".

Because this port scheme is sometimes unhandy, umldir provides a second, multiplexing TCP socket. This is the so−called TCP redirector. This socket has also a defined name ("umlredir" in /etc/services). When connecting to it, one can announce to which VM one would like to talk, and umldir arranges that all further data over this connection are forwarded to the right monitor daemon. This way, one needs only two defined TCP ports to control all VMs on the system. This scheme is advantegous when there are firewalls or tunnels between client and server.

In order to send commands to the RPC interface, one can use the command−line tool umlwatch. It is called as

umlwatch vmname[@host]

Without "@" or with "@localhost", umlwatch uses the Unix Domain socket to establish the control connection. Otherwise, it connects to the "umldir" port on the given host, and finds out how to create the control connection.

The RPC interface is fully documented in the file umlmonctrl.x (with classic RPC IDL syntax).

The umlmon daemon prepares the allocation of virtual memory by creating a tmpfs filesystem of the appropriate size. The filesystem is mounted at the directory

/var/lib/umlmon/vmname/jail/tmp

just before the UML kernel is executed. The kernel will create the backing file for its VM memory in this directory. (This file is immediately deleted, so you cannot see it.) Furthermore, this directory also contains the "mconsole socket" over which certain control commands can be sent to the running kernel.

The umlmon daemon can establish channels that connect the virtual consoles and virtual serial lines with the outer world. The RPC interface provides procedures allowing to read data from these channels or write data into them. The umlwatch program is a simple front−end for these procedures, so one can interactively connect with the consoles, and have a user session. The data flow is usually:

termemu <−[pty]−> umlwatch <−[RPC]−> umlmon <−[pty]−> UML kernel

The termemu on the left side is where the user currently is logged in on the host system (e.g. an xterm). umlwatch (with −connect argument) passes all interactive I/O over the RPC interface to umlmon, which finally drives the consoles or serial lines in the UML kernel.

The umlmon daemon allocates pseudo terminals (pty) to communicate with the running VM kernel. A pseudo terminal is a kernel channel with two endpoints, the master device and the slave device. The file descriptor accessing the slave device is passed to the UML kernel, and the master device can be opened by umlmon itself. When one calls the mentioned RPC procedures, the master device is opened, and the data exchange is performed.

It is possible to log all data flowing from the VM to the host.

In order to support the web interface, the umlmon daemon even contains its own implementation of a terminal emulator. The command set of a normal Linux console is supported.

The UML kernel supports two kinds of virtual disks: Flat images and copy−on−write images. Flat images can be regular files containing the blocks of the disk to emulate in ascending order, or they can be real partitions of real disks. Copy−on−write images are files that are backed by flat images, and only the differences to the backing file are stored.

In order to simplify the handling of these formats, UMLMON implements a number of features. First, the umlmon daemon has administrative commands to create, delete, copy and resize images. These commands can be executed over the RPC interface (e.g. use umlwatch to perform them). Second, it is defined in which directories the disk files may be contained (see above). Third, the disk files have defined extensions: the suffix .dsk is used for flat images in regular files, .dev is used for block devices, and .cow is used for copy−on−write images. Fourth, the daemon determines the kernel arguments.

The daemon tracks which disk files are open for reading and which are open for writing. Furthermore, the daemon knows which backing files and which COW files refer to each other. The daemon prevents operations that would destroy data, e.g. because the same image is open r/w by two VMs.

The VM can create virtual network interfaces. It is, however, quite difficult to connect these interface with the real network. UMLMON tries to simplify this by providing special support for two kinds of setup.

The first setup is "host to host" networking. On the host system, a proxy network interface is created that has a point−to−point connection with the virtual interface existing within the VM. Every packet sent from the proxy interface appears at the virtual interface and vice versa. Both interfaces need IP addresses. This setup is simply configured with a single line:

eth0 = host_to_host(proxy_if=proxy1, proxy_addr=192.168.2.1, guest_addr=192.168.2.2)

The monitor performs the complete setup at the host side. The user only has to configure networking within the VM in the right way (by setting the IP address and the gateway).

Optionally, one can announce the guest address in the real network (by ARP proxying, just add host_if=eth0 to this configuration).

The disadvantage of this setup is that two addresses are needed and that the connection is restricted to IP version 4. A more powerful setup is "bridged" networking.

Here, the host administrator must already have set up a bridge interface. The bridge can remain a purely virtual LAN segment, or it can be connected with the real network. This part of the setup is outside of the scope of UMLMON. The daemon includes the necessary knowledge to attach a proxy interface to the bridge (which is again magically connected with the virtual interface within the VM). The configuration looks like:

eth0 = bridged(bridge_if=br0, proxy_if=proxy1)

For both setups the monitor manages a pool of MAC addresses. One problem of existing UML is that the MAC addresses of the proxy interfaces are randomly selected, and that the MAC addresses of the virtual interfaces are derived from the IP addresses − which might not work for some configurations. The pool contains a range of reserved MAC addresses, and when the mentioned interfaces are configured, the MAC addresses may be fetched from the pool.

User Mode Linux (UML) is a special port of the Linux kernel that does not run directly on a certain hardware platform but as a user−space process on a host system (which must be, at the time of this writing, also a Linux system). Of course, you need a UML kernel binary which is normally a statically linked executable called linux.

Furthermore, you need a disk image, i.e. the file system that will be used for the virtual system. This should be a flat image (i.e. a sequence of disk blocks) without partition table and MBR. The image has the right format when you can loop−mount it. In the following, we assume that the image has the file name root.dsk (note that UMLMON needs certain file name suffixes like .dsk to identify file types). You can use any type of filesystem (ext2, ext3, reiserfs) as long as the UML kernel binary supports it.

The disk image must have a few special properties:

There should be block devices for the ubd driver in the /dev directory. You can create them using

for n in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do
mknod <mnt>/dev/ubd$n b 98 $((n*16));
done

Here, <mnt> is the path where the disk image is mounted.

You may need to adjust <mnt>/etc/fstab to use these block devices.

The CTRL−ALT−DEL event should halt the system instead of rebooting it. Edit in <mnt>/etc/inittab the line where the string "ctrlaltdel" occurs. It should look like

ca:12345:ctrlaltdel:/sbin/shutdown −t1 −a −h now

The point is that the −h switch will halt the system.

If the system on disk image uses TLS and NPTL, you should disable that, because UML does not support these features yet (when this text was written). Often, this can be achieved my moving the library <mnt>/lib/tls away,e.g.

mv <mnt>/lib/tls <mnt>/lib/tls.disabled

This assumes, however, that the system has another version of the multi−threading library in <mnt>/lib without TLS support (i.e. "LinuxThreads"). This is the case for all major Linux distributions.

Now you need a configuration file for UMLMON. Create the file /etc/umlmon with the contents:

[GLOBAL]

[vm1]
vmuser = nobody
logfile = monitor
start = manual
kernel = /shared/kernels/linux
mem = 64
ubd0 = /disks/root.dsk
con0 = pty:con0
con1 = pty
con2 = pty
con3 = pty
con4 = pty

"vm1" is the name of the virtual machine. As vmuser you can use any valid user of the host system, instead of "nobody" this could be simply your personal user account, or a special user created for VMs. Do not use "root", however, because this is insecure.

You may wonder why there are the strange paths /shared/kernels/linux and /disks/root.dsk that do not exist. Well, they do not yet exist, but we arrange that now.

Now we start UMLMON. There is a script in /etc/init.d that performs this:

/etc/init.d/umlmon start

You get a message that a service called umldir is started. This is the directory service, i.e. a special daemon that tracks which VMs are defined on the host and that can be used to connect to these VMs over the network.

Furthermore, this script says it has started a monitor for vm1. You can see this monitor in the process listing (ps −ef) as "umlmon" process. There is one such process for every inactive VM, and two such processes for every running VM.

The VM will run in a chroot jail. The monitor prepared a directory for this purpose: /var/lib/umlmon/vm1/jail. The strange paths you observed above are interpreted relative to this directory, i.e. /disks/root.dsk is /var/lib/umlmon/vm1/jail/disks/root.dsk in reality.

Put now the files root.dsk and linux in its places:

mv root.dsk /var/lib/umlmon/vm1/jail/disks/root.dsk
mv linux /var/lib/umlmon/vm1/jail/shared/kernels/linux

If you are a very good observer, you will notice that /var/lib/umlmon/vm1/jail/shared is physically the same directory as /var/lib/umlmon/shared. This is a kind of symbolic link that isn’t a link (it is a so−called bind mount). The "shared" directory is linked into the chroot jails of all virtual machines, so files common of all VMs can be placed there. The UML kernel is an example.

Now you can start the virtual machine. The program umlwatch connects to the monitor and passes commands to it:

umlwatch vm1 −info

This outputs the current configuration, and the current runtime state of the VM.

umlwatch vm1 −start

The output of the UML kernel is logged in /var/lib/umlmon/vm1/log/con0. If this looks good, you can login:

umlwatch vm1 −connect con1

This connects to the first console, and you can start an interactive session in the VM. Use the key CTRL−] to suspend the session.

Note that it is not necessary to configure networking in order to have a login session on the VM.

To shut down the VM send the command

umlwatch vm1 −ctrlaltdel

This simulates a CTRL−ALT−DEL event. As the VM should be configured such that CTRL−ALT−DEL shuts the system down (see above), this command will initiate the shutdown.

UMLMON was written by Gerd Stolpmann.

Report bugs to gerd@gerd−stolpmann.de

You can get commercial support for UMLMON. Please ask Gerd Stolpmann <gerd@gerd−stolpmann.de>.

Copyright (C) 2005 Informatikbuero Gerd Stolpmann. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

umlwatch(1), umlmon(5), umlmon(8), umldir(8), umladmin(8)