RancherOS: A simpler Linux for Docker lovers

Like the various Linux server and desktop distributions, the container-oriented Linux distributions mix and match various projects and components to construct a complete container infrastructure. These distros generally combine a minimal OS kernel, an orchestration framework, and an ecosystem of container services. RancherOS not only fits the mold, but takes the minimal kernel and the container paradigm to extremes.

RancherOS is a container infrastructure platform that runs Docker directly on top of a reduced footprint (20MB) Linux kernel. Rancher’s take on a minimalist OS is unique in that even the init process is a Dockerized service container. Likewise, traditional system level services, like NTP and DNS, have been replaced with containerized equivalents.

Minimal operating systems have several advantages in production. Removing unnecessary packages and libraries makes for faster booting, easier version management, and a smaller attack surface, which means fewer security patches. RancherOS takes “updates” a step further and distributes all system services as Docker containers. A security update simply means downloading a new image and restarting the container, a process that takes just a few seconds with no downtime for the service.

As an OS designed for containers, RancherOS will be hardly recognizable to someone coming from a traditional Unix background. Only a small subset of the kernel is preserved—everything else is done in containers. But if you are familiar with Docker, you’ll feel right at home in RancherOS.

Designed for Docker

To understand the RancherOS design, you need to recall that traditional Unix systems are configured infrequently, with applications layered on top of a stable base image. In a container infrastructure, the OS is in a sense disposable and will likely be changed often. Although you could use a tool such as Ansible,SaltStack, Puppet, or Chef to configure and maintain container hosts, when running at scale it is easier to use the same API for the OS as you do for containers and launch a new instance of the OS.

Thus the RancherOS strips out everything but the essentials for running Docker and hosting agents for container management platforms like Rancher or Kubernetes. In fact RancherOS is so minimal that only two users are supported: root and rancher. To understand this design, it is best to start with an illustration of the system architecture:

Rancher

RancherOS has been containerized to the point that even the init process, PID 1, from which all other processes are spawned, is a Docker daemon. Although seemingly a small detail, replacing the traditional init system such as Sysvinit or Systemd with a Dockerized init process is one of the core features of RancherOS, as it overcomes some of the incompatibilities in the architectures of Systemd and Docker in a clean way. Although progress on resolving these incompatibilities is being made by the Systemd developers, the RancherOS design ensures zero problems here, now and in the future, albeit at the cost of managing things a bit differently.

RancherOS runs two Docker daemons, the System Docker and the User Docker. All system level services such as the console, device management, NTP, and DHCP are managed by the system-docker command, while the container workloads are managed with the traditional docker command. These commands are identical, except for the types of containers they can operate on. So, if you want to see what system level services are running, you would enter system-docker ps.

If you do, you will notice that the names on the far right – Syslog, NTP, Udev, etc. – are all Linux system services. Stopping, starting, and updating a system service is handled the same way as with any other container, using the Docker API.

Note also that the system service container called docker, which is started by System Docker, is a separate Docker daemon for managing user containers. This is an important separation of privileges. Because all user containers run inside the User Docker container, deleting all user containers, for example, will not bring down the system containers running RancherOS services.

You can see what the user container workloads are just as you normally would with Docker, by entering docker ps. Even the shell is a container (the one named console), so you can choose which one you want to run. The currently available shells are BusyBox (the default), Alpine, CentOS, Debian, Fedora, and Ubuntu.

Because system services are containers, there is no package management. To upgrade a service, you simply stop the old container, pull a new version, and restart the service, all using the same Docker API.

RancherOS installation and configuration

The first thing a traditional Unix admin needs to do is stop thinking “multiuser OS” and starting thinking “infrastructure platform.” RancherOS is designed to be provisioned across a variety of environments – including bare metal, virtual machine, and a number of clouds including AWS and Google – in a predictable, automated fashion.

I installed to a virtual machine using the ISO image and did not encounter any issues in the basic install. There are no options or configuration screens. You boot the OS (which automatically logs you in), set a disk partition for RANCHER_STATE, and then reboot and configure.

If you take a look at the configuration directory, /etc, you will see there are no /etc/rc files, or much else, and what is there is not modified with a text editor. RancherOS has the equivalent of three run levels in the traditional sense that correspond to System Docker, User Docker, and Container. Configuration is done by bootcmd, which runs before System Docker, and runcmd, which runs in System Docker, executing before User Docker is started.

RancherOS is configured in two ways: manually by using the ros config command, and at boot-time using a cloud-config configuration file. I found this a bit fiddly at first, especially if you have never used cloud-config (hint: use a YAML-aware editor, like Emacs and Tramp Mode, for the initial setup). Once I was able to SSH into the machine, it was relatively easy to use ros config to get the configuration I wanted and write out the corresponding cloud-config YAML file to take effect at the next reboot. Because booting is fast, the development cycle here is also fast.

Anything you want to configure can be done with either the YAML file or ros config including loading kernel modules, TLS configuration, and kernel tuning parameters. It would be nice to have a way to preserve existing investments in configuration management tools like SaltStack and Puppet, at least partially, to ease the transition to container platforms. SaltStack’s Reactor seems like it would be good for this use case. As it is now, most people will be learning an entirely new way to manage machine configurations. Most container administrators will use the Rancher container management web UI, so this task is likely easier at that level.

RancherOS storage and networking

I mentioned previously that you install RancherOS by specifying a device to store persistent state. In most situations, this is the only disk that RancherOS will use. Because all services run in Docker containers, they will use Docker volumes for persistent storage, roughly mirroring the system architecture. System volumes provide persistent storage for system containers, user volumes for console services, and command volumes for binaries used by system services. The large number of volumes and services means that the mount command will not be much help: there is a full page of incomprehensible output. I wish the documentation explained this a bit better, because persistence is critically important topic to understand.

Rancher supports live snapshots and backup of Docker volumes, enabling users to back up stateful containers and stateful services. This is not included in RancherOS, but is part of the the Convoy feature of the Rancher container management system. Using Convoy you can take snapshots of volumes, incrementally back up snapshots to object stores such as Amazon S3, and restore volumes to running hosts.

The ZFS file system is supported, but I did not have an opportunity to try it. Given the state of the documentation and the restriction that the zpool can only be mounted on /mnt, I would recommend thorough testing before using this in production.

You can do all of the usual network configuration in RancherOS, but using YAML configuration files or ros commands. Network settings live in the rancher.network.interfaces namespace, where you configure the likes of DHCP, gateways, MTU, etc. Multiple NICs, bonding, bridges, and VLANS can be configured the same way. DNS is in the rancher.networks.dns keyspace. Until you become accustomed to the namespace mappings, expect to do a bit of digging around.

RancherOS upgrades and downgrades

In-place upgrades and downgrades could hardly be easier. You will need to upgrade (or downgrade) two systems: the OS and the Docker engine. Managing either is easy, and takes mere seconds using, you guessed it, the ros command. All you need to do is select the version of the OS you want to run and reboot. I wish all Unix upgrades and downgrades went so smoothly. For example, downgrading from 1.0.1 to 1.0.0 took less than a minute:

Rancher

Nice. If you want to switch Docker engines, you can do that just as easily:

ros engine switch docker-1.11.2

RancherOS is a nice little operating system. Its approaches to system configuration and package management will be new and different to traditional system administrators, and the documentation isn’t always what it could be. But if you know Docker, you know most of what you need to run RancherOS.

The new breed of containers is reaching maturity and production deployment of containers is a genuine possibility for early adopters. RancherOS makes it easy to build a container hosting infrastructure, but new skills are required.