Livepatch: Linux kernel updates without rebooting

If you run a Linux server, software patching is a task that will have to be performed on a regular basis. Although most programs can be auto-restarted with a tool like needrestart, there is one exception: the kernel.

Wouldn’t it be a nice if we could update the kernel without the mandatory reboot? Here is livepatch, the feature of the Linux kernel that makes it possible. Let’s discover how it works and if you can use it on your system.

Maximize uptime with livepatch

What is live kernel patching?

Live kernel patching is the process of applying security patches to a running Linux kernel without the need for a system reboot. The implementation for Linux is named livepatch. The process of patching a live kernel is a fairly complex process. It can be compared to an open heart surgery. The patient is the kernel itself, and precision and care are needed to get things right. One wrong move and it is game over.

One benefit of live patching is the ability to postpone a reboot until scheduled maintenance can be done. This means that the availability of a system can be maximized. Another benefit is that security updates are not just installed but also active immediately. While the live patching has its own risks, at least the known vulnerabilities can be mitigated.

Requirements for the patch process

To allow live patching to work, several requirements need to be met. First of all, the kernel itself requires to support livepatch. Initial support was added in 4.x, so you need an up-to-date kernel. Secondly, your system needs a client tool to retrieve kernel patches and load them. To allow loading the kernel patches, your system needs to be configured to allow loading kernel modules. The kernel patches are typically created by the Linux distribution. It requires some expertise to know how to redirect instruction sets.

How does live kernel patching work?

There are three features that enable patching a kernel while it is running:

Kernel probes (Kprobes)
Function tracing (Ftrace)
Livepatching (livepatch)

These features each have their own role and work closely together. As the process of live patching has its risks, each needs to be careful. Responsibility will be passed from one to another until the full patching cycle has finished.

The holy trinity: Kprobes, Function Tracer, and Livepatch

Let’s have a look into the three kernel features that make the process of patching possible.

Kprobes

Kprobes or kernel probes is a kernel feature that is used by developers to measure the Linux kernel and perform debugging. Kprobes allows to break into kernel routines and at many code addresses. This called a breakpoint and allow the developer to take an action. Such action could be to run a new set of instructions.

Ftrace

The next feature is named Function Tracer or Ftrace. It is a powerful framework to measure several aspects within the kernel like events and interrupts. For example, it can measure the latency of specific functions like writing to disk.

Livepatch

Livepatch is the third component. It is also the latest addition to the kernel. With a custom Ftrace handler, it can redirect routines and jump to a patched set of instructions.

Kernel patch creation

Live patching starts with making a patch. This means that a specific kernel function needs to be changed. The creation of the patch can be done with a tool like kpatch-build. The result is a kernel module, that is then distributed. When this module is loaded, it ensures that processes that use a particular system call are using the patched version of it. It is similar to a traffic diversion.

History of live patching implementations

Although the livepatch functionality was the last missing link to allow live patching, it took some years of development to reach this point. The first working implementation of kernel patching was Ksplice. This project was part of university research by MIT. Four students created the company Ksplice, Inc. to market this new technology. Ksplice (the company) was acquired by Oracle and sold as a separate service for their own Linux distribution.

In 2014, Red Hat created kpatch and released it under the GPLv2 license. In the same year, SUSE announced kGraft. Both technologies are very similar with some minor differences. Red Hat’s implementation stops the kernel to apply live patching, while kGraft does lazy patching. Where kGraft requires manual patch creation, kpatch allows both manual and automatic patch creation.

Commercial offerings

As the feature is in high demand, most Linux distributions offer the option only as a paid add-on. Technologies like Ksplice, kpatch, and kGraft, are commercially interesting for the vendors. The typical user of the live patching feature is willing to pay a good amount of money for it. Although there are a few exceptions, most users won’t have direct access to this technology. Slowly this might be changing, especially now that livepatch landed in the kernel.

Kernel Live Patching Core

The implementation that landed in the Linux kernel source tree is named livepatch. It is the best of both worlds from kpatch and kGraft. It is named the Kernel Live Patch Core and therefore available for everyone. Because this feature is now one of the components of the kernel, no custom patches are needed anymore.

Which distributions support live patching currently?

At this moment it is not easy to test livepatch, as not all kernels are built with support for it, or have the client tooling to add and apply patches. There are different technologies around, like kpatch, ksplice, kGraft, and livepatch. Here is a quick overview of some of the technologies used and their status.

Arch Linux (livepatch, kpatch-git tool)
Debian (unknown, maybe Debian 9?)
Gentoo (kpatch or ksplice)
Oracle Linux (ksplice)
Red Hat Enterprise Linux 7 (kpatch or ksplice)
SUSE (kGraft)
Ubuntu 16.04 and higher (livepatch)

(Something outdated in this list? Let it know in the comments)

How to check if livepatch is supported with your kernel?

To check if you have support for livepatch in your kernel, check if the CONFIG_HAVE_LIVEPATCH setting is enabled. There are different ways to check for this support, depending on your Linux distribution.

Arch Linux

zcat /proc/config.gz | grep LIVEPATCH

If it is enabled it will show you CONFIG_HAVE_LIVEPATCH=y.

Ubuntu

For systems running Ubuntu, have a look in your /boot directory.

cat /boot/config-$(uname -r) | grep LIVEPATCH

Livepatch status via sysfs

If livepatch support in the kernel is enabled, then there is also another way to check it. The livepatch entries and patches can be found in the pseudo file system sysfs. Look for the /sys/kernel/livepatch directory.

ls -ld /sys/kernel/livepatch

Another option is to peek in the parent kernel directory to see all kernel-related options.

If the livepatch directory is present it means you have kernel support enabled. What is next? A patching client!

Live patching of the Linux kernel

To enable live patching, we need a client to perform this duty. The client has the instructions on how to operate on a specific kernel. As said, this is a delicate job, so it cannot be universally applied. For this reason, it is Linux distribution specific. For this blog post, we will use an Ubuntu 16.04 (LTS) system. The client utility is commercial (provided by Canonical). Fortunately, they allow free users to patch up to three systems to be live patched. Before we can use the software, we need to create a token first.

Installing the client

Using canonical-livepath (Ubuntu)

The first step is to install the livepatch utility named canonical-livepatch with snap.

sudo snap install canonical-livepatch

Then enable livepatch:

sudo canonical-livepatch enable [token]

You should get a positive response saying “Successfully enabled device. Using machine-token: [token]”. If not, have a look at the common issues at the bottom of this post.

Now that the client tool is installed, it is time to use it. Run it with the status command.

canonical-livepatch status

You can also use the -verbose option to see more information about any applied patches. For example which CVE was involved.

How to know if the kernel is patched properly?

So it says it is doing its job, as it is fully patched. Great, but how do we know? As this is an old kernel, we know there are some patches available.

1. Using the livepatch directory

The first option is to look in the earlier mentioned directory /sys/kernel/livepatch and see if there are any entries in it.

So it says it is doing its job, as it is fully patched. Great, but how do we know? As this is an old kernel, we know there are some patches available. The earlier referenced directory** /sys/kernel/livepatch** has the answer.

We can see there is a patch applied. It has the same kernel version. The last number in the directory name refers to the version number displayed in the canonical-livepatch output.

2. Using the tainted flag

Another way to know that a kernel has been patched is via /proc/sys/kernel/tainted and check if the kernel is ’tainted’ (value higher than zero). This tells debugger and other tools that the kernel has been altered or adjusted.

cat /proc/sys/kernel/tainted

To get more information who tainted the kernel, use the dmesg command.

dmesg -T | grep tainted

We can use the same command to see more information about livepatch and related details. Interestingly it shows a failed verification during our testing.

Livepatch caveats

If you have your system well-hardened and disabled loading kernel modules, then livepatch won’t work. This is because a kernel module is loaded to apply the patching.

To determine if your kernel allows loading modules, have a look at the file /proc/sys/kernel/modules_disabled.

cat /proc/sys/kernel/modules_disabled

If this gives you a ‘1’, then kernel modules cannot be loaded and livepatch won’t work.

Troubleshooting livepatch errors

Connection to daemon failed (Ubuntu)

$ canonical-livepatch
2016/10/19 17:01:26 Error executing enable.
Connection to the daemon failed: Get http://127.0.0.1/enable: dial unix /var/snap/canonical-livepatch/15/livepatchd.sock: connect: no such file or directory

This issue is most likely caused because your snapd package is outdated. Upgrade it first with sudo apt install snapd.

Command not found (Ubuntu)

sudo: canonical-livepatch: command not found

Most likely the binary directory for snaps are not in your PATH variable defined. A workaround is referring to the tool by its full path (/snap/bin/canonical-livepatch).

Relevant reads

Interested in learning more about live patching? Here are some good resources:

Linux kernel documentation
Canonical livepatch service (by Dustin Kirkland)
Kpatch
LWN article about Kernel live patching

Interested in learning more about the kernel and its features? We are working on a project to get them all listed our Linux security features page.

Enjoyed this article and like to do something in return? Share it with others so more people can read it. Happy patching!