Livepatch: Linux kernel updates without rebooting
If you run a Linux server, software patching is a task that will have to be performed on a regular basis. Although most programs can be auto-restarted with a tool like needrestart, there is one exception: the kernel.
Wouldn’t it be a nice if we could update the kernel without the mandatory reboot? Here is livepatch, the feature of the Linux kernel that makes it possible. Let’s discover how it works and if you can use it on your system.
Table of Contents
Maximize uptime with livepatch
What is live kernel patching?
Live kernel patching is the process of applying security patches to a running Linux kernel without the need for a system reboot. The implementation for Linux is named livepatch. The process of patching a live kernel is a fairly complex process. It can be compared to an open heart surgery. The patient is the kernel itself, and precision and care are needed to get things right. One wrong move and it is game over.
One benefit of live patching is the ability to postpone a reboot until scheduled maintenance can be done. This means that the availability of a system can be maximized. Another benefit is that security updates are not just installed but also active immediately. While the live patching has its own risks, at least the known vulnerabilities can be mitigated.
Requirements for the patch process
To allow live patching to work, several requirements need to be met. First of all, the kernel itself requires to support livepatch. Initial support was added in 4.x, so you need an up-to-date kernel. Secondly, your system needs a client tool to retrieve kernel patches and load them. To allow loading the kernel patches, your system needs to be configured to allow loading kernel modules. The kernel patches are typically created by the Linux distribution. It requires some expertise to know how to redirect instruction sets.
How does live kernel patching work?
There are three features that enable patching a kernel while it is running:
- Kernel probes (Kprobes)
- Function tracing (Ftrace)
- Livepatching (livepatch)
These features each have their own role and work closely together. As the process of live patching has its risks, each needs to be careful. Responsibility will be passed from one to another until the full patching cycle has finished.
The holy trinity: Kprobes, Function Tracer, and Livepatch
Let’s have a look into the three kernel features that make the process of patching possible.
Kprobes or kernel probes is a kernel feature that is used by developers to measure the Linux kernel and perform debugging. Kprobes allows to break into kernel routines and at many code addresses. This called a breakpoint and allow the developer to take an action. Such action could be to run a new set of instructions.
The next feature is named Function Tracer or Ftrace. It is a powerful framework to measure several aspects within the kernel like events and interrupts. For example, it can measure the latency of specific functions like writing to disk.
Livepatch is the third component. It is also the latest addition to the kernel. With a custom Ftrace handler, it can redirect routines and jump to a patched set of instructions.
Kernel patch creation
Live patching starts with making a patch. This means that a specific kernel function needs to be changed. The creation of the patch can be done with a tool like kpatch-build. The result is a kernel module, that is then distributed. When this module is loaded, it ensures that processes that use a particular system call are using the patched version of it. It is similar to a traffic diversion.
History of live patching implementations
Although the livepatch functionality was the last missing link to allow live patching, it took some years of development to reach this point. The first working implementation of kernel patching was Ksplice. This project was part of university research by MIT. Four students created the company Ksplice, Inc. to market this new technology. Ksplice (the company) was acquired by Oracle and sold as a separate service for their own Linux distribution.
In 2014, Red Hat created kpatch and released it under the GPLv2 license. In the same year, SUSE announced kGraft. Both technologies are very similar with some minor differences. Red Hat’s implementation stops the kernel to apply live patching, while kGraft does lazy patching. Where kGraft requires manual patch creation, kpatch allows both manual and automatic patch creation.
As the feature is in high demand, most Linux distributions offer the option only as a paid add-on. Technologies like Ksplice, kpatch, and kGraft, are commercially interesting for the vendors. The typical user of the live patching feature is willing to pay a good amount of money for it. Although there are a few exceptions, most users won’t have direct access to this technology. Slowly this might be changing, especially now that livepatch landed in the kernel.
Kernel Live Patching Core
The implementation that landed in the Linux kernel source tree is named livepatch. It is the best of both worlds from kpatch and kGraft. It is named the Kernel Live Patch Core and therefore available for everyone. Because this feature is now one of the components of the kernel, no custom patches are needed anymore.
Which distributions support live patching currently?
At this moment it is not easy to test livepatch, as not all kernels are built with support for it, or have the client tooling to add and apply patches. There are different technologies around, like kpatch, ksplice, kGraft, and livepatch. Here is a quick overview of some of the technologies used and their status.
- Arch Linux (livepatch, kpatch-git tool)
- Debian (unknown, maybe Debian 9?)
- Gentoo (kpatch or ksplice)
- Oracle Linux (ksplice)
- Red Hat Enterprise Linux 7 (kpatch or ksplice)
- SUSE (kGraft)
- Ubuntu 16.04 and higher (livepatch)
(Something outdated in this list? Let it know in the comments)
How to check if livepatch is supported with your kernel?
To check if you have support for livepatch in your kernel, check if the CONFIG_HAVE_LIVEPATCH setting is enabled. There are different ways to check for this support, depending on your Linux distribution.
zcat /proc/config.gz | grep LIVEPATCH
If it is enabled it will show you
For systems running Ubuntu, have a look in your /boot directory.
cat /boot/config-$(uname -r) | grep LIVEPATCH
Livepatch status via sysfs
If livepatch support in the kernel is enabled, then there is also another way to check it. The livepatch entries and patches can be found in the pseudo file system sysfs. Look for the /sys/kernel/livepatch directory.
ls -ld /sys/kernel/livepatch
Another option is to peek in the parent kernel directory to see all kernel-related options.
If the livepatch directory is present it means you have kernel support enabled. What is next? A patching client!
Live patching of the Linux kernel
To enable live patching, we need a client to perform this duty. The client has the instructions on how to operate on a specific kernel. As said, this is a delicate job, so it cannot be universally applied. For this reason, it is Linux distribution specific. For this blog post, we will use an Ubuntu 16.04 (LTS) system. The client utility is commercial (provided by Canonical). Fortunately, they allow free users to patch up to three systems to be live patched. Before we can use the software, we need to create a token first.
Installing the client
Using canonical-livepath (Ubuntu)
The first step is to install the livepatch utility named canonical-livepatch with snap.
sudo snap install canonical-livepatch
Then enable livepatch:
sudo canonical-livepatch enable [token]
You should get a positive response saying “Successfully enabled device. Using machine-token: [token]”. If not, have a look at the common issues at the bottom of this post.
Now that the client tool is installed, it is time to use it. Run it with the status command.
You can also use the –verbose option to see more information about any applied patches. For example which CVE was involved.
How to know if the kernel is patched properly?
So it says it is doing its job, as it is fully patched. Great, but how do we know? As this is an old kernel, we know there are some patches available.
1. Using the livepatch directory
The first option is to look in the earlier mentioned directory /sys/kernel/livepatch and see if there are any entries in it.
So it says it is doing its job, as it is fully patched. Great, but how do we know? As this is an old kernel, we know there are some patches available. The earlier referenced directory /sys/kernel/livepatch has the answer.
We can see there is a patch applied. It has the same kernel version. The last number in the directory name refers to the version number displayed in the canonical-livepatch output.
2. Using the tainted flag
Another way to know that a kernel has been patched is via /proc/sys/kernel/tainted and check if the kernel is ‘tainted’ (value higher than zero). This tells debugger and other tools that the kernel has been altered or adjusted.
To get more information who tainted the kernel, use the dmesg command.
dmesg -T | grep tainted
We can use the same command to see more information about livepatch and related details. Interestingly it shows a failed verification during our testing.
If you have your system well-hardened and disabled loading kernel modules, then livepatch won’t work. This is because a kernel module is loaded to apply the patching.
To determine if your kernel allows loading modules, have a look at the file /proc/sys/kernel/modules_disabled.
If this gives you a ‘1’, then kernel modules cannot be loaded and livepatch won’t work.
Troubleshooting livepatch errors
Connection to daemon failed (Ubuntu)
2016/10/19 17:01:26 Error executing enable.
Connection to the daemon failed: Get http://127.0.0.1/enable: dial unix /var/snap/canonical-livepatch/15/livepatchd.sock: connect: no such file or directory
This issue is most likely caused because your snapd package is outdated. Upgrade it first with
sudo apt install snapd.
Command not found (Ubuntu)
sudo: canonical-livepatch: command not found
Most likely the binary directory for snaps are not in your PATH variable defined. A workaround is referring to the tool by its full path (/snap/bin/canonical-livepatch).
Interested in learning more about live patching? Here are some good resources:
- Linux kernel documentation
- Canonical livepatch service (by Dustin Kirkland)
- LWN article about Kernel live patching
Interested in learning more about the kernel and its features? We are working on a project to get them all listed our Linux security features page.
Enjoyed this article and like to do something in return? Share it with others so more people can read it. Happy patching!
If I tested this correctly, it seems that Debian (stretch) also supports live patching currently:
# uname -a
Linux lenovo-daniel 4.7.0-1-amd64 #1 SMP Debian 4.7.6-1 (2016-10-07) x86_64 GNU/Linux
# cat /etc/issue
Debian GNU/Linux stretch/sid n l
# lsb_release -a
No LSB modules are available.
Description:Debian GNU/Linux unstable (sid)
# cat /boot/config-$(uname -r) | grep LIVEPATCH
Do you also have a client already?
I am not 100% sure, but I couldn’t easily find it by searching for it, so it seems like a no:
# apt-cache search livepatch
kpatch-build – Build Tools for Kpatch and Livepatch (https://packages.debian.org/stretch/kpatch-build)
# apt-cache search kpatch
dh-kpatches – Debhelper script to help packaging kernel patches (https://packages.debian.org/stretch/dh-kpatches)
kernel-patch-scripts – Scripts to help dealing with packaged kernel patches (https://packages.debian.org/stretch/kernel-patch-scripts)
kpatch – Runtime tools for Kpatch (https://packages.debian.org/stretch/kpatch)
kpatch-build – Build Tools for Kpatch and Livepatch (https://packages.debian.org/stretch/kpatch-build)
kpatch-dkms – DKMS module for Kpatch (https://packages.debian.org/stretch/kpatch-dkms)
Thanks! Considering it is not easy to find yet, most people won’t be able to use it. As Mehdi commented, it is in Debian Stretch.
kpatch is packaged in Debian Stretch (See https://tracker.debian.org/pkg/kpatch).
Arch Linux has the kpatch tool (as “kpatch-git”) in the Arch User Repository, so Arch Linux does have a tool – it just isn’t part of the official Arch software repository.
Thanks, will update the article.
I still get
2016/11/07 14:34:32 Error executing enable.
Connection to the daemon failed: Get http://127.0.0.1/enable: dial unix /var/snap/canonical-livepatch/17/livepatchd.sock: connect: no such file or directory
even though snapd is up to date.
Possibly you need to update more software packages first.
I am using the (free) Ubuntu livepatch service with Ubuntu 16.04. Like you, I see this error message in my logs just before a patch is applied, but AFAIK the patch still works ok e.g:
2016-11-30 14:30:18 dl1 kernel:[630069.958790] kpatch_livepatch_Ubuntu_4_4_0_47_68_generic_14: module verification failed: signature and/or required key missing – tainting kernel
2016-11-30 14:30:18 dl1 kernel:[630069.959078] livepatch: tainting kernel with TAINT_LIVEPATCH
2016-11-30 14:30:18 dl1 kernel:[630069.959080] livepatch: enabling patch ‘kpatch_livepatch_Ubuntu_4_4_0_47_68_generic_14’
2016-11-30 14:30:28 dl1 canonical-livepatch: Applied patch version 14.1 to 4.4.0-47.68-generic
To my understanding, it means the module is not signed. The “tainting” means the loaded kernel has been altered, which is obviously the goal with live patching. Make sure to check if the patch is applied. I wouldn’t be surprised if the error message disappears with future patches.
help when i try to enable livepatch it gives this error
error executing enable: cannot enable machine: Unknown Auth-Token
Did you fully configure it?