Livepatch: Linux kernel updates without rebooting
Maximize uptime with livepatch
If you run a Linux server, software patching is a task that will have to be performed on a weekly (or daily) basis. Although most programs can be auto-restarted with a tool like needrestart, there is one exception: the kernel. Wouldn’t it be a nice if we could patch the kernel, without the mandatory reboot? Here is livepatch, the feature of the Linux kernel that makes it possible.
Kernel Live Patching Core
Although there are more options available to limit reboots, they are commonly not available to those who not pay for running a Linux distribution. This is because technologies like Ksplice are commercially guarded by the bigger vendors. You get access is you open up your wallet.
Things are different now with livepatch. Instead of a set of custom patches op top of the Linux kernel, this is one is part of the basics now. It is named the Kernel Live Patch Core and therefore available for everyone.
What is live patching actually?
Before we dive into the specifics of using live patching, let’s get a better understanding of the complexity involved. The process of patching a live kernel is not an easy task. What happens is a little bit like open heart surgery. The patient is the kernel itself, and precision and care are needed to get things right. One wrong move and it is game over.
Live patching starts with making a patch. This means that a specific function (system call) in the kernel needs to be changed (e.g. to solve a security issue). The creation of the patch can be done with a tool like kpatch-build. The result is a kernel module, that is then distributed. When this module is loaded, it ensures that processes that use a particular system call are using the patched version of it. It is similar to a traffic diversion.
Which distributions support live patching currently?
At this moment it is not easy to test livepatch, as not all kernels are build with support for it, or have the client tooling to add and apply patches. There are different technologies around, like kpatch, ksplice, kGraft and livepatch. Here is a quick overview of some of the technologies used and their status.
- Arch Linux (livepatch, kpatch-git tool)
- Debian (unknown)
- Gentoo (unknown)
- Oracle Linux (ksplice)
- Red Hat Enterprise Linux (kpatch or ksplice)
- SUSE (kGraft)
- Ubuntu 16.04 (livepatch)
(Something outdated in this list? Let it know in the comments)
Livepatch support in Linux kernel
To check if you have support for livepatch enabled, the CONFIG_HAVE_LIVEPATCH should be there. There are different ways to check for this support, depending on your Linux distribution.
zcat /proc/config.gz | grep LIVEPATCH
If it is enabled it will show you
For systems running Ubuntu, have a look in your /boot directory.
cat /boot/config-$(uname -r) | grep LIVEPATCH
Livepatch status via sysfs
If support for the kernel is enabled, then there should be some evidence of this on disk. The livepatch entries and patches can be found in the pseudo file system sysfs which resides in /sys/kernel/livepatch and below.
Great, so you have kernel support enabled and the livepatch directory is there. What is next? A client!
Live patching of the Linux kernel
To enable live patching, we need a client to perform this duty. The client has the instructions on how to operate on a specific kernel. As said, this is a delicate job, so it can not be universally applied. For this reason, it is Linux distribution specific. For this blog post, we will use an Ubuntu 16.04 (LTS) system. The client utility is commercial (provided by Canonical). Fortunately, they allow free users to patch up to three systems to be live patched. Before we can use the software, we need to create a token first.
Installing the client
Using canonical-livepath (Ubuntu)
Install the utility with the
sudo snap install canonical-livepatch
Then enable livepatch:
sudo canonical-livepatch enable [token]
You should get a positive return saying “Successfully enabled device. Using machine-token: [token]”. If not, have a look at the common issues at the bottom of this post.
Now that the client tool is installed, it is time to use it. Run
canonical-livepatch with the status command.
You can also use the –verbose option to see more information about any applied patches. For example which CVE was involved.
Determine kernel patch status
So it says it is doing its job, as it is fully patched. Great, but how do we know? As this is an old kernel, we know there are some patches available.
1. Using the livepatch directory
The first option is to look in the earlier mentioned directory /sys/kernel/livepatch and see if there are any entries in it.
So it says it is doing its job, as it is fully patched. Great, but how do we know? As this is an old kernel, we know there are some patches available. The earlier referenced directory /sys/kernel/livepatch has the answer.
We can see there is a patch applied. It has the same kernel version. The last number in the directory name refers to the version number displayed in the canonical-livepatch output.
2. Using the tainted flag
Another way to know that a kernel has been patched is via /proc/sys/kernel/tainted and check if the kernel is ‘tainted’ (value higher than zero). This tells debugger and other tools that the kernel has been altered or adjusted.
To get more information who tainted the kernel, use the
dmesg -T | grep tainted
We can use the same command to see more information about livepatch and related details. Interestingly it shows a failed verification during our testing.
If you have your system well-hardened and disabled loading kernel modules, then livepatch won’t work. This is because a kernel module is loaded to apply the patching.
To determine if your kernel allows loading modules, have a look at the file /proc/sys/kernel/modules_disabled.
If this gives you a ‘1’, then kernel modules can not be loaded and livepatch won’t work.
Typical errors with livepatch
Connection to daemon failed (Ubuntu)
2016/10/19 17:01:26 Error executing enable.
Connection to the daemon failed: Get http://127.0.0.1/enable: dial unix /var/snap/canonical-livepatch/15/livepatchd.sock: connect: no such file or directory
This issue is most likely caused because your snapd package is outdated. Upgrade it first with
sudo apt install snapd.
Command not found (Ubuntu)
sudo: canonical-livepatch: command not found
Most likely the binary directory for snaps are not in your PATH variable defined. A workaround is referring to the tool by its full path (/snap/bin/canonical-livepatch).
Interested in learning more about live patching? Here are some good resources:
Enjoyed this article? Share it with the rest of the world.