Livepatch: Linux kernel updates without rebooting

Maximize uptime with livepatch

If you run a Linux server, software patching is a task that will have to be performed on a weekly (or daily) basis. Although most programs can be auto-restarted with a tool like needrestart, there is one exception: the kernel. Wouldn’t it be a nice if we could patch the kernel, without the mandatory reboot? Here is livepatch, the feature of the Linux kernel that makes it possible.

Kernel Live Patching Core

Although there are more options available to limit reboots, they are commonly not available to those who not pay for running a Linux distribution. This is because technologies like Ksplice are commercially guarded by the bigger vendors. You get access is you open up your wallet.

Things are different now with livepatch. Instead of a set of custom patches op top of the Linux kernel, this is one is part of the basics now. It is named the Kernel Live Patch Core and therefore available for everyone.

What is live patching actually?

Before we dive into the specifics of using live patching, let’s get a better understanding of the complexity involved. The process of patching a live kernel is not an easy task. What happens is a little bit like open heart surgery. The patient is the kernel itself, and precision and care are needed to get things right. One wrong move and it is game over.

Live patching starts with making a patch. This means that a specific function (system call) in the kernel needs to be changed (e.g. to solve a security issue). The creation of the patch can be done with a tool like kpatch-build. The result is a kernel module, that is then distributed. When this module is loaded, it ensures that processes that use a particular system call are using the patched version of it. It is similar to a traffic diversion.

Which distributions support live patching currently?

At this moment it is not easy to test livepatch, as not all kernels are build with support for it, or have the client tooling to add and apply patches. There are different technologies around, like kpatch, ksplice, kGraft and livepatch. Here is a quick overview of some of the technologies used and their status.

  • Arch Linux (livepatch, kpatch-git tool)
  • Debian (unknown)
  • Gentoo (unknown)
  • Oracle Linux (ksplice)
  • Red Hat Enterprise Linux (kpatch or ksplice)
  • SUSE (kGraft)
  • Ubuntu 16.04 (livepatch)

(Something outdated in this list? Let it know in the comments)

Livepatch support in Linux kernel

To check if you have support for livepatch enabled, the CONFIG_HAVE_LIVEPATCH should be there. There are different ways to check for this support, depending on your Linux distribution.

Arch Linux

zcat /proc/config.gz | grep LIVEPATCH

If it is enabled it will show you CONFIG_HAVE_LIVEPATCH=y.

Ubuntu

For systems running Ubuntu, have a look in your /boot directory.

cat /boot/config-$(uname -r) | grep LIVEPATCH

Livepatch status via sysfs

If support for the kernel is enabled, then there should be some evidence of this on disk. The livepatch entries and patches can be found in the pseudo file system sysfs which resides in /sys/kernel/livepatch and below.

Screenshot of livepatch directories in /sys/kernel

Great, so you have kernel support enabled and the livepatch directory is there. What is next? A client!

Live patching of the Linux kernel

To enable live patching, we need a client to perform this duty. The client has the instructions on how to operate on a specific kernel. As said, this is a delicate job, so it can not be universally applied. For this reason, it is Linux distribution specific. For this blog post, we will use an Ubuntu 16.04 (LTS) system. The client utility is commercial (provided by Canonical). Fortunately, they allow free users to patch up to three systems to be live patched. Before we can use the software, we need to create a token first.

Installing the client

Using canonical-livepath (Ubuntu)

Install the utility with the snap tooling.

sudo snap install canonical-livepatch

Then enable livepatch:

sudo canonical-livepatch enable [token]

You should get a positive return saying “Successfully enabled device. Using machine-token: [token]”. If not, have a look at the common issues at the bottom of this post.

Now that the client tool is installed, it is time to use it. Run canonical-livepatch with the status command.

canonical-livepatch status

Screenshot of live patching status from canonical-livepatch on Ubuntu Linux

The Linux kernel fully patched on Ubuntu

You can also use the –verbose option to see more information about any applied patches. For example which CVE was involved.

Determine kernel patch status

So it says it is doing its job, as it is fully patched. Great, but how do we know? As this is an old kernel, we know there are some patches available.

1. Using the livepatch directory

The first option is to look in the earlier mentioned directory /sys/kernel/livepatch and see if there are any entries in it.

So it says it is doing its job, as it is fully patched. Great, but how do we know? As this is an old kernel, we know there are some patches available. The earlier referenced directory /sys/kernel/livepatch has the answer.

Livepatch in action on Ubuntu Linux

We can see there is a patch applied. It has the same kernel version. The last number in the directory name refers to the version number displayed in the canonical-livepatch output.

2. Using the tainted flag

Another way to know that a kernel has been patched is via /proc/sys/kernel/tainted and check if the kernel is ‘tainted’ (value higher than zero). This tells debugger and other tools that the kernel has been altered or adjusted.

cat /proc/sys/kernel/tainted

To get more information who tainted the kernel, use the dmesg command.

dmesg -T | grep tainted

We can use the same command to see more information about livepatch and related details. Interestingly it shows a failed verification during our testing.

Dmesg tool showing that kernel is tainted by livepatch utility

Livepatch caveats

If you have your system well-hardened and disabled loading kernel modules, then livepatch won’t work. This is because a kernel module is loaded to apply the patching.

Livepatch module visible in lsmod output

To determine if your kernel allows loading modules, have a look at the file /proc/sys/kernel/modules_disabled.

cat /proc/sys/kernel/modules_disabled

If this gives you a ‘1’, then kernel modules can not be loaded and livepatch won’t work.

Typical errors with livepatch

Connection to daemon failed (Ubuntu)

$ canonical-livepatch
2016/10/19 17:01:26 Error executing enable.
Connection to the daemon failed: Get http://127.0.0.1/enable: dial unix /var/snap/canonical-livepatch/15/livepatchd.sock: connect: no such file or directory

This issue is most likely caused because your snapd package is outdated. Upgrade it first with sudo apt install snapd.

Command not found (Ubuntu)

sudo: canonical-livepatch: command not found

Most likely the binary directory for snaps are not in your PATH variable defined. A workaround is referring to the tool by its full path (/snap/bin/canonical-livepatch).

Relevant reads

Interested in learning more about live patching? Here are some good resources:

Enjoyed this article? Share it with the rest of the world.

Happy patching!

Lynis Enterprise

Lynis Enterprise screenshot to help with system hardening

This blog post is part of our Linux security series and the mission to get Linux and Unix-based systems more secure.

Does system hardening take a lot of time, or do you have any compliance in your company? Have a look at Lynis Enterprise.

Or start today with the open source security scanner Lynis (GitHub)


12 comments

  • Daniel van DorpDaniel van Dorp

    If I tested this correctly, it seems that Debian (stretch) also supports live patching currently:

    # uname -a
    Linux lenovo-daniel 4.7.0-1-amd64 #1 SMP Debian 4.7.6-1 (2016-10-07) x86_64 GNU/Linux

    # cat /etc/issue
    Debian GNU/Linux stretch/sid n l

    # lsb_release -a
    No LSB modules are available.
    Distributor ID:Debian
    Description:Debian GNU/Linux unstable (sid)
    Release:unstable
    Codename:sid

    # cat /boot/config-$(uname -r) | grep LIVEPATCH
    CONFIG_HAVE_LIVEPATCH=y
    CONFIG_LIVEPATCH=y

    Reply
  • KevinKevin

    Arch Linux has the kpatch tool (as “kpatch-git”) in the Arch User Repository, so Arch Linux does have a tool – it just isn’t part of the official Arch software repository.

    Reply
  • Siva MachinaSiva Machina

    I still get
    canonical-livepatch
    2016/11/07 14:34:32 Error executing enable.
    Connection to the daemon failed: Get http://127.0.0.1/enable: dial unix /var/snap/canonical-livepatch/17/livepatchd.sock: connect: no such file or directory

    even though snapd is up to date.

    Reply
  • Dominic RaferdDominic Raferd

    I am using the (free) Ubuntu livepatch service with Ubuntu 16.04. Like you, I see this error message in my logs just before a patch is applied, but AFAIK the patch still works ok e.g:

    2016-11-30 14:30:18 dl1 kernel:[630069.958790] kpatch_livepatch_Ubuntu_4_4_0_47_68_generic_14: module verification failed: signature and/or required key missing – tainting kernel
    2016-11-30 14:30:18 dl1 kernel:[630069.959078] livepatch: tainting kernel with TAINT_LIVEPATCH
    2016-11-30 14:30:18 dl1 kernel:[630069.959080] livepatch: enabling patch ‘kpatch_livepatch_Ubuntu_4_4_0_47_68_generic_14’
    2016-11-30 14:30:28 dl1 canonical-livepatch[672]: Applied patch version 14.1 to 4.4.0-47.68-generic

    Reply
    • To my understanding, it means the module is not signed. The “tainting” means the loaded kernel has been altered, which is obviously the goal with live patching. Make sure to check if the patch is applied. I wouldn’t be surprised if the error message disappears with future patches.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *