Understand and configure core dumps on Linux

Linux and core dumps

Every system needs running processes to fulfill its primary goal. But sometimes things go wrong and a process may crash. Depending on the configuration of the system a core dump is created. In other words, a memory snapshot of the crashed process is stored, usually on a disk drive. The term “core” actually refers to the old magnetic core memory from older systems. Although this type of memory is no longer being used, we still use this term on Linux systems. Enough for history, let’s configure our system to properly handle core dumps.

Disable or enable?

Most systems have core dumps enabled by default. As always, there is a tradeoff to make. This time it is between gathering data for improved stability, or limit the debug data to avoid leaking sensitive data. The first option is good for machines where unstable programs need to be investigated. The second option is better suited for production systems dealing with our precious data. Let’s check both options more in detail and learn how to configure for those scenarios.

Disable core dumps

It makes sense to disable any core dumps by default for all systems. This is because the files take up disk space and may contain sensitive data. So if you don’t need the core dumps for troubleshooting purposes, disable them.

Option 1: ulimit via configuration file

To disable core dumps we need to set a ulimit value. This is done via the /etc/security/limits.conf file and defines some shell specific restrictions.

Good to know is that there are soft and hard limits. A hard limit is something can never can be overrided, while a soft limit might only be applicable for specific users. If we would like to ensure that no process can create a core dump, we can set them both to zero. Although it may look like a boolean (0 = False, 1 = True), it actually indicates the allowed size.

* soft core 0
* hard core 0

The asterix means it applies to all users. The second column states if we want to use a hard or soft limit, followed by the columns stating the setting and the value.

Option 2: configure ulimit via profile

The values for ulimit can also be set via /etc/profile or a custom file in the /etc/profile.d directory. The latter is preferred when it is available. For example by creating a file named /etc/profile.d/disable-coredumps.sh.

echo “ulimit -c 0 > /dev/null 2>&1” > /etc/profile.d/disable-coredumps.sh

This command adds the setting to a new file and sets both the soft and hard limit to zero. Each user gets this value when logging in.

Besides ulimit settings, there are also kernel settings to consider. So choosing one of the options is the first step.

Disable setuid processes dumping their memory

Processes with elevated permissions (or the setuid bit), might be still able to perform a core dump, depending on your other settings. As these processes usually have more access, they might contain more sensitive data segments in memory. So time to change this as well. The behavior can be altered with a sysctl key, or directly via the /proc file system. Usually for permanent settings the sysctl command and configuration is used. A setting is called a ‘key’, which has a related value attached to it (also known as a key-value pair).

To disable program with the setuid bit to dump, set the fs.suid_dumpable to zero.

echo “fs.suid_dumpable=0” >> /etc/sysctl.conf

Reload the sysctl configuration to activate the change.

sysctl -p

Just want to test without making permanent changes? Use sysctl -w followed by the key=value.

Side-note: Using sysctl keys is a good way to harden the Linux kernel.

 

Enable core dumps

The primary reason to allow core dumps is for troubleshooting purposes. The dumped memory of the process can be used for debugging issues, usually by more experienced developers. A software vendor may ask to enable core dumps. Usually to discover why a process crashed in the first place and find the related routine that caused it.

Enabling core dumps is similar to disabling them, except that a few specific details should be configured. For example, if you only need a particular program to be troubleshooted, you can use soft limits. By default, we set the configured to disable it for all. This is done by using -Swhich indicates that it is a soft limit. The -c denotes the size of a core dump.

ulimit -S -c 0

Next step is to only allow ‘my-program-to-troubleshoot’ to create a core dump.

ulimit -S -c unlimited my-program-to-troubleshoot

If you want to allow all processes to use core dumps, use the line above without the program, or set a system limit in /etc/security/limits.conf

* soft core unlimited

Troubleshoot setuid binaries

If you like to troubleshoot programs with a setuid bit set, you can temporarily change the fs.suid_dumpable to 1 or 2. Setting it to 2 is preferred, as this will be similar to zero, where dumping is disabled. However, now a core dump will be readable by root only. This is a good alternative for systems with sensitive data. Setting the option to 1 is best suited for personal development systems.

 

Create normal dump files

Linux has a trick in place to capture core dumps. This is set via /proc/sys/kernel/core_pattern. Most systems will have a pipe (‘|’) in this setting to indicate that a program needs to take care of the generated data. For Ubuntu this might be Apport, and for Red Hat based systems it may be redirected to Automatic Bug Reporting Tool (ABRT).

You can temporary change this setting, by echoing “core” to that file, or use the sysctl utility.

sysctl -w kernel.core_pattern=core

An important note is that this change might not be enough. It depends also on your fs.suid_dumpable setting. A warning will be logged to your kernel logger if that is the case.

Sep 06 15:51:18 hardening kernel: Unsafe core_pattern used with suid_dumpable=2. Pipe handler or fully qualified core dump path required.

When needed set your core_pattern to a full path, optionally with parameters defining who was running it, the PID, etc.

sysctl -w kernel.core_pattern=/var/crash/core.%u.%e.%p

In this example our dumps will contain the user id, program name and process id.

Systemd and core dumps

When using a modern Linux distribution you will most likely have systemd enabled. You might need to override settings via /etc/sysctl.d/50-coredump.conf and define how and where you want to store your core dumps.

Testing your configuration

Most other tutorials just give you the settings to be configured. But how would you know things work as expected? The answer is short: testing.

Let’s create a simple program that will crash when being executed. Install gcc on your system and create a file crash.c in your home directory.

int main()
{
    return 1/0;
}

This program will start the main function and return an integer value (number). However, it is dividing 1 by zero, which is not allowed and will crash. The next step is compiling our little buggy program.

Example program to test core dumps

Our unstable little program

As you can see the compiler even shows our program contains a serious issue and displays a warning about it. Now let’s run it and see if this is the case.

Program crash with core dump

A nice crash!

From this single line we can actually learn a few things. First of all that it quit with an exception, specifically referring to floating points. This is a decimal number format for programs, so it may indicate that something happened while doing some math. Another conclusion is that the core is dumped due to the “(core dumped)” addition at the end. If core dumps were disabled, this would not appear.

Great, so with this crash above we have now a dumped file, right? Not exactly. Depending on your Linux distribution things might not as simple as it looks. Each distribution deals differently with core dumps and the default settings. Most recent Linux distributions also use systemd now and the rules have slightly been changed with that as well. Depending on your configuration, you might need to search for your core dumps. So here are some tips to ensure everything is configured correctly.

Check ulimit settings

The ulimit settings define what may happen when a program crashes. So it is safe to first check this, for both root and a normal non-privileged user.

Check hard limit for core dumps:

ulimit -H -c

Check soft limit as well:

ulimit -S -c

Check the core pattern

Use the /proc file system to gather the value and change it temporarily during testing.

cat /proc/sys/kernel/core_pattern

It might show something like this:

|/usr/share/apport/apport %p %s %c %P

In this case, a crash will be piped to the apport utility. So this means that crashes are going to be analyzed by Apport. Normally crashes are found in /var/crash, but may also be in /var/spool or /var/lib/systemd/coredump on other Linux distributions.

Check the journal (systemd)

In our case journalctl shows our crash, so that’s a start.

Sep 06 15:19:23 hardening kernel: traps: crash[22832] trap divide error ip:4004e5 sp:7fff4c2fc650 error:0 in crash[400000+1000]

Other tips

Instead of using a test program, you can also terminate an existing program.

kill -s SIGSEGV PID

If you replace PID with “$$” the current program (most likely your shell) will crash. Everything for science, right?

After checking all these settings you should be able to create a nice core dump.

Conclusion

Core dumps can be useful for troubleshooting, but a disaster for leaking sensitive data. Disable core dumps when possible, and only enable them when really needed. In such case check if the files are stored safely, so normal users can’t see the data. And indepently of what choice you made, always test if your configuration does work exactly as you expect it to work.

Do you have other tips regarding core dumps? Share them in the comments!

Lynis Enterprise

Lynis Enterprise screenshot to help with system hardening

This blog post is part of our Linux security series and the mission to get Linux and Unix-based systems more secure.

Does system hardening take a lot of time, or do you have any compliance in your company? Have a look at Lynis Enterprise.

Or start today with the open source security scanner Lynis (GitHub)


Leave a Reply

Your email address will not be published. Required fields are marked *