Understand and configure core dumps on Linux

Every system needs running processes to fulfill its primary goal. But sometimes things go wrong and a process may crash. Depending on the configuration of the system a core dump is created. In other words, a memory snapshot of the crashed process is stored. The term core actually refers to the old magnetic core memory from older systems. Although this type of memory is no longer being used, we still use this term on Linux systems. Enough for history, let’s configure our Linux system to properly handle core dumps.

Linux and core dumps

Most Linux systems have core dumps enabled by default. As always, there is a tradeoff to make here. On one hand, we want to gather data for improved stability and troubleshooting. On the other, we want to limit the debug data and avoid leaking sensitive data.

The first option is good for machines where unstable programs need to be investigated, like the workstation of a developer. The second option is better suited for production systems storing or processing sensitive data.

Disable core dumps

It makes sense to disable any core dumps on Linux by default for all your systems. This is because the files take up disk space and may contain sensitive data. So if you don’t need the core dumps for troubleshooting purposes, disabling them is a safe option.

Option 1: ulimit via the configuration file

To disable core dumps we need to set a ulimit value. This is done via the /etc/security/limits.conf file and defines some shell specific restrictions.

Good to know is that there are soft and hard limits. A hard limit is something that never can be overridden, while a soft limit might only be applicable for specific users. If we would like to ensure that no process can create a core dump, we can set them both to zero. Although it may look like a boolean (0 = False, 1 = True), it actually indicates the allowed size.

• soft core 0
• hard core 0

The asterisk sign means it applies to all users. The second column states if we want to use a hard or soft limit, followed by the columns stating the setting and the value.

Option 2: configure ulimit via profile

The values for ulimit can also be set via /etc/profile or a custom file in the /etc/profile.d directory. The latter is preferred when it is available. For example by creating a file named /etc/profile.d/disable-coredumps.sh.

echo "ulimit -c 0 > /dev/null 2>&1" > /etc/profile.d/disable-coredumps.sh

This command adds the setting to a new file and sets both the soft and hard limit to zero. Each user gets this value when logging in.

Besides ulimit settings, there are also kernel settings to consider. So choosing one of the options is the first step.

Option 3: disable via systemd

When using systemd and the systemd-coredump service, change the coredump.conf file. This file is most likely located at /usr/lib/sysctl.d/50-coredump.conf. As systemd has a set of files, ensure to check the others like:

/etc/systemd/coredump.conf
/etc/systemd/coredump.conf.d/*.conf
/run/systemd/coredump.conf.d/*.conf
/usr/lib/systemd/coredump.conf.d/*.conf

Set the Storage setting to ’none’. Then configure ProcessSizeMax to limited the maximum size to zero.

[Coredump]

Storage=none
ProcessSizeMax=0

Typically it is sufficient to just reload the systemd configuration.

systemctl daemon-reload

If this still creates a core dump, then reboot the system.

Disable setuid processes dumping their memory

Processes with elevated permissions (or the setuid bit), might be still able to perform a core dump, depending on your other settings. As these processes usually have more access, they might contain more sensitive data segments in memory. So time to change this as well. The behavior can be altered with a sysctl key, or directly via the /proc file system. For permanent settings, the sysctl command and configuration is typically used. A setting is called a ‘key’, which has a related value attached to it (also known as a key-value pair).

To disable program with the setuid bit to dump, set the fs.suid_dumpable to zero.

echo "fs.suid_dumpable=0" >> /etc/sysctl.conf

Reload the sysctl configuration with the -p flag to activate any changes you made.

sysctl -p

Just want to test without making permanent changes? Use sysctl -w followed by the key=value.

Tip: Using sysctl you can tune your system and is a good way to harden the Linux kernel.

Enable core dumps

The primary reason to allow core dumps is for troubleshooting purposes. The dumped memory of the process can be used for debugging issues, usually by more experienced developers. A software vendor may ask to enable core dumps. Usually to discover why a process crashed in the first place and find the related routine that caused it.

Enabling core dumps on Linux is similar to disabling them, except that a few specific details should be configured. For example, if you only need details from a particular program, you can use soft limits. This is done by using -Swhich indicates that it is a soft limit. The -c denotes the size of a core dump.

ulimit -S -c 0

Next step is to only allow ‘my-program-to-troubleshoot’ to create a core dump.

ulimit -S -c unlimited my-program-to-troubleshoot

If you want to allow all processes to use core dumps, use the line above without the program, or set a system limit in /etc/security/limits.conf

• soft core unlimited

Troubleshoot setuid binaries

Binaries that have a setuid bit set, can run with root permissions. This special type of access needs to be restricted as much as possible. Also for the creation of core dumps, it needs to be configured properly. This is done with the sysctl fs.suid_dumpable key.

So if you like to troubleshoot programs with a setuid bit set, you can temporarily change the fs.suid_dumpable to 1 or 2. Setting it to 2 is preferred as this makes the core dumps only readable to the root user. This is a good alternative for systems with sensitive data. Setting the option to 1 is better suited for personal development systems.

Create normal dump files

One of the big mysteries with Linux systems is where the core dumps are located. Linux has a trick in place to capture core dumps. This particular setting is done via the sysctl kernel.core_pattern setting or /proc/sys/kernel/core_pattern. Most systems will have a pipe (|) in this setting to indicate that a program needs to take care of the generated data. So if you wonder where your core dump goes, follow the pipe!

Core dumps on Ubuntu systems are typically going to Apport. For Red Hat based systems it may be redirected to Automatic Bug Reporting Tool (ABRT).

You can temporarily change this setting, by echoing “core” to that file, or use the sysctl utility.

sysctl -w kernel.core_pattern=core

An important note is that this change might not be enough. It depends also on your fs.suid_dumpable setting. A warning will be logged to your kernel logger if that is the case.

Sep 06 15:51:18 hardening kernel: Unsafe core_pattern used with suid_dumpable=2. Pipe handler or fully qualified core dump path required.

When needed set your core_pattern to a full path, optionally with variables defining who was running it, the PID, etc.

sysctl -w kernel.core_pattern=/var/crash/core.%u.%e.%p

In this example, our dumps will contain the user id, program name, and process id.

Systemd core dumps

When using a modern Linux distribution you will most likely have systemd enabled. You might need to override settings via /etc/sysctl.d/50-coredump.conf and define how and where you want to store your core dumps.

Using systemd-coredump

Your kernel.core_pattern may be defined to use the systemd-coredump utility. The default path where core dumps are stored is then in /var/lib/systemd/coredump.

Testing your configuration

Most other tutorials just give you the settings to be configured. But how would you know things work as expected? You will need to test it!

Create a core dump

Option 1: Create an unstable program

Let’s create a simple program. Its primary goal is to crash when being executed and then optionally create a core dump. Install gcc on your system and create a file crash.c in your home directory.

int main()
{
    return 1/0;
}

This program will start the main function and return an integer value (number). However, it is dividing 1 by zero, which is not allowed and will crash. The next step is compiling our little buggy program.

Even the compiler shows our program contains a serious issue and displays a warning about it. Now let’s run it and see if this is the case.

From this single line, we can actually learn a few things. First of all that it quit with an exception, specifically referring to floating points. This is a decimal number format for programs, so it may indicate that something happened while doing some math. Another conclusion is that the core is dumped due to the (core dumped) addition at the end. If core dumps were disabled, this would not appear.

Great, so with this crash above we have now a dumped file, right? Not exactly. Depending on your Linux distribution things might not as simple as it looks. Each distribution deals differently with core dumps and the default settings. Most recent Linux distributions also use systemd now and the rules have slightly been changed with that as well. Depending on your configuration, you might need to search for your core dumps. So here are some tips to ensure everything is configured correctly.

Option 2: kill a running process

Instead of using a test program, you can also terminate an existing process. This is done by using the SIGSEGV, which is short for s_egmentation violation_ and also known as a segmentation fault.

kill -s SIGSEGV PID

If you replace PID with “$$” the current program (most likely your shell) will crash. Everything for science, right?

Option 3: using gdb

If you have the developer debugging tool gdb installed, then attach to a process of choice using its process ID (PID).

gdb -p 1234

Then when at the gdb prompt, generate the core dump by invoking the generate-core-file instruction. After using this command, it should return you output.

Saved corefile core.1234

Check ulimit settings

The ulimit settings define what may happen when a program crashes. So it is safe to first check this, for both root and a normal non-privileged user.

Check hard limit for core dumps:

ulimit -H -c

Check soft limit as well:

ulimit -S -c

Check the core pattern

Use the /proc file system to gather the value and change it temporarily during testing. If you prefer using sysctl, then query the kernel.core_pattern key.

cat /proc/sys/kernel/core_pattern

It might show something like this:

|/usr/share/apport/apport %p %s %c %P

In this case, a crash will be piped to the apport utility. So this means that crashes are going to be analyzed by Apport. Normally crashes are found in /var/crash, but may also be in /var/spool or /var/lib/systemd/coredump on other Linux distributions.

Check the journal (systemd)

In our case journalctl shows our crash, so that’s a start.

Sep 06 15:19:23 hardening kernel: traps: crash[22832] trap divide error ip:4004e5 sp:7fff4c2fc650 error:0 in crash[400000+1000]

After checking all these settings you should be able to create a nice core dump.

Conclusion

Core dumps can be useful for troubleshooting, but a disaster for leaking sensitive data. Disable core dumps when possible, and only enable them when really needed. In such case check if the files are stored safely, so normal users can’t see the data. And independently of what choice you made, always test if your configuration does work exactly as you expect it to work.

Do you have other tips regarding core dumps? Share them in the comments!