Hardening nginx with systemd security features

Introduction

Nginx is still a popular web server and powering a part of the web. Wouldn’t it be great if we could secure it a little bit more? In this article we use the security features to secure systemd units and services and apply it to nginx.

If you are not familiar yet with the unit settings of systemd, then this document would be a good introduction into the subject. Another useful resource is the nginx hardening guide, which has a focus on the nginx configuration itself.

File paths

Like most services, nginx uses files to read and write. Typically web site content is read from the disk, while log files are created and written to. We can use the systemd unit settings to restrict access to only those paths that are strictly required for nginx to run. But how to find out what those paths are?

Configuration files

The first step is to look at the paths configured in the nginx configuration. Usually the /etc/nginx/nginx.conf is a good start.

Next step is following any configuration includes. If they are located in or as a subdirectory of /etc/nginx, then we can write down this path as well. Configuration files are typically read-only by the nginx process itself. Only the system administrator makes changes to it, but not from within the nginx process.

Log files

Log files are fairly easy with nginx, as they are mentioned in the configuration files.

Another common path is /var/log/nginx. Log files require write permissions, otherwise we can’t append new log entries to it.

Content

A web server without content is just a boring service. The locations of the content is typically defined within the configuration files. In most cases we need read-only access to these locations as well.

Other types or locations

Now we might already have discovered 90% of our locations, but maybe there are more?

We can use the strings command to discover words in a file, including a compiled binary. That is very useful, as on a Linux systems paths typically start with a slash followed by three characters (e.g. /var).

# strings /usr/sbin/nginx | grep -E "^/[a-z]{3}"
/lib64/ld-linux-x86-64.so.2
/var/log/nginx/error.log
/etc/nginx/
/usr/share/nginx/
/etc/nginx/nginx.conf
/run/nginx.pid
/var/lock/nginx.lock
/dev/null
/var/lib/nginx/body
/var/log/nginx/access.log
/temp
/index.html
/var/lib/nginx/proxy
/var/lib/nginx/scgi
/var/lib/nginx/uwsgi
/var/lib/nginx/fastcgi

In this output we can see the default log files, a library (in /lib64), a configuration path /etc/nginx, a path for the modules and content (/usr/share/nginx), and a device (/dev/null). With this information we now have a better.

Step by step approach

Before we start adding our paths, let’s practice first what would happen if we tighten down our security profile a little bit too much. This way we know what to expect and know the actions that we can take to troubleshoot if our service is no longer working.

Known good state

First we want to make sure that everything is working as expected, by restarting nginx. This way we have a “known good” state and know what to expect after restarting the nginx service in the future.

systemctl restart nginx.service

Does it run?

# pidof nginx
36836 36835

We see two processes run, so that is good. Now we look in the journal to see what “good” output looks like.

# journalctl -u nginx.service
Jun 17 17:40:23 test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 17 17:40:23 test systemd[1]: Started A high performance web server and a reverse proxy server.

Time to make changes and on purpose reach a failed state!

Failing on purpose

We start the configured system editor by using systemctl with the edit subcommand.

systemctl edit nginx.service

Important: when making changes, don’t remove the comment lines, only apply changes between the comment lines. This way an override file is created, then is then combined with the vendor-supplied configuration file.

Add ProtectSystem setting with the ‘strict’ value.

[Service]
ProtectSystem=strict

Save the file, restart nginx.

# systemctl restart nginx.service 
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.

That is not surprising, so let’s have a look what failed.

Jun 17 17:47:27 test nginx[36880]: nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (30: Read-only file system)
Jun 17 17:47:27 test nginx[36880]: 2024/06/17 17:47:27 [emerg] 36880#36880: open() "/run/nginx.pid" failed (30: Read-only file system)
Jun 17 17:47:27 test nginx[36880]: nginx: configuration file /etc/nginx/nginx.conf test failed
Jun 17 17:47:27 test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 17 17:47:27 test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 17 17:47:27 test systemd[1]: Failed to start A high performance web server and a reverse proxy server.

The first three lines in this output give a few good hints. The first one is that a log file could not be created or written to. The second one is that it could not create a file to store the process ID. Let’s give nginx write access to those paths using the ReadWritePaths setting.

[Service]
ProtectSystem=strict
ReadWritePaths=/run /var/log/nginx 

Restart the nginx service and if all is good, no error should be shown. If that is the case, our first pieces of hardening is done!

Hardening measures

Basics

Nginx uses a master process with one or more workers. These workers are child processes and need to be spawned using the system call fork(2).

To avoid making the profile way too complicated or non-functioning, the generic system call set @system-service is used. This way nginx can use common systems functions, such as writing to files, forking itself, define process priority, and receiving process signals.

Related system calls for basic functionality

• bind() - Assigns address to a socket that was created with socket()
• dup2() - Same as dup(), duplicate file descriptor; difference is that it uses file descriptor number specified in newfd
• fork() - Create a new child process by duplicating the calling process, with caller becoming the parent process
• sendfile() - Copies data between one file descriptor and another
• socketpair() - Create a pair of connected sockets, for example for communication between parent and child process

Capabilities and system calls used by nginx

A process providing HTTP or HTTPS typically binds to port 80 and/or 443. To make this possible, the related system calls like bind(2) are need. On top of that, the capability CAP_NET_BIND_SERVICE is needed. Let’s have a look what else is needed to provide basic functionality.

CPU resources

• sched_setaffinity(), covered by @resources (part of @system-service)
• setpriority(), covered by @resources (part of @system-service)

No entries of realtime scheduling policies were found in the source (e.g. SCHED_DEADLINE, SCHED_FIFO, SCHED_RR). So RestrictRealtime is set to yes.

User and creating log files

When the parent process spawns the workers, it runs them as a non-privileged user (e.g. www-data). Due to this change, the parent process needs to be able to change ownership of the user-ID and group-ID. For this reason, it requires the capabilities CAP_SETGID and CAP_SETUID.

Another important area is the ownership of the access and error files. To be able to create them and adjust ownership, the capabilities CAP_CHOWN and CAP_DAC_OVERRIDE are typically needed.

Note: while testing, nginx create the access/error log with the root user as owner. When running nginx -s reopen it properly corrected them to the non-privileged user. Looks like a bug, so this part may require additional testing.

Device files

The source code of nginx makes a reference to different files in /dev, like /dev/null, /dev/poll, and /dev/zero. When looking at an active system, only /dev/null is opened.

Use the lsof command to validate what devices are used by your nginx processes.

lsof -a -c nginx /dev

With that, we could throw multiple options into the mix.

[Service]
PrivateDevices=yes
DevicePolicy=strict
DeviceAllow=/dev/null

Unfortunately, this will not restrict access to /dev/null alone. The PrivateDevices setting will change DevicePolicy to closed. To simplify matters, we therefore set PrivateDevices to strict.

Namespaces

Linux namespaces create an abstraction layer around processes. To determine if this is used, look at the source code for system calls like clone(2), ioctl, setns, unshare.

Syscall clone() is mentioned in the source code, but not used. Also, it requires the a flag that starts with ‘CLONE_NEW’, like CLONE_NEWCGROUP for a new cgroup. With that, we can be fairly certain nginx does not create namespaces.

To be continued…

The remaining parts on hardening will be added later.

In the meantime, we created a predefined hardening profile for nginx.