Systemd

Systemd uses different types of units. One of them is the timer unit, which can be used to schedule a periodic task. This timer unit is linked to an existing service unit and will activate the service according to the defined schedule. The timer unit can be defined using the information about systemd timers.

Timers use one or more OnCalendar definitions to specify when execution of the timer should happen. As systemd timers are very versatile and provide good monitoring options, they are a good replacement of cronjobs. The only downside is that slightly more configuration is needed when a just a single shell script needs to be executed, as you need to create both the timer as service unit.

Systemd is nowadays a common system and service manager for Linux systems. But how do you know for sure that it is being active? The easiest way is to have a look at PID number 1. This is the first process started after the kernel itself. With the help of ps we can determine the underlying command behind this initial process.

ps -p 1 -o comm=

This command defines what columns should be part of the output, where only shows the actual command. Adding the ‘=’ removes the header, so if systemd is being used, then the string ‘systemd’ will be returned.

The systemctl command show active unit. To see only services that are enabled, we need to filter the output. This can be achieved using the list-unit-files subcommand and combined with the --state= option. As we are interested in enabled services only, set the value to enabled.

Want to parse the output with a script? Consider adding --legend=false to remove the header and footer text (X unit files listed.).

Usage

systemctl list-unit-files --type=service --state=enabled
UNIT FILE                              STATE   VENDOR PRESET
apparmor.service                       enabled enabled      
blk-availability.service               enabled enabled      
cloud-config.service                   enabled enabled      
cloud-final.service                    enabled enabled      
cloud-init-local.service               enabled enabled      
cloud-init.service                     enabled enabled      
console-setup.service                  enabled enabled      
cron.service                           enabled enabled      
dmesg.service                          enabled enabled      
e2scrub_reap.service                   enabled enabled      
finalrd.service                        enabled enabled      
getty@.service                         enabled enabled      
gpu-manager.service                    enabled enabled      
grub-common.service                    enabled enabled      
grub-initrd-fallback.service           enabled enabled      
irqbalance.service                     enabled enabled      
keyboard-setup.service                 enabled enabled      
lvm2-monitor.service                   enabled enabled      
lxd-agent.service                      enabled enabled      
ModemManager.service                   enabled enabled      
multipathd.service                     enabled enabled      
networkd-dispatcher.service            enabled enabled      
nginx.service                          enabled enabled      
open-iscsi.service                     enabled enabled      
open-vm-tools.service                  enabled enabled      
pollinate.service                      enabled enabled      
rsyslog.service                        enabled enabled      
secureboot-db.service                  enabled enabled      
setvtrgb.service                       enabled enabled      
snap.lxd.activate.service              enabled enabled      
snapd.apparmor.service                 enabled enabled      
snapd.autoimport.service               enabled enabled      
snapd.core-fixup.service               enabled enabled      
snapd.recovery-chooser-trigger.service enabled enabled      
snapd.seeded.service                   enabled enabled      
snapd.service                          enabled enabled      
snapd.system-shutdown.service          enabled enabled      
ssh.service                            enabled enabled      
systemd-networkd-wait-online.service   enabled disabled     
systemd-networkd.service               enabled enabled      
systemd-pstore.service                 enabled enabled      
systemd-resolved.service               enabled enabled      
systemd-timesyncd.service              enabled enabled      
thermald.service                       enabled enabled      
ua-reboot-cmds.service                 enabled enabled      
ubuntu-advantage.service               enabled enabled      
udisks2.service                        enabled enabled      
ufw.service                            enabled enabled      
unattended-upgrades.service            enabled enabled      
vgauth.service                         enabled enabled      

50 unit files listed.

Introduction

This is a hardening profile to help securing nginx by using systemd unit configuration. It’s goal is to restrict what nginx can do and make it harder for any possible vulnerability to be misused.

The rationale for the selected settings is based on the analysis as part of the article Hardening nginx with systemd security features.

Introduction

Systemd has a range of security features to help securing services running on your system. That is the good part. The big challenge with so many features is that it is hard to find out which ones you could or should apply, without breaking a service. That is why we started working on hardening profiles.

The hardening profiles are pre-defined templates that are documented and tested against a default installation of a piece of software. System administrators then can use this as the foundation of their service and tune it slightly to their specific configuration.

Why and when to use SocketBindAllow

The setting SocketBindAllow is used together with SocketBindDeny and defines restrictions on the usage of the system call bind on a network socket.

Settings

Both SocketBindAllow and SocketBindDeny use a bind-rule. See SocketBindDeny for the details.

Generic advice

This setting is useful in combination with SocketBindDeny to create an allow-list.

Examples

Allow binding on TCP port 80

Why and when to use SocketBindDeny

The setting SocketBindDeny can be used alone or together with SocketBindAllow to set restrictions on the usage of the system call bind on a network socket.

Settings

If the SocketBindDeny list is used alone, then it is a deny-list. Everything except the defined ports/protocols will be allowed.

By defining the value ‘any’, all combinations are denied. This is typically used in combination with SocketBindAllow to open up one or more ports.

Why and when to use DevicePolicy

The setting DevicePolicy aims to reduce access to devices in /dev. By default, there is no limitation to access devices.

Settings

The value strict is the most strict, as the name implies. This is suitable for services that do not need any access, like custom shell scripts. Unless output is being redirect to /dev/null, then access could be granted with DeviceAllow=/dev/null.

Generic advice

Aim for using ‘strict’ when possible and define entries that should be allowed. To discover files used by a binary, consider inspecting it with the strings command or look at open files from a running process with lsof.

Why and when to use DeviceAllow

The setting DeviceAllow aims to reduce access or its level to devices in /dev. By default, there is no limitation to access devices.

Settings

Define DeviceAllow with the path and access level.

DeviceAllow=/dev/sda3 r

General advice

For most services it might be easier to use ProtectDevices=yes to reduce the devices that can be access.

Discover the reasons why a systemd unit went into a failed state

Systemd stores the configuration for units, like services, in individual unit files. When changes are made to these units, a reload might be needed. This is where systemctl daemon-reload comes into play. But what exactly does the daemon-reload subcommand really do? In short: rerun generators, reload units files, recreate the dependency tree. Let’s have a look at the more detailed answer.

Running generators

Generators are helper scripts to convert non-native scripts to unit files that are usuable by systemd. These generators are used during the boot, but also when the daemon configuration is reloaded. In multiple phases all generators are started in parrallel, with the goal to minize the time tasks are waiting for each other to finish.

Systemd may need to reload a part of the unit configuration if changes were made. To find out if the related systemctl daemon-reload command is needed, the state of the individual units can be tested. This is done by querying the property using the --property=NeedDaemonReload option.

Testing a single service like nginx, can be done this way:

# systemctl show --property=NeedDaemonReload --value nginx.service
yes

This output will return a ‘yes’ or ’no’ value.

Systemd can restrict services from using particular syscalls with the help of the unit setting SystemCallFilter. Instead of mentioning all individual syscalls, systemd has predefined sets that can be used. These sets group functions that are related. To see which syscalls are part of a set, use the systemd-analyze command.

# systemd-analyze syscall-filter @ipc
@ipc
    # SysV IPC, POSIX Message Queues or other IPC
    ipc
    memfd_create
    mq_getsetattr
    mq_notify
    mq_open
    mq_timedreceive
    mq_timedreceive_time64
    mq_timedsend
    mq_timedsend_time64
    mq_unlink
    msgctl
    msgget
    msgrcv
    msgsnd
    pipe
    pipe2
    process_madvise
    process_vm_readv
    process_vm_writev
    semctl
    semget
    semop
    semtimedop
    semtimedop_time64
    shmat
    shmctl
    shmdt
    shmget

See systemd syscall filtering for all details.

Why and when to use SystemCallFilter

The setting SystemCallFilter aims to prevent misuse of syscalls that are not needed for normal functioning of a process. This powerful filtering restricts the abilities of a process, but requires understanding of processes by the system administrator. See the overview of Linux syscalls for more details.

Configuration

This setting takes a space-separated list and may be specified multiple times.

Overview of syscalls in systemd by group

Systemd and its services can be in several states, such as enabled, disabled, failed, running. If you no longer need a particular service to run, then the first step is to stop a service.

systemctl stop nginx.service

But stopping a service is not the same as disabling a service. With that comes a very frequently asked question: what is the difference between a service that is disabled and one that is masked?

Systemd allows service units to be configured using a drop-in file, which is often called override.conf. It overrides the vendor-supplied version of a service to customize it. Instead of duplicating the configuration, the override file contains the differences.

Editing service file

Changing a service can be done using systemctl, followed by the edit subcommand and service unit. The editor that is configured on the system will be opened and any changes can be made between the comment section at the top and the comment section a little bit lower. Do not remove these comment sections, as systemctl uses these to see what changes you made.

The systemctl command will normally all active units. To filter this output to just the running services, we can combine the options --type= and --state=. For this particular case we set the type to service and the type state to running.

Usage

# systemctl --type=service --state=running --legend=false
  accounts-daemon.service   loaded active running Accounts Service
  avahi-daemon.service      loaded active running Avahi mDNS/DNS-SD Stack
  colord.service            loaded active running Manage, Install and Generate Color Profiles
  dbus-broker.service       loaded active running D-Bus System Message Bus
  firewalld.service         loaded active running firewalld - dynamic firewall daemon
  gdm.service               loaded active running GNOME Display Manager
  NetworkManager.service    loaded active running Network Manager
  polkit.service            loaded active running Authorization Manager
  qemu-guest-agent.service  loaded active running QEMU Guest Agent
  rtkit-daemon.service      loaded active running RealtimeKit Scheduling Policy Service
  spice-vdagentd.service    loaded active running Agent daemon for Spice guests
  sshd.service              loaded active running OpenSSH Daemon
  systemd-journald.service  loaded active running Journal Service
  systemd-logind.service    loaded active running User Login Management
  systemd-timesyncd.service loaded active running Network Time Synchronization
  systemd-udevd.service     loaded active running Rule-based Manager for Device Events and Files
  systemd-userdbd.service   loaded active running User Database Manager
  udisks2.service           loaded active running Disk Manager
  upower.service            loaded active running Daemon for power management
  user@1000.service         loaded active running User Manager for UID 1000
  wpa_supplicant.service    loaded active running WPA supplicant

This command will show just services that are running. The legend is set to false, meaning no header and footer are displayed.

Elevating permissions

Elevating permissions

Systemd introduced run0 as its alternative to sudo. One of the features if a colored background when your privileges are elevated.

To disable this behaviour, use the option --background= with an empty value.

run0 --background=

The red background now will be gone, which can be useful if the color conflicts with the output or when it is unwanted.

Why and when to use MemoryDenyWriteExecute

The setting MemoryDenyWriteExecute will block the creation or alteration of a memory segment to become writable and executable as well. By enabling this limitation, it will increase the bar software exploits to change running code dynamically.

Usage

[Service]
MemoryDenyWriteExecute=yes
InaccessiblePaths=/dev/shm
SystemCallFilter=~memfd_create

Caveats

To prevent circumvention of this setting, access to /dev/shm and the syscall memfd_create should be blocked as well.

Generic advice

For most common services this option can be implemented and will increase the security of a service. That is, if used together with InaccessiblePaths and SystemCallFilter.

Why and when to use InaccessiblePaths

The setting InaccessiblePaths defines paths that should never be accessible. Instead of using the principles of an allow list, it is an explicit deny list. It can be used to block access by a process to a location with sensitive data or a path commonly misused for exploits.

The systemctl command has multiple options to show the memory usage. With the status subcommand followed by the service, it will show the basics, including memory usage.

To retrieve the information that easier to parse, then use show followed by --property=MemoryCurrent and the service name.

Usage

The status output will include memory usage.

systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nginx.service.d
             └─override.conf
     Active: active (running) since Mon 2024-06-17 17:59:45 UTC; 3h 54min ago
       Docs: man:nginx(8)
   Main PID: 36971 (nginx)
      Tasks: 2 (limit: 1012)
     Memory: 2.6M
        CPU: 26ms
     CGroup: /system.slice/nginx.service
             ├─36971 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             └─36972 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""

For pulling in the information that can be parsed or scripted, consider retrieving the actual property from a running service.

The systemctl command can be used to show all settings of an unit, such as a service. To display the full list of applicable settings, use the show subcommand followed by the unit name.

Besides the settings, the output will also include actual runtime information, such as memory usage, when the unit was started, etc.

Usage

Just provide the unit file to see all available information.

# systemctl show nginx.service
Type=forking
Restart=no
PIDFile=/run/nginx.pid
NotifyAccess=none
RestartUSec=100ms
TimeoutStartUSec=1min 30s
TimeoutStopUSec=5s
TimeoutAbortUSec=5s
TimeoutStartFailureMode=terminate
TimeoutStopFailureMode=terminate
<snip>

The systemctl command can show settings of a systemd unit, such as a service. It can also assist in overriding these settings by using the edit subcommand followed by the unit name. This will open the editor that is configured on the system and create the override file.

Usage

Run the edit command with the unit, and the editor like vim or nano will show up.

### Editing /etc/systemd/system/nginx.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Service]
ProtectSystem=strict
ReadWritePaths=/run /var/log/nginx

### Lines below this comment will be discarded
<snip>

Important: Do not remove the comments and only insert or change between the specified comment lines.

Why and when to use ReadWritePaths

The setting ReadWritePaths grants read and write permissions to defined paths. It can be used in combination with other settings like ‘ProtectSystem=strict’ to make the full file system read-only, and then open up a few paths that are required for a service to run correctly.

Values

Define the paths that are granted write access.

[Service]
ProtectSystem=strict
ReadWritePaths=/run /var/log/nginx

• When a path is prefixed with a minus (-), it is ignored if it does not exist
• When a path is prefixed with a plus (+), the path is considered relative to root of directory (e.g. configured with RootDirectory)

Caveats

This setting will not have effect if a process is missing the normal file permissions or ownership. For additional sandboxing, consider CapabilityBoundingSet=CAP_SYS_ADMIN or SystemCallFilter=@mount.

Introduction

Nginx is still a popular web server and powering a part of the web. Wouldn’t it be great if we could secure it a little bit more? In this article we use the security features to secure systemd units and services and apply it to nginx.

If you are not familiar yet with the unit settings of systemd, then this document would be a good introduction into the subject.

File paths

Like most services, nginx uses files to read and write. Typically web site content is read from the disk, while log files are created and written to. We can use the systemd unit settings to restrict access to only those paths that are strictly required for nginx to run. But how to find out what those paths are?

Secure services with these features

Why and when to use ProcSubset

The setting ProcSubset controls the “subset” mount option of /proc for the unit.

Caveats

This function does not if the “subnet” option for procfs is not supported.

Generic advice

The Linux kernel shares information from various kernel APIs via /proc. When activating this setting, these kernel APIs are also made unavailable, which might break common software, unless it is a trivial process. So this option is to be used with care. Typically it may be better to implement the ProtectProc setting.

Why and when to use RestrictAddressFamilies

The setting RestrictAddressFamilies aims to restrict what socket address families can be used. When using it, the default is that it is used as an allow-list and define what address families can be used.

Settings

When this setting is not configured, there are no restrictions to what address families can be used.

Setting the value to none will block all address families.

To block specific address families only, a ~ can be used to turn the allow-list into a deny-list.

Why and when to use ProtectProc

The setting ProtectProc aims to protect information that normally can be retrieved from /proc.

Settings

The value default, which is also the default, will not restrict access. Value invisible will hide information, where ptraceable restrict the set to only processes that be monitored with the system call ptrace(). The value noaccess is the most strict option.

Caveats

This setting will not have effect if the kernel does not support the hidepid mount option per individual mount point.

Why and when to use ProtectHome

The setting ProtectHome aims to protect home directories. These three paths are included:

• /home
• /root
• /run/user

Settings

The default no will not restrict access to the home directories. Using yes will active full protection, not allowing access.

The value read-only will make the paths read-only, so no data can be written to it.

With tmpfs a temporary file system is being used, also read-only, yet it hides the actual home directories. It will still allow access to the actual directories when using BindPaths or BindReadOnlyPaths. For rare situations this might be useful to protect the home directories, while still allowing some very specific access.

Background

The Linux kernel exposes its kernel log ring buffer to userspace via /dev/kmsg and /proc/kmsg.

When this setting is defined as yes, the capability CAP_SYS_MODULE will be removed from the capability bounding set. This means that all processes in the unit will no longer have access to the kernel log ring buffer.

Explanation

Kernel modules can provide additional functionality when using a modular Linux kernel, which is applicable to most systems. When this setting is set to yes, it tries to prevent the unit from loading kernel modules. This is achieved by removing the CAP_SYS_MODULE from the capability bounding set.

Generic advice

Most units do not need the permission to load kernel modules, so typically a unit can be configured with ProtectKernelModules=true.

The timedatectl command can show the time, time zone information, and its status. Add the timesync-status subcommand to see synchronization details.

Usage

Use timedatectl with the timesync-status command to see the actual status. Under normal conditions, the leap should show ’normal'.

# timedatectl timesync-status
       Server: 185.125.190.56 (ntp.ubuntu.com)
Poll interval: 34min 8s (min: 32s; max 34min 8s)
         Leap: normal
      Version: 4
      Stratum: 2
    Reference: 4FF33C32
    Precision: 1us (-25)
Root distance: 762us (max: 5s)
       Offset: +882us
        Delay: 15.169ms
       Jitter: 7.098ms
 Packet count: 711
    Frequency: +2.731ppm

With the hostnamectl command basic system information like the operating system, hostname, and machine ID can be displayed.

Usage

Run the command without any parameters to get the status displayed, including the machine ID.

hostnamectl

The systemctl command can be used to show dependencies between units with the list-dependencies subcommand. A nicely human-readable output will be displayed showing the selected unit, followed by the dependencies that rely on this unit. This is useful when a unit is in a failed state due to a dependency on another unit.

Usage

To see which units require the multi-user target to be active:

# systemctl list-dependencies multi-user.target
multi-user.target
● ├─apport.service
● ├─console-setup.service
● ├─cron.service
● ├─dbus.service
○ ├─dmesg.service
○ ├─e2scrub_reap.service
○ ├─grub-common.service
○ ├─grub-initrd-fallback.service
○ ├─irqbalance.service
<snip>

Want to move up the other way and see on which our unit requires? Add the --reverse option.

The systemctl command can show the available systemd unit types when using the option --type=help.

Usage

# systemctl --type=help
Available unit types:
service
mount
swap
socket
target
device
automount
timer
path
slice
scope

The systemctl command will show by default all active units. To filter down on a particular unit type, use the --type= option, followed by the type. Not sure what types are available? Run systemctl --type=help.

Usage

systemctl list-units --type=target

To limit the maximum size that journals may use on the system, define the setting SystemMaxUse in /etc/systemd/journald.conf. Save the file, confirm that the settings are correct, then restart the journal daemon.

Configuration

Open /etc/systemd/journald.conf, copy the commented line, remove the hash, and assign it a value.

SystemMaxUse=256M

Note: depending on how many events happen on a system, this value might be too small. Make sure that the size for logs is big enough.

The journalctl command can be used to show the journal. By using the --disk-usage option, the size of the journal is displayed. This includes the archived and active journal files. When the journal is using too much disk space, consider performing a vacuum task.

Usage

Showing the disk usage is quick and easy.

# journalctl --disk-usage
Archived and active journals take up 56.0M in the file system.

Does the journal take up too much space?

The journalctl command can show all events related to the kernel itself usig the --dmesg option. This option will filter out kernel messages and has a similar output as the dmesg command.

Usage

Use the full or shorter option to query the kernel messages.

journalctl -k

Looking for only the kernel messages of today? Combine it with the --since= option.

journalctl -k -S "today"

Systemd units define resources that can be used by the system. Examples of these units are a service, path, socket, and timer. Each unit type has its own basic set of properties that then individually can be configured. Unit types can be recognized by their file extension. A service will use the ‘.service’ extension, making it easy to recognize. Units are usually managed with the systemctl command.

See systemd unit types and their purpose for a full overview of the units.

The journalctl command shows by default the oldest entries it has in the journal. Typically we are not interested in that, for that purpose there is the --since= option. This option defines that entries should be after the specified moment in time. Besides using an actual date, a shortened name like ’today’ can also be used that automatically defines the date and time.

Usage

To see the entries of today, use the aptly named ’today'.

The journalctl command can show continuously new log entries with the --follow option. When new entries are added to the journal, they are automatically shown.

Usage

The follow option is a great option to continuously monitor a particular unit.

journalctl --follow --unit=nginx.service

Without providing a unit, all system events will be shown and followed.

The journalctl command can show the events from its journal by --unit= followed by the service or its unit name. This way events will be filtered, making it much easier to troubleshoot a particular service.

Example

journalctl -u nginx.service

Reload systemd

Systemd is a system and service manager. The name is short for ‘system daemon’, an ongoing service that manages the system. As it is also a service manager, it is responsible for start, stopping, and monitoring services. Systemd replaces the SysV init system and focuses on performance and resource management. It was created by Lennart Poettering in 2010, with Fedora Linux being the first to adopt it in May 2011. In 2015, several major Linux distributions started shipping with systemd.

Systemd units that are in a masked state are administratively disabled. While being in this state, they can not be started until they are unmasked. Typically a unit is masked when it should not start by default or manually, to prevent it causing issues or running an unwanted service. With systemctl and the subcommand mask, a systemd unit can be masked.

Relevant FAQ: How to see all masked units with systemctl?

Systemd has a range of commands to interact with the systemd services and functions. Typically daemons will have a ’d’ in their name, such as systemd-networkd, while client tools end with ‘ctl’ (systemctl). Here is an overview of the available commands and their purpose.

Primary commands

Journalctl

Events are stored in a journal, similar to what syslog does. The biggest exception is the way log information is stored and how it is queried. As data is stored in a binary format, the journalctl command can filter out relevant information.

Learn about the available systemd unit types

Clear DNS cache using resolvectl

Systemd allows fine-grained customization of units by defining so-called properties. These properties or settings influence what a unit, such as a service, can or can not do. As their is a wide range of settings, this page has the goal to present them, including a quick reference to each of them.

Introduction

Systemd is a system and service manager for Linux. For many Linux distributions it replaced the existing SysV init system, modernizing how services are started and monitored.

Some basics about systemd:

• Author: Lennart Poettering
• First release: 2010
• First adopter: Fedora Linux
• Common usage by major Linux distributions: 2015

Learn more: What is systemd?

Systemd units

To monitor and manage services on a system using systemd, unit files are used. These text-based files define what to run or do, relevant conditions, and any applicable dependencies.

Show all masked units

Perform smarter queries when requesting information from journalctl

Disable a service or specific unit with systemctl

Start and enable a unit with one command

Show failed systemd units with systemctl

Make a new friend?

Learn about the available systemd unit types

Control those processes and timers

Query the journal and find the needle

Systemd used a binary log to store information about specific events. These events include the boot sequence and the related output. In this article we have a look at finding our boot logs in systemd journals.

Binary logging

When using systemd, boot data is stored in journals, a binary format. There is big benefit of saving boot data in a binary format: log information of each boot can be stored separately, linked to other pieces of information, and queried easier and quicker. For example, different boots can be compared, as they are individually available.

Solving failed units with systemctl

Systemd is an alternative service manager to the more traditional init system. To ensure the system is healthy, failed units should be investigated on a regular basis. Sooner or later a unit might fail and showing up the systemctl listing. In this article we have a look at how to solve it.

Why do services fail?

During the start of the system, enabled services are started and queued to be executed. Most processes will start correctly and systemd logs the related status in the journal. However, in some cases a service might enter a “failed state”, as a result of another command not finishing properly.