Systemd

Systemd uses different types of units. One of them is the timer unit, which can be used to schedule a periodic task. This timer unit is linked to an existing service unit and will activate the service according to the defined schedule. The timer unit can be defined using the information about systemd timers. Timers use one or more OnCalendar definitions to specify when execution of the timer should happen. As systemd timers are very versatile and provide good monitoring options, they are a good replacement of cronjobs.

Systemd is nowadays a common system and service manager for Linux systems. But how do you know for sure that it is being active? The easiest way is to have a look at PID number 1. This is the first process started after the kernel itself. With the help of ps we can determine the underlying command behind this initial process. ps -p 1 -o comm= This command defines what columns should be part of the output, where only shows the actual command.

The systemctl command show active unit. To see only services that are enabled, we need to filter the output. This can be achieved using the list-unit-files subcommand and combined with the --state= option. As we are interested in enabled services only, set the value to enabled. Want to parse the output with a script? Consider adding --legend=false to remove the header and footer text (X unit files listed.). Usage systemctl list-unit-files --type=service --state=enabled UNIT FILE STATE VENDOR PRESET apparmor.

Introduction This is a hardening profile to help securing nginx by using systemd unit configuration. It’s goal is to restrict what nginx can do and make it harder for any possible vulnerability to be misused. The rationale for the selected settings is based on the analysis as part of the article Hardening nginx with systemd security features.

Introduction Systemd has a range of security features to help securing services running on your system. That is the good part. The big challenge with so many features is that it is hard to find out which ones you could or should apply, without breaking a service. That is why we started working on hardening profiles. The hardening profiles are pre-defined templates that are documented and tested against a default installation of a piece of software.

Why and when to use SocketBindAllow The setting SocketBindAllow is used together with SocketBindDeny and defines restrictions on the usage of the system call bind on a network socket. Settings Both SocketBindAllow and SocketBindDeny use a bind-rule. See SocketBindDeny for the details. Generic advice This setting is useful in combination with SocketBindDeny to create an allow-list. Examples Allow binding on TCP port 80 [Service] SocketBindDeny=any SocketBindAllow=tcp:80 Allow binding on port 443 (IPv4/IPv6, TCP/UDP)

Why and when to use SocketBindDeny The setting SocketBindDeny can be used alone or together with SocketBindAllow to set restrictions on the usage of the system call bind on a network socket. Settings If the SocketBindDeny list is used alone, then it is a deny-list. Everything except the defined ports/protocols will be allowed. By defining the value ‘any’, all combinations are denied. This is typically used in combination with SocketBindAllow to open up one or more ports.

Why and when to use DevicePolicy The setting DevicePolicy aims to reduce access to devices in /dev. By default, there is no limitation to access devices. Settings The value strict is the most strict, as the name implies. This is suitable for services that do not need any access, like custom shell scripts. Unless output is being redirect to /dev/null, then access could be granted with DeviceAllow=/dev/null. Generic advice Aim for using ‘strict’ when possible and define entries that should be allowed.

Why and when to use DeviceAllow The setting DeviceAllow aims to reduce access or its level to devices in /dev. By default, there is no limitation to access devices. Settings Define DeviceAllow with the path and access level. DeviceAllow=/dev/sda3 r General advice For most services it might be easier to use ProtectDevices=yes to reduce the devices that can be access.

Discover the reasons why a systemd unit went into a failed state

Systemd stores the configuration for units, like services, in individual unit files. When changes are made to these units, a reload might be needed. This is where systemctl daemon-reload comes into play. But what exactly does the daemon-reload subcommand really do? In short: rerun generators, reload units files, recreate the dependency tree. Let’s have a look at the more detailed answer. Running generators Generators are helper scripts to convert non-native scripts to unit files that are usuable by systemd.

Systemd may need to reload a part of the unit configuration if changes were made. To find out if the related systemctl daemon-reload command is needed, the state of the individual units can be tested. This is done by querying the property using the --property=NeedDaemonReload option. Testing a single service like nginx, can be done this way: # systemctl show --property=NeedDaemonReload --value nginx.service yes This output will return a ‘yes’ or ’no’ value.

Systemd can restrict services from using particular syscalls with the help of the unit setting SystemCallFilter. Instead of mentioning all individual syscalls, systemd has predefined sets that can be used. These sets group functions that are related. To see which syscalls are part of a set, use the systemd-analyze command. # systemd-analyze syscall-filter @ipc @ipc # SysV IPC, POSIX Message Queues or other IPC ipc memfd_create mq_getsetattr mq_notify mq_open mq_timedreceive mq_timedreceive_time64 mq_timedsend mq_timedsend_time64 mq_unlink msgctl msgget msgrcv msgsnd pipe pipe2 process_madvise process_vm_readv process_vm_writev semctl semget semop semtimedop semtimedop_time64 shmat shmctl shmdt shmget See systemd syscall filtering for all details.

Why and when to use SystemCallFilter The setting SystemCallFilter aims to prevent misuse of syscalls that are not needed for normal functioning of a process. This powerful filtering restricts the abilities of a process, but requires understanding of processes by the system administrator. See the overview of Linux syscalls for more details. Configuration This setting takes a space-separated list and may be specified multiple times. Allow-listing By default the list contains the entries of allowed system call names.

Overview of syscalls in systemd by group

Systemd and its services can be in several states, such as enabled, disabled, failed, running. If you no longer need a particular service to run, then the first step is to stop a service. systemctl stop nginx.service But stopping a service is not the same as disabling a service. With that comes a very frequently asked question: what is the difference between a service that is disabled and one that is masked?

Systemd allows service units to be configured using a drop-in file, which is often called override.conf. It overrides the vendor-supplied version of a service to customize it. Instead of duplicating the configuration, the override file contains the differences. Editing service file Changing a service can be done using systemctl, followed by the edit subcommand and service unit. The editor that is configured on the system will be opened and any changes can be made between the comment section at the top and the comment section a little bit lower.

The systemctl command will normally all active units. To filter this output to just the running services, we can combine the options --type= and --state=. For this particular case we set the type to service and the type state to running. Usage # systemctl --type=service --state=running --legend=false accounts-daemon.service loaded active running Accounts Service avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack colord.service loaded active running Manage, Install and Generate Color Profiles dbus-broker.

Elevating permissions

Elevating permissions

Systemd introduced run0 as its alternative to sudo. One of the features if a colored background when your privileges are elevated. To disable this behaviour, use the option --background= with an empty value. run0 --background= The red background now will be gone, which can be useful if the color conflicts with the output or when it is unwanted.

Why and when to use MemoryDenyWriteExecute The setting MemoryDenyWriteExecute will block the creation or alteration of a memory segment to become writable and executable as well. By enabling this limitation, it will increase the bar software exploits to change running code dynamically. Usage [Service] MemoryDenyWriteExecute=yes InaccessiblePaths=/dev/shm SystemCallFilter=~memfd_create Caveats To prevent circumvention of this setting, access to /dev/shm and the syscall memfd_create should be blocked as well. Generic advice For most common services this option can be implemented and will increase the security of a service.

Why and when to use InaccessiblePaths The setting InaccessiblePaths defines paths that should never be accessible. Instead of using the principles of an allow list, it is an explicit deny list. It can be used to block access by a process to a location with sensitive data or a path commonly misused for exploits. Values Define the paths that are granted write access. [Service] InaccessiblePaths=/dev/shm When a path is prefixed with a minus (-), it is ignored if it does not exist When a path is prefixed with a plus (+), the path is considered relative to root of directory (e.

The systemctl command has multiple options to show the memory usage. With the status subcommand followed by the service, it will show the basics, including memory usage. To retrieve the information that easier to parse, then use show followed by --property=MemoryCurrent and the service name. Usage The status output will include memory usage. systemctl status nginx ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.

The systemctl command can be used to show all settings of an unit, such as a service. To display the full list of applicable settings, use the show subcommand followed by the unit name. Besides the settings, the output will also include actual runtime information, such as memory usage, when the unit was started, etc. Usage Just provide the unit file to see all available information. # systemctl show nginx.service Type=forking Restart=no PIDFile=/run/nginx.

The systemctl command can show settings of a systemd unit, such as a service. It can also assist in overriding these settings by using the edit subcommand followed by the unit name. This will open the editor that is configured on the system and create the override file. Usage Run the edit command with the unit, and the editor like vim or nano will show up. ### Editing /etc/systemd/system/nginx.service.d/override.conf ### Anything between here and the comment below will become the new contents of the file [Service] ProtectSystem=strict ReadWritePaths=/run /var/log/nginx ### Lines below this comment will be discarded <snip> Important: Do not remove the comments and only insert or change between the specified comment lines.

Why and when to use ReadWritePaths The setting ReadWritePaths grants read and write permissions to defined paths. It can be used in combination with other settings like ‘ProtectSystem=strict’ to make the full file system read-only, and then open up a few paths that are required for a service to run correctly. Values Define the paths that are granted write access. [Service] ProtectSystem=strict ReadWritePaths=/run /var/log/nginx When a path is prefixed with a minus (-), it is ignored if it does not exist When a path is prefixed with a plus (+), the path is considered relative to root of directory (e.

Introduction Nginx is still a popular web server and powering a part of the web. Wouldn’t it be great if we could secure it a little bit more? In this article we use the security features to secure systemd units and services and apply it to nginx. If you are not familiar yet with the unit settings of systemd, then this document would be a good introduction into the subject.

Secure services with these features

Why and when to use ProcSubset The setting ProcSubset controls the “subset” mount option of /proc for the unit. Caveats This function does not if the “subnet” option for procfs is not supported. Generic advice The Linux kernel shares information from various kernel APIs via /proc. When activating this setting, these kernel APIs are also made unavailable, which might break common software, unless it is a trivial process. So this option is to be used with care.

Why and when to use RestrictAddressFamilies The setting RestrictAddressFamilies aims to restrict what socket address families can be used. When using it, the default is that it is used as an allow-list and define what address families can be used. Settings When this setting is not configured, there are no restrictions to what address families can be used. Setting the value to none will block all address families. To block specific address families only, a ~ can be used to turn the allow-list into a deny-list.

Why and when to use ProtectProc The setting ProtectProc aims to protect information that normally can be retrieved from /proc. Settings The value default, which is also the default, will not restrict access. Value invisible will hide information, where ptraceable restrict the set to only processes that be monitored with the system call ptrace(). The value noaccess is the most strict option. Caveats This setting will not have effect if the kernel does not support the hidepid mount option per individual mount point.

Why and when to use ProtectHome The setting ProtectHome aims to protect home directories. These three paths are included: /home /root /run/user Settings The default no will not restrict access to the home directories. Using yes will active full protection, not allowing access. The value read-only will make the paths read-only, so no data can be written to it. With tmpfs a temporary file system is being used, also read-only, yet it hides the actual home directories.

Background The Linux kernel exposes its kernel log ring buffer to userspace via /dev/kmsg and /proc/kmsg. When this setting is defined as yes, the capability CAP_SYS_MODULE will be removed from the capability bounding set. This means that all processes in the unit will no longer have access to the kernel log ring buffer. Generic advice For most common services access to the kernel log ring buffer is not need, therefore safe to disable (ProtectKernelLogs=yes).

Explanation Kernel modules can provide additional functionality when using a modular Linux kernel, which is applicable to most systems. When this setting is set to yes, it tries to prevent the unit from loading kernel modules. This is achieved by removing the CAP_SYS_MODULE from the capability bounding set. Generic advice Most units do not need the permission to load kernel modules, so typically a unit can be configured with ProtectKernelModules=true.

The timedatectl command can show the time, time zone information, and its status. Add the timesync-status subcommand to see synchronization details. Usage Use timedatectl with the timesync-status command to see the actual status. Under normal conditions, the leap should show ’normal'. # timedatectl timesync-status Server: 185.125.190.56 (ntp.ubuntu.com) Poll interval: 34min 8s (min: 32s; max 34min 8s) Leap: normal Version: 4 Stratum: 2 Reference: 4FF33C32 Precision: 1us (-25) Root distance: 762us (max: 5s) Offset: +882us Delay: 15.

With the hostnamectl command basic system information like the operating system, hostname, and machine ID can be displayed. Usage Run the command without any parameters to get the status displayed, including the machine ID. hostnamectl

The systemctl command can be used to show dependencies between units with the list-dependencies subcommand. A nicely human-readable output will be displayed showing the selected unit, followed by the dependencies that rely on this unit. This is useful when a unit is in a failed state due to a dependency on another unit. Usage To see which units require the multi-user target to be active: # systemctl list-dependencies multi-user.target multi-user.target ● ├─apport.

The systemctl command can show the available systemd unit types when using the option --type=help. Usage # systemctl --type=help Available unit types: service mount swap socket target device automount timer path slice scope

The systemctl command will show by default all active units. To filter down on a particular unit type, use the --type= option, followed by the type. Not sure what types are available? Run systemctl --type=help. Usage systemctl list-units --type=target

To limit the maximum size that journals may use on the system, define the setting SystemMaxUse in /etc/systemd/journald.conf. Save the file, confirm that the settings are correct, then restart the journal daemon. Configuration Open /etc/systemd/journald.conf, copy the commented line, remove the hash, and assign it a value. SystemMaxUse=256M Note: depending on how many events happen on a system, this value might be too small. Make sure that the size for logs is big enough.

The journalctl command can be used to show the journal. By using the --disk-usage option, the size of the journal is displayed. This includes the archived and active journal files. When the journal is using too much disk space, consider performing a vacuum task. Usage Showing the disk usage is quick and easy. # journalctl --disk-usage Archived and active journals take up 56.0M in the file system. Does the journal take up too much space?

The journalctl command can show all events related to the kernel itself usig the --dmesg option. This option will filter out kernel messages and has a similar output as the dmesg command. Usage Use the full or shorter option to query the kernel messages. journalctl -k Looking for only the kernel messages of today? Combine it with the --since= option. journalctl -k -S "today"

Systemd units define resources that can be used by the system. Examples of these units are a service, path, socket, and timer. Each unit type has its own basic set of properties that then individually can be configured. Unit types can be recognized by their file extension. A service will use the ‘.service’ extension, making it easy to recognize. Units are usually managed with the systemctl command. See systemd unit types and their purpose for a full overview of the units.

The journalctl command shows by default the oldest entries it has in the journal. Typically we are not interested in that, for that purpose there is the --since= option. This option defines that entries should be after the specified moment in time. Besides using an actual date, a shortened name like ’today’ can also be used that automatically defines the date and time. Usage To see the entries of today, use the aptly named ’today'.

The journalctl command can show continuously new log entries with the --follow option. When new entries are added to the journal, they are automatically shown. Usage The follow option is a great option to continuously monitor a particular unit. journalctl --follow --unit=nginx.service Without providing a unit, all system events will be shown and followed.

The journalctl command can show the events from its journal by --unit= followed by the service or its unit name. This way events will be filtered, making it much easier to troubleshoot a particular service. Example journalctl -u nginx.service

Reload systemd

Systemd is a system and service manager. The name is short for ‘system daemon’, an ongoing service that manages the system. As it is also a service manager, it is responsible for start, stopping, and monitoring services. Systemd replaces the SysV init system and focuses on performance and resource management. It was created by Lennart Poettering in 2010, with Fedora Linux being the first to adopt it in May 2011. In 2015, several major Linux distributions started shipping with systemd.

Systemd units that are in a masked state are administratively disabled. While being in this state, they can not be started until they are unmasked. Typically a unit is masked when it should not start by default or manually, to prevent it causing issues or running an unwanted service. With systemctl and the subcommand mask, a systemd unit can be masked. Relevant FAQ: How to see all masked units with systemctl?

Systemd has a range of commands to interact with the systemd services and functions. Typically daemons will have a ’d’ in their name, such as systemd-networkd, while client tools end with ‘ctl’ (systemctl). Here is an overview of the available commands and their purpose. Primary commands Journalctl Events are stored in a journal, similar to what syslog does. The biggest exception is the way log information is stored and how it is queried.

Learn about the available systemd unit types

Clear DNS cache using resolvectl

Systemd allows fine-grained customization of units by defining so-called properties. These properties or settings influence what a unit, such as a service, can or can not do. As their is a wide range of settings, this page has the goal to present them, including a quick reference to each of them.

Introduction Systemd is a system and service manager for Linux. For many Linux distributions it replaced the existing SysV init system, modernizing how services are started and monitored. Some basics about systemd: Author: Lennart Poettering First release: 2010 First adopter: Fedora Linux Common usage by major Linux distributions: 2015 Learn more: What is systemd? Systemd units To monitor and manage services on a system using systemd, unit files are used. These text-based files define what to run or do, relevant conditions, and any applicable dependencies.

Show all masked units

Perform smarter queries when requesting information from journalctl

Disable a service or specific unit with systemctl

Start and enable a unit with one command

Show failed systemd units with systemctl

Make a new friend?

Learn about the available systemd unit types

Control those processes and timers

Query the journal and find the needle

Systemd used a binary log to store information about specific events. These events include the boot sequence and the related output. In this article we have a look at finding our boot logs in systemd journals. Binary logging When using systemd, boot data is stored in journals, a binary format. There is big benefit of saving boot data in a binary format: log information of each boot can be stored separately, linked to other pieces of information, and queried easier and quicker.

Solving failed units with systemctl Systemd is an alternative service manager to the more traditional init system. To ensure the system is healthy, failed units should be investigated on a regular basis. Sooner or later a unit might fail and showing up the systemctl listing. In this article we have a look at how to solve it. Why do services fail? During the start of the system, enabled services are started and queued to be executed.