Systemd
How to schedule a periodic task with systemd
Schedule a repeating task via systemd by using a timer. Learn how to configure and use it.
Summary
Systemd uses different types of units. One of them is the timer unit, which can be used to schedule a periodic task. This timer unit is linked to an existing service unit and will activate the service according to the defined schedule. The timer unit can be defined using the information about systemd timers.
Timers use one or more OnCalendar definitions to specify when execution of the timer should happen. As systemd timers are very versatile and provide good monitoring options, they are a good replacement of cronjobs. The only downside is that slightly more configuration is needed when a just a single shell script needs to be executed, as you need to create both the timer as service unit.
Systemd-analyze
The command systemd-analyze helps analyzing systemd components to optimize the system including performance and security.
Summary
How to check if systemd is being used or active
Learn how to quickly confirm that systemd is being used as your system and service manager.
Summary
Systemd is nowadays a common system and service manager for Linux systems. But how do you know for sure that it is being active? The easiest way is to have a look at PID number 1. This is the first process started after the kernel itself. With the help of ps we can determine the underlying command behind this initial process.
ps -p 1 -o comm=
This command defines what columns should be part of the output, where only shows the actual command. Adding the ‘=’ removes the header, so if systemd is being used, then the string ‘systemd’ will be returned.
How to see all enabled services with systemctl
The systemctl command can be used to show all service units and filter those that are enabled.
Summary
The systemctl command show active unit. To see only services that are enabled, we need to filter the output. This can be achieved using the list-unit-files subcommand and combined with the --state= option. As we are interested in enabled services only, set the value to enabled.
Want to parse the output with a script? Consider adding --legend=false to remove the header and footer text (X unit files listed.).
Usage
systemctl list-unit-files --type=service --state=enabled
UNIT FILE STATE VENDOR PRESET
apparmor.service enabled enabled
blk-availability.service enabled enabled
cloud-config.service enabled enabled
cloud-final.service enabled enabled
cloud-init-local.service enabled enabled
cloud-init.service enabled enabled
console-setup.service enabled enabled
cron.service enabled enabled
dmesg.service enabled enabled
e2scrub_reap.service enabled enabled
finalrd.service enabled enabled
getty@.service enabled enabled
gpu-manager.service enabled enabled
grub-common.service enabled enabled
grub-initrd-fallback.service enabled enabled
irqbalance.service enabled enabled
keyboard-setup.service enabled enabled
lvm2-monitor.service enabled enabled
lxd-agent.service enabled enabled
ModemManager.service enabled enabled
multipathd.service enabled enabled
networkd-dispatcher.service enabled enabled
nginx.service enabled enabled
open-iscsi.service enabled enabled
open-vm-tools.service enabled enabled
pollinate.service enabled enabled
rsyslog.service enabled enabled
secureboot-db.service enabled enabled
setvtrgb.service enabled enabled
snap.lxd.activate.service enabled enabled
snapd.apparmor.service enabled enabled
snapd.autoimport.service enabled enabled
snapd.core-fixup.service enabled enabled
snapd.recovery-chooser-trigger.service enabled enabled
snapd.seeded.service enabled enabled
snapd.service enabled enabled
snapd.system-shutdown.service enabled enabled
ssh.service enabled enabled
systemd-networkd-wait-online.service enabled disabled
systemd-networkd.service enabled enabled
systemd-pstore.service enabled enabled
systemd-resolved.service enabled enabled
systemd-timesyncd.service enabled enabled
thermald.service enabled enabled
ua-reboot-cmds.service enabled enabled
ubuntu-advantage.service enabled enabled
udisks2.service enabled enabled
ufw.service enabled enabled
unattended-upgrades.service enabled enabled
vgauth.service enabled enabled
50 unit files listed.
Nginx hardening profile
Harden the nginx configuration with the help of systemd sandboxing capabilities and restricting resources.
Summary
Introduction
This is a hardening profile to help securing nginx by using systemd unit configuration. It’s goal is to restrict what nginx can do and make it harder for any possible vulnerability to be misused.
The rationale for the selected settings is based on the analysis as part of the article Hardening nginx with systemd security features.
Hardening profiles for systemd
Hardening profiles for systemd that can be used to secure your applications.
Summary
Introduction
Systemd has a range of security features to help securing services running on your system. That is the good part. The big challenge with so many features is that it is hard to find out which ones you could or should apply, without breaking a service. That is why we started working on hardening profiles.
The hardening profiles are pre-defined templates that are documented and tested against a default installation of a piece of software. System administrators then can use this as the foundation of their service and tune it slightly to their specific configuration.
SocketBindAllow setting
Allow systemd units to use system call bind() on sockets specified with the unit setting SocketBindAllow.
Summary
Why and when to use SocketBindAllow
The setting SocketBindAllow is used together with SocketBindDeny and defines restrictions on the usage of the system call bind on a network socket.
Settings
Both SocketBindAllow and SocketBindDeny use a bind-rule. See SocketBindDeny for the details.
Generic advice
This setting is useful in combination with SocketBindDeny to create an allow-list.
Examples
Allow binding on TCP port 80
SocketBindDeny setting
Restrict systemd units to use system call bind() on sockets specified with the unit setting SocketBindDeny.
Summary
Why and when to use SocketBindDeny
The setting SocketBindDeny can be used alone or together with SocketBindAllow to set restrictions on the usage of the system call bind on a network socket.
Settings
If the SocketBindDeny list is used alone, then it is a deny-list. Everything except the defined ports/protocols will be allowed.
By defining the value ‘any’, all combinations are denied. This is typically used in combination with SocketBindAllow to open up one or more ports.
DevicePolicy setting
Restrict systemd units to access devices in the /dev directory with the unit setting DevicePolicy.
Summary
Why and when to use DevicePolicy
The setting DevicePolicy aims to reduce access to devices in /dev. By default, there is no limitation to access devices.
Settings
The value strict is the most strict, as the name implies. This is suitable for services that do not need any access, like custom shell scripts. Unless output is being redirect to /dev/null, then access could be granted with DeviceAllow=/dev/null.
Generic advice
Aim for using ‘strict’ when possible and define entries that should be allowed. To discover files used by a binary, consider inspecting it with the strings command or look at open files from a running process with lsof.
DeviceAllow setting
Restrict systemd units to access devices in the /dev directory with the unit setting DeviceAllow.
Summary
Why and when to use DeviceAllow
The setting DeviceAllow aims to reduce access or its level to devices in /dev. By default, there is no limitation to access devices.
Settings
Define DeviceAllow with the path and access level.
DeviceAllow=/dev/sda3 r
General advice
For most services it might be easier to use ProtectDevices=yes to reduce the devices that can be access.
Troubleshooting a failed systemd unit (with examples)
Learn how to troubleshoot failed systemd units, examples, possible causes, and how to resolve them.
Summary
Discover the reasons why a systemd unit went into a failed state
What does systemctl daemon-reload do?
When making changes to systemd unit files, you may need to use systemctl daemon-reload. This article explains what happens next.
Summary
Systemd stores the configuration for units, like services, in individual unit files. When changes are made to these units, a reload might be needed. This is where systemctl daemon-reload
comes into play. But what exactly does the daemon-reload subcommand really do? In short: rerun generators, reload units files, recreate the dependency tree. Let’s have a look at the more detailed answer.
Running generators
Generators are helper scripts to convert non-native scripts to unit files that are usuable by systemd. These generators are used during the boot, but also when the daemon configuration is reloaded. In multiple phases all generators are started in parrallel, with the goal to minize the time tasks are waiting for each other to finish.
How to check if 'systemctl daemon-reload' is needed
When systemd units are changed, a 'systemctl daemon-reload' might be needed. Need to know if this is the case? Let's test for that.
Summary
Systemd may need to reload a part of the unit configuration if changes were made. To find out if the related systemctl daemon-reload
command is needed, the state of the individual units can be tested. This is done by querying the property using the --property=NeedDaemonReload option.
Testing a single service like nginx, can be done this way:
# systemctl show --property=NeedDaemonReload --value nginx.service
yes
This output will return a ‘yes’ or ’no’ value.
How to see which syscalls are part of a systemd syscall filter set
Learn how to see what syscalls are part of a particular syscall filter set in systemd.
Summary
Systemd can restrict services from using particular syscalls with the help of the unit setting SystemCallFilter. Instead of mentioning all individual syscalls, systemd has predefined sets that can be used. These sets group functions that are related. To see which syscalls are part of a set, use the systemd-analyze command.
# systemd-analyze syscall-filter @ipc
@ipc
# SysV IPC, POSIX Message Queues or other IPC
ipc
memfd_create
mq_getsetattr
mq_notify
mq_open
mq_timedreceive
mq_timedreceive_time64
mq_timedsend
mq_timedsend_time64
mq_unlink
msgctl
msgget
msgrcv
msgsnd
pipe
pipe2
process_madvise
process_vm_readv
process_vm_writev
semctl
semget
semop
semtimedop
semtimedop_time64
shmat
shmctl
shmdt
shmget
See systemd syscall filtering for all details.
SystemCallFilter setting
Define if systemd units are allowed to use specific syscalls or groups with the unit setting SystemCallFilter.
Summary
Why and when to use SystemCallFilter
The setting SystemCallFilter aims to prevent misuse of syscalls that are not needed for normal functioning of a process. This powerful filtering restricts the abilities of a process, but requires understanding of processes by the system administrator. See the overview of Linux syscalls for more details.
Configuration
This setting takes a space-separated list and may be specified multiple times.
Systemd syscall filtering
Learn more about the system calls (syscalls) that systemd may use in commands and unit files, such as with SystemCallFilter property.
Summary
Overview of syscalls in systemd by group
What is the difference between systemctl disable and systemctl mask?
Want to disable a service, but wondering the difference between systemctl disable and systemctl mask? This article shows the differences between the two.
Summary
Systemd and its services can be in several states, such as enabled, disabled, failed, running. If you no longer need a particular service to run, then the first step is to stop a service.
systemctl stop nginx.service
But stopping a service is not the same as disabling a service. With that comes a very frequently asked question: what is the difference between a service that is disabled and one that is masked?
How to use systemctl edit to change a service?
Learn how to edit an existing systemd service unit with the systemctl edit command.
Summary
Systemd allows service units to be configured using a drop-in file, which is often called override.conf. It overrides the vendor-supplied version of a service to customize it. Instead of duplicating the configuration, the override file contains the differences.
Editing service file
Changing a service can be done using systemctl, followed by the edit subcommand and service unit. The editor that is configured on the system will be opened and any changes can be made between the comment section at the top and the comment section a little bit lower. Do not remove these comment sections, as systemctl uses these to see what changes you made.
How to see only running services with systemctl
The systemctl command can be used to filter its output and only show all running services.
Summary
The systemctl command will normally all active units. To filter this output to just the running services, we can combine the options --type= and --state=. For this particular case we set the type to service and the type state to running.
Usage
# systemctl --type=service --state=running --legend=false
accounts-daemon.service loaded active running Accounts Service
avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack
colord.service loaded active running Manage, Install and Generate Color Profiles
dbus-broker.service loaded active running D-Bus System Message Bus
firewalld.service loaded active running firewalld - dynamic firewall daemon
gdm.service loaded active running GNOME Display Manager
NetworkManager.service loaded active running Network Manager
polkit.service loaded active running Authorization Manager
qemu-guest-agent.service loaded active running QEMU Guest Agent
rtkit-daemon.service loaded active running RealtimeKit Scheduling Policy Service
spice-vdagentd.service loaded active running Agent daemon for Spice guests
sshd.service loaded active running OpenSSH Daemon
systemd-journald.service loaded active running Journal Service
systemd-logind.service loaded active running User Login Management
systemd-timesyncd.service loaded active running Network Time Synchronization
systemd-udevd.service loaded active running Rule-based Manager for Device Events and Files
systemd-userdbd.service loaded active running User Database Manager
udisks2.service loaded active running Disk Manager
upower.service loaded active running Daemon for power management
user@1000.service loaded active running User Manager for UID 1000
wpa_supplicant.service loaded active running WPA supplicant
This command will show just services that are running. The legend is set to false, meaning no header and footer are displayed.
Run0 cheat sheet
Learn how to get everything out of the run0 tool to increase your privilege level.
Summary
Elevating permissions
Run0: introduction and usage
Learn the goal and purpose of run0 and how to use it for elevating privileges.
Summary
Elevating permissions
How to disable the background color of run0
Learn how to disable the change of the background color when using run0.
Summary
Systemd introduced run0 as its alternative to sudo. One of the features if a colored background when your privileges are elevated.
To disable this behaviour, use the option --background= with an empty value.
run0 --background=
The red background now will be gone, which can be useful if the color conflicts with the output or when it is unwanted.
MemoryDenyWriteExecute setting
Block the ability for systemd units to create or alter memory segments to become writable and executable as well with the unit setting MemoryDenyWriteExecute.
Summary
Why and when to use MemoryDenyWriteExecute
The setting MemoryDenyWriteExecute will block the creation or alteration of a memory segment to become writable and executable as well. By enabling this limitation, it will increase the bar software exploits to change running code dynamically.
Usage
[Service]
MemoryDenyWriteExecute=yes
InaccessiblePaths=/dev/shm
SystemCallFilter=~memfd_create
Caveats
To prevent circumvention of this setting, access to /dev/shm and the syscall memfd_create should be blocked as well.
Generic advice
For most common services this option can be implemented and will increase the security of a service. That is, if used together with InaccessiblePaths and SystemCallFilter.
InaccessiblePaths setting
Block systemd units to access specified paths with the unit setting InaccessiblePaths.
Summary
Why and when to use InaccessiblePaths
The setting InaccessiblePaths defines paths that should never be accessible. Instead of using the principles of an allow list, it is an explicit deny list. It can be used to block access by a process to a location with sensitive data or a path commonly misused for exploits.
How to see memory usage of a service with systemctl?
The systemctl command can be used to show the memory usage of a service managed by systemd.
Summary
The systemctl command has multiple options to show the memory usage. With the status subcommand followed by the service, it will show the basics, including memory usage.
To retrieve the information that easier to parse, then use show followed by --property=MemoryCurrent and the service name.
Usage
The status output will include memory usage.
systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/nginx.service.d
└─override.conf
Active: active (running) since Mon 2024-06-17 17:59:45 UTC; 3h 54min ago
Docs: man:nginx(8)
Main PID: 36971 (nginx)
Tasks: 2 (limit: 1012)
Memory: 2.6M
CPU: 26ms
CGroup: /system.slice/nginx.service
├─36971 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
└─36972 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
For pulling in the information that can be parsed or scripted, consider retrieving the actual property from a running service.
How to see the active settings of a systemd unit
The systemctl command can be used to show the settings of a systemd unit, like a service.
Summary
The systemctl command can be used to show all settings of an unit, such as a service. To display the full list of applicable settings, use the show subcommand followed by the unit name.
Besides the settings, the output will also include actual runtime information, such as memory usage, when the unit was started, etc.
Usage
Just provide the unit file to see all available information.
# systemctl show nginx.service
Type=forking
Restart=no
PIDFile=/run/nginx.pid
NotifyAccess=none
RestartUSec=100ms
TimeoutStartUSec=1min 30s
TimeoutStopUSec=5s
TimeoutAbortUSec=5s
TimeoutStartFailureMode=terminate
TimeoutStopFailureMode=terminate
<snip>
How to override the settings of a systemd unit
The systemctl command can be used to override settings of a systemd unit, like a service.
Summary
The systemctl command can show settings of a systemd unit, such as a service. It can also assist in overriding these settings by using the edit subcommand followed by the unit name. This will open the editor that is configured on the system and create the override file.
Usage
Run the edit command with the unit, and the editor like vim or nano will show up.
### Editing /etc/systemd/system/nginx.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file
[Service]
ProtectSystem=strict
ReadWritePaths=/run /var/log/nginx
### Lines below this comment will be discarded
<snip>
Important: Do not remove the comments and only insert or change between the specified comment lines.
ReadWritePaths setting
Grant systemd units to specified paths to read from and write to new or existing files with the unit setting ReadWritePaths.
Summary
Why and when to use ReadWritePaths
The setting ReadWritePaths grants read and write permissions to defined paths. It can be used in combination with other settings like ‘ProtectSystem=strict’ to make the full file system read-only, and then open up a few paths that are required for a service to run correctly.
Values
Define the paths that are granted write access.
[Service]
ProtectSystem=strict
ReadWritePaths=/run /var/log/nginx
- When a path is prefixed with a minus (-), it is ignored if it does not exist
- When a path is prefixed with a plus (+), the path is considered relative to root of directory (e.g. configured with RootDirectory)
Caveats
This setting will not have effect if a process is missing the normal file permissions or ownership. For additional sandboxing, consider CapabilityBoundingSet=CAP_SYS_ADMIN or SystemCallFilter=@mount.
Hardening nginx with systemd security features
Secure your nginx service by using security features provided by systemd.
Summary
Introduction
Nginx is still a popular web server and powering a part of the web. Wouldn’t it be great if we could secure it a little bit more? In this article we use the security features to secure systemd units and services and apply it to nginx.
If you are not familiar yet with the unit settings of systemd, then this document would be a good introduction into the subject.
File paths
Like most services, nginx uses files to read and write. Typically web site content is read from the disk, while log files are created and written to. We can use the systemd unit settings to restrict access to only those paths that are strictly required for nginx to run. But how to find out what those paths are?
Systemd features to secure units and services
Learn more about systemd features that help in securing units and services.
Summary
Secure services with these features
ProcSubset setting
Restrict systemd units to access information from the /proc directory with the unit setting ProcSubset.
Summary
Why and when to use ProcSubset
The setting ProcSubset controls the “subset” mount option of /proc for the unit.
Caveats
This function does not if the “subnet” option for procfs is not supported.
Generic advice
The Linux kernel shares information from various kernel APIs via /proc. When activating this setting, these kernel APIs are also made unavailable, which might break common software, unless it is a trivial process. So this option is to be used with care. Typically it may be better to implement the ProtectProc setting.
RestrictAddressFamilies setting
Restrict systemd units using only specified socket address families with the unit setting RestrictAddressFamilies.
Summary
Why and when to use RestrictAddressFamilies
The setting RestrictAddressFamilies aims to restrict what socket address families can be used. When using it, the default is that it is used as an allow-list and define what address families can be used.
Settings
When this setting is not configured, there are no restrictions to what address families can be used.
Setting the value to none will block all address families.
To block specific address families only, a ~ can be used to turn the allow-list into a deny-list.
ProtectProc setting
Restrict systemd units to access information from the /proc directory with the unit setting ProtectProc.
Summary
Why and when to use ProtectProc
The setting ProtectProc aims to protect information that normally can be retrieved from /proc.
Settings
The value default, which is also the default, will not restrict access. Value invisible will hide information, where ptraceable restrict the set to only processes that be monitored with the system call ptrace(). The value noaccess is the most strict option.
Caveats
This setting will not have effect if the kernel does not support the hidepid mount option per individual mount point.
ProtectHome setting
Restrict systemd units to access data in home directories with the unit setting ProtectHome.
Summary
Why and when to use ProtectHome
The setting ProtectHome aims to protect home directories. These three paths are included:
- /home
- /root
- /run/user
Settings
The default no will not restrict access to the home directories. Using yes will active full protection, not allowing access.
The value read-only will make the paths read-only, so no data can be written to it.
With tmpfs a temporary file system is being used, also read-only, yet it hides the actual home directories. It will still allow access to the actual directories when using BindPaths or BindReadOnlyPaths. For rare situations this might be useful to protect the home directories, while still allowing some very specific access.
ProtectKernelLogs setting
Restrict systemd units to read or write to the kernel log ring buffer with the unit setting ProtectKernelLogs.
Summary
Background
The Linux kernel exposes its kernel log ring buffer to userspace via /dev/kmsg and /proc/kmsg.
When this setting is defined as yes, the capability CAP_SYS_MODULE will be removed from the capability bounding set. This means that all processes in the unit will no longer have access to the kernel log ring buffer.
ProtectKernelModules setting
Restrict systemd units to load kernel modules with the ProtectKernelModules unit setting.
Summary
Explanation
Kernel modules can provide additional functionality when using a modular Linux kernel, which is applicable to most systems. When this setting is set to yes, it tries to prevent the unit from loading kernel modules. This is achieved by removing the CAP_SYS_MODULE from the capability bounding set.
Generic advice
Most units do not need the permission to load kernel modules, so typically a unit can be configured with ProtectKernelModules=true.
How to see the time synchronization details with timedatectl
Show time synchronization details with the systemd timedatectl command and related subcommands.
Summary
The timedatectl command can show the time, time zone information, and its status. Add the timesync-status subcommand to see synchronization details.
Usage
Use timedatectl with the timesync-status command to see the actual status. Under normal conditions, the leap should show ’normal'.
# timedatectl timesync-status
Server: 185.125.190.56 (ntp.ubuntu.com)
Poll interval: 34min 8s (min: 32s; max 34min 8s)
Leap: normal
Version: 4
Stratum: 2
Reference: 4FF33C32
Precision: 1us (-25)
Root distance: 762us (max: 5s)
Offset: +882us
Delay: 15.169ms
Jitter: 7.098ms
Packet count: 711
Frequency: +2.731ppm
How to show the systemd machine ID
Find the machine ID that was generated by systemd.
Summary
With the hostnamectl command basic system information like the operating system, hostname, and machine ID can be displayed.
Usage
Run the command without any parameters to get the status displayed, including the machine ID.
hostnamectl
How to see the dependencies of a systemd unit
The systemctl command has the list-dependencies option to show dependencies between units. But there are more options to query a little bit more information.
Summary
The systemctl command can be used to show dependencies between units with the list-dependencies subcommand. A nicely human-readable output will be displayed showing the selected unit, followed by the dependencies that rely on this unit. This is useful when a unit is in a failed state due to a dependency on another unit.
Usage
To see which units require the multi-user target to be active:
# systemctl list-dependencies multi-user.target
multi-user.target
● ├─apport.service
● ├─console-setup.service
● ├─cron.service
● ├─dbus.service
○ ├─dmesg.service
○ ├─e2scrub_reap.service
○ ├─grub-common.service
○ ├─grub-initrd-fallback.service
○ ├─irqbalance.service
<snip>
Want to move up the other way and see on which our unit requires? Add the --reverse option.
How to see the available systemd unit types
The systemctl command can be used to show all available systemd unit types.
Summary
The systemctl command can show the available systemd unit types when using the option --type=help.
Usage
# systemctl --type=help
Available unit types:
service
mount
swap
socket
target
device
automount
timer
path
slice
scope
How to see all active systemd units of one type
The systemctl command can be used to show all active systemd units of one particular type with the --type option.
Summary
The systemctl command will show by default all active units. To filter down on a particular unit type, use the --type= option, followed by the type. Not sure what types are available? Run systemctl --type=help
.
Usage
systemctl list-units --type=target
How to limit the disk usage of the systemd journal
Learn how to define the maximum size that the systemd journal daemon may use for storing journals.
Summary
To limit the maximum size that journals may use on the system, define the setting SystemMaxUse in /etc/systemd/journald.conf. Save the file, confirm that the settings are correct, then restart the journal daemon.
Configuration
Open /etc/systemd/journald.conf, copy the commented line, remove the hash, and assign it a value.
SystemMaxUse=256M
Note: depending on how many events happen on a system, this value might be too small. Make sure that the size for logs is big enough.
How to see the size of the systemd journal
Summary
The journalctl command can be used to show the journal. By using the --disk-usage option, the size of the journal is displayed. This includes the archived and active journal files. When the journal is using too much disk space, consider performing a vacuum task.
Usage
Showing the disk usage is quick and easy.
# journalctl --disk-usage
Archived and active journals take up 56.0M in the file system.
Does the journal take up too much space?
How to see kernel messages with journalctl
Learn how to show all kernel events by using journalctl and filter out the kernel entries in the journal.
Summary
The journalctl command can show all events related to the kernel itself usig the --dmesg option. This option will filter out kernel messages and has a similar output as the dmesg
command.
Usage
Use the full or shorter option to query the kernel messages.
journalctl -k
Looking for only the kernel messages of today? Combine it with the --since= option.
journalctl -k -S "today"
What is a systemd unit?
Learn more about systemd units and what they do.
Summary
Systemd units define resources that can be used by the system. Examples of these units are a service, path, socket, and timer. Each unit type has its own basic set of properties that then individually can be configured. Unit types can be recognized by their file extension. A service will use the ‘.service’ extension, making it easy to recognize. Units are usually managed with the systemctl command.
See systemd unit types and their purpose for a full overview of the units.
How to see only recent journal entries
Learn how to filter journal entries by specifying a date or time interval.
Summary
The journalctl command shows by default the oldest entries it has in the journal. Typically we are not interested in that, for that purpose there is the --since= option. This option defines that entries should be after the specified moment in time. Besides using an actual date, a shortened name like ’today’ can also be used that automatically defines the date and time.
Usage
To see the entries of today, use the aptly named ’today'.
How to see new log entries automatically with journalctl
Learn how to continuously show new log entries with journalctl like the tail -f command.
Summary
The journalctl command can show continuously new log entries with the --follow option. When new entries are added to the journal, they are automatically shown.
Usage
The follow option is a great option to continuously monitor a particular unit.
journalctl --follow --unit=nginx.service
Without providing a unit, all system events will be shown and followed.
How to see logging for a specific unit or service
Limit the number of log entries from the systemd journal by filtering journalctl output by unit.
Summary
The journalctl command can show the events from its journal by --unit= followed by the service or its unit name. This way events will be filtered, making it much easier to troubleshoot a particular service.
Example
journalctl -u nginx.service
How to reload the systemd configuration
How can systemd be instructed to reload its configuration?
Summary
Reload systemd
What is systemd?
Learn what systemd is and the main components of this system and service manager.
Summary
Systemd is a system and service manager. The name is short for ‘system daemon’, an ongoing service that manages the system. As it is also a service manager, it is responsible for start, stopping, and monitoring services. Systemd replaces the SysV init system and focuses on performance and resource management. It was created by Lennart Poettering in 2010, with Fedora Linux being the first to adopt it in May 2011. In 2015, several major Linux distributions started shipping with systemd.
What is a masked systemd unit?
What does it mean when a systemd unit is masked? Learn about this state.
Summary
Systemd units that are in a masked state are administratively disabled. While being in this state, they can not be started until they are unmasked. Typically a unit is masked when it should not start by default or manually, to prevent it causing issues or running an unwanted service. With systemctl and the subcommand mask, a systemd unit can be masked.
Relevant FAQ: How to see all masked units with systemctl?
Systemd commands
All commands related to systemd in one overview. Learn about their purpose and when to use them.
Summary
Systemd has a range of commands to interact with the systemd services and functions. Typically daemons will have a ’d’ in their name, such as systemd-networkd, while client tools end with ‘ctl’ (systemctl). Here is an overview of the available commands and their purpose.
Primary commands
Journalctl
Events are stored in a journal, similar to what syslog does. The biggest exception is the way log information is stored and how it is queried. As data is stored in a binary format, the journalctl command can filter out relevant information.
Systemd timers
Learn about systemd timers, the unit type for scheduled tasks and how it differs from cron.
Summary
Learn about the available systemd unit types
Show to clear the DNS cache with systemd
Learn how to inspect and clear the DNS cache when using the systemd resolver daemon.
Summary
Clear DNS cache using resolvectl
Resolvectl
The command resolvectl provides details about systemd-resolved, the name resolution daemon.
Summary
Settings for systemd units
Units in systemd have their own set of configuration settings. This overview shows the availability and their purpose.
Summary
Systemd allows fine-grained customization of units by defining so-called properties. These properties or settings influence what a unit, such as a service, can or can not do. As their is a wide range of settings, this page has the goal to present them, including a quick reference to each of them.
Systemd
Everything related to systemd in one place. From the basics like the different units tips, up to advanced troubleshooting.
Summary
Introduction
Systemd is a system and service manager for Linux. For many Linux distributions it replaced the existing SysV init system, modernizing how services are started and monitored.
Some basics about systemd:
- Author: Lennart Poettering
- First release: 2010
- First adopter: Fedora Linux
- Common usage by major Linux distributions: 2015
Learn more: What is systemd?
Systemd units
To monitor and manage services on a system using systemd, unit files are used. These text-based files define what to run or do, relevant conditions, and any applicable dependencies.
Systemd settings
Units in systemd have their own set of configuration settings. This overview shows the availability and their purpose.
Summary
How to see all masked units with systemctl
Want to find all masked unit files? In this article we show how to do this with systemctl and query those units.
Summary
Show all masked units
How to see the last X lines with journalctl
Limit the output from journalctl by defining the number of lines you want to see.
Summary
Perform smarter queries when requesting information from journalctl
How to disable a systemd unit with systemctl
Want to disable a service or specific systemd unit? Use systemctl to configure units and disable it on boot or completely.
Summary
Disable a service or specific unit with systemctl
How to start and enable a unit with systemctl
Combine the start and enable command when using systemctl to get a unit like a service started at boot and right away.
Summary
Start and enable a unit with one command
How to show failed units with systemctl
Want to check the system for failed systemd units? In this article we show how to do this with systemctl and query the units with a failure state.
Summary
Show failed systemd units with systemctl
Systemd: Frequently Asked Questions
Frequently asked questions about systemd, systemctl, and journalctl. Learn by pratical examples how to use these tools.
Summary
Systemd cheat sheet
Increase your system administration skills with this systemd cheat sheet, including how to configure and monitor systemd units.
Summary
Make a new friend?
Systemd units and their purpose
Which systemd unit types are available and what is their goal? In this article we cover them and show some useful commands related to these units.
Summary
Learn about the available systemd unit types
Systemctl cheat sheet
Learn how to get every piece of information from systemd units, such as services and timers, including its configuration and status.
Summary
Control those processes and timers
Journalctl cheat sheet
Learn how to get every piece of information from systemd journals with the journalctl command. This cheat sheet will help you with the task.
Summary
Query the journal and find the needle
Finding boot logs in systemd journals
This article shows how to find boot logs in the systemd journal. Learn the commands to query all relevant information.
Summary
Systemd used a binary log to store information about specific events. These events include the boot sequence and the related output. In this article we have a look at finding our boot logs in systemd journals.
Binary logging
When using systemd, boot data is stored in journals, a binary format. There is big benefit of saving boot data in a binary format: log information of each boot can be stored separately, linked to other pieces of information, and queried easier and quicker. For example, different boots can be compared, as they are individually available.
Auditing systemd: solving failed units with systemctl
Sometimes systemd units like services and timers may fail. Learn how to troubleshoot such issues and resolve them much easier.
Summary
Solving failed units with systemctl
Systemd is an alternative service manager to the more traditional init system. To ensure the system is healthy, failed units should be investigated on a regular basis. Sooner or later a unit might fail and showing up the systemctl listing. In this article we have a look at how to solve it.
Why do services fail?
During the start of the system, enabled services are started and queued to be executed. Most processes will start correctly and systemd logs the related status in the journal. However, in some cases a service might enter a “failed state”, as a result of another command not finishing properly.