System administration

Show directory contents sorted by modification date and time

Systemd uses different types of units. One of them is the timer unit, which can be used to schedule a periodic task. This timer unit is linked to an existing service unit and will activate the service according to the defined schedule. The timer unit can be defined using the information about systemd timers. Timers use one or more OnCalendar definitions to specify when execution of the timer should happen. As systemd timers are very versatile and provide good monitoring options, they are a good replacement of cronjobs.

Systemd is nowadays a common system and service manager for Linux systems. But how do you know for sure that it is being active? The easiest way is to have a look at PID number 1. This is the first process started after the kernel itself. With the help of ps we can determine the underlying command behind this initial process. ps -p 1 -o comm= This command defines what columns should be part of the output, where only shows the actual command.

When the SSH agent is running, the ssh-add command can be used to load a SSH key. The SSH agent then will request a password when needed, and load the key details in memory. Adding the SSH key just requires the path to the private key. ssh-add ~/.ssh/id_ed25519 When the key is loaded, use the -l or -L option to show the identities that the SSH agent has. ssh-add -l

The systemctl command show active unit. To see only services that are enabled, we need to filter the output. This can be achieved using the list-unit-files subcommand and combined with the --state= option. As we are interested in enabled services only, set the value to enabled. Want to parse the output with a script? Consider adding --legend=false to remove the header and footer text (X unit files listed.). Usage systemctl list-unit-files --type=service --state=enabled UNIT FILE STATE VENDOR PRESET apparmor.

Discover the reasons why a systemd unit went into a failed state

Systemd stores the configuration for units, like services, in individual unit files. When changes are made to these units, a reload might be needed. This is where systemctl daemon-reload comes into play. But what exactly does the daemon-reload subcommand really do? In short: rerun generators, reload units files, recreate the dependency tree. Let’s have a look at the more detailed answer. Running generators Generators are helper scripts to convert non-native scripts to unit files that are usuable by systemd.

Systemd may need to reload a part of the unit configuration if changes were made. To find out if the related systemctl daemon-reload command is needed, the state of the individual units can be tested. This is done by querying the property using the --property=NeedDaemonReload option. Testing a single service like nginx, can be done this way: # systemctl show --property=NeedDaemonReload --value nginx.service yes This output will return a ‘yes’ or ’no’ value.

Systemd can restrict services from using particular syscalls with the help of the unit setting SystemCallFilter. Instead of mentioning all individual syscalls, systemd has predefined sets that can be used. These sets group functions that are related. To see which syscalls are part of a set, use the systemd-analyze command. # systemd-analyze syscall-filter @ipc @ipc # SysV IPC, POSIX Message Queues or other IPC ipc memfd_create mq_getsetattr mq_notify mq_open mq_timedreceive mq_timedreceive_time64 mq_timedsend mq_timedsend_time64 mq_unlink msgctl msgget msgrcv msgsnd pipe pipe2 process_madvise process_vm_readv process_vm_writev semctl semget semop semtimedop semtimedop_time64 shmat shmctl shmdt shmget See systemd syscall filtering for all details.

Overview of syscalls in systemd by group

Systemd and its services can be in several states, such as enabled, disabled, failed, running. If you no longer need a particular service to run, then the first step is to stop a service. systemctl stop nginx.service But stopping a service is not the same as disabling a service. With that comes a very frequently asked question: what is the difference between a service that is disabled and one that is masked?

Systemd allows service units to be configured using a drop-in file, which is often called override.conf. It overrides the vendor-supplied version of a service to customize it. Instead of duplicating the configuration, the override file contains the differences. Editing service file Changing a service can be done using systemctl, followed by the edit subcommand and service unit. The editor that is configured on the system will be opened and any changes can be made between the comment section at the top and the comment section a little bit lower.

The systemctl command will normally all active units. To filter this output to just the running services, we can combine the options --type= and --state=. For this particular case we set the type to service and the type state to running. Usage # systemctl --type=service --state=running --legend=false accounts-daemon.service loaded active running Accounts Service avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack colord.service loaded active running Manage, Install and Generate Color Profiles dbus-broker.

Elevating permissions

Elevating permissions

Systemd introduced run0 as its alternative to sudo. One of the features if a colored background when your privileges are elevated. To disable this behaviour, use the option --background= with an empty value. run0 --background= The red background now will be gone, which can be useful if the color conflicts with the output or when it is unwanted.

The systemctl command has multiple options to show the memory usage. With the status subcommand followed by the service, it will show the basics, including memory usage. To retrieve the information that easier to parse, then use show followed by --property=MemoryCurrent and the service name. Usage The status output will include memory usage. systemctl status nginx ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.

The systemctl command can be used to show all settings of an unit, such as a service. To display the full list of applicable settings, use the show subcommand followed by the unit name. Besides the settings, the output will also include actual runtime information, such as memory usage, when the unit was started, etc. Usage Just provide the unit file to see all available information. # systemctl show nginx.service Type=forking Restart=no PIDFile=/run/nginx.

The systemctl command can show settings of a systemd unit, such as a service. It can also assist in overriding these settings by using the edit subcommand followed by the unit name. This will open the editor that is configured on the system and create the override file. Usage Run the edit command with the unit, and the editor like vim or nano will show up. ### Editing /etc/systemd/system/nginx.service.d/override.conf ### Anything between here and the comment below will become the new contents of the file [Service] ProtectSystem=strict ReadWritePaths=/run /var/log/nginx ### Lines below this comment will be discarded <snip> Important: Do not remove the comments and only insert or change between the specified comment lines.

Introduction Nginx is still a popular web server and powering a part of the web. Wouldn’t it be great if we could secure it a little bit more? In this article we use the security features to secure systemd units and services and apply it to nginx. If you are not familiar yet with the unit settings of systemd, then this document would be a good introduction into the subject.

Secure services with these features

The control group of a process can be retrieved from the /proc directory. We only need to know the PID of the process, which can be found using ps or pidof. Usage If we know that our PID is 1234, then showing the cgroup is as easy as using cat to see the contents of the ‘cgroup’ file. cat /proc/1234/cgroup To see the cgroup for the nginx process (or one of them), we could something like this.

The ps command can show the control group of a process using the -o option, followed by the right column names. Usage To show processes and the control group, we can filter the output columns. # ps -e -o pid,cgroup:64,args PID CGROUP COMMAND 1 0::/init.scope /lib/systemd/systemd --system --deserialize 58 2 - [kthreadd] 3 - [rcu_gp] <snip> 576 - [xprtiod] 634 0::/system.slice/dbus.service @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only 640 0::/system.slice/networkd-dispatcher.service /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers 645 - [nfsiod] 653 0::/system.

The timedatectl command can show the time, time zone information, and its status. Add the timesync-status subcommand to see synchronization details. Usage Use timedatectl with the timesync-status command to see the actual status. Under normal conditions, the leap should show ’normal'. # timedatectl timesync-status Server: 185.125.190.56 (ntp.ubuntu.com) Poll interval: 34min 8s (min: 32s; max 34min 8s) Leap: normal Version: 4 Stratum: 2 Reference: 4FF33C32 Precision: 1us (-25) Root distance: 762us (max: 5s) Offset: +882us Delay: 15.

With the hostnamectl command basic system information like the operating system, hostname, and machine ID can be displayed. Usage Run the command without any parameters to get the status displayed, including the machine ID. hostnamectl

The systemctl command can be used to show dependencies between units with the list-dependencies subcommand. A nicely human-readable output will be displayed showing the selected unit, followed by the dependencies that rely on this unit. This is useful when a unit is in a failed state due to a dependency on another unit. Usage To see which units require the multi-user target to be active: # systemctl list-dependencies multi-user.target multi-user.target ● ├─apport.

The systemctl command can show the available systemd unit types when using the option --type=help. Usage # systemctl --type=help Available unit types: service mount swap socket target device automount timer path slice scope

The systemctl command will show by default all active units. To filter down on a particular unit type, use the --type= option, followed by the type. Not sure what types are available? Run systemctl --type=help. Usage systemctl list-units --type=target

To limit the maximum size that journals may use on the system, define the setting SystemMaxUse in /etc/systemd/journald.conf. Save the file, confirm that the settings are correct, then restart the journal daemon. Configuration Open /etc/systemd/journald.conf, copy the commented line, remove the hash, and assign it a value. SystemMaxUse=256M Note: depending on how many events happen on a system, this value might be too small. Make sure that the size for logs is big enough.

The journalctl command can be used to show the journal. By using the --disk-usage option, the size of the journal is displayed. This includes the archived and active journal files. When the journal is using too much disk space, consider performing a vacuum task. Usage Showing the disk usage is quick and easy. # journalctl --disk-usage Archived and active journals take up 56.0M in the file system. Does the journal take up too much space?

The journalctl command can show all events related to the kernel itself usig the --dmesg option. This option will filter out kernel messages and has a similar output as the dmesg command. Usage Use the full or shorter option to query the kernel messages. journalctl -k Looking for only the kernel messages of today? Combine it with the --since= option. journalctl -k -S "today"

Systemd units define resources that can be used by the system. Examples of these units are a service, path, socket, and timer. Each unit type has its own basic set of properties that then individually can be configured. Unit types can be recognized by their file extension. A service will use the ‘.service’ extension, making it easy to recognize. Units are usually managed with the systemctl command. See systemd unit types and their purpose for a full overview of the units.

The journalctl command shows by default the oldest entries it has in the journal. Typically we are not interested in that, for that purpose there is the --since= option. This option defines that entries should be after the specified moment in time. Besides using an actual date, a shortened name like ’today’ can also be used that automatically defines the date and time. Usage To see the entries of today, use the aptly named ’today'.

The journalctl command can show continuously new log entries with the --follow option. When new entries are added to the journal, they are automatically shown. Usage The follow option is a great option to continuously monitor a particular unit. journalctl --follow --unit=nginx.service Without providing a unit, all system events will be shown and followed.

The journalctl command can show the events from its journal by --unit= followed by the service or its unit name. This way events will be filtered, making it much easier to troubleshoot a particular service. Example journalctl -u nginx.service

Reload systemd

Systemd is a system and service manager. The name is short for ‘system daemon’, an ongoing service that manages the system. As it is also a service manager, it is responsible for start, stopping, and monitoring services. Systemd replaces the SysV init system and focuses on performance and resource management. It was created by Lennart Poettering in 2010, with Fedora Linux being the first to adopt it in May 2011. In 2015, several major Linux distributions started shipping with systemd.

Systemd units that are in a masked state are administratively disabled. While being in this state, they can not be started until they are unmasked. Typically a unit is masked when it should not start by default or manually, to prevent it causing issues or running an unwanted service. With systemctl and the subcommand mask, a systemd unit can be masked. Relevant FAQ: How to see all masked units with systemctl?

The ProxyJump defines a bastion host (jump host, jump server, jump box) to use. Values Value Meaning none Disable ProxyJump functionality HOST Define the hostname of the bastion host [USER]HOST[:PORT] Define one or more parameters of the bastion host URI Define parameters in URI format Hostname Format: hostname User Format: user@hostname Port Format: hostname:port Command-line usage ssh -J bastion destinationsystem These parameters can also be specified in the format as a URI .

The ForwardAgent option specifies if SSH agent forwarding is allowed or not. ForwardAgent values Value Meaning Yes Agent forwarding is allowed No (default) Agent forwarding is not allowed PATH Path to the agent socket $VARIABLE Environment variable that stores the path Security caution Agent forwarding should not be used if not strictly needed. Any user that can access the agent’s socket stored in SSH_AUTH_SOCK may have access through the forwarded connection.

The agent forwarding feature in SSH allows using your local SSH agent to be reached through an existing SSH connection. This way you don’t have to store copies of your private keys on intermediate systems to use them for authentication. While SSH agent forward simplifies things, it also introduces a new risk related to Unix domain socket. If a user on the intermediate system can access the related socket, then it may abuse this connection back to the SSH agent to authenticate on your behalf.

The ssh-agent command is started manually using eval $(ssh-agent). This will initiate the SSH agent and make it available for clients, such as ssh, to use it. To confirm that the agent is running is by looking at the SSH_AUTH_SOCK environment variable. Automatic start of SSH agent Gnome Keyring SSH Agent When using Gnome, it typically comes with its SSH agent as part of Keyring. This will automatically load any files in ~/.

The ssh-agent command starts the SSH agent, a helper utility to store private keys when using public key authentication. The ssh-agent process is usually started at the the beginning of a login session and then can be connected to by a SSH client. Clients can detect the environment variable named SSH_AUTH_SOCK. Related settings on the client IdentityAgent

Disable usage of SSH agent identities

The IdentityAgent option specifies what UNIX-domain socket to use to communicate with the authentication agent. When configured, it overrides the environment variable SSH_AUTH_SOCK and provides the option to select a specific agent. Besides the option to define a socket, the location of the socket can also be provided by the SSH_AUTH_SOCK environment variable. In that case the value should defined as “SSH_AUTH_SOCK” (without quotes). If the value starts with a ‘$’, it indicates that another enviroment variable is to be used.

The file /etc/ssh/ssh_config is the main configuration file of the OpenSSH server daemon (sshd). Purpose It defines the system-wide settings for the SSH client. It can be overriden by storing settings in ~/.ssh/config that takes precedence. Configuration To learn more about the available configuration settings, have a look at the section OpenSSH client configuration.

Configuration files /etc/ssh/ssh_config ~/.ssh/config Configuration settings The client knows a lot of individual settings that may be configured in a configuration file or via the command-line. IdentityAgent Configure the socket to use for the SSH agent. See IdentityAgent Port Defines the port to be used. By default, the SSH server is configured with port 22 (TCP). ProxyJump Defines a bastion host (jump host, jump server, jump box) to connect to the destination system.

For the SSH client there are typically two places where configuration files are stored: in the home directory of the user and a global configuration file. User configuration Location: ~/.ssh/config This file is stored in the home directory of an user. It is optional and by default no file is available. System-wide configuration Location: /etc/ssh/ssh_config Overrides via: /etc/ssh/ssh_config.d/*.conf The default settings are in the ssh_config file. The system administrator may add customizations to this file, but typically it is advised to override settings using a separate configuration file.

Querying pacman

The SSH client won’t connect to a system when it sees that host key changed since the initial connection it made. This helps against MitM attacks. The client knows when the host key is different by comparing it with the related values in the ~/.ssh/known_hosts file. Values Value Automatically save new host keys Action if host key changed yes No Refuse ask No, ask Refuse accept-new Yes Refuse no | off Yes Connect When connecting to many different systems, the accept-new value can help reducing the manual step to accept keys.

One of the common methods to authenticate with a SSH server is using the combination of a username and password. With the option PasswordAuthentication we can define if we want to use this type of authentication. While yes is the default, it might be useful to disable it for hosts that require public key authentication. This way we instruct the client to only try that.

Killing processes with a filter

Disable check for host authenticity

Why change your SSH port? Systems that are available via the internet and can’t be fully protected with a firewall, they might benefit from running on a different TCP port than the default 22. This way automated scanners will less likely probe your system(s), as they don’t know what port you use for SSH. Changing your SSH port won’t make a system more secure in itself, and therefore is often called security through obscurity.

Configure a Message of the Day The MOTD is typically stored in /etc/motd or a related directory, such as /etc/update-motd.d or /etc/motd.d/. The related message or messages are then displayed after a user is logged in. Another option is that the MOTD is displayed using the PAM configuration. To see if this is the case, perform a grep in your PAM configuration directory. grep -ir motd /etc/pam.d | grep -v ":#"

Escape sequences are a patterns of keys that are recognized to send special instructions. To see the ones that are supported, press tilde (~) followed by a question mark (?). Supported escape sequences: ~. - terminate connection (and any multiplexed sessions) ~B - send a BREAK to the remote system ~C - open a command line ~R - request rekey ~V/v - decrease/increase verbosity (LogLevel) ~^Z - suspend ssh ~# - list forwarded connections ~& - background ssh (when waiting for connections to terminate) ~?

Use an escape sequence to terminate a connection that is stuck

Remove a passphrase from existing SSH key

How to see the available SSH keys in the OpenSSH authentication agent

The kill command is used on Linux to send a process signals. This can be a numeric value or its defined name (e.g. SIGTERM).

Zombies…

Killing zombies, for fun?

Search for PID and process name

Retrieve PIDs for a service

Linux uses signals to interact and define the state of a process. It uses POSIX reliable and real-time signals. The first are considered standard signals. Many programs are build using glibc and therefore use functions like kill(2) to send a signal to a process or processes group, or even all processes on the system. A process can decide to ignore a signal or take an action after it is received by a signal handler, a routine to catch incoming signals.

Stop a process by searching for its name

Show your local IP address

Query the IP address of your internet connection

Show the active DNS server

Find the files that are writable

Managing packages

Show disk usage by files and directories

Show hidden files

Show files bigger or smaller than a specified size

Show when the last modification was made within a directory

Show used and free disk space

Show used and free disk space

Find symbolic links pointing to a directory

Compare two directories, find their differences and what they have in common

The file /etc/ssh/sshd_config is the main configuration file of the OpenSSH server daemon (sshd). Purpose It defines the primary set of settings, with the option of being overridden by configuration files from /etc/ssh/sshd_config.d/*.conf. Configuration To learn more about the available configuration settings, have a look at the section OpenSSH server configuration.

Learn about the available systemd unit types

Show number of open connections per protocol

Show start time of a process

Show uptime of the system

Show actual bandwidth usage

Got a busy system that comes to a halt due it being too busy? In this article we look at troubleshooting issues related to CPU usage. Monitoring CPU usage The tool top might be the most familiar tool to monitor CPU or memory usage. A good alternative is the pidstat tool. It can be using an interval and easily show active processes, followed by a summary. # pidstat 3 Linux 6.

Clear DNS cache using resolvectl

Network connectivity starts at a device that links the system to a network, and for Linux systems that is no different. Depending on the physical layer, such as Ethernet of Wi-Fi, the transactions between be better or worse. Besides physical limitations, there is congestion and packet loss that may introduce issues. In this section we look at troubleshooting network performance issues, tooling, and examples. Interfaces The first step is to learn what links we have towards the network.

Show TCP connection statistics

The lscpu command retrieves information about the the CPU architecture. The sources to retrieve this information include /proc/cpuinfo, pseudo file system sysfs, and available libraries. Examples of information includes: Cores CPUs BogoMIPS Byte order Cache sharing Caches Family Model Stepping NUMA nodes Threads Sockets Virtualization details Vulnerabilities The tool is both suitable for using in the terminal and manual interpretation, but also for automated processing. When its output is piped to other another application, it is aware that any screen markup should be left out.

Show CPU details

Users of the command-line are typically aware of the netstat command. The package that netstat belongs to is getting deprecated and slowly replaced with tools like ip. This is also where nstat comes in, a more modern approach to pulling in data from the kernel. Another benefit is that it can retrieve more information than netstat. This is due to the static list of metrics that netstat looks for, while nstat will parse output files from /proc.

No network packet will remain hidden

Find the Linux distribution name and version

All hardware exposed

Show memory details

Learn how to purge data before deleting a file

Find out when a file was initially created

The stat command can be used to show file statistics, such as file size, ownership, type, and several timestamps. It is a great addition to ls. Time to let it work for us! Basic example If we use the command on our /etc/passwd file, we might get output like below. # stat /etc/passwd File: /etc/passwd Size: 3387 Blocks: 8 IO Block: 4096 regular file Device: 10303h/66307d Inode: 47186412 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-08 15:17:01.

Archiving all the data

Learn what it means when the kernel is tainted

Learn about the specific cause that tainted the kernel

No more networking secrets

Show network link statistics to discover errors or dropped packets

Show network table to discover the default gateway

Show which process is listening to a port

Show open file information

Show open network ports such as TCP and UDP

Reveal all those sockets

Query DNS to reveal the TTL value of a DNS record.

Query tools like dpkg to show installed packages

Show installed package on the most common Linux distributions

Retrieve USB device information using lsusb

Query the available hard disk(s)

More in-depth information about the available hard disks

Show BIOS information using dmidecode

Find out who is using up that disk space

Leverage the du command to find the biggest directories

Show all masked units

Perform smarter queries when requesting information from journalctl

Disable a service or specific unit with systemctl

Start and enable a unit with one command

Show failed systemd units with systemctl

Learn to search and to find

Make a new friend?

Learn how to use curl to test if your web server is properly caching static files

Learn about the available systemd unit types

Control those processes and timers

Query the journal and find the needle

Learn how to define the Expires header in nginx to improve the caching of static assets.

Download files and troubleshoot issues faster with curl

Parse files quicker with smarter expressions

How to display the configured hosts by filtering out the server_name entries

SELinux is a well-known security framework on Linux systems like Red Hat Enterprise Linux. Let’s have a look at the basics of this framework, such as its purpose and features. History SELinux was developed by the National Security Agency (NSA) and integrated into many Linux distributions, including Red Hat Enterprise Linux (RHEL) and CentOS. Purpose SELinux is a MAC framework that enforces fine-grained access controls. SELinux achieves this by labeling files, processes, and network ports with security contexts.

Linux security frameworks restrict unauthorized access by defining a policy and a set of predefined rules. In this article we look their purpose and how they work. Purpose of security frameworks Linux security frameworks aim to enforce mandatory access controls (MAC). This is a way of saying to restrict the actions that processes and users can perform on the system. Unlike discretionary access controls (DAC), which rely on the discretion of users and processes, MAC frameworks enforce policies defined by the system administrator.

AppArmor is the profile-based security security framework and available on many Debian-based distributions. In this article we look at the basics of this security framework, its purpose and features. History AppArmor was developed by Canonical Ltd. and therefore included in their own Ubuntu distribution. included in Ubuntu and other Debian-based distributions What is AppArmor? AppArmor is a profile-based MAC framework. As the name implies, profiles are used that focus on simplifying security management.

Changing file permissions: chmod The primary command to change file permissions on a Linux system is chmod. It’s a basic system administration utility and pre-installed on the system. To make changes to an existing directory or file, it is first good to look up the existing permissions. This can be done using the ls -l command, that lists them with the long format. ls -l /etc/hosts -rw-r--r-- 1 root root 241 Feb 2 19:10 /etc/hosts There are two syntax styles to tell chmod what the new value should be.

File permissions are stored together with the data on a disk. The Linux kernel uses them to decide which users and processes can access what file. This page can be considered as a good cheat sheet, while the underlying articles explain how to use this information. Main permissions Read (r): Allows users to view the contents of a file or directory Write (w): Grants users the ability to modify the contents of a file or directory Execute (x): Enables users to execute a file or access the contents of a directory Permission Abbreviation Octal value Read r 4 Write w 2 Execute x 1 Possible combinations:

Every file that is stored has a set of file permissions stored within the filesystem. This data about the actual data, it called meta-data. Let have a look at how file permissions work on Linux systems and how to read and understand them. Read, Write, and Execute Linux file permissions are divided into three main categories: Read (r): Allows users to view the contents of a file or directory Write (w): Grants users the ability to modify the contents of a file or directory Execute (x): Enables users to execute a file or access the contents of a directory User, Group, and Others These permissions are each assigned to three entities:

If you want to know which hard links are present, the find utility can give you the answer. In this article we have a look at a few ways to discover more information about hard links. Good to know: a hard link shares the same inode, where a symbolic link has its own inode and just points from one to another. Show all hard links within a specific file system or directory When we have a directory with hard links, we can discover by looking at the link count of each file.

Linux systems use a file system to store and process data. In this section we have a look at what a file system and supporting articles to learn more about it. Purpose of a file system At its core, a file system is the method by which data is organized and stored on a storage device. The storage device is typically a hard drive or solid-state drive (SSD), but it can also be memory.

Shell scripting is a powerful way to assist in automation of repeating tasks. While fairly easy to learn, the shell scripting language has a lot of caveats. In this area we collect tips and tricks to help you creating better shell scripts.

A Linux systems without processes is not possible. So we collect tips to deal with processes and improve your skills.

Sometimes a process gets stuck and how often you try, it won’t respond to the combination of CTRL+C. One option is to open a second shell, then perform a kill. kill 1234 Pushing a job to the background While this works, there is usually a much easier way. This involves pushing a running process into the background by pressing CTRL+Z. [1]+ Stopped ./runserver Kill the process To get it back to the foreground, we would normally run fg.

Rnr The first tool to cover is called rnr and is written in Rust. It can be downloaded on GitHub where also some good examples can be found on how to use the tool. Let’s try it out on a directory that we have with Markdown files. Due to a conversion, the file names include a date. As this is no longer needed, we want to strip out the date and only get the bit after the third hyphen.

A regular issue with systems running Ubuntu is that may fill up the /boot partition. You might have discovered it when running apt, which refused to work. That is unfortunate, as you also need apt to resolve the issue. After trying several options, we found a way to resolve this catch 22, with just three steps. Opposed to other solutions, you don’t need to move files or do other tricky things on your system.

Introduction Each Linux system has a bunch of processes running. Most of these processes might be familiar to you if you regularly use a command like ps or top to display them. Processes may look like just an item in a list. They are actually complicated pieces of code that are tamed by a memory manager. To truly understand how your system is running, knowledge of process (or memory) management is of great help.

Having the right time set on a Linux system is important for data synchronization, forensics, and troubleshooting. Next step is to configure the correct time zone. This article will help you: See the current time configuration Learn how to configure the time zone Time zone information We will have a look on how to check and configure the time zone on Linux systems. Show current time zone Most new Linux distributions use systemd now.

Learn how to use nftables in this introduction guide to the tool. With common examples, frequently asked questions, and generic tips.

Determine Oracle Linux version Oracle Linux is based on Red Hat Enterprise Linux. At first, it may be confusing to determine what specific operating system is running. This is because both have the /etc/redhat-release file. If that file exists, use the cat command to display the contents. Next step is to determine if there is a /etc/oracle-release file as well. If so, then you can be sure that Oracle Linux is running.

The system hardening process of a system is critical during and after installation. It helps the system to perform its duties properly. This blog post shows you several tips for Ubuntu system hardening. It will dive into the most critical steps to take first. Then more specific hardening steps can be added on top of these. As most security guides only tell you what to do, we will also go into more detail on why a specific security measure is important.

Feeling overwhelmed with the resources available to secure your Linux system? This security guide will provide you the points where to start. With this Linux security guide, we walk step-by-step through the options, tools, and resources. After reading this article, you will be able to make educated decisions about what Linux security defenses to implement for your systems. You will be introduced to the right tools that help you automate and test your improvements.

Discover quickly which file(s) and package are matched together.

The grep command is one of the oldest tools for Linux and other platforms. Actually, it is older than Linux itself. It was written by Ken Thompson more than 45 years ago! The name grep stands for “globally regular expression print”. This name comes from its predecessor ed and the specific mode in which you would globally search, using a regular expression, and print the output. The related command was “g/re/p”.

Software updates and package management is easy with systems based on Debian or Ubuntu. Just apt-get update (or apt update) and run an upgrade. But sometimes you may encounter the following situation: a KEYEXPIRED message. KEYEXPIRED message # apt-get update && apt-get upgrade Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB] Hit:2 http://nl.archive.ubuntu.com/ubuntu xenial InRelease Get:3 http://nl.archive.ubuntu.com/ubuntu xenial-updates InRelease [95.7 kB] Hit:4 http://nl.archive.ubuntu.com/ubuntu xenial-backports InRelease Hit:5 https://packages.cisofy.com/community/lynis/deb stable InRelease Get:6 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [373 kB] Ign:7 http://nginx.

Time Synchronization The network time protocol helps computer systems to synchronize their time. We know this protocol by its shorter name NTP. In the past, it was not really a big issue if your system was a few minutes off. This changed with the interconnected world we are now living in. One of the better examples is networks relying on the authentication protocol Kerberos. If your system time is not correct, you may not be able to authenticate.

Mosh, or mobile shell, is the ideal tool for remote system administration. While SSH is great, Mosh beats it in several areas. Let’s dive into the reasons why it makes sense to learn about Mosh. Pros Session Resumption Remember the last time your connection was interrupted? It it frustrating and sometimes even leads to losing some of your work. The stable TCP connection is not always a blessing. Mosh comes to the rescue, especially for less stable connections.

The Dandified YUM tool, DNF, has become a powerful package manager for systems running Fedora. As it looks now, it will become also the default package manager for CentOS 8 and RHEL 8. One of the benefits from dnf is the option to retrieve security information very easily. This allows us to use it for automatic security patching of our Linux systems. Let’s explore the options and see how dnf-automatic can help us with fully automated patching.

Sometimes programs crash, usually for a different variety of reasons. While it is good to do research and find the underlying cause, sometimes you simply want to disable any reporting. Clean up /var/crash First thing to do is check your /var/crash directory and see if there are any “crash” files. These are just normal text files and include details about a process. If you have a process crashing regularly, you most likely want to report it, so the vendor can implement a fix.

Lately I saw a great feature request for Lynis, to detect differences between two runs of Lynis. Wouldn’t it be great to run Lynis daily and then see if anything changes and act upon those differences? While our auditing tool doesn’t have such an option itself, it is very easy to implement something and fine-tune it to your needs. Report Lynis has two important files to which is logs data:

If you are in the business of system administration, you know the big dilemma when it comes to installing software: missing packages. Yes, a lot of packages are available in the repositories of your Linux distribution, but not the one you need. Or when it is, it is horribly outdated. So you reach out to external resources, like community maintained repositories, right? With Lynis, we face this same issue. While most of the distributions have Lynis in the repository, it is often outdated.

Processes are the running workforce on a Linux system. Each process has a particular goal, like forking child processes, handling incoming user requests of monitoring other processes. As a system administrator or IT auditor, you might want to know at some point what disk activity occurs in a process. In this article, we have a look at a few options to quickly reveal what is occuring in a process, including disk and file activity.

Including the usage of Freshclam To get ClamAV on CentOS installed, we have to use the EPEL repository (Extra Packages for Enterprise Linux). Fortunately, the Fedora project provides this with an easy installation. Unfortunately the default configuration is not properly working. In this post we collect some of the issues and required changes. Let’s start with installing the EPEL support. yum install epel-release Next step is installing all ClamAV components.

Proper software patch management helps reducing weaknesses on your systems. But even if you patched an outdated system, old processes and libraries can continue to run in memory. For example when a library is updated, an active program might still use the old version. To really finish the process of software patching, we have to do more. This includes preparation, performing the update and finally check if we need a restart of software components.

Linux systems are usually managed remotely with SSH (secure shell). Still many administrators are using passwords, instead of keys. Keys not only boost security, it also makes managing systems much easier. Instead of entering your password for each server, you only have to do it once per session. When managing several systems per day, you will be wondering why you ever used password based authentication before. Generating the SSH key Depending on your desktop platform, we first have to create a key pair.

SSH daemon The OpenSSH daemon is named sshd and typically this is also the process visible in a process listing. The main listener usually has a full path visible, while active SSH sessions get their own child process. Configuration files /etc/ssh/sshd_config /etc/ssh/sshd_config.d/*.conf The configuration file sshd_config is the primary file for the settings. Any settings in a configuration file specified in /etc/ssh/sshd_config.d will overwrite those listed in the primary configuration file.

Recently we changed our corporate website into a “HTTPS only” version. Most of the content is not secret information, still we have some sensitive areas. The ordering section and downloads, and additional our portal. While some areas were already covered with a lock, we felt it was time to make the jump to cover it all. Additionally, we believe that we doing everything we can on our website, practicing security hardening ourselves.

To counter the biggest threat to software packages, they should be updated on a regular basis. Vulnerabilities are discovered on a daily basis, which also requires we monitor daily. Software patching takes time, especially when testing and reboots are needed. Fortunately, systems running Debian and Ubuntu can use unattended-upgrades to achieve automated patch management for security updates. Installation With most software packages, unattended-upgrades has to be installed. apt install unattended-upgrades

Using pkg_add Keeping your systems stable and secure Every system needs to stay up-to-date with its packages, including OpenBSD. Most OpenBSD users already use pkg_add for the installation of packages. This utility can also be used for package upgrades. Option 1: Use /etc/installurl Newer OpenBSD versions use the file /etc/installurl to select the mirror for pkg_add. Option 2: PKG_PATH The first thing to do is defining your PKG_PATH. This will usually be the address of a FTP or HTTP server, which has the latest packages available.

Maximum Linux security with proper software patch management Software upgrades are almost as old as the first lines of software code. Still companies struggle to properly update software, also when it comes to security patching. In this article we have a look at the reason behind patching and some methods to keep your systems humming, with fresh packages. Why Update? To most of us, it instantly makes sense to keep the software on your systems up-to-date.

Starting with Linux auditing can be overwhelming. Fortunately, there is a great feature in the Linux kernel to watch events and log them for us. To give you a quick start to use the Linux Audit Framework, we have collected some basic rules for configuring the audit daemon and its rules. Main Configuration By default the configuration values in /etc/audit/audit.conf are suitable for most systems. If you know your system is very low or very high (e.

The Secure Alternative for ntpd, ntpdate and rdate The common protocol to synchronize the time, is named Network Time Protocol, or NTP. While this protocol works great for synchronizing systems to one or more multiple time sources, it is not always easy to set-up. One alternative is using tlsdate, a secure replacement to keep your systems in sync. About the Project The software is written in 2012 by Jacob Appelbaum and can be found at GitHub: tlsdate.

By default Arch will install the kernel in /boot with the name vmlinuz-linux. To determine if the system is running the latest kernel, we can compare the running kernel and the one on disk. Running kernel One way to determine the running kernel is with the uname command. By default installed and with the -r parameter it will provide the kernel release version. # uname -r 3.17.4-1-ARCH Kernel on disk Checking the latest kernel on disk is almost as easy.

Security audit of NetBSD software packages with pkg_admin NetBSD is especially known for it’s diverse platforms it can run on. What is less known is the ability to audit the installed packages. In this article we have a look on how to audit NetBSD and ensure the file integrity of your packages. Performing a security audit is easy, as long as you use the right tool! Packages When using packages, their metadata will be installed in directory within /var/db/pkg.

Systemd used a binary log to store information about specific events. These events include the boot sequence and the related output. In this article we have a look at finding our boot logs in systemd journals. Binary logging When using systemd, boot data is stored in journals, a binary format. There is big benefit of saving boot data in a binary format: log information of each boot can be stored separately, linked to other pieces of information, and queried easier and quicker.

The ss tool helps system administrators and security professionals to display information about network connected applications. Learn how to maximize its potential and get everything out of this nifty tool.

Security of Linux systems and applications can be greatly improved by using hardening measures. One of these measures is called Linux capabilities. Capabilities are supported by the kernel for some while now. Using capabilities we can strengthen applications and containers. Unfortunately, this powerful tool is still underutilized. Time to change that! This article helps to understand and apply them. What are Linux capabilities? Normally the root user (or any ID with UID of 0) gets a special treatment when running processes.

Solving failed units with systemctl Systemd is an alternative service manager to the more traditional init system. To ensure the system is healthy, failed units should be investigated on a regular basis. Sooner or later a unit might fail and showing up the systemctl listing. In this article we have a look at how to solve it. Why do services fail? During the start of the system, enabled services are started and queued to be executed.

There are several reasons when you might need to clear your ARP cache. There are two common ways on Linux systems, typically using the arp or ip utility. Depending on your Linux distribution and the availability, we suggest using the ip tool. Clearing cache with ip Newer Linux distributions have the ip utility. The ip tool has a more advanced way to clear out the full ARP cache. ip -s -s neigh flush all

Hardening Linux binaries by removing setuid Normally Unix based systems use two kind of processes: privileged and unprivileged. The first category is usually used for administrative purposes, like starting and stopping other processes, tuning the kernel and opening sockets. Root permissions The command ping is a great example why even small programs needs root permissions. In a first glance you might consider this tool to be simple: send a package to a host and see if it responds.

Anyone who wants to create a new key set via GnuPG (GPG) may run into this error: We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. Not enough random bytes available. Please do some other work to give the OS a chance to collect more entropy!

Many people used Bastille Linux to harden their Linux systems. Unfortunately the website of Bastille seems very outdated, including the tool. This resulted in people searching for a great alternative to replace this tool. We found the alternative by actually combining different solutions, being more powerful. Security automation is hot, so forget Bastille and do it the right way. Automatic hardening makes sense Most system administrators can’t keep up with the new technologies and security threats.

Installing from source Security automation is hot and we love it. One way is using the OpenSCAP toolkit. Unfortunately it is not mature enough, so you might want to build and install it from source. We share our findings while creating our test environment. Install required components On our minimum installed CentOS 7 system, we need to install a few components. Most are related to compiling C++ and parsing XML files.

Enhancing yum Determine available plugins and built-in security support To enhance the support in our auditing tool Lynis, we wanted to know if yum supports security related functions by using a plugin or having it as built-in functionality. Yum Yum, or Yellowdog Updater Modified, is a software management tool for Linux based systems. Usually it is used on systems running SuSE or Red Hat based (like RHEL, Fedora or CentOS). Plugins extend the functionality of yum, to improve its functionality.

What is the Poodle vulnerability ? The “Poodle” vulnerability is basicly an attack on the SSL 3.0 protocol. It was discovered in October 2014. The flaw is in the protocol itself (not implementation), which makes the issue applicable for all products using SSL 3.0. TLS 1.0 and later are considered safe against the attack. How does the attack work? While we won’t go into too much depth of encryption and ciphers, we will share some basics.

Tutorial for Lynis installation on Arch Linux Pacman Arch Linux is getting more popular due to its great community support and the way it is organized. Being a “rolling release” system, it is continuously up-to-date. Still, you want to make sure your security defenses are equally up-to-date, so that’s where Lynis comes in. Normally pacman is used for installing new packages. Unfortunately, the lynis package does not show up. # pacman -Ss lynis # pacman -Ss rkhunter community/rkhunter 1.

Protect against Shellshock Shellshock is a serious software weakness, or vulnerability, in Bash. This shell is used on almost all Unix based systems, including Debian and Ubuntu. As it can be used without much effort and remotely exploit systems, it has a maximum vulnerability score according to CVSS. Upgrade Bash First update the software repository with apt-get, using the update parameter. apt-get update && apt-get install -only-upgrade bash Your system should now have a newer version of bash.

During the last years the role of DevOps evolved. This person could be described as the hybrid: a system administrator with development skills, or the developer which is also infrastructure savvy. With Linux and so many available tooling, it is becoming easier for people to learn both development and managing infrastructures. We are especially interested in Linux security for DevOps and what they can apply. Automation is key Repeating work is not only boring, but also a waste of time.

From the initial start of the Linux operating system, the first processes are already born. In this article we have a look on dealing with processes. In particular we look at how to do process auditing. Whenever you are an auditor, system administrator or just a Linux enthusiast, you can’t ignore processes and should know how to deal with them. Process listing For most people working on Linux systems, it might be obvious to display running processes with ps.