Troubleshooting
You got 99 problems? This range of troubleshooting articles will help you reducing them. From securing your Linux systems up to standard system administration tasks.
Troubleshooting CPU usage
Articles and information about troubleshooting system performance issues with focus on CPU usage.
Summary
Got a busy system that comes to a halt due it being too busy? In this article we look at troubleshooting issues related to CPU usage.
Monitoring CPU usage
The tool top might be the most familiar tool to monitor CPU or memory usage. A good alternative is the pidstat tool. It can be using an interval and easily show active processes, followed by a summary.
# pidstat 3
Linux 6.5.0-28-generic (workstation) 20-05-24 _x86_64_ (8 CPU)
13:45:47 UID PID %usr %system %guest %wait %CPU CPU Command
13:45:50 0 638 0,00 0,66 0,00 0,00 0,66 3 irq/204-nvidia
13:45:50 1000 1744 0,33 1,00 0,00 0,00 1,33 6 pulseaudio
13:45:50 1000 2050 1,66 0,33 0,00 0,00 1,99 2 gnome-shell
13:45:50 1000 3767 1,99 1,33 0,00 0,00 3,32 3 firefox
13:45:50 1000 3985 1,00 0,66 0,00 0,00 1,66 7 Isolated Web Co
13:45:50 1000 4277 0,33 0,00 0,00 0,00 0,33 5 WebExtensions
13:45:50 1000 25736 0,00 0,33 0,00 0,00 0,33 5 Isolated Web Co
13:45:50 1000 560859 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co
13:45:50 1000 657165 0,33 0,00 0,00 0,00 0,33 3 Isolated Web Co
13:45:50 1000 858923 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co
13:45:50 1000 1235407 0,33 0,00 0,00 0,00 0,33 1 Isolated Web Co
13:45:50 0 1284255 0,00 0,33 0,00 0,00 0,33 7 kworker/7:1-events
13:45:50 0 1284904 0,00 0,33 0,00 0,00 0,33 0 kworker/0:2-pm
13:45:50 0 1285798 0,00 0,33 0,00 0,00 0,33 3 kworker/3:0-events
13:45:50 1000 1286455 1,00 0,00 0,00 0,00 1,00 0 Isolated Web Co
13:45:50 0 1287603 0,00 0,33 0,00 0,00 0,33 0 kworker/0:0-events
13:45:50 UID PID %usr %system %guest %wait %CPU CPU Command
13:45:53 0 638 0,00 0,67 0,00 0,00 0,67 3 irq/204-nvidia
13:45:53 1000 1744 0,00 1,00 0,00 0,00 1,00 6 pulseaudio
13:45:53 1000 2050 0,33 0,00 0,00 0,00 0,33 2 gnome-shell
13:45:53 1000 3767 7,67 1,67 0,00 0,00 9,33 3 firefox
13:45:53 1000 3985 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co
13:45:53 1000 4277 0,67 0,00 0,00 0,00 0,67 7 WebExtensions
13:45:53 1000 18502 0,33 0,00 0,00 0,00 0,33 5 virt-manager
13:45:53 1000 560859 1,00 0,00 0,00 0,00 1,00 0 Isolated Web Co
13:45:53 1000 656930 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co
13:45:53 1000 1235407 0,33 0,00 0,00 0,00 0,33 6 Isolated Web Co
13:45:53 1000 1286455 0,67 0,67 0,00 0,00 1,33 7 Isolated Web Co
13:45:53 1000 1287953 0,00 0,33 0,00 0,00 0,33 1 pidstat
^C
Average: UID PID %usr %system %guest %wait %CPU CPU Command
Average: 0 638 0,00 0,67 0,00 0,00 0,67 - irq/204-nvidia
Average: 0 945 0,00 0,11 0,00 0,00 0,11 - libvirtd
Average: 1000 1744 0,11 1,00 0,00 0,00 1,11 - pulseaudio
Average: 1000 2050 0,67 0,11 0,00 0,00 0,78 - gnome-shell
Average: 1000 3767 5,77 1,44 0,00 0,00 7,21 - firefox
Average: 1000 3985 0,55 0,22 0,00 0,00 0,78 - Isolated Web Co
Average: 1000 3989 0,00 0,11 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 4277 0,55 0,00 0,00 0,00 0,55 - WebExtensions
Average: 1000 18502 0,11 0,00 0,00 0,00 0,11 - virt-manager
Average: 1000 25736 0,00 0,11 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 492449 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 560859 0,55 0,00 0,00 0,00 0,55 - Isolated Web Co
Average: 1000 656930 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 657165 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 858923 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 1202040 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 1235407 0,33 0,00 0,00 0,00 0,33 - Isolated Web Co
Average: 0 1284255 0,00 0,11 0,00 0,00 0,11 - kworker/7:1-events
Average: 0 1284904 0,00 0,11 0,00 0,00 0,11 - kworker/0:2-pm
Average: 0 1285798 0,00 0,11 0,00 0,00 0,11 - kworker/3:0-events
Average: 1000 1286455 0,78 0,55 0,00 0,00 1,33 - Isolated Web Co
Average: 0 1287603 0,00 0,11 0,00 0,00 0,11 - kworker/0:0-events
Average: 1000 1287953 0,00 0,22 0,00 0,00 0,22 - pidstat
Filter by process
To zoom in on a particular process or task, the option -C or -G can be used. It filters on the provided string and looks if that is part of the command name. When needed, a regular expression can be used.
Network
Articles and information about troubleshooting network performance issues and monitoring network statistics
Summary
Network connectivity starts at a device that links the system to a network, and for Linux systems that is no different. Depending on the physical layer, such as Ethernet of Wi-Fi, the transactions between be better or worse. Besides physical limitations, there is congestion and packet loss that may introduce issues. In this section we look at troubleshooting network performance issues, tooling, and examples.
Interfaces
The first step is to learn what links we have towards the network.
Troubleshooting guide for Lynis
Troubleshooting Lynis. This document helps with solving most common issues experienced when running Lynis.
Summary
Troubleshooting Lynis
This document helps with solving most common issues experienced when running Lynis.
Common Lynis errors
No hostid and/or hostid2 found
Some systems do not have the OpenSSH server package installed. In this case, the hostid2 value may be missing. During the upload it may result in an error.
Error: No hostid and/or hostid2 found. Can not upload report file.
To see what Lynis discovered, use the show command.
Tuning auditd: high-performance Linux Auditing
To achieve better performance with a auditd configuration, it needs to be tuned. See performance boosters like events exclusion, rule ordering, and more.
Summary
The Linux Audit framework is a powerful tool to audit system events. From running executables up to system calls, everything can be logged. However, all this audit logging comes at the price of decreased system performance. Let’s have a look at how we can optimize our audit rules.
Performance tips
Good auditd performance will reduce stress on the Linux kernel and lower its impact. Before changing anything to your system, we suggest benchmarking your system performance before and after. This way you can see the benefits of your tuning efforts.
Linux Audit Framework 101 – Basic Rules for Configuration
The Linux audit framework is a very powerful tool to monitor files, directories, and system calls. Learn how to configure it.
Summary
Starting with Linux auditing can be overwhelming. Fortunately, there is a great feature in the Linux kernel to watch events and log them for us. To give you a quick start to use the Linux Audit Framework, we have collected some basic rules for configuring the audit daemon and its rules.
Main Configuration
By default the configuration values in /etc/audit/audit.conf are suitable for most systems. If you know your system is very low or very high (e.g. mainframe) on resources, then you might want to adjust some file sizes or buffers.