Troubleshooting CPU usage
Got a busy system that comes to a halt due it being too busy? In this article we look at troubleshooting issues related to CPU usage.
Monitoring CPU usage
The tool top might be the most familiar tool to monitor CPU or memory usage. A good alternative is the pidstat tool. It can be using an interval and easily show active processes, followed by a summary.
# pidstat 3
Linux 6.5.0-28-generic (workstation) 20-05-24 _x86_64_ (8 CPU)
13:45:47 UID PID %usr %system %guest %wait %CPU CPU Command
13:45:50 0 638 0,00 0,66 0,00 0,00 0,66 3 irq/204-nvidia
13:45:50 1000 1744 0,33 1,00 0,00 0,00 1,33 6 pulseaudio
13:45:50 1000 2050 1,66 0,33 0,00 0,00 1,99 2 gnome-shell
13:45:50 1000 3767 1,99 1,33 0,00 0,00 3,32 3 firefox
13:45:50 1000 3985 1,00 0,66 0,00 0,00 1,66 7 Isolated Web Co
13:45:50 1000 4277 0,33 0,00 0,00 0,00 0,33 5 WebExtensions
13:45:50 1000 25736 0,00 0,33 0,00 0,00 0,33 5 Isolated Web Co
13:45:50 1000 560859 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co
13:45:50 1000 657165 0,33 0,00 0,00 0,00 0,33 3 Isolated Web Co
13:45:50 1000 858923 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co
13:45:50 1000 1235407 0,33 0,00 0,00 0,00 0,33 1 Isolated Web Co
13:45:50 0 1284255 0,00 0,33 0,00 0,00 0,33 7 kworker/7:1-events
13:45:50 0 1284904 0,00 0,33 0,00 0,00 0,33 0 kworker/0:2-pm
13:45:50 0 1285798 0,00 0,33 0,00 0,00 0,33 3 kworker/3:0-events
13:45:50 1000 1286455 1,00 0,00 0,00 0,00 1,00 0 Isolated Web Co
13:45:50 0 1287603 0,00 0,33 0,00 0,00 0,33 0 kworker/0:0-events
13:45:50 UID PID %usr %system %guest %wait %CPU CPU Command
13:45:53 0 638 0,00 0,67 0,00 0,00 0,67 3 irq/204-nvidia
13:45:53 1000 1744 0,00 1,00 0,00 0,00 1,00 6 pulseaudio
13:45:53 1000 2050 0,33 0,00 0,00 0,00 0,33 2 gnome-shell
13:45:53 1000 3767 7,67 1,67 0,00 0,00 9,33 3 firefox
13:45:53 1000 3985 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co
13:45:53 1000 4277 0,67 0,00 0,00 0,00 0,67 7 WebExtensions
13:45:53 1000 18502 0,33 0,00 0,00 0,00 0,33 5 virt-manager
13:45:53 1000 560859 1,00 0,00 0,00 0,00 1,00 0 Isolated Web Co
13:45:53 1000 656930 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co
13:45:53 1000 1235407 0,33 0,00 0,00 0,00 0,33 6 Isolated Web Co
13:45:53 1000 1286455 0,67 0,67 0,00 0,00 1,33 7 Isolated Web Co
13:45:53 1000 1287953 0,00 0,33 0,00 0,00 0,33 1 pidstat
^C
Average: UID PID %usr %system %guest %wait %CPU CPU Command
Average: 0 638 0,00 0,67 0,00 0,00 0,67 - irq/204-nvidia
Average: 0 945 0,00 0,11 0,00 0,00 0,11 - libvirtd
Average: 1000 1744 0,11 1,00 0,00 0,00 1,11 - pulseaudio
Average: 1000 2050 0,67 0,11 0,00 0,00 0,78 - gnome-shell
Average: 1000 3767 5,77 1,44 0,00 0,00 7,21 - firefox
Average: 1000 3985 0,55 0,22 0,00 0,00 0,78 - Isolated Web Co
Average: 1000 3989 0,00 0,11 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 4277 0,55 0,00 0,00 0,00 0,55 - WebExtensions
Average: 1000 18502 0,11 0,00 0,00 0,00 0,11 - virt-manager
Average: 1000 25736 0,00 0,11 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 492449 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 560859 0,55 0,00 0,00 0,00 0,55 - Isolated Web Co
Average: 1000 656930 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 657165 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 858923 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 1202040 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co
Average: 1000 1235407 0,33 0,00 0,00 0,00 0,33 - Isolated Web Co
Average: 0 1284255 0,00 0,11 0,00 0,00 0,11 - kworker/7:1-events
Average: 0 1284904 0,00 0,11 0,00 0,00 0,11 - kworker/0:2-pm
Average: 0 1285798 0,00 0,11 0,00 0,00 0,11 - kworker/3:0-events
Average: 1000 1286455 0,78 0,55 0,00 0,00 1,33 - Isolated Web Co
Average: 0 1287603 0,00 0,11 0,00 0,00 0,11 - kworker/0:0-events
Average: 1000 1287953 0,00 0,22 0,00 0,00 0,22 - pidstat
Filter by process
To zoom in on a particular process or task, the option -C or -G can be used. It filters on the provided string and looks if that is part of the command name. When needed, a regular expression can be used.
# pidstat -C firefox 3
Linux 6.5.0-28-generic (ws03) 20-05-24 _x86_64_ (8 CPU)
13:50:10 UID PID %usr %system %guest %wait %CPU CPU Command
13:50:13 1000 3767 7,97 1,99 0,00 0,00 9,97 2 firefox
13:50:13 UID PID %usr %system %guest %wait %CPU CPU Command
13:50:16 1000 3767 1,67 1,00 0,00 0,00 2,67 5 firefox
13:50:16 UID PID %usr %system %guest %wait %CPU CPU Command
13:50:19 1000 3767 3,33 1,00 0,00 0,00 4,33 2 firefox
13:50:19 UID PID %usr %system %guest %wait %CPU CPU Command
13:50:22 1000 3767 5,00 2,33 0,00 0,00 7,33 5 firefox
^C
Average: UID PID %usr %system %guest %wait %CPU CPU Command
Average: 1000 3767 4,50 1,58 0,00 0,00 6,08 - firefox
To get more details, add the options -l for long format (command with arguments) or -t for task details. The last one will include a tree-based output.
# pidstat -l -t
12:55:31 UID TGID TID %usr %system %guest %wait %CPU CPU Command
12:55:31 0 1 - 0.00 0.00 0.00 0.00 0.00 0 /sbin/init
12:55:31 0 - 1 0.00 0.00 0.00 0.00 0.00 0 |__systemd
12:55:31 0 2 - 0.00 0.00 0.00 0.00 0.00 0 kthreadd
12:55:31 0 - 2 0.00 0.00 0.00 0.00 0.00 0 |__kthreadd
12:55:31 0 13 - 0.00 0.00 0.00 0.00 0.00 0 ksoftirqd/0
12:55:31 0 - 13 0.00 0.00 0.00 0.00 0.00 0 |__ksoftirqd/0
12:55:31 0 14 - 0.00 0.00 0.00 0.00 0.00 0 rcu_sched
12:55:31 0 - 14 0.00 0.00 0.00 0.00 0.00 0 |__rcu_sched
12:55:31 0 15 - 0.00 0.00 0.00 0.00 0.00 0 migration/0
12:55:31 0 - 15 0.00 0.00 0.00 0.00 0.00 0 |__migration/0
12:55:31 0 22 - 0.00 0.00 0.00 0.00 0.00 0 khungtaskd
12:55:31 0 - 22 0.00 0.00 0.00 0.00 0.00 0 |__khungtaskd
12:55:31 0 25 - 0.00 0.00 0.00 0.00 0.00 0 kcompactd0
12:55:31 0 - 25 0.00 0.00 0.00 0.00 0.00 0 |__kcompactd0
12:55:31 0 83 - 0.00 0.00 0.00 0.00 0.00 0 kworker/0:1H-kblockd
12:55:31 0 - 83 0.00 0.00 0.00 0.00 0.00 0 |__kworker/0:1H-kblockd
12:55:31 0 294 - 0.00 0.00 0.00 0.00 0.00 0 jbd2/dm-0-8
12:55:31 0 - 294 0.00 0.00 0.00 0.00 0.00 0 |__jbd2/dm-0-8
12:55:31 0 369 - 0.00 0.00 0.00 0.00 0.00 0 /lib/systemd/systemd-journald
12:55:31 0 - 369 0.00 0.00 0.00 0.00 0.00 0 |__systemd-journal
12:55:31 0 406 - 0.00 0.00 0.00 0.00 0.01 0 /sbin/multipathd -d -s
12:55:31 0 - 406 0.00 0.00 0.00 0.00 0.00 0 |__multipathd
12:55:31 0 - 413 0.00 0.00 0.00 0.00 0.00 0 |__multipathd
12:55:31 0 - 414 0.00 0.00 0.00 0.00 0.00 0 |__multipathd
Start and monitor a single process
Sometimes you want to zoom in on a single process. With pidstat we can start the process and monitor it, until we manually stop it. This way the process can be monitored from beginning till the end.
# pidstat 3 -e /opt/google/chrome/chrome
Linux 6.5.0-28-generic (workstation) 20-05-24 _x86_64_ (8 CPU)
14:20:29 UID PID %usr %system %guest %wait %CPU CPU Command
14:20:32 1000 1290789 42,00 6,00 0,00 2,00 48,00 7 chrome
14:20:35 1000 1290789 65,00 18,33 0,00 7,67 83,33 0 chrome
14:20:38 1000 1290789 1,00 0,67 0,00 0,00 1,67 7 chrome
14:20:41 1000 1290789 0,33 0,00 0,00 0,00 0,33 6 chrome
14:20:44 1000 1290789 1,67 0,33 0,00 0,00 2,00 5 chrome
14:20:47 1000 1290789 3,67 0,67 0,00 0,33 4,33 2 chrome
14:20:50 1000 1290789 8,00 4,00 0,00 0,33 12,00 2 chrome
14:20:53 1000 1290789 12,00 2,00 0,00 0,33 14,00 1 chrome
14:20:56 1000 1290789 5,00 1,67 0,00 0,00 6,67 2 chrome
14:20:59 1000 1290789 11,00 3,67 0,00 0,33 14,67 4 chrome
14:21:02 1000 1290789 9,67 3,33 0,00 0,33 13,00 2 chrome
14:21:05 1000 1290789 6,67 2,00 0,00 0,33 8,67 2 chrome
Average: 1000 1290789 13,83 3,56 0,00 0,97 17,39 - chrome
Got more useful commands to help troubleshooting performance issues? Contribute and make this article better.
