Auditing Linux processes: The Deep Dive!
Auditing Linux processes
From the initial start of the Linux operating system, the first processes are already born. In this article we have a look on dealing with processes. In particular we look at how to do process auditing. Whenever you are an auditor, system administrator or just a Linux enthusiast, you can’t ignore processes and should know how to deal with them.
For most people working on Linux systems, it might be obvious to display running processes with ps. For Linux it’s common to use “ps -ef”, which shows effectively a list of all processes with a full listing. Those who are used to work on BSD machines will prefer using “ps aux”. On Linux with the POSIX tools, both will work, however with a slightly different output.
Every process, except init has a parent process. This is the process which started another one. Usually when a program consists of only one process (no children), it will be spawned with init as its parent process. In such case the PPID column of ps will show the ID value 1.
root@server:~# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 16343 1 0 Apr14 ? 00:00:00 nginx: master process /usr/sbin/nginx
www-data 16344 16343 0 Apr14 ? 00:00:13 nginx: worker process
www-data 16345 16343 0 Apr14 ? 00:00:12 nginx: worker process
www-data 16346 16343 0 Apr14 ? 00:00:16 nginx: worker process
www-data 16347 16343 0 Apr14 ? 00:00:14 nginx: worker process
From this example we can clearly see that there is one master process (PID 16343, parent: init) and having 4 children (worker processes).
The virtual /proc filesystem provides us with additional information about the kernel and running processes. While most of the information can be extracted via ps or other tools, the information in /proc is easily accessible. Let’s dive into a few common files and their information treasure.
Most processes will be started with some parameters. If so, these parameters will be listed in the cmdline file. If there are no parameters, the binary itself will be displayed.
The exe file is a symlink to the actual binary on disk. In case some process is running, this might help finding the related binary.
Most processes need to open a file or a socket. This happens with a system call like fopen, to open a file descriptor (fd). The fd directory within the /proc file system shows all file descriptors. When displaying a file listing, the related files (or sockets) will be displayed.
root@server:/proc/16343/fd# ls -l
lrwx—— 1 root root 64 Apr 18 18:07 0 -> /dev/null
lrwx—— 1 root root 64 Apr 18 18:07 1 -> /dev/null
lrwx—— 1 root root 64 Apr 18 18:07 10 -> socket:
lrwx—— 1 root root 64 Apr 18 18:07 11 -> socket:
lrwx—— 1 root root 64 Apr 18 18:07 12 -> socket:
lrwx—— 1 root root 64 Apr 18 18:07 13 -> socket:
lrwx—— 1 root root 64 Apr 18 18:07 14 -> socket:
lrwx—— 1 root root 64 Apr 18 18:07 15 -> socket:
lrwx—— 1 root root 64 Apr 18 18:07 16 -> socket:
lrwx—— 1 root root 64 Apr 18 18:07 17 -> socket:
lrwx—— 1 root root 64 Apr 18 18:07 18 -> socket:
l-wx—— 1 root root 64 Apr 18 18:07 19 -> /var/log/nginx/cisofy.local.access.log
l-wx—— 1 root root 64 Apr 18 18:07 2 -> /var/log/nginx/error.log
lr-x—— 1 root root 64 Apr 18 18:07 3 -> /proc/16342/auxv
lrwx—— 1 root root 64 Apr 18 18:07 4 -> socket:
l-wx—— 1 root root 64 Apr 18 18:07 6 -> /var/log/nginx/access.log
l-wx—— 1 root root 64 Apr 18 18:07 7 -> /var/log/nginx/error.log
l-wx—— 1 root root 64 Apr 18 18:07 8 -> /var/log/nginx/localhost.access.log
lrwx—— 1 root root 64 Apr 18 18:07 9 -> socket:
Within the fdinfo subdirectory, we can additionally find more information about the file descriptor itself.
root@server:/proc/16343/fdinfo# cat 3
Pos defines the position, where flags describe the related “parameters” used when opening the file (read-only, append, write, etc).
The syscall file is available with newer kernels. It displays the current status by sharing the system call it last performed. Knowing the most important system calls is valuable for auditing purposes. For example they are also used when monitoring file access or other system events, together with the Linux audit framework.
Displaying the output of the file might look like:
root@server:/proc/16343# cat syscall
130 0x7fff714df300 0x8 0x0 0x0 0xcccccccd 0x7ff6148d7400 0x7fff714df2a8 0x7ff6147cb77a
Well, this doesn’t give much information at a first glance, except that the first identifier is the syscall ID. In this case we can easily lookup number 130 by using the ausyscall tool (if installed).
root@server:~# ausyscall 130
Alternative option: For x86_64 based systems, look for the file unistd_64.h and grep for the related ID. Make sure to determine the proper machine type with uname -m.
Another option to determine the latest system call is by checking the stack. This file (/proc/pid/stack) will display something like:
root@server:/proc/16343# cat stack
The top row displays the current system call and should be similar to what is in the syscall file.
As can be seen, there is a lot of information available about processes.