Auditing Linux processes: The Deep Dive!

From the initial start of the Linux operating system, the first processes are already born. In this article we have a look on dealing with processes. In particular we look at how to do process auditing. Whenever you are an auditor, system administrator or just a Linux enthusiast, you can’t ignore processes and should know how to deal with them.

Process listing

For most people working on Linux systems, it might be obvious to display running processes with ps. For Linux it’s common to use ps -ef, which shows effectively a list of all processes with a full listing. Those who are used to work on BSD machines will prefer using ps aux. On Linux with the POSIX tools, both will work, however with a slightly different output.

Parent process

Every process, except init has a parent process. This is the process which started another one. Usually when a program consists of only one process (no children), it will be spawned with init as its parent process. In such case the PPID column of ps will show the ID value 1.

# ps -ef  
UID        PID  PPID  C STIME TTY          TIME CMD  
root     16343     1  0 Apr14 ?        00:00:00 nginx: master process /usr/sbin/nginx  
www-data 16344 16343  0 Apr14 ?        00:00:13 nginx: worker process  
www-data 16345 16343  0 Apr14 ?        00:00:12 nginx: worker process  
www-data 16346 16343  0 Apr14 ?        00:00:16 nginx: worker process  
www-data 16347 16343  0 Apr14 ?        00:00:14 nginx: worker process

From this example we can clearly see that there is one master process (PID 16343, parent: init) and having 4 children (worker processes).

/proc file system

The virtual /proc file system provides us with additional information about the kernel and running processes. While most of the information can be extracted via ps or other tools, the information in /proc is easily accessible. Let’s dive into a few common files and their information treasure.

/proc/pid/cmdline

Most processes will be started with some parameters. If so, these parameters will be listed in the cmdline file. If there are no parameters, the binary itself will be displayed.

/proc/pid/exe

The exe file is a symlink to the actual binary on disk. In case some process is running, this might help finding the related binary.

/proc/pid/fd/

Most processes need to open a file or a socket. This happens with a system call like fopen, to open a file descriptor (fd). The fd directory within the /proc file system shows all file descriptors. When displaying a file listing, the related files (or sockets) will be displayed.

# ls -l /proc/16343/fd/
total 0  
lrwx-- 1 root root 64 Apr 18 18:07 0 -> /dev/null  
lrwx-- 1 root root 64 Apr 18 18:07 1 -> /dev/null  
lrwx-- 1 root root 64 Apr 18 18:07 10 -> socket:[1310432]  
lrwx-- 1 root root 64 Apr 18 18:07 11 -> socket:[1310433]  
lrwx-- 1 root root 64 Apr 18 18:07 12 -> socket:[1310834]  
lrwx-- 1 root root 64 Apr 18 18:07 13 -> socket:[1310835]  
lrwx-- 1 root root 64 Apr 18 18:07 14 -> socket:[1310836]  
lrwx-- 1 root root 64 Apr 18 18:07 15 -> socket:[1310837]  
lrwx-- 1 root root 64 Apr 18 18:07 16 -> socket:[1310838]  
lrwx-- 1 root root 64 Apr 18 18:07 17 -> socket:[1310839]  
lrwx-- 1 root root 64 Apr 18 18:07 18 -> socket:[1310840]  
l-wx-- 1 root root 64 Apr 18 18:07 19 -> /var/log/nginx/cisofy.local.access.log  
l-wx-- 1 root root 64 Apr 18 18:07 2 -> /var/log/nginx/error.log  
lr-x-- 1 root root 64 Apr 18 18:07 3 -> /proc/16342/auxv  
lrwx-- 1 root root 64 Apr 18 18:07 4 -> socket:[1310833]  
l-wx-- 1 root root 64 Apr 18 18:07 6 -> /var/log/nginx/access.log  
l-wx-- 1 root root 64 Apr 18 18:07 7 -> /var/log/nginx/error.log  
l-wx-- 1 root root 64 Apr 18 18:07 8 -> /var/log/nginx/localhost.access.log  
lrwx-- 1 root root 64 Apr 18 18:07 9 -> socket:[1310431]

/proc/pid/fdinfo/

Within the fdinfo subdirectory, we can additionally find more information about the file descriptor itself.

# cat /proc/16343/fdinfo/3  
pos:    304  
flags:  0100000

Pos defines the position, where flags describe the related “parameters” used when opening the file (read-only, append, write, etc).

/proc/pid/syscall

The syscall file is available with newer kernels. It displays the current status by sharing the system call it last performed. Knowing the most important system calls is valuable for auditing purposes. For example they are also used when monitoring file access or other system events, together with the Linux audit framework.

Displaying the output of the file might look like:

# cat /proc/16343/syscall
130 0x7fff714df300 0x8 0x0 0x0 0xcccccccd 0x7ff6148d7400 0x7fff714df2a8 0x7ff6147cb77a

Well, this doesn’t give much information at a first glance, except that the first identifier is the syscall ID. In this case we can easily lookup number 130 by using the ausyscall tool (if installed).

# ausyscall 130
> rt_sigsuspend

Alternative option: For x86_64 based systems, look for the file unistd_64.h and grep for the related ID. Make sure to determine the proper machine type with uname -m.

/proc/id/stack

Another option to determine the latest system call is by checking the stack. This file (/proc/pid/stack) will display something like:

# cat /proc/16343stack**  
[<ffffffff8107f269>] sys_rt_sigsuspend+0x89/0xc0  
[<ffffffff81669b82>] system_call_fastpath+0x16/0x1b  
[<ffffffffffffffff>] 0xffffffffffffffff

The top row displays the current system call and should be similar to what is in the syscall file.

As can be seen, there is a lot of information available about processes.