The ultimate strace cheat sheet

Strace cheat sheet

The strace utility is very powerful to learn what a new or running process is doing. Due to its diversity of monitoring options, the tool is less accessible at first. This strace cheat sheet helps with getting the best out of this tool.

Normally cheat sheets come in a single 1 page PDF. In this case, we combined it all within a blog post. First section shows an explanation per area, the bottom of the post contains all useful commands for quick reference.

Troubleshooting with strace

One of options of the strace utility is to help as a troubleshooting utility. If you want to know what a process is doing, or why it hangs, strace will definitely help. By running strace without any parameters, it will already show why a process is doing. You can trace a running process, or instruct strace to start it for you.

screenshot of strace utility with -c parameter

All syscalls listed by amount of time

Monitoring file activity

Strace can monitor file related activity. There are two useful parts. The first is file, which shows file interactions. The other one allows tracing file descriptors. Both can be used to monitor for actions like opening files, reading/writing and closing. Usually using “trace=file” provides enough insights. If you really need more insights in the way a program deals with file descriptors, then use the second one.

  • Monitor opening of files: strace -e open
  • See all file activity: strace -e trace=file -p 1234 or strace -e trace=desc -p 1234

If you want to track specific paths, use 1 or more times the -P parameter, following by the path.

# sudo strace -P /etc/cups -p 2261
Process 2261 attached
— SIGHUP {si_signo=SIGHUP, si_code=SI_USER, si_pid=6149, si_uid=0} —
lstat(“/etc/cups”, {st_mode=S_IFDIR|0755, st_size=4096, …}) = 0
openat(AT_FDCWD, “/etc/cups”, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
getdents(7, /* 11 entries */, 32768) = 336
getdents(7, /* 0 entries */, 32768) = 0
close(7) = 0
openat(AT_FDCWD, “/etc/cups”, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
getdents(7, /* 11 entries */, 32768) = 336
getdents(7, /* 0 entries */, 32768) = 0
close(7) = 0

Common calls:

  • access
  • close (close file handle)
  • fchmod (change file permissions)
  • fchown (change file ownership)
  • fstat (retrieve details)
  • lseek (move through file)
  • open (open file for reading/writing)
  • read (read a piece of data)
  • statfs (retrieve file system related details)

A related example screen output:

screenshot of strace monitoring file access and activity

Monitoring file access and activity with strace

Monitoring the network

Strace definitely can be useful for revealing more details about network traffic. Very useful to determine what network related connections are used, like when building your Docker image.

strace -e trace=network

Common syscalls:

  • bind – link the process to a network port
  • listen – allow to receive incoming connections
  • socket – open a local or network socket
  • setsockopt – define options for an active socket

Monitoring memory calls

To get better insights on the memory usage and system calls, strace can monitor for these as well. They are nicely grouped in the memory group.

strace -e trace=memory

Common syscalls:

  • mmap
  • munmap

Strace Cheat Sheet – Overview

Useful options and examples

  • -c – See what time is spend and where (combine with -S for sorting)
  • -f – Track process including forked child processes
  • -o my-process-trace.txt – Log strace output to a file
  • -p 1234 – Track a process by PID
  • -P /tmp – Track a process when interacting with a path
  • -T – Display syscall duration in the output

Track by specific system call group

  • -e trace=ipc – Track communication between processes (IPC)
  • -e trace=memory – Track memory syscalls
  • -e trace=network – Track memory syscalls
  • -e trace=process – Track process calls (like fork, exec)
  • -e trace=signal – Track process signal handling (like HUP, exit)
  • -e trace=file – Track file related syscalls

Trace multiple syscalls

  • strace -e open,close

Got other clever stracing tips? Use the comments for inclusion!


Enjoyed reading the article? Share your discovery with others:
twittergoogle_plusreddit

Monitor file access by Linux processes

Process Auditing: Disk and File Activity

Processes are the running workforce on a Linux system. Each process has a particular goal, like forking child processes, handling incoming user requests of monitoring other processes. As a system administrator or IT auditor, you might want to know at some point what disk activity occurs in a process. In this article, we have a look at a few options to quickly reveal what is occuring in a process, including disk and file activity.

Monitor syscalls

The kernel uses system calls, or syscalls for short. These are specific functions, which perform a low-level system function. Think of activities like reserving a memory section, or in this case opening a file from disk. The first utility to provide insights in active syscalls, is the strace utility. By tracking the right system call, we can see exactly what files are opened while it happens. Great for tracking required file access, dependencies, and troubleshooting purposes.

The usage of strace is simple. Just run a command you normally would execute, prepended with the strace utility.

# strace ls

If you run the same command on the CUPS daemon, this would be the output:

 

screenshot of strace capturing syscalls on cups daemon

Lots of output after starting a strace on a running process

While this provides interesting information, it might actually flood your screen, making it hard to work with. As we are interested in file access, we want to see only the open syscall.

# strace -f -e open ls 2>&1

So let’s first check what chain does: startstrace, track forked childs (-f) for the open system call (-e open). Ascommand we track the ls utility and redirect any errors to the screen output. For other interesting system calls, see the man 2 syscalls page.

If you want a clean output which only shows , here is a trick to only list the files:

# strace -f -e open ls 2>&1 | grep ^open\( | grep “[[:digit:]]\+$” | cut -d\” -f2

We can also apply monitoring system calls to a running process. Provide the -p and define the process ID you want to monitor. If you also want to monitor any forked child processes like in previous example, add the -f parameter.

# strace -f -p 4121

Most systems have the strace utility already installed by default. If you have a minimal installation without it, use your package manager.

Tracking system calls

The second option to check what system calls are used, is by monitoring the libraries used. Libraries are similar to a toolbox, filled with individual functional tools. In the case of Linux, the library is filled with functions, including indirect system functions.

Monitoring these functions can be done with the ltrace utility. Its usage is similar strace, but with the focus on libraries.

To get a first impression what kind of functions are used, use the -c parameter. It lists the functions, how often it was used (calls) and the time involved with that function. Great for troubleshooting why a process is taking a while to respond.

# ltrace -c ls

An example output of a trace on a Chrome process:

screenshot of ltrace tool to track used system calls of a process

Using ltrace to track system calls (syscalls)

Like strace we can attach to a process. For example tracking what the cron process does:

# ltrace -p 1275
time(0) = 1435911661
localtime(0x60b300) = 0x7f36b92ecde0
__xstat(1, “crontabs”, 0x7fffb2700c60) = 0
__xstat(1, “/etc/crontab”, 0x7fffb2700cf0) = 0
__xstat(1, “/etc/cron.d”, 0x7fffb2700d80) = 0
__sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 16
__xstat(1, “/etc/cron.d/php5″, 0x7fffb2700e10) = 0
__sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 19
__xstat(1, “/etc/cron.d/anacron”, 0x7fffb2700e10) = 0
__sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 23
__xstat(1, “/etc/cron.d/amavisd-new”, 0x7fffb2700e10) = 0
gmtime(0x7fffb2702fa8) = 0x7f36b92ecde0
time(0) = 1435911661
sleep(60

It will do nothing for a while and suddenly it shows up. It looks in several common cron related files (like /etc/crontab and /etc/cron.d). The __xstat function in this case monitors the files and tries avoiding opening each of them, unless it file meta information changed (e.g. modification date). The output suddenly looks different:

 # ltrace -p 1275
time(0) = 1435912201
localtime(0x60b300) = 0x7f36b92ecde0
__xstat(1, “crontabs”, 0x7fffb2700c60) = 0
__xstat(1, “/etc/crontab”, 0x7fffb2700cf0) = 0
__xstat(1, “/etc/cron.d”, 0x7fffb2700d80) = 0
__sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 16
__xstat(1, “/etc/cron.d/php5″, 0x7fffb2700e10) = 0
__sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 19
__xstat(1, “/etc/cron.d/anacron”, 0x7fffb2700e10) = 0
__sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 23
__xstat(1, “/etc/cron.d/amavisd-new”, 0x7fffb2700e10) = 0
gmtime(0x7fffb2702fa8) = 0x7f36b92ecde0
time(0) = 1435912201
sleep(60) = 0
time(0) = 1435912261
localtime(0x60b300) = 0x7f36b92ecde0
__xstat(1, “crontabs”, 0x7fffb2700c60) = 0
__xstat(1, “/etc/crontab”, 0x7fffb2700cf0) = 0
__xstat(1, “/etc/cron.d”, 0x7fffb2700d80) = 0
__sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 16
__xstat(1, “/etc/cron.d/php5″, 0x7fffb2700e10) = 0
__sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 19
__xstat(1, “/etc/cron.d/anacron”, 0x7fffb2700e10) = 0
__sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 23
__xstat(1, “/etc/cron.d/amavisd-new”, 0x7fffb2700e10) = 0
__lxstat(1, “/etc/crontab”, 0x7fffb2700cf0) = 0
open(“/etc/crontab”, 0, 00) = 4
__fxstat(1, 4, 0x7fffb2700cf0) = 0

After seeing the change, it uses a __lxstat and then the open function. The difference between normal functions and those prepended with two underscores, is that the latter are wrappers. Usually around the equally named function name, to provide a transparent wrapper and ensure the correct input and output.

Couldn’t find .dynsym or .dynstr in …

Some files can not be traced with ltrace and may result something like:

Couldn’t find .dynsym or .dynstr in “/proc/2098/exe”

If you encounter this error, you most likely have a statically linked binary. All functions are then inside the binary and not in a library. As ltrace can track libraries only, you have to use strace instead.

List Open Files

Everything on disk is a file. Normal files, devices and even directories are all presented as a file. The file system marks each of these entries in a file table, with the related type. While you might normally not even notice due to colored screen output, directories have a different type. It is the “d” in the first column in a long listing (-l) output of ls. Common types include a directory (d), a block or character based device (b/c) or a normal file (-, minus).

To see open files, we can use the lsof utility. It stands for “list open files” and definitely reveals its purpose. It can really show any type of open files, from the earlier mentioned special files (block and character devices), to tracking open network connections.

One big disadvantage of the lsof utility should be shared up front: there are way too many options to remember. The man page doesn’t make things easier, especially if you don’t know exactly what to look for. So getting exactly the right output is usually experimenting. At the bottom we have share some common used examples, to make this process easier.

Lsof cheat sheet

Processes

  • lsof -c cupsd = show open files for cups daemon

Networking

  • lsof -i -n =  show open network connections (without name resolving)
  • lsof -i4 or lsof -i6 = show IPv4 or IPv6 traffic
  • lsof -N = show NFS

Tracking syscalls with Linux Audit

We already have written some posts about the powerful Linux audit framework. This built-in kernel feature allows tracking files and system calls. Of course we can combine both. We define what process we want to track and the related system call. Similar to strace we use the “open” system call.

Create rule: open

# auditctl -a always,exit -F arch=b64 -F pid=8175 -S open -k cups-open-files

This rule adds a system call monitor on “open” (with 64 bits architecture), for PID 8175. Now when this process uses the open system call, it will be logged in the audit log. We give it a key “cups-open-files”.

Search for file activity

We can easily find them by referring to the earlier defined key “cups-open-files”.

# ausearch -k cups-open-files
—-
time->Fri Jul 3 15:31:20 2015
type=CONFIG_CHANGE msg=audit(1435930280.293:390): auid=4294967295 ses=4294967295 op=”add rule” key=”cups-open-files” list=4 res=1
—-
time->Fri Jul 3 15:31:31 2015
type=PATH msg=audit(1435930291.853:392): item=0 name=”/etc/group” inode=5377472 dev=08:05 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=CWD msg=audit(1435930291.853:392): cwd=”/”
type=SYSCALL msg=audit(1435930291.853:392): arch=c000003e syscall=2 success=yes exit=4 a0=7fd5c0909351 a1=80000 a2=1b6 a3=0 items=1 ppid=1 pid=8175 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm=”cupsd” exe=”/usr/sbin/cupsd” key=”cups-open-files”
—-
time->Fri Jul 3 15:31:31 2015
type=PATH msg=audit(1435930291.853:391): item=0 name=”/etc/passwd” inode=5377470 dev=08:05 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=CWD msg=audit(1435930291.853:391): cwd=”/”
type=SYSCALL msg=audit(1435930291.853:391): arch=c000003e syscall=2 success=yes exit=4 a0=7fd5c090935c a1=80000 a2=1b6 a3=0 items=1 ppid=1 pid=8175 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm=”cupsd” exe=”/usr/sbin/cupsd” key=”cups-open-files”
—-
time->Fri Jul 3 15:31:31 2015
type=PATH msg=audit(1435930291.853:393): item=0 name=”/etc/cups/cups-files.conf” inode=5376875 dev=08:05 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=CWD msg=audit(1435930291.853:393): cwd=”/”
type=SYSCALL msg=audit(1435930291.853:393): arch=c000003e syscall=2 success=yes exit=4 a0=7fd5c55ce090 a1=0 a2=0 a3=23 items=1 ppid=1 pid=8175 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm=”cupsd” exe=”/usr/sbin/cupsd” key=”cups-open-files”
—-
time->Fri Jul 3 15:31:31 2015
type=PATH msg=audit(1435930291.853:394): item=0 name=”/etc/group” inode=5377472 dev=08:05 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=CWD msg=audit(1435930291.853:394): cwd=”/”
type=SYSCALL msg=audit(1435930291.853:394): arch=c000003e syscall=2 success=yes exit=5 a0=7fd5c0909351 a1=80000 a2=1b6 a3=2f items=1 ppid=1 pid=8175 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm=”cupsd” exe=”/usr/sbin/cupsd” key=”cups-open-files”
—-

The Linux audit framework is a great alternative to strace, but might be less friendly to configure. Especially on a system which already has watches going on, you might want to skip inserting a few test rules. In that case use strace instead.

Got some others tools to track disk and file activity on running processes? Share it in the comments!


Enjoyed reading the article? Share your discovery with others:
twittergoogle_plusreddit

3 Lessons for Security Professionals: Be a Donkey!

Security Professionals: Be a Donkey!

How being a donkey can help you succeed in your daily work.

 

The donkey is considered to be a stupid animal in most of the western world. Actually we might be pretty wrong about our current believe. As security professionals, we can definitely learn something from these friendly-looking creatures.

1) Learn Quickly

We have a Dutch saying A donkey does not bump into the same stone twice”. The thought is that if you make the same mistake twice, you are stupid. If we reverse this though: the donkey is actually smart. It quickly learns from experiences and tries to avoid making mistakes over and over. This is one of the areas in which donkeys are much smarter than most animals, including humans.

As security professionals, we tend to be busy all day. We have to deal with a lot of information, resistance and politics. On top of that we have end-users which don’t want to comply with security policies, including system administrators, developers, and even upper management.

Lesson

What we can learn from our furry friends is that we should adapt more quickly to our environment and learn from mistakes made in the past.

2) Fight or Flight

The donkey is a versatile animal, helping the human for centuries to transport, plowing land and actually protecting sheep against predators. It has a very logical way of thinking, in that it will fight or fly. It does not hesitate and makes quickly a decision on the current circumstances.

As security professionals, we are sometimes trapped between hidden agendas and different beliefs. Instead of clearly make a decision, we hide behind e-mail and policies.

Lesson

The behavior of the donkey is similar to the fight-or-flight mechanisms we humans had in the past. Nowadays we have too many choices, believes and internal company politics. Instead, we should quicker decide if we fight a situation or choose to step out, and let the person with the right authority make a decision.

3) Weaknesses

Donkeys have a weakness, which is the lack of being waterproof. This makes them vulnerable for rain and cold. The donkey knows this and needs a place to take shelter and keep itself protected against mother nature.

As security professionals, we have each our weaknesses. Unfortunately most of the times we forget about this, especially when answering (difficult) questions from colleagues or customers. Too often we want to stay outside and answer everything, instead of taking shelter. It makes us vulnerable, possibly even annoying.

Lesson

What we can learn from this example is that we should go back to a safe haven, especially when we don’t know. We also should involve other specialists, instead of battling alone.

Image of a donkey learning a new trick

Like humans, the donkey has to learn by doing.

Conclusion

Donkeys are not stupid, but a little bit stubborn and cautious (like us). Yet they can be very flexible, and take action when required. We should adopt their flexibility, to get our work done. Learn from questions, cases and mistakes from the past.

We also should know our personal weaknesses, and accept help from others when needed. Last but not least, information security is about protecting assets and knowing risks. We should sooner make decisions and decide to fight, or flight. This way we ensure our efforts are helping the company and business. After all security is not just a price tag to protect the business, but also to stay in business.

 


Enjoyed reading the article? Share your discovery with others:
twittergoogle_plusreddit

« Older Entries