Linux security: Reviewing log files

Log files are the precious collection of system events. Still many people don’t use them, until it is really needed. Let’s go from the reactive use of log files to a proactive stance.

The Logging Dilemma

Capturing events helps in troubleshooting. By defining what events are ignored and which ones are logged, we get a quick overview on the status of a system. The dilemma is usually in how much logging is enough to get a fair picture and when is it too much. Too less and you lose valuable information, too much and information is hard to find.

What to capture

Depending on the goal of the machine, it helps to create a small matrix with the main services running on the system. Then define what the bare minimum information is required per service to be logged. For example when running nginx, you might want to log all major events of nginx itself (crashes, configuration errors), while limiting debug information. Additionally you want to define per website a log file to capture access requests for data analysis purposes. One log file for the errors (per website) for troubleshooting and optimizing the particular website.

Looking for the needle

Log files are usually simple text based files. With your favorite text manipulation tools (e.g. grep, awk, sed) you can quickly search through log files and find the events you are looking for.

Log files and Automation

The handling of log files can be automated fairly easy. Yet there are some options to consider:

Location:

  • Keep all log files local
  • Store local and send to remote syslog host
  • Log events to SIEM solution

Data handling:

  • Store as-is
  • Store and forward data with filtering

With rsyslog you can directly filter events and store them in separated files. This makes reviewing logs much easier. Tools like logcheck, swatch and logwatch may be helpful as well for monitoring logs and filtering out “noise”.

Feedback

Small picture of Michael Boelen

This article has been written by our Linux security expert Michael Boelen. With focus on creating high-quality articles and relevant examples, he wants to improve the field of Linux security. No more web full of copy-pasted blog posts.

Discovered outdated information or have a question? Share your thoughts. Thanks for your contribution!

Mastodon icon