Linux security: Reviewing log files
Log files are the precious collection of system events. Still many people don’t use them, until it is really needed. Let’s go from the reactive use of log files to a proactive stance.
The Logging Dilemma
Capturing events helps in troubleshooting. By defining what events are ignored and which ones are logged, we get a quick overview on the status of a system. The dilemma is usually in how much logging is enough to get a fair picture and when is it too much. Too less and you lose valuable information, too much and information is hard to find.
What to capture
Depending on the goal of the machine, it helps to create a small matrix with the main services running on the system. Then define what the bare minimum information is required per service to be logged. For example when running nginx, you might want to log all major events of nginx itself (crashes, configuration errors), while limiting debug information. Additionally you want to define per website a log file to capture access requests for data analysis purposes. One log file for the errors (per website) for troubleshooting and optimizing the particular website.
Looking for the needle
Log files are usually simple text based files. With your favorite text manipulation tools (e.g. grep, awk, sed) you can quickly search through log files and find the events you are looking for.
Log files and Automation
The handling of log files can be automated fairly easy. Yet there are some options to consider:
Location:
- Keep all log files local
- Store local and send to remote syslog host
- Log events to SIEM solution
Data handling:
- Store as-is
- Store and forward data with filtering
With rsyslog you can directly filter events and store them in separated files. This makes reviewing logs much easier. Tools like logcheck, swatch and logwatch may be helpful as well for monitoring logs and filtering out “noise”.
