How to deal with a compromised Linux system

One day your web hoster or yourself may discover that your Linux system is slow. Upon logging in, you see a high load consumed by a suspicious process name or maybe just the Apache web server. Is your system compromised? How do you know it is? Let’s have a look at how to deal with security breaches and incident response.

Recognizing a security breach

Not all security breaches are directly visible. Attackers may have compromised your system a while ago and just planted some seeds at the time. Then finally when the resources of your system may be abused, your system could be part of a botnet.

Looking for signs

Before we dive deep into this subject of dealing with a compromised Linux system, we have to answer the biggest question: how do I know my system is compromised?

Sometimes there are signs that a clear giveaway. Examples:

• Pages on your website were replaced with a “You have been hacked” text
• The system is missing essential binaries or they crash (segfault) after executing
• Unauthorized users have been created
• The system is hosting movies, music, or other pirated data

If one of these examples apply to your system, then you know there is work to do.

Not so obvious

Unfortunately, too often the signs are not so clear. The system might misbehave and yet it is not compromised. For example, it may have been placed on a blacklist (RBL). E-mails are queuing up and web application might get stuck on delivering their email. It is also possible that the CPU is spending a lot of time on a process in trouble. For example, a disk partition is full and the process is retrying to write to disk.

Consult a professional

If you are still in doubt that a system is compromised, don’t wait and consult a security professional. Waiting might reduce the chance of securing data and gather evidence. Waiting also delays getting the system back to a healthy state.

Every system has a function and you should consider the consequences it might have on the integrity and confidentiality of the data stored or processed by the system. Even if the system does not have a business critical purpose, consulting with a professional will give you the best action to take. Just pulling the network or power plug isn’t always the best choice.

Breach confirmed

If you have confirmed that the Linux system is really compromised, the first step is: don’t panic.

Don’t panic!

Whatever you do from here on, stay calm and focused. It is time to take actions and to execute them according to a plan. The plan helps you to stay on the course. It is easy to take the wrong actions or overlook the consequences. Too often, a system administrator reboots the system in the hope that the issues will disappear. While this is a good approach for a healthy system, it may turn the compromised system into a total mess. All data may be gone, the provided services are down, and evidence may be destructed.

Incident response plan

Let’s assume your system is compromised. We need an action plan to deal with a security incident. While many steps will be similar to most situations, your plan might need some customization. This mainly depends on the importance of your system and the services it provides. For example, if the system provides services to your customers, then you might need to get them involved. This means most likely informing them about the intrusion and letting them know what steps they need to take.

Action steps

Assign someone to take notes

While things are hectic, it is easy to forget important details. Make notes of what technical actions have been performed. Also include important decisions, especially when management is involved. Document who has given you the permission to take a particular action.

Good note-taking will be rewarding at the moment of reflection. It helps to better understand what happened and which of the actions were good or bad. Especially if more systems are involved, it may also help in troubleshooting or even reduce the impact of the security incident.

Draft a communication plan

Decide who needs to be informed, when, and how often. Make someone responsible for the communication. Preferably this is someone from the department that deals with marketing, PR, or communication. It might also be your CEO, as he or she has the right stature to address customers, employees, and partners.

Try to be as clear as possible in your communication. Explain what happened and what actions you have taken so far. Share the steps you expect to take next and when more updates will follow.

Include the security team or security officer

Don’t try to solve everything alone, even if you are the most technical person in the company. A security breach might impact the whole company and your actions can be costly. Get someone involved and have them help you during the decision making process.

Inform the legal department

If you have a Legal department, get them involved. A security breach might have to be reported to the authorities.

Report to the police department or CERT

The police were never very good with digital crimes or cybercrime. That is changing, especially now more specific departments are created to deal with this. So consider reporting it at your local police station or a digital equivalent of it. During resolving the issue, you might need to collect evidence, especially if you consider taking legal actions later on. Also, make sure that any actions you take are according to the law in your country.

Think ahead

From a technical point of view, it’s wise to think ahead. How are you going to rebuild the services? Is a replacement system needed, or is the current pool of machines capable of handling it? What if more systems are compromised?

Prepare

Taking preparation steps to make things go smoother later on, can be a very wise decision. For example, rolling out some fresh virtual machines in the background and have them at the latest update level. If the decision is made to switch over to new systems, you saved some time. Especially with the flexibility nowadays, decide where you can take some preparations upfront and save you some time.

Rebuilding the system

Perform a fresh installation

The first step now is the installation of the new system. With virtual machine technology, it may be fine to reuse the system. Simply detach the existing disk and add a new one. The detached disk can be used for forensics when needed.

This is also a good time to learn from the breach. So while you are doing a new install, choose for a “minimal installation”. Any package that is not installed, can’t become an issue.

Security updates

Before bringing any system into production, make sure the system is up-to-date and has all security patches. It’s a waste of time and other resources to activate a new system, finding out it’s compromised again in just a matter of hours or even minutes!

Software

Time to install the required software. Only installing the required software will speed up the installation process, but also keeps the system clean. This is the time to avoid clutter!

Restore of data

After the system is reinstalled, data can be restored. Additional measures might be needed to prevent restoring data with anything related to the previous breach. For example, if you would restore binaries, you might bring back backdoored system utilities. Only restore the data than you can fully trust.

Deploy the system

When everything is clean and the system is tested, it can be deployed to the production farm again. As a follow-up, it’s wise to keep this system carefully monitored. The attacker might want to break the security defenses again.

Audit

The point is to learn from every experience, including a compromised Linux system. So after installation of new software, it is time to do another round of auditing. Run Lynis on the system to check if any vulnerabilities exist and fix them before deploying the system.

Additional thoughts

Depending on your organization, you might have to deal with forensics as well. In this case, it is important that each step you take, does not make the work of the forensics team harder (or even impossible). Before altering the system, determine if your actions are authorized and do not obstruct with any forensics at a later stage.

Collecting evidence and data is a completely field in its own. Still, there are some great guidelines and valuable lessons, which can be found in RFC3227: Guidelines for Evidence Collection and Archiving .