Dealing with a compromised Linux system
Compromised Linux system
Before we dive deep into this subject of dealing with a compromised Linux system, we have the answer the biggest question: how do we know we are compromised?
Usually some signs are a clear give-away:
- The website hosted was altered and replaced with a “You have been hacked” page
- The system is missing essential binaries, or they all crash after executing
- Unauthorized users have been created and the system is hosting movies and music, which is not a main goal of the system.
Sometimes the signs are present, but it’s not directly clear if the system is fully compromised or just victim of an bad decision (e.g. blacklist).
- Strangely named processes are running and many network connections are visible
- The system is blacklisted on the internet in several RBLs (blacklists)
If you are still in doubt that a system is compromised, don’t wait and consult a security professional. Waiting reduces the time to act, gather evidence and only delays getting back to a clear state. Every system has a function and you should consider the consequences it might have on the integrity and confidentiality of the data stored or processed by the system. Even if the system not have a business critical purpose, consulting with a professional will give you the best action to take. Just pulling the network or power plug isn’t always the best choice.
If you have confirmed that the Linux system is really compromised, the first step is: don’t panic. Whatever you do, stay calm and focused. Some pointers which might help you in dealing with the compromise:
Assign someone to take notes
Write down the date and time of each action. This helps in troubleshooting steps, problem analysis and collecting data in case multiple machines are compromised.
Draft a small communication plan
Decide who needs to be informed, when, how often and who is responsible for the communication.
Inform the legal and communication department
Inform legal, if needed. Bigger companies might need to do a press release.
Report to police department
You might need to collect evidence, or want to prosecute later on. Make sure that the actions you take are according the rules.
From a technical point of view it’s wise to think ahead. How are you going to rebuild the services. Is a replacement system needed, or is the current pool of machines capable of handling it? What if more systems are compromised?
Taking preparation steps to make things go smoother later on, can be a very wise decision. For example rolling out some fresh virtual machines in the background and have them at the latest update level. If the decision is made to switch over to new systems, you saved some time. Especially with the flexibility nowadays, decide where you can take some preparations upfront and save you some time.
Rebuilding the system
First step now is the installation of the new system. In case the existing system is used, a different hard disk could be used when the old one is needed for (additional) forensics.
Choose for a “minimal installation” when reinstalling the system. Packages that are not installed, won’t affect your system in any way. Also from a security point of view it is wise to use a minimal installation, as this limits the amount of vulnerable packages.
Before bringing any system into production, make sure the system is fully patched. It’s a waste of time and other resources to activate a new system, finding out it’s compromised again in just a matter of hours, or even minutes!
Time to install the required software. Only installing the required software will speed up the installation process, but also keeps the system clean. This is the time to avoid clutter!
Restore of data
After the system is reinstalled, the data can be restored. Care should be taken that this data is not compromised. Restoring binaries for example might increase the risk that backdoored binaries are used again. Where possible, only restore the data itself.
The point is to learn from every experience, including a compromised Linux system. So after installation of new software, it is time to do another round of auditing. Run Lynis on the system to check if any vulnerabilities exist and fix them before deploying the system.
When everything is clean and the system is tested, it can be deployed into the production farm again. As a follow-up it’s wise to keep this system carefully monitored, as the attacker might want to break the security defenses again.
Depending on your organization, you might have to deal with forensics as well. In this case it is important that each step you take, does not make the work of the forensics team harder (or even impossible). Before altering the system, determine if your actions are authorized and do not obstruct with any forensics at a later stage.
Collecting evidence and data is a completely field in its own. Still, there are some great guidelines and valuable lessons, which can be found in RFC3227: Guidelines for Evidence Collection and Archiving.