How to become a Linux security expert?

Years ago it was a challenge to find screenshots of devices running Linux. Nowadays, Linux can power phones, TV’s, computer systems, mainframes, and many more devices. With more devices, the demand for Linux knowledge will continue to grow. At the same time, the demand for security is higher than ever. All the media attention and regulations like GDPR, asks for more Linux security specialists. In this post, the goal is to answer the question: How to become a Linux security expert?

What is actually an expert?

Personally, I find it hard to call myself an expert on any subject. The more you learn about a subject, the better you realize that there is still much more to learn. So I would consider myself not an expert. However, I do know a few things about Linux, security, and the combination of the two. With this post, let’s discover the minimum someone should know to get close to the title of Linux security expert. Before we continue, let me warn you first: the road to be a true expert is long and requires a good amount of persistence. I feel there are no shortcuts possible, as you will have to gain both a technical understanding and gain practical experience. Some of which is hard to gain in your current work, resulting in limited testing in your own lab. Learning can be lonely, but it doesn’t have to be. Share your knowledge in the format you prefer and make friends along the way. Are you in? Great! Let’s have a look at the required basics of Linux.

Know your Linux basics

Linux is more than just the kernel. It is typically a synonym for the GNU/Linux kernel, system administrations utilities, and the distribution that packaged it. Whatever Linux distribution is your favorite, all are sufficient to learn about the subject. However, it does help if you have access to source code and documentation. This way you can retrieve more information about some subjects like system calls (or syscalls).

Subtle differences might be the key to learning

To gain a good understanding of Linux, it is actually a good idea to play with several different Linux distributions. Each of them does things slightly different, from the installation process, up to package management. These differences typically reveal also the discussions that people have about so-called best practices. Should you pull in packages via HTTP and then check their signature, or should the transport be also be encrypted by using HTTPS? I will let you make the choice, so let’s go back to the subject of Linux distributions. Get some virtual machines ready! If you want to master Linux, then at least try these: Arch, CentOS, Debian, Gentoo, Ubuntu.

Go with the flow

After you installed a Linux system, it becomes more powerful if you add useful packages to it. The selection of the packages mainly depends on the goal of your system. Your Linux desktop system will most likely have a graphical interface, where your test web server may lacking any X Window package. When we want to become better at Linux, it helps to learn the most common tools. One of the reasons is that some packages can be found on most systems, as they provide tools like ps, ls, passwd, sed, or cat. Surprisingly, many veteran Linux users still don’t know all of the default available tools. So have a look at your GNU coreutils package, and see what tools it provides. I would not be surprised if you find a tool in there that you never heard of before. No worries, that is exactly the reason to keep learning.

It is a popularity contest

Besides the essential tools for system administration, there is typically more software installed. If you like to learn more about common tools, the popularity contest of Debian is a good start. It shows what packages are installed the most. This list gives also a good idea of the basic skills that an expert should be knowing. For example, in the top results you will find file system utilities. A basic knowledge in this area is required to set up the system (partitioning, file system, usage of LVM). It also provides a good gateway to security measures, like file system encryption with LUKS. In other words, focus on popular packages and tools, as this gives some guidance on where to start.

Getting to know all aspects of a Linux system will take time. Some subjects are easy to learn, yet to master. File permissions is a great example. Setting the basic file permissions and understanding them is not that hard. Most users will be able to answer the question if user Michael can access a particular file. But if you add a umask, file ACLs, and a MAC framework like SELinux to the mix, then it becomes harder to answer. That is actually fine, as we typically can be learning by doing things in our own lab, or by learning from the experience of others.

Forget education, go for skills that matter

Those who are less experienced in a particular field, often have the idea that you need to remember everything. My take on this is simple: you don’t have to. One of the best skills that you can have in this age of time, is searching skills. There is so much to find on the internet, that typically time and our imagination are our only restrictions. If you can find the right search phrases and apply the right filter and selections, almost everything can be found. The power of searching can often also outperform education. And yes, education is useful and can learn you the deeper understanding of a particular field. Still, you will need to search a lot of your life, so you better improve this skill. Now that we covered some of the Linux basics, let’s get into security.

Specialization in modern times

The field of information security is also neverending. So many things to learn and a daily stream of news surrounding the subject. I feel that it is actually a blessing that there is so much information available. This restricts us to try and specialize in particular subjects like Linux security. After all, the more generic you are, the easier you are to be replaced, forgotten, or ignored. With a specialization in a niche subject, you may be easier recognized as someone who is passionate about the subject. So the overwhelming amount of information can definitely be an advantage. Let’s have a look and see how this applies to Linux security in particular.

Like Linux, information security has its own foundations. Some are related to concepts, others to the human aspects. Technical parts cannot be ignored, including techniques and tooling. A true Linux security expert definitely will need this in daily activities. One of the biggest challenges might be finding the right sources. Some valuable ones will follow later in this article. Let’s focus first on the security skills.

Security skills 101

Information security is a field that relies on integrity, confidentiality, and trust. The people that entered the field because you could earn good money, often discovered that it won’t work that easily. The typical security person is a little bit paranoid and does not just do business with everyone. We like to gain confidence in the capabilities of a company and determine the level of trust with the people. Integrity is key, so is confidentiality. You don’t simply share details about other customers or people.

Trust, trust, trust

As a founder of a security firm (CISOfy), our best customers are the ones that actually took a while before they made the jump and pay for our service and software. Not that they didn’t trust us, they were careful. They tested the waters by using the open source tooling, read something on the blog, then got in contact. This might also apply to the learning curve that comes with information security. One does not simply become good in security overnight. You have to show it. Or like the Americans say: walk the talk.

So it is good to remember that confidentiality, integrity, and especially are trust is very important. While looking at the work of others, I see so many that have a bad ‘profile’. They are good at what they do. At the same time, they lack in their presentation. With that I mean the overall picture they provide to the rest of the world, like what they tweet, listed on their LinkedIn page, or put up online. If you want to have others trust you, get your story right and use your personal name. No more hiding behind nicknames and pseudonyms.

Certifications: time-wasters or needed in a field of trust?

If you are technical and want to become better in information security, then consider doing some certifications. Most of the certifications won’t give you superpowers, but they will provide you with a good foundation. The CISSP certification is a good example and typically a nice addition to your resume. And sure, some people will ridicule the certification or tell you its peanuts. Nevertheless, it gives you a good foundation to build on. Most of the covered principles and areas even have their own specialists. Some know everything about risk management, while others do penetration testing with a focus on physical security. In other words, do the exams that you enjoy and increase your skillset.

Technical security skills are in high demand. Interestingly, Linux is often used during security assessments, yet not part of the project scope itself. So a security professional may use Kali Linux, yet only to scan Windows hosts. Still, this professional will really benefit from having good Linux skills, including shell scripting and doing some Python development.

Linux security topics

So if we combine Linux and security, let’s have a look at the particular topics that one should know. I discovered them over a period of 10+ year and it looks like most are still applicable and will remain applicable.

Database security
Digital forensics and incident response
Events and logging
File and data security
Identification, Authentication, Authorization
Kernel security
Malware
Network traffic filtering
Remote administration
Software patch management
Time and scheduling

So these are the topics. Let’s discuss them briefly first. After that, I will provide you with some resources to continue your research or training.

Database security

Proper database administration is a specialty in itself. The average system administrator will be familiar with the basic concepts of databases. For those aiming to learn Linux security, you have to know at least these concepts as well. Building on that knowledge, you then can zoom into the security aspects. Think of securing the database connection, data encryption, transport encryption, and user management. When setting up some test systems in your lab, go at least for MySQL or MariaDB, MongoDB, PostgreSQL, and SQLite. That typically covers 80-90% of the database engines that you may find in the wild. If you are working for a bigger company, consider diving into Oracle databases.

Digital forensics and incident response

Companies and individuals will be implementing system hardening, yet computers will fall to attackers. From backdooring to crypto mining, systems will always be an interesting target. One of the skills that a Linux security expert could use is that of digital forensics. Learn how file systems and memory can be analyzed to find interesting artifacts. Also, you may want to learn about incident response and dealing with break-ins. It will happen one day, so better be prepared for it.

Events and logging

All systems deal with events, with everything between the initial boot up to the shutdown of the system. Most of the events are not very interesting, but now and then some are. The storage and analysis do truly matter within this area. It is useful to learn about the basics of syslog, how systemd stores it data, and how to forward useful events to a SIEM solution.

File and data security

Like databases, sensitive data is typically clustered in a few places on the disk. Learn how to find where data is stored, with tools like lsof. Apply file permissions and take care of the ownership. Where possible, reduce the default permissions by setting a strict umask. Data security also involves looking at the used storage and protocols. Consider encrypting a disk with LUKS, when sensitive data is being stored. When using protocols like SMB and NFS, learn about the specifics of these protocols. They can be configured to reduce who can access what and the related permissions.

Identification, Authentication, Authorization

Upon connecting to a system, users typically should show some proof before they get access. For Linux systems, it is the Pluggable Authentication Modules (PAM) that play an important role. Although PAM configuration is not an easy subject, it is an important one. Most systems will work fine without adjustments, but if you like to set up multi-factor authentication (MFA), you will have to get to learn how PAM really works.

Kernel security

Although Linux is typically more than just the kernel, it is still the kernel that has a huge impact on the security posture of the system. It acts as an agent when accessing hardware. It is the kernel that does traffic filtering, allowing access or denying it, and decide which processes get priority over the others. As part of kernel security, it is good to learn about system calls (syscalls) and what they do. They are often referred to in security modules (e.g. seccomp). Speaking about Linux security modules, or LSM, these are useful additions to fine-tuning permissions. Some are added during the compilation of the kernel, yet most of them can be loaded manually as well. The Linux kernel can also be tuned with the so-called sysctl settings. They can be found in the /proc pseudo-filesystem or by using the sysctl command itself.

Malware

One of the most debated subjects for Linux security is the need for anti-malware solutions. From a basic virus scanner, up to full endpoint solutions, it is hard to answer if a system requires them. This is where your expertise comes in. By following the news and latest threats, it is up to you to decide what is required. With ransomware that previously hijacked MongoDB data, we can safely say that today’s opinion can change tomorrow. The Linux security expert recognizes this fact and has familiarity with backdoors, rootkits, worms, and other types of malware.

Network traffic filtering

Most systems are connected to the network and benefit from some traffic filtering. This can be done with iptables, nftables, or with BPF. Learn the principles behind firewalls and how they apply to Linux.

Use a default deny policy
Learn how to read a firewall configuration
Configure proper logging

Remote administration

Within this area, extensive knowledge of SSH and sudo is very useful. Both are commonly used, especially within enterprise environments. SSH can be optimized and secured in several areas. Become familiar with the main concepts and set up different types of accounts in your test lab.

Public key authentication
Avoid root logins
Set time-outs for active connections

Software patch management

One of the areas that deserve a lot of attention is software patching. This is the practice of updating existing packages and reduce the number of known vulnerabilities. A good understanding of the available package managers is key here. Familiarity with apt, DNF, pacman, and yum, will cover already a good number of Linux distributions. Besides manual patching, also learn how to optimize systems by applying automatic updates and using livepatch.

Time and scheduling

If there is one topic overlooked easily, it is time. We take it for granted, yet so many things rely on it. This also applies to Linux systems and security in particular. Time synchronization is required to validate authentication services like Kerberos tickets and one-time passwords (OTP and TOTP). It improves the quality of logs and event data, as we need the time to do our forensics and incident response.

Resources to continue your journey

There are many websites and tools available to do something with Linux or security. And although the combination of the two reveals fewer resources, there are still quite some good resources left. I’ve collected the most important ones that show up regularly or focus on quality.

Blogs and articles

Qualitative articles are a great way to learn about technical aspects. Unfortunately, most articles on the web are not in-depth. They might instruct one to change the system, but don’t properly explain the ‘why’ behind the change. Here are some resources that are in-depth and are good starters:

Linux audit (this page, Linux hardening, security, compliance)
Blog by Paul Moore, about SELinux
Blog by Kees Cook, one of the kernel developers working on Linux security
LWN security pages, they contain news and articles

Training and courses

The SEC506 course of SANS is one of the few courses that are available on Linux security. The materials of SANS are known for their high quality, intensive learning, and being highly technical. The possible downside is that these courses are often limited in time. This is because they are given at a particular location, or to be followed online.

For those who rather do lab-based training, you might want to follow developments of the aptly named Linux Security Expert project (disclaimer: I’m involved). This website has the goal to provide an extensive training program to learn Linux security by doing practical labs. It does not stop there, as it provides tool collections, security professionals, how-tos, and many other tips and tricks.

Follow some professionals

While there is not one true expert on Linux (ok, maybe Linus is), there are many that specialize in some area of Linux. Then are a few that also care about security, making them great specialists to follow.

Binni Shah (kernel development, malware, shares many good discoveries at Twitter)
Daniel Walsh (SELinux, containers, presenter, works for Red Hat)
Hal Pomeranz (forensics, Linux security, instructor at SANS)
James Tarala (IT auditing, presenter, instructor at SANS)
Jay Beale (tool author, instructor, presenter)
Jessie Frazelle (Linux kernel development, container technology, presenter)
Kees Cook (kernel security)
Michael Boelen (author of this article, also tool developer, presenter)

More useful online resources

Besides these professionals, there are more useful online resources. One of them is @ToolsWatch on Twitter, which is founded by Nabil Ouchn. He covers security tools and is part of the Black Hat Arsenal. If you want to see tools being demoed by the original authors, this is the place to be. Two other resources to discover tools are @KitPloit, which have a huge following on Twitter. Their reviews are minimal and sometimes the tools are not of high quality. If you want to focus more on Linux security tools, then go for @LSELabs on Twitter.

Conclusion

If you reached this part of the article, you took a good amount of time to read. Well done! It is fair to say that becoming a true Linux security expert takes time. Even if you master all aspects listed in this post, there is still so much more to learn. This can be achieved by continuous education, including the reading of in-depth articles. Another valuable resource is by actively participating, including writing a blog post or submitting an article. Not only can it boost your own name, but during the research of writing the article, you will (re)learn things.

I hope that I have inspired you with this blog post. If so, go ahead and share with other students, your teacher, your colleagues, or any social medium. Do you have additional useful resources for those who want to learn about Linux security? Contribute and submit a comment or send me a message via Twitter (@mboelen).