Three kernel developers describe what it's really like to work on the kernel, how they interact with developers from other companies, some pet peeves and how to get started.

Like most Linux users, I rarely touch the actual code for the Linux kernel. Sure, I've looked at it. I've even compiled the kernel myself on a handful of occasions—sometimes to try out something new or simply to say I could do it ("Linux From Scratch" is a bit of a right of passage).

But, unless you're one of the Linux kernel developers, odds are you just don't get many opportunities to truly look "under the hood".

Likewise, I think for many Linux users (even the pro users, sysadmins and developers), the wild world of kernel development is a bit of a mystery. Sure, we have the publicly available Linux Kernel Mailing List (LKML.org) that anyone is free to peruse for the latest features, discussions and (sometimes) shenanigans, but that gives only a glimpse at one aspect of being a kernel developer.

And, let's be honest, most of us simply don't have time to sift through the countless pull requests (and resulting discussions of said pull requests) that flood the LKML on a daily basis.

With that in mind, I reached out to three kernel developers—each working at some of the most prominent Linux contributing companies today—to ask them some basic questions that might provide a better idea of what being a Linux kernel developer is truly like: what their days look like and how they work with kernel developers at other companies.

Those three developers (in no particular order):

• Dave Hansen, Principal Engineer, System Software Products at Intel.
• Josh Poimboeuf, Principal Software Engineer on Red Hat Enterprise Linux.
• Jeff Mahoney, Team Lead of Kernel Engineering at SUSE Labs.

Intel, Red Hat and SUSE—three of the top contributors of code to the Linux kernel. If anyone knows what it's like being a kernel developer, it's them.

I asked all three the exact same questions. Their answers are here, completely unmodified.

Bryan Lunduke: How long have you been working with the Linux kernel? What got you into it?

Dave Hansen (Intel): My first experience for the Linux kernel was a tiny little device driver to drive the eight-character display on an IBM PS/2, probably around 20 years ago. I mentioned the project on my college resume, which eventually led to a job with IBM's Linux Technology Center in 2001. IBM is where I started doing the Linux kernel professionally.

Josh Poimboeuf (Red Hat): My first introduction to Linux happened around 2001, when I was a software engineer at IBM working on server firmware. IBM had recently embraced Linux, and I was placed on a team that was responsible for replacing legacy proprietary firmware with a new embedded PowerPC platform based on Linux.

Once I discovered Linux, I was hooked. I installed it on my laptop immediately. It was mind-boggling that all the source code was freely available, and that you could control every single bit of code that ran on your laptop. And, unbelievably, it was free.

At IBM, I started out by doing hardware bringup and Linux application development. But I was always especially fascinated by the kernel. So my curiosity gradually led me to work my way down the stack. My first real kernel experience came when I started writing device drivers in 2004.

By 2008 I was the "kernel guy" on the team, responsible for porting the kernel to our proprietary HW, and for resolving all kernel issues found in the field. That was a bit of a trial by fire, but it was a great way to learn about the entire kernel tree.

The kernel is so big that my learning process still continues to this day. That's always been one of my favorite things about the kernel. There's always more to learn.

These days I work at Red Hat, where I do a lot more upstream work. I work on a wide variety of things: live patching, the objtool static analysis tool, the x86 unwinder, speculative CPU vulnerability mitigations and more.

Jeff Mahoney (SUSE): I've been working with the Linux kernel for 20 years. I got into it initially in college because I was interested in systems software. I happened to buy some hardware for which there was no driver, and I wrote a small one. Before working on the kernel, I was a systems admin for UNIX systems, and a coworker and I then decided to try writing a clustered filesystem ourselves. It turned out that the hardware to do that was much too expensive for us, so we ended up contributing to ReiserFS instead. That led both of us into careers working on Linux.

Bryan: What does a regular day as a Linux kernel developer look like?

Dave (Intel): My days can vary a lot. The one constant is probably email—lots and lots of email. It might be internal or external code reviews, or answering a question from another Intel team or an external customer. The best, most satisfying days are the ones where you start with a problem or a kernel crash, and have a patch posted by the end of the day.

Josh (Red Hat): It might actually be surprising to learn the variety of things a kernel developer does on a daily basis. Each day—and week and month—is different. It's often "choose your own adventure".

Obviously, one of the main things a kernel developer does is write code. Sometimes I can (mostly) disappear for a week or month (or two!) to hack away on a new feature. Those days/weeks/months are my favorite part.

But writing code is only part of it. There's also debugging, reading code, collaborating, testing, code review, code-related discussions, reading papers, research and meetings. It's good to mix things up. And you get to interact with some really smart people from all over the world, which is really interesting.

Most communication happens through email, but many kernel developers also attend conferences, like Linux Plumbers Conference or Kernel Recipes. A lot of good discussion happens at conferences. They're also useful for putting faces to names, which makes a big difference when you're mostly interacting with people over email.

Jeff (SUSE): It's a mix of communication, coding, building and testing. Lots of email—bug reports, code review, design discussions either internally, on public mailing lists or IRC.

Bryan: With the kernel work your team does, how much of it is working with others within your own company, and how much is working with developers working on Linux at other companies? Possibly companies that compete with each other in some ways?

Dave (Intel): Because of the incredible variety in the way that our customers use Linux, work with the upstream kernel is an absolute imperative. Virtually all of that work results in work with folks in the community from other companies. There is also a lot of work that goes on behind the scenes to support the work with the community. A great comment from someone in the community might result in us going off for a week or a month to revise our work. Although we might not be sending mail on LKML on a given day, we are actively working with the community.

Josh (Red Hat): It really varies by person. Some people spend 100% of their time working upstream with the Linux community. Others spend 100% of their time internally, backporting and resolving issues in RHEL. Many of us are somewhere in the middle, dividing our time in both worlds.

Red Hat's policy is "upstream first". So any features or fixes in Red Hat's kernel have to be accepted by the upstream community first. That gives us plenty of opportunities to work with the Linux community.

Jeff (SUSE): It depends on what we're doing—bug reports tend to be handled within SUSE, by our developers and support teams. Part of that is contractual, while part of it is practical. When we release a product, we've chosen a particular kernel version and build on top of that. Any fixes must also be against that version, and the upstream community isn't usually interested in those. Once we've created a fix, if the bug still exists in the latest release, we'll do that work in public.

Feature development happens on public mailing lists, where the participants may work at SUSE, may work for other companies, or may be doing it out of personal interest. One of the most enjoyable parts of working on Linux is that even though there are developers from hundreds of different companies who may be competing with one another, we get to collaborate as if we were on a single team. In addition to the mailing lists used to do code review and discussions, many subsystems have IRC channels where developers (and users) chat about projects and socialize.

Bryan: When you need to work with other companies (be it Intel, SUSE, Red Hat, Canonical, IBM and so on) on kernel issues—such as security vulnerabilities—how does that work? Is there an established process?

Dave (Intel): There are really two complementary processes that happen. Intel has formal company-to-company communication channels that are really great for synchronizing the business side of things. A challenge on the security front has been creating communication channels that support the coordinated disclosure process and simultaneously support normal community processes, like mailing lists. Both avenues have matured quickly and continue to evolve to help us meet the evolving security landscape.

Josh (Red Hat): When there's an embargoed security vulnerability, we do have strict processes in place for secure collaboration with other companies.

Luckily, such embargoes are rare, and they're the exception to the rule for how we normally operate. We typically work closely with developers from other companies all the time on the Linux kernel mailing list, with no special processes needed.

One good example is live kernel patching. My team at Red Hat created the kpatch technology, but at the same time, a team at SUSE created kGraft. Instead of going forward with competing approaches, we worked closely with the SUSE team at conferences and through mailing lists to create livepatch, which actually turned out to be a better technology than both kpatch and kGraft.

In fact, cross-company collaborations like that happen every day on the upstream mailing lists. It's really just business as usual. Our interactions are always focused on what's best for upstream. In the end, what's best for upstream is also what's best for the companies that rely on it. That independent company-agnostic attitude is strongly reflected in the upstream Linux culture.

Jeff (SUSE): For security vulnerabilities that aren't yet public, our security team coordinates with their counterparts with other companies. Otherwise, unless there's a compelling reason not to, the collaboration all happens on public mailing lists. There, the process is to post your code, listen and respond to review and feedback, perform the required changes, re-post, and repeat.

When the feedback is positive, the process is complete, the maintainer for the subsystem will pick it up (according to their timeline) and pull it into the git repository for their subsystem. Then the maintainer asks Linus to pull those changes into the mainline repository.

Bryan: Every software developer has a pet peeve with the projects they work on. What's your pet peeve—the thing that you really wish you could change—with Linux?

Dave (Intel): I really wish developers would focus on making reviewers' lives easier. First, communicating what you are doing, why you are doing it and why it matters is critical. Then, making sure that the code and its supporting comments are as self-explanatory as possible. Too often, we focus on making sure the code functions, then call it a day. To me, that's only half of the job.

Josh (Red Hat): If I had a magic Linux wand, I would:

1. Eliminate CPU speculation—no more Spectre/Meltdown-type bugs!
2. Get rid of the need for security embargoes—but to be clear, I believe that such embargoes are necessary in the real world.
3. More broadly diversify the Linux kernel development population. More differing perspectives can produce better ideas. I think we're already slowly moving in that direction.

Of those, #1 and #2 aren't realistic, but maybe we can achieve #3.

Jeff (SUSE): The lack of diversity in the community, especially the gender gap. Women are underrepresented in computer science fields generally, but especially so in the Linux kernel community. There has been some outreach efforts, but more needs to be done.

Bryan: The Linux kernel is, at this point, more than a quarter of a century old. In software terms, it's certainly been around a while! Do you see the need for it to be replaced any time soon? If so, with what? If not, why?

Dave (Intel): I don't really see Linux as a 25-year-old project. The Linux of 25 years ago was not the de facto OS on servers, routers or phones. There's no need to replace something that's continually changing, growing and improving as fast as Linux.

Josh (Red Hat): These days, tech trends are fickle, and most technologies have a very short lifetime. But I don't see Linux going anywhere. Its true strength is in its development model. It's not perfect, but it's still the best way to produce software at scale that I've ever seen. I wouldn't be surprised to see Linux thrive well into the next quarter century.

Jeff (SUSE): While the project is more than 25 years old, it hasn't stood still. The kernel itself is constantly evolving to meet new needs. New kinds of users are coming to the Linux community every year. The kernel has had between 10,000 and 15,000 commits in each release for at least the past ten years. The community is still growing. I don't think it will be replaced any time soon, but it will continue to evolve.

Bryan: What would you tell folks thinking of getting into Linux kernel development?

Dave (Intel): Please do! Linux is only becoming more important to companies like Intel. It's a challenge to find folks with the technical skills to work on the kernel and the skills necessary to navigate the community. The most successful folks who join the community are the ones that have a problem to work on. Maybe it's some device that Linux doesn't support, or a bug that's driving you crazy on your laptop. The folks that come with patches that don't solve a clear problem generally have a tough time getting those patches accepted.

Josh (Red Hat): First, I'd say to just find a way to dive in and see if you like it.

One good way to get started is to pick a small area of the kernel you're interested in and dedicate yourself to becoming an expert on that little piece of code. Read the code until you understand it. Tweak it and see how it affects your system. Start reviewing patches related to it on the mailing list. After a while, you'll start seeing opportunities for patches, like bug fixes or code improvements.

When you do eventually post a patch, don't get overly attached to your code. Try not to take feedback personally. Our common goal is to produce the best code. It's ok to make mistakes. Put your ego aside, be humble, be respectful, and listen to feedback with an open mind and try to learn from it. That's how the code gets better. It also helps you earn respect from others in the community.

Kernel development can take a lot of patience, humility and persistence. It's not uncommon for code to be thrown away or rewritten several times. The process can seem inefficient at times. But in my experience, the end result is always better than anything proprietary development can produce.

The kernel codebase is huge, so diving into code you've never seen will be a common occurrence. Whenever you have a question about how something works, the answer is always in the code somewhere. Get familiar with cscope. For vim users, I'd recommend the vim cscope plugin.

Also, work on your written communication skills, as most of your non-coding time will be spent in email. And, of course, learning to make your code easily readable by others is also very important.

Finally, finding a mentor (or mentors) can be valuable. I never had an official mentor per se, but I've been lucky enough to have had many more experienced people guide me through the years.

Jeff (SUSE): It can be a lot of fun, but it takes some effort to get started. Start with something you're interested in, find something small to fix, and post your work. Read about and understand the process. Listen and respond to feedback. Experienced developers are usually willing to spend some time helping new developers if they're willing to listen.