Detecting vulnerabilities and managing the associated patching is challenging even in a small-scale Linux environment. Scale things up and the challenge becomes almost unsurmountable. There are approaches that help, but these approaches are unevenly applied.

In our survey, State of Enterprise Vulnerability Detection and Patch Management, we set out to investigate how large organizations handle the dual, linked security concerns of vulnerability detection and patch management.

The results produced interesting insights into the tools that organizations depend on to effectively deal with vulnerability and patch management at scale, how these tools are used, and which restrictions organizations face in their battle against threat actors. Download the copy of the report here.

Vulnerability management is an enterprise responsibility

Before we dive into the results of our survey, let’s take a quick look at why vulnerability management operations matter so much in large organizations.

Vulnerabilities are widespread and a major cybersecurity headache. In fact, vulnerabilities are such a critical problem that laws and regulations are in place to ensure that covered organizations adequately perform vulnerability management tasks – because the failure to do so can hurt a company’s customers.

Each industry has different rules that apply to it – with organizations that handle personal data such as healthcare records and financial service firms operating under the strictest rules. It has an impact on day-to-day vulnerability management operations – some organizations must act much faster and more thoroughly than others.

This is one of the points we explored in the survey, trying to understand how different industry compliance requirements affect vulnerability operations on the ground.

The survey

Early in 2021, we kicked off a survey with the intention to study three key factors in vulnerability and patch management operations. We examined patch deployment practices, how maintenance windows are handled, and tried to get a view into the overall level of security awareness of the organizations that responded.

The survey was advertised publicly to IT professionals around the world and it continues to run, even though we have published the initial results.

An interesting observation we encountered right at the start is that vulnerability management and patching are handled in similar ways around the globe. The geographical location of a respondent did not have any observable correlation to the response we received – we couldn’t find a meaningful link. That said, the industry in which an organization operates did have an impact.

An initial look at the survey results 

So what did we find? Several interesting facts emerged in our survey. First, automated patching is in widespread use – with 76% of respondents stating that they are implementing automated patching across their server fleets.

Live patching was also commonly used, with almost half of respondents depending on live patching to fix vulnerabilities without the downtime usually associated with patching. This is not surprising given the volume of vulnerabilities that are discovered and patched every week – there are simply too many patches to apply to do so in a manual manner.

That said, we did find it interesting that manually researching vulnerabilities online is the most commonly used tool in the vulnerability management arsenal. It suggests that, while automation has a place, some organizations have not fully embraced automation – and that automation may not cover all aspects of vulnerability management.

We made a notable observation when it comes to server fleets as 73% of our respondents said that they rely on single-OS server fleets. It suggested to us that organizations value the ease of maintenance of using a single Linux distribution for all server roles – rather than utilizing a specialized Linux distribution for each server role. CentOS or another CentOS fork was the most commonly used OS.

Different industries exhibited varying practices

The results emphasized how vulnerability and patch management practices varied from industry to industry. The technology sector, for example, spent more than three times as many hours per week monitoring for vulnerabilities when compared to the banking and financial services sector. This may be because tech companies are more in touch with threats – or more frequently targeted.

In another interesting observation, the tolerance – or perhaps the need – for downtime varied dramatically from industry to industry. Across the transport and logistics industries, our respondents suggested that their organizations tolerated on average 15 hours a week in downtime in order to accommodate patching. But healthcare organizations only reported on average one hour a week in downtime.

There were also significant differences in how organizations in different sectors spent staff hours to accommodate vulnerability and patch management. For example, respondents working in public and social services, as well as those in banking and financial services, reported spending a significant proportion of staff hours on monitoring efforts, but industrial firms spend comparatively little time monitoring for vulnerabilities.

Resourcing is a significant issue

Staff hours are a limited resource and organizations must choose carefully how they allocate available resources. When we examined what our respondents reported in aggregate, two interesting facts emerged. First, documenting the patching process takes up relatively little time compared to other patching-related tasks.

In contrast, our respondents suggested that arranging for a maintenance window to apply patches takes the most time – possibly because of the number of stakeholders involved, and the inevitable disappointment as maintenance windows cause disruption.

It also became clear that there are challenges around resourcing. 38% of respondents said that they want to increase IT security staff in order to improve their organization’s patch management – while 29% of respondents said that patch installation was delayed due to a lack of resources.

We’re not surprised, then, that more than half of respondents – 54.5% – said that the staff resources available to them is not sufficient to meet the patching workload, while 27.2% indicated that they intend to hire more staff to accommodate vulnerability and patch management tasks.

Capable tools can boost resources

Human resources underpin the patching process, but access to the right tools and the right features is equally important. Our survey revealed the need for a couple of key features that make vulnerability management and patching more efficient than it would otherwise be.

We asked our respondents what features they would like to see in a patch management tool. Fast responses to new CVEs, live patching and automated comprehensive reporting were features that were requested in almost equal measure.

The question was left open-ended, and some respondents requested features we did not list. Logging was one suggestion, indicating that many of the vulnerability management tools in use do not offer sufficient transparency into the way that the tool works – and how it affects systems.

Phased rollouts are another feature that was requested which points to the need to manage patching in a way that avoids catastrophic disruption by enabling the roll-out of patches in a more controlled manner.

What does it mean for Linux users?

Linux vulnerabilities keep coming hard and fast, and the associated exploits are becoming more and more common – in part because threat actors make use of automated tools to probe for vulnerabilities.

Even the best-resourced security team will get stuck trying to battle threat automation, with the only viable route being security automation. The majority of our respondents already used patching automation, and it is clear that using vulnerability management tools with the right feature set can help teams get more out of the hours that they have available to them.

Your chance to win a Kubernetes course

Earlier in this article, I suggested that while we’ve received a meaningful number of responses, the survey is still running, and we are very eager to build on the number of responses we’ve received so that we can build a more complete picture of vulnerability and patch management in the enterprise environment.

To encourage more people to complete our survey we’re awarding ten free Certified Kubernetes Administrator (CKA) certifications from The Linux Foundation to survey participants. You can stand the chance to win by completing the survey on this link. Survey results are informative and drive opinion: your contribution will help shape the future of vulnerability and patch management, driving good practice across industries.

Interested in the full results? You can download the State of Enterprise Vulnerability Detection and Patch Management report here.