The Red Hat family of operating systems addressed Meltdown and Spectre in its v3.10 kernel quickly, but relied too much upon Intel's flawed microcode and was forced to revert from a complete solution. Oracle implemented alternate approaches more suited to its v4.1 UEK, but both kernels continue to lack full Spectre coverage while they wait for Intel. Conspicuously absent from either Linux branch is Google's retpoline, which offers far greater and more efficient coverage for all CPUs. Auditing this status is a challenge. This article presents the latest tools for vulnerability assessments.

A frenzy of patch activity has surrounded this year's Meltdown and Spectre CPU vulnerability disclosures. Normally quiet microcode packages for Intel chips have seen four updates in the month of January, one of which was finally to roll back flawed code that triggers random reboots. For enterprise-grade hardware, Intel's quality control has left much to be desired.

It is likely premature to deploy new monitoring and compliance tools, and a final solution for this set of vulnerabilities will wait until correct microcode is obtained. Still, it may be important for many organizations to evaluate the patch status of servers running Linux kernels packaged by Oracle and/or Red Hat.

Meltdown patches exist now and should be deployed immediately on vulnerable servers. Remediating all Spectre vulnerabilities requires not only the latest kernels, but also a patched GCC to compile the kernel that is capable of implementing "retpolines", or compatible microcode from your CPU vendor.

Red Hat was one of the first Linux distributions to publish guidance on Meltdown and Spectre. It established three files as "kernel tunables" in the /sys/kernel/debug/x86 directory to monitor and control these patches: pti_enabled for Meltdown, ibpb_enabled for Spectre v1 and ibrs_enabled for Spectre v2. Only the root user can access these files.

When these files contain a numerical zero, the patches are not active. If allowed for the CPU, a numerical one may be written to the file to enable the relevant remediation, and a zero may be written later to disable it. This is not always allowed—AMD processors are not vulnerable to Meltdown, and the value in the pti_enabled file is locked to zero and cannot be changed. If the fixes are active and show 1, the performance of the CPU may be reduced. Compatible microcode is required to enable all patches on vulnerable CPUs, which adds new assembler/machine language op codes that erase vulnerable kernel data from CPU pipelines and caches to close the exploit.

It is not generally understood that, although the BIOS is responsible for providing a base microcode image, the Linux kernel is able to update some CPUs at boot with a volatile, runtime upgrade for Intel microcode. The update must come from the CPU vendor, carrying its digital signature; it cannot be produced independently by the OS maintainers. This is accomplished on Intel CPUs with the help of the following RPM (microcode for AMD CPUs is contained within the larger "linux-firmware" package):

# rpm -qi microcode_ctl
Name        : microcode_ctl
Epoch       : 2
Version     : 2.1
Release     : 22.5.0.3.el7_4
Architecture: x86_64
Install Date: Sun 21 Jan 2018 12:58:36 PM CST
Group       : System Environment/Base
Size        : 1001060
License     : GPLv2+ and Redistributable, no modification permitted
Signature   : RSA/SHA256, Sat 20 Jan 2018 08:32:03 PM CST, Key ID
72f97b74ec551f03
Source RPM  : microcode_ctl-2.1-22.5.0.3.el7_4.src.rpm
Build Date  : Sat 20 Jan 2018 08:31:55 PM CST
Build Host  : x86-ol7-builder-02.us.oracle.com
Relocations : (not relocatable)
Vendor      : Oracle America
URL         : http://fedorahosted.org/microcode_ctl
Summary     : Tool to transform and deploy CPU microcode update for x86.
Description :
The microcode_ctl utility is a companion to the microcode driver written
by Tigran Aivazian <tigran@aivazian.fsnet.co.uk>.

The microcode update is volatile and needs to be uploaded on each system
boot i.e. it doesn't reflash your cpu permanently, reboot and it reverts
back to the old microcode.

Red Hat has "washed its hands" of Intel's recent flawed microcode, reverted to previous packages, and is advising customers to obtain BIOS updates for Spectre remediation. Although Red Hat very likely will publish new Intel microcode updates again at some point in the distant future, there is clear and obvious frustration that could delay full security compliance.

Oracle distributes the Unbreakable Enterprise Kernel (UEK) as its recommended Linux. Where used within its distribution, it has retroactively renamed the Red Hat-Compatible Kernel (RHCK), which now contains the files in /sys/kernel/debug/x86.

The UEK is a popular choice in enterprise Linux, and it brings many new features to RPM distributions. It is the easiest and most stable method to implement a v4.1 performance-tuned kernel in a supported manner on a Red Hat-derivative OS (which usually runs v3.10 at best).

The UEK is especially notable as it works with Ksplice (as does the RHCK), which allows for online kernel patches and upgrades without rebooting, and it even can extend this ability to select userland libraries (glibc and OpenSSL). This service is offered for free to Fedora and Ubuntu users, and as a paid-support option for both CentOS (after a scripted conversion to Oracle Linux) and Red Hat. Ksplice is the oldest and most mature high-availability option for Linux security upgrades.

The UEK implements a different set of world-readable monitor files, and they do not appear to allow adjustment. This is not generally well known (the information was obtained from Oracle support).

Oracle suggests that you can learn the Meltdown and Spectre status under a running UEK from the following:

# awk '{print FILENAME":"$0}' /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/meltdown:Mitigation: PTI
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Not affected
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Vulnerable

The UEK also adds commentary to /proc/cpuinfo regarding the bug status of the running CPU (which does not appear when booted on the RHCK):

# egrep '(model name|bugs)' /proc/cpuinfo | sort -u
bugs            : cpu_meltdown spectre_v2
model name      : Intel(R) Core(TM)2 Duo CPU     E4600  @ 2.40GHz

Oracle further recommends that Document ID 2348448.1 be consulted for full details on Meltdown and Spectre remediation under Linux. Oracle Solaris is also vulnerable and is addressed in the latest Critical Patch Update. Note that, since updated microcode will be required to remediate Spectre on supported Intel CPUs, a reboot will be required even for systems on Ksplice.

Stephane Lesimple published an elaborate detection script on Github The script is a tour de force that tests for all known mitigations and microcode. The latest version is now (mostly) able to detect the UEK properly, and it has output options for JSON and NRPE. This is likely to remain the authoritative audit tool for speculative attacks on Linux for the immediate future.

Lesimple's script also probes for the presence of "retpolines" in the kernel. Retpolines concern a CPU's "indirect addressing modes" when a register is involved in computing the target of a branch ("jump tables" often are mentioned). The retpoline essentially plays out the Spectre exploit for every vulnerable jump in the kernel binary, and a capable compiler is required to implement it. A Linux kernel that implements retpolines is not vulnerable to Spectre even when running on older microcode and maintains complete control of speculative execution. The performance impact has been demonstrated to be far less dramatic than a kernel using the microcode fixes on Intel's own Clear Linux distribution.

Google (who developed and introduced the concept) has pushed retpolines into Android's January security advisory and deployed retpoline-enabled kernels throughout its data centers. It likely will be some time before retpolines begin to appear outside of supported Nexus and Pixel devices—LineageOS (known for its speed in pushing patches) has only a single device with a retpoline-enabled kernel at this time.

Unfortunately, neither the UEK nor the RHCK appear to have retpoline support at present. The backporting effort into legacy GCC appears to be an obstacle.

In any case, here are demonstration runs of Lesimple's script on various architectures. To start is an Oracle Linux system on the first Meltdown-aware RHCK with the first January microcode release. You can see from the RHCK tunables that all patches are active:

# for x in /sys/kernel/debug/x86/*_enabled; do echo "$x:$(<$x)"; done
/sys/kernel/debug/x86/ibpb_enabled:1
/sys/kernel/debug/x86/ibrs_enabled:1
/sys/kernel/debug/x86/pti_enabled:1

The script warns that the microcode is not stable, but confirms the system is (mostly) patched—it is not detecting the Spectre V1 fix:

# ./spectre-meltdown-checker.sh
Spectre and Meltdown mitigation detection tool v0.34

Checking for vulnerabilities on current system
Kernel is Linux 3.10.0-693.11.6.el7.x86_64 #1 SMP Wed Jan 3 18:59:47
 ↪PST 2018 x86_64
CPU is Intel(R) Pentium(R) CPU G3220T @ 2.60GHz
We're missing some kernel info (see -v), accuracy might be reduced

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates IBRS capability:  YES  (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available:  YES
    * CPU indicates IBPB capability:  YES  (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates STIBP capability:  YES
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability:  NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability:  NO
  * CPU explicitly indicates not being vulnerable to Meltdown
     ↪(RDCL_NO):  NO
  * CPU microcode is known to cause stability problems:  YES  (model 60
  * stepping 3 ucode 0x23)
The microcode your CPU is running on is known to cause instability
 ↪problems, such as intempestive reboots or random crashes.
You are advised to either revert to a previous microcode version
 ↪(that might not have the mitigations for Spectre), or
 ↪upgrade to a newer one if available.

* CPU vulnerability to the three speculative execution attacks variants
  * Vulnerable to Variant 1:  YES
  * Vulnerable to Variant 2:  YES
  * Vulnerable to Variant 3:  YES

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Kernel has array_index_mask_nospec:  UNKNOWN  (couldn't check (couldn't
* find your kernel image in /boot, if you used netboot, this is normal))
* Checking count of LFENCE instructions following a jump in kernel:
* UNKNOWN (couldn't check (couldn't find your kernel image in /boot, if
* you used netboot, this is normal))
> STATUS:  UNKNOWN  (Couldn't find kernel image or tools missing to
> execute the checks)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigation 1
  * Kernel is compiled with IBRS/IBPB support:  YES
  * Currently enabled features
    * IBRS enabled for Kernel space:  YES
    * IBRS enabled for User space:  NO
    * IBPB enabled:  YES
* Mitigation 2
  * Kernel compiled with retpoline option:  UNKNOWN  (couldn't read
  * your kernel configuration)
  * Kernel compiled with a retpoline-aware compiler:  NO
  * Retpoline enabled:  NO
> STATUS:  NOT VULNERABLE  (IBRS/IBPB are mitigating the vulnerability)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Kernel supports Page Table Isolation (PTI):  YES
* PTI enabled and active:  YES
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (PTI mitigates the vulnerability)

A false sense of security is worse than no security at all,
 ↪see --disclaimer

Running the latest version of the script under the UEK now yields reasonable results. The UEK files are specifically mentioned in the script, with the following repeated commentary: # this kernel has the /sys interface, trust it over everything. Here are the results:

# ./spectre-meltdown-checker.sh
Spectre and Meltdown mitigation detection tool v0.34

Checking for vulnerabilities on current system
Kernel is Linux 4.1.12-112.14.13.el7uek.x86_64 #2 SMP Thu Jan 18
 ↪11:38:29 PST 2018 x86_64
CPU is Intel(R) Core(TM)2 Duo CPU     E4600  @ 2.40GHz

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available:  NO
    * CPU indicates IBRS capability:  NO
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available:  NO
    * CPU indicates IBPB capability:  NO
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available:  NO
    * CPU indicates STIBP capability:  NO
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability:  NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability:  NO
  * CPU explicitly indicates not being vulnerable to Meltdown
     ↪(RDCL_NO):  NO
  * CPU microcode is known to cause stability problems:  NO  (model 15
  * stepping 13 ucode 0xa4)
* CPU vulnerability to the three speculative execution attacks variants
  * Vulnerable to Variant 1:  YES
  * Vulnerable to Variant 2:  YES
  * Vulnerable to Variant 3:  YES

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface:  YES  (kernel confirms
* that your CPU is unaffected)
* Kernel has array_index_mask_nospec:  NO
* Checking count of LFENCE instructions following a jump in kernel:
   ↪YES  (34
* jump-then-lfence instructions found, which is >= 30 (heuristic))
> STATUS:  NOT VULNERABLE  (Not affected)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface:  NO  (kernel confirms your
* system is vulnerable)
* Mitigation 1
  * Kernel is compiled with IBRS/IBPB support:  YES
  * Currently enabled features
    * IBRS enabled for Kernel space:  NO
    * IBRS enabled for User space:  NO
    * IBPB enabled:  UNKNOWN
* Mitigation 2
  * Kernel compiled with retpoline option:  NO
  * Kernel compiled with a retpoline-aware compiler:  NO
  * Retpoline enabled:  NO
> STATUS:  VULNERABLE  (IBRS hardware + kernel support OR kernel with
> retpoline are needed to mitigate the vulnerability)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface:  YES  (kernel confirms
* that the mitigation is active)
* Kernel supports Page Table Isolation (PTI):  YES
* PTI enabled and active:  YES
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

A false sense of security is worse than no security at all,
 ↪see --disclaimer

Note that the UEK's Spectre V1 fix is detected via "heuristic" counts of jump-then-lfence instructions; hopefully this is reliable.

It is important for OS kernel vendors to understand that the user community wants these security fixes with a minimum loss of performance and stability. Intel's microcode updates have failed to meet this requirement, and it now falls upon operating system maintainers to close the exploits. There should be continuing questions pointed at Intel regarding why faulty firmware emerged, why it was slower than retpolines, and how a redesigned microcode solution could improve upon retpoline performance. From the perspective of performance, nothing is literally preferable to Intel's January efforts.

However, if Red Hat continues to profess that it has "washed its hands" of this issue, there is a clear opening for Oracle to establish a retpoline-enabled UEK as a widely adopted option in the user community. Although one would hope that all vendors are attentive and responsive to their users, the UEK would become a much more visible and formidable option in the RPM world if retpolines were added to its already considerable advantages. Even a (clearly labeled) beta test version would take it far in this regard.

What is certain is that this tale is far from over. The right people have not spoken, and the right words have not been said. Hopefully, it ends quickly and well.

The views and opinions expressed in this article are those of the author and do not necessarily reflect those of Linux Journal.