Red Hat's Spectre remediation currently requires new microcode for a complete fix, which leaves most x86 processors vulnerable as they lack this update. Oracle has released new retpoline kernels that completely remediate Meltdown and Spectre on all compatible CPUs, which I show how to install and test on CentOS here.

The Red Hat community has patiently awaited a retpoline kernel implementation that remediates CVE-2017-5715 (Spectre v2) and closes all Meltdown and Spectre vulnerabilities that have captured headlines this year.

Red Hat's initial fixes rely upon microcode updates for v2 remediation, a decision that leaves the vast majority of AMD64-capable processors in an exploitable state. Intel's new microcode has proven especially problematic; it performs badly and the January 2018 versions were plagued with stability issues that crashed many systems. It is a poor solution to a pressing problem.

Google's retpolines—an ingenious approach to v2—essentially play out the exploit within the Linux kernel in any and all vulnerable code. This method allows the kernel to retain complete control over speculative execution hardware in all architectures upon which it runs. It is faster and more generic than Intel's microcode and seems in every way a superior solution.

Oracle appears to be the first major member of the Red Hat community that has delivered a Linux kernel with retpoline support. The latest version of the Unbreakable Enterprise Kernel preview release 5 (UEKr5) now offers complete remediation for Meltdown and Spectre regardless of CPU microcode status. The UEKr5 is based on the mainline Linux kernel's long-term support branch v4.14 release. In late-breaking news, Oracle has issued a production release of the UEKr4 that also includes retpolines, details below. For corporate users and others with mandated patch schedules over a large number of servers, the UEK now seems to be the only solution for complete Spectre coverage on all CPUs. The UEK brings a number of other advantages over the "Red Hat-Compatible Kernel" (RHCK), but this patch response is likely to drive Oracle deeply into the Red Hat community should they remain the single source.

CentOS, Scientific and Oracle Linux are all based off the upstream provider Red Hat. CentOS is a popular variant and is likely the best demonstration for loading the UEK on a peer distro.

It appears that Red Hat itself has been laying groundwork for a retpoline kernel. A January 2018 gcc compiler update implies a successful backport:

# rpm -q --changelog gcc | head -2
* Wed Jan 24 2018 Jeff Law ... 4.8.5-16.2
- retpoline support for spectre mitigation (#1538487)

Oracle announced a yum repository for the UEKr5 on February 25, 2018. There are detailed instructions for users of Oracle Linux to add this repo, but the installation procedure on CentOS is slightly different. Alternately, the RPMs can be manually downloaded and installed from the repository. Below I describe proceeding with the yum repo.

First, define the repo for yum with the exact contents that are found in Oracle's installation instructions:

echo '[ol7_developer_UEKR5]
name=Oracle Linux $releasever UEK5 Development Packages
 ↪($basearch)
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/
↪developer_UEKR5/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1' > /etc/yum.repos.d/public-yum-ol7_uek.repo

Note that the GPG entry above will not normally exist on CentOS. If you want to install it, the contents of the file must be:

# cat /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.5 (GNU/Linux)

mQENBEwtJWoBCACpiY2rGA6T0ceBi92X88/QclytVBjtDRohOVzs3pmIPh3ULqsW
G323nmyKbKQBBSjh9OnuO9Y09VG8mzr/w9YV0Ix4cI9/HDTERZ2+TR5u+VNn5J5h
yvwQSN/FEK6oH2+mz7O0yUNleN7UltR4MOEkHIoAhIvv+1AQQKN3OM8oalz+3gv/
zz9rAoQczQzT7QWOtBrsRMZgBrKXY/TpJrpVSO3Hx8CnbhKGtLxCCrxZ5v7hh1ht
3CRAr2+h5bDA9KP6vBZWKEs7NgcvtZFDY6EJc7AoApr3phX9hHE2+snTxe82DkFT
uA69C8wLyjPCoSy+tcaCqJKOZelNy9RN/WKRABEBAAG0RE9yYWNsZSBPU1MgZ3Jv
dXAgKE9wZW4gU291cmNlIFNvZnR3YXJlIGdyb3VwKSA8YnVpbGRAb3NzLm9yYWNs
ZS5jb20+iQE8BBMBAgAmAhsDBgsJCAcDAgQVAggDBBYCAwECHgECF4AFAlNhkUEF
CSaOl9cACgkQcvl7dOxVHwNHFQf9G2ZI5ZH8T1VASvQteTyUR7uNgqXEbJhi9zZO
7EO26+wzkj9EGLmH1COdUApZ1cINsYfGGgUJT5mfcRV3yYZbwc4AZJbJe0z7C5Yu
ZLs5W0ryV4bzIqcWnVphIAOwmxMxIVGz8Cr1Dsyyal7ORgYzdfOppYetwtZ+J+Wn
/oVgFkh9v19l/CltBkRh2SqgUZYfCoELo7hUzLNh7cw2av8rcSUKSH3ra9MvpYfS
ANCnouzbgKix1gD6niT3pm1s5II3VuQ2vEcJunnoRFci9FzLXelTuL00MvuxERr7
Fsqm1/D2JfKDbE09qy5bLWrWaTM6zOFQKN7F2edY2uaukLT6/w==
=Djed
-----END PGP PUBLIC KEY BLOCK-----

Assuming the GPG key has not been loaded, install the kernel-uek package with a flag to skip the GPG check:

# yum install --nogpgcheck kernel-uek
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: www.gtlib.gatech.edu
 * extras: www.gtlib.gatech.edu
 * updates: www.gtlib.gatech.edu
Resolving Dependencies
--> Running transaction check
---> Package kernel-uek.x86_64 0:4.14.26-1.el7uek will be installed
--> Processing Dependency: linux-firmware >= 20180113-60.git65b1c68
     for package: kernel-uek-4.14.26-1.el7uek.x86_64
--> Running transaction check
---> Package linux-firmware.noarch
      0:20170606-58.gitc990aae.el7_4 will be updated
---> Package linux-firmware.noarch
      0:20180113-60.git65b1c68.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================
 Package         Arch    Version                     Repository
=========================================================================
Installing:
 kernel-uek      x86_64  4.14.26-1.el7uek            ol7_developer_UEKR5
Updating for dependencies:
 linux-firmware  noarch  20180113-60.git65b1c68.el7  ol7_developer_UEKR5

Transaction Summary
=========================================================================
Install  1 Package
Upgrade             ( 1 Dependent package)

Total size: 108 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : linux-firmware-20180113-60.git65b1c68.el7.noarch       1/3
  Installing : kernel-uek-4.14.26-1.el7uek.x86_64                     2/3
  Cleanup    : linux-firmware-20170606-58.gitc990aae.el7_4.noarch     3/3
  Verifying  : kernel-uek-4.14.26-1.el7uek.x86_64                     1/3
  Verifying  : linux-firmware-20180113-60.git65b1c68.el7.noarch       2/3
  Verifying  : linux-firmware-20170606-58.gitc990aae.el7_4.noarch     3/3

Installed:
  kernel-uek.x86_64 0:4.14.26-1.el7uek                                         

Dependency Updated:
  linux-firmware.noarch 0:20180113-60.git65b1c68.el7                           

Complete!

After the installation on a clean CentOS system where all past kernel updates have been removed, the following kernel packages will be present:

# rpm -qa | grep ^kernel | sort
kernel-3.10.0-693.21.1.el7.x86_64
kernel-devel-3.10.0-693.21.1.el7.x86_64
kernel-headers-3.10.0-693.21.1.el7.x86_64
kernel-tools-3.10.0-693.21.1.el7.x86_64
kernel-tools-libs-3.10.0-693.21.1.el7.x86_64
kernel-uek-4.14.26-1.el7uek.x86_64

Reboot to this kernel. The GRUB menu entry will appear as:

CentOS Linux (4.14.26-1.el7uek.x86_64) 7 (Core)

After booting into CentOS-uek, run the spectre-meltdown-checker.sh script by Stephane Lesimple. It should confirm retpoline support:

# ./spectre-meltdown-checker.sh
Spectre and Meltdown mitigation detection tool v0.35

Checking for vulnerabilities on current system
Kernel is Linux 4.14.26-1.el7uek.x86_64 #2 SMP
 Tue Mar 13 01:14:49 PDT 2018 x86_64
CPU is Intel(R) Core(TM)2 Duo CPU     E6550  @ 2.33GHz

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available:  NO
    * CPU indicates IBRS capability:  NO
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available:  NO
    * CPU indicates IBPB capability:  NO
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available:  NO
    * CPU indicates STIBP capability:  NO
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability:  NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability:  NO
  * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO):
     NO
  * CPU microcode is known to cause stability problems:
     NO  (model 15 stepping 11 ucode 0xba)
* CPU vulnerability to the three speculative execution attacks variants
  * Vulnerable to Variant 1:  YES
  * Vulnerable to Variant 2:  YES
  * Vulnerable to Variant 3:  YES

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface:
     YES  (kernel confirms that the mitigation is active)
* Kernel has array_index_mask_nospec:
     YES  (1 occurrence(s) found of 64 bits array_index_mask_nospec())
* Kernel has the Red Hat/Ubuntu patch:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: __user pointer sanitization)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface:
     YES  (kernel confirms that the mitigation is active)
* Mitigation 1
  * Kernel is compiled with IBRS/IBPB support:  YES
  * Currently enabled features
    * IBRS enabled for Kernel space:  UNKNOWN
    * IBRS enabled for User space:  UNKNOWN
    * IBPB enabled:  UNKNOWN
* Mitigation 2
  * Kernel compiled with retpoline option:  YES
  * Kernel compiled with a retpoline-aware compiler:
     YES  (kernel reports full retpoline compilation)
> STATUS:  NOT VULNERABLE  (Mitigation: Full generic retpoline)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface:
     YES  (kernel confirms that the mitigation is active)
* Kernel supports Page Table Isolation (PTI):  YES
* PTI enabled and active:  YES
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

A false sense of security is worse than no security at all,
 see --disclaimer

On the day of this article's submission (March 15, 2018), Oracle issued a new UEKr4 including retpoline support. Production Oracle Linux has now completely mitigated Meltdown and Spectre:

# ./spectre-meltdown-checker.sh
Spectre and Meltdown mitigation detection tool v0.35

Checking for vulnerabilities on current system
Kernel is Linux 4.1.12-112.16.4.el7uek.x86_64
     #2 SMP Mon Mar 12 23:57:12 PDT 2018 x86_64
CPU is Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates IBRS capability:  YES  (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available:  YES
    * CPU indicates IBPB capability:  YES  (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates STIBP capability:  YES
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability:  NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability:  NO
  * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO):
  * NO
  * CPU microcode is known to cause stability problems:
     NO  (model 45 stepping 7 ucode 0x713)
* CPU vulnerability to the three speculative execution attacks variants
  * Vulnerable to Variant 1:  YES
  * Vulnerable to Variant 2:  YES
  * Vulnerable to Variant 3:  YES

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface:
     YES  (kernel confirms that the mitigation is active)
* Kernel has array_index_mask_nospec:  NO
* Kernel has the Red Hat/Ubuntu patch:  YES
> STATUS:  NOT VULNERABLE  (Mitigation: lfence)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface:
     YES  (kernel confirms that the mitigation is active)
* Mitigation 1
  * Kernel is compiled with IBRS/IBPB support:  YES
  * Currently enabled features
    * IBRS enabled for Kernel space:  NO
    * IBRS enabled for User space:  NO
    * IBPB enabled:  YES
* Mitigation 2
  * Kernel compiled with retpoline option:  YES
  * Kernel compiled with a retpoline-aware compiler:
     YES  (kernel reports full retpoline compilation)
> STATUS:  NOT VULNERABLE
     (Mitigation: Full generic retpoline, IBRS_FW, IBPB)


CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface:
     YES  (kernel confirms that the mitigation is active)
* Kernel supports Page Table Isolation (PTI):  YES
* PTI enabled and active:  YES
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

A false sense of security is worse than no security at all,
     see --disclaimer

The yum repo entry for the UEKr4 is below:

[ol7_UEKR4]
name=Latest Unbreakable Enterprise Kernel Release 4 for Oracle Linux
$releasever ($basearch)
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1

The UEK is a popular choice in enterprise Linux, and it brings many new features to RPM distributions. It is the easiest and most stable method to implement a v4 performance-tuned kernel in a supported manner on a Red Hat-derivative OS (which usually runs v3.10 at best).

The UEK is especially notable as it works with Ksplice (as does Oracle's RHCK), which allows for online kernel patches and upgrades without rebooting, and Ksplice can even extend this ability to select userland libraries (glibc and OpenSSL). This service is offered for free to Fedora and Ubuntu users, and as a paid support option for both CentOS (after a scripted conversion to Oracle Linux) and Red Hat. Ksplice is the oldest and most mature high-availability option for Linux security upgrades.

While the UEKr5 remains in preview release status, it is likely not appropriate to use it on critical production servers. The UEKr4 is now a production option for those who require immediate remediation. Users on peer distributions should certainly begin testing the UEKr5 now if their timetables permit. Those with critical systems should consider commercial (paid) support and potentially implement Ksplice for high availability.

Many customers of peer distros would likely prefer to keep their systems "pure" from a single supplier. Still, the community has long been making use of EPEL, ElRepo, Nux and others. Oracle simply offers another repository with useful packages. The EPEL maintainer Fedora was launched as an organization by Red Hat; administrators who forbid third-party yum repositories likely do not understand the historical relationships between these organizations.

There is also occasional hostile sentiment toward Oracle for various reasons. Still, Oracle remains an important contributor to the Linux community, and its software meets the needs of a large base of customers. Quality software with useful features should not be rejected because of organizational bias.

In any case, this is a major accomplishment for Oracle Linux, and all members of the Red Hat community stand to benefit from these efforts regardless of their level of participation. It is unlikely that Oracle will forever maintain the technical advantages discussed here, but it has earned and deserve accolades for these advances.

The views and opinions expressed in this article are those of the author and do not necessarily reflect those of Linux Journal.