Mender is an open-source (Apache 2.0) project to address over-the-air (OTA) software update management for Linux-based IoT devices. When we researched this five years ago, there were no open-source end-to-end (device-to-server) options to manage the lifecycle of OTA updates for connected devices. Some open-source options were available, but they either had a proprietary management server, or they were client-only and required integration with another back-end server.

In short, the options available to IoT device-makers either had vendor lock-in or simply were too kludgy. Thus, we created Mender, which has two components: the runtime client integrated into the device and the management server with an intuitive user interface to manage updates at scale for large fleets.

Figure 1. The Mender Server's User Interface

We found in our initial research phase that many embedded systems developers created their own remote update mechanism, which usually took risky shortcuts around security and robustness. Embedded development traditionally has been a very diverse space, and the lack of technology standardization generates a lot of custom work for device-makers. Unlike web development and accepted standards, such as the LAMP stack, device-makers had to create much of their stack. This includes the fundamental capability of remote updates. And, most developers had no other choice but to build their own, given how exotic hardware and OS combinations could be for connected devices. We created a community repository called Mender Hub to allow developers to create and reuse tested and validated integrations to enable OTA updates for any combination of hardware and OS.

A consequence of the growth of IoT devices is the increase of easy targets for malicious actors, evident in the proliferation of malware targeting poorly secured IoT devices. There have been an increasing number of malware attacks infecting poorly secured connected devices. The 2016 Dyn DDoS attack was one of the clearest examples of the ramifications of poorly secured IoT devices, which was executed through the Mirai malware infecting a large number of IoT devices and enslaved them into a botnet. The IoT botnet attack caused major outages across internet platforms and services, including Amazon, GitHub and Netflix.

The increasing connectivity of cars, medical devices and more is making IoT security a serious public health issue. We created Mender to help with baseline security-hardening, and security patching is fundamental. But remote updates is quite challenging and has a lot of nuances to consider to establish a secure and robust OTA process.

There are many real-world examples of connected devices bricking or otherwise becoming unusable due to a brittle update mechanism. Devices can be bricked if an update is interrupted for any reason, including power loss on the device or poor network connectivity. Lockstate, a smart lock company recommended by Airbnb, bricked their devices after a software update and their customers were required to ship back their locks to be repaired manually. The underlying reason is as follows: "A feature update for a different set of locks accidentally included this subset of locks from a first generation 6000i WiFi lock we stopped making a year ago."

Mender has a concept of device types to make sure software can be deployed only to compatible hardware. In the situation with Lockstate, the software simply wouldn't have been able to be deployed to an incorrect version of the device, as it would have crashed at boot time, and Mender would automatically roll back to the last working version.

Fiat Chrysler also had an issue with an OTA software update causing its UConnect infotainment system to go into a reboot loop and in some cases caused the eventual draining of the vehicle's battery. Mender has adopted a dual root filesystem approach to avoid this issue, where an update would be installed in the passive rootfs partition with sanity checks to ensure it is working properly before making that partition active. In this situation, Mender's post-install scripts would have avoided this situation entirely, as Mender has automatic rollback built in.

Mender has full image updates today in order to avoid partially updated devices. The typical output of an embedded Linux CI build is a complete root filesystem, and we wanted to avoid the unmanageability of caring for individual packages. Atomic, full image updates help make deployments reproducible, as all devices will get the same version of all subcomponents. In a fleet of devices, having some with untested configurations because there was a package-based partial installation would become chaotic very quickly.

Figure 2. General IoT Software Update Workflow

Mender's security features include requiring a secure communication channel between the device and server with TLS. Mender also has code-signing for the verification of update artifacts, a feature that industry-leader Tesla implemented after being hacked by Tencent's Keen Security Lab, who was able to get through the vehicle's WiFi connection of a Model S and was able to reach the driving systems and manipulate the brakes while it moved.

Mender can be deployed on-premises or can be used as a service with Hosted Mender. We also have collaborated with Google to integrate Mender into Cloud IoT. Other capabilities include device groupings for controlled update rollouts and an integration to the Yocto Project, a popular build system for embedded Linux. Mender also provides out-of-the-box support for binary distributions including Debian, Raspbian and Ubuntu, and it has the Beaglebone Black and Raspberry Pi 3 as reference devices. The Mender team also is working on the ability to install updates to smaller devices as well as delta updates.