At one point in time, when you wanted to attach an external block storage device to a server, you mapped it as a Logical Unit (LU) across a Storage Area Network (SAN). In the early days, you would do this over the Fibre Channel (FC) protocol. More recently, iSCSI (SCSI over IP) has usurped FC in most data centers. Although, they're stable, feature-rich and fully functional, these protocols are built on top of multiple layers and are extremely complex. It requires a certain level of expertise to truly master these technologies. That is why the Brantley Coile Company wrote the ATA over Ethernet (AoE) specification. The standard has been published for a bit longer than a decade. It was Brantley Coile himself who used this technology as the base framework for his then new startup company, Coraid (later rebranded to the Brantley Coile Company). Since then, that same framework has been open-sourced under the General Public License version 2 (GPLv2) and made available to the general public.

What makes AoE attractive is mainly its simplicity. It was written to run in the Data Link Layer (Layer 2) of the networking OSI model. This means that it's not impacted by any of the Internet Protocol (IP) overhead (that is, the Network Layer or Layer 3). Translation: block devices exported via AoE cannot be accessed by IP. Without this additional overhead, network performance does improve when accessing the exported block device(s). This non-routability also adds to the security of the technology. In order to access the volumes, you need to be physically plugged in to the Ethernet switch hosting them.

As for how data is transferred across the line, AoE encapsulates traditional ATA commands inside Ethernet frames and sends them across an Ethernet network as opposed to a SATA or 40-pin ribbon cable.

For the following example, you are required to have at least two computing nodes running any Linux distribution. One of these nodes will export the block devices. This node will be referred to as the Target. The second node will import the block devices and will be referred to as the Initiator.

Most modern Linux distributions will provide binary packages to the entire AoE suite in their local repos, but if you prefer to install from source, you can visit the OpenAoE project's new home on GitHub.

To configure the Target, you must install the AoE server dæmon: vblade. On a distribution like Debian or Ubuntu, run the following command:

$ sudo aptitude install vblade

The next step is to identify a block device to export and the Ethernet port to export through. The vblade application service is very flexible and can work with traditional block devices (such as sdb, sdc and so on), loopback devices (loop0, loop1, etc.), and even files on top of a traditional filesystem, the latter of which appears like any other block device to the Initiator.

For instance, if you want to export /dev/sdb through Ethernet port enps0s3, you would run the following command:

$ sudo vblade 1 1 enps0s3 /dev/sdb

Note: the numeric values following the vblade execution binary define the block device's major and minor numbers. This is sort of the equivalent of a SCSI Logical Unit Number (LUN). Each device needs to have a unique major/minor combination to function properly on the Initiator.

Now, create a generic empty file and export it:

$ dd if=/dev/zero of=aoe.block0 bs=1M count=128
$ sudo vblade 1 2 enps0s3 aoe.block0

Note: the vblade binary does not run as a background dæmon. In runs in the foreground. If you interrupt its execution, it will stop exporting the block devices. In order to avoid this, you can either background the service with the traditional ampersand (&) or just run the dæmonized version of the binary (with the same arguments): vbladed.

Install the AoE initiator utilities. On Debian/Ubuntu run the following command:

$ sudo aptitude install aoetools

Load the kernel module:

$ sudo /sbin/modprobe aoe

This module functions a lot like the Network Block Device (NBD) kernel module in that it gives the client an accessible block device, but all requests to that block device are translated across a network. It also may be beneficial to spend some time looking into the kernel module options. Refer to the modinfo excerpt below:

$ modinfo aoe
filename:       /lib/modules/4.4.0-72-generic/kernel/drivers/block/aoe/aoe.ko
version:        85
description:    AoE block/char driver for 2.6.2 and newer 2.6 kernels
author:         Sam Hopkins 
license:        GPL
srcversion:     668418429AE7969D1660B64
depends:
intree:         Y
vermagic:       4.4.0-72-generic SMP mod_unload modversions
parm:           aoe_iflist:aoe_iflist=dev1[,dev2...] (string)
parm:           aoe_dyndevs:Use dynamic minor numbers for devices. (int)
parm:           aoe_deadsecs:After aoe_deadsecs seconds, give up and fail dev. (int)
parm:           aoe_maxout:Only aoe_maxout outstanding packets for every MAC on eX.Y.
(int)
parm:           aoe_maxsectors:When nonzero, set the maximum number of sectors per I/O
request (int)

For instance, if the dæmon on the Target server is interrupted or exits, pending requests to the Initiator's AoE block devices will hang, indefinitely—well, at least until the dæmon is restarted. To avoid this, you easily can set a timeout value. In the following example, I set this timeout value to 10 seconds:

$ sudo /sbin/modprobe aoe aoe_deadsecs=10

Once that 10 seconds elapses, I/O errors are issued for anything pending and travel back up the software stack to the calling application(s).

So, you have a Target server exporting AoE target volumes. Using the AoE userspace management utilities, define the desired interface to scan/import those volumes from:

$ sudo aoe-interfaces enp0s3

You can avoid this easily with every reboot by updating the INTERFACESfield of the /etc/default/aoetools configuration file and replace the word "none" with that interface. In my case, this is enp0s3:

$ sudo sed -i "s/INTERFACES=.*/INTERFACES='enp0s3'/" /etc/default/aoetools

And to have it take affect, start the service:

$ sudo service aoetools start

Making this service start on every reboot will differ across Linux platforms. It also will differ between traditional init and systemd init services. I advise that you refer to your distribution's documentation for these instructions.

Anyway, it's time to scan for any AoE volumes across the network and import them:

$ sudo aoe-discover

To list what has been discovered, type the following command on the command line:

$ sudo aoe-stat
  e1.1   0.134GB  enp0s3 1024 up
  e1.2   0.134GB  enp0s3 1024 up

By default, these block devices are accessed through the /dev/etherd directory path (take notice of the major/minor numbers):

$ ls /dev/etherd/e1*
/dev/etherd/e1.1        /dev/etherd1.2

And, it will function like any other block device:

$ sudo mkfs.ext4 /dev/etherd/e1.1
$ sudo mount /dev/etherd/e1.1 /mnt

Pretty simple, right? It is no wonder that many storage and data-center administrators have opted for this technology over the more complex SANs.