How to mount an EBS volume on NVMe based instance types

Mounting an persistent EBS volume at boot time is no trivial task. Since the introduction of the Nitro based instance types, EBS volumes no longer appear under the device name specified during the attach command. Instead, the device name will be determined by the kernel and depend upon the order in which it is attached. As a result, switching EC2 instance type or volumes, may cause an incorrect volume to be mounted or not be mounted at all. In this blog we present the details for mounting an EBS volume on any EC2 instance type.

mounting an EBS volume

In cloud migrations, we often run into an application that require a persistent volume to store state. For instance, WebsphereMQ.
In these cases, We provision the relevant EC2 instances with a detachable network interfaces and EBS volumes: In that way, we can still
destroy and replace the instance while preserving the ip address and data.
In order to mount such an EBS volume on the EC2 instance, we need to:
* Attach the disk
* Wait for the disk
* Format the disk
* Mount the disk

Attach the disk

Attaching the disk is quite easy. In CloudFormation we specify the volume id and device name as one of the volumes of the ec2 instance:

EC2Instance:
  Properties:
    Volumes:
      - Device: /dev/xvdd
    VolumeId: !Ref 'Storage'

Wait for the disk

Even though the disk is part of the ec2 instance definition, the ec2 instance is booted before
the attachment has completed. To avoid the boot to complete before the volume is mounted, we
need to wait for the block device to appear:

while [[ ! -b $(readlink -f /dev/xvdd) ]]; do
    echo "waiting for the disk to appear..">&2;
    sleep 5;
done

On NVMe based instances, the disk will not appear as the specified device name ‘/dev/xvdd’, but rather
as ‘/dev/nvme[0-9]n1’. When the disk is attached, the device manager will create a symbolic link ‘/dev/xvdd’
pointing to the actual device. Unfortunately, the mount command does not accept symbolic links as device.
By using the readlink command we always get the actual device name, whether it is on a Nitro-based instance or
a classic instance.

Format the disk

Now the device has appeared, we format the disk only if it is unformatted:

blkid $(readlink -f /dev/xvdd) || mkfs -t ext4 $(readlink -f /dev/xvdd)

Labeling the disk

To ensure that we can create a mount instruction without specifying a
physical device name, we label the disk:

e2label $(readlink -f /dev/xvdd) wmq-data

Mount the disk

Now the disk is attached, formatted and labeled, we generate the fstab entry:

grep -q ^LABEL=wmq-data /etc/fstab || echo "LABEL=wmq-data /var/mqm ext4 defaults" >> /etc/fstab

Remove all the old-style mount entries in /etc/fstab:

sed -i -e '/^[\/][^ \t]*[ \t]*\/var\/mqm[ \t]/d' /etc/fstab

Finally we mount the disk, if it was not mounted:

grep -q "$(readlink -f /dev/xvdd) /var/mqm " /proc/mounts || mount /var/mqm

When the machine reboots, the /etc/fstab entry will already have mounted the disk for us.

Usage

When you want to add your own EBS volume mount, you can use the generate-mount-ebs-volume-bootcmd script to generate the required commands. Type:

$ ./generate-mount-ebs-volume-bootcmd /dev/xvdd /var/mqm wmq-data

The first parameter is your device name, the second the mount point and the third the label.
It will generate a bootcmd snippet, you can add to your machine’s cloud-config.

  bootcmd:
    - mkdir -p /var/mqm
    - while [ ! -b $(readlink -f /dev/xvdd) ]; do echo "waiting for device /dev/xvdd"; sleep 5 ; done
    - blkid $(readlink -f /dev/xvdd) || mkfs -t ext4 $(readlink -f /dev/xvdd)
    - e2label $(readlink -f /dev/xvdd) wmq-data
    - sed  -e '/^[\/][^ \t]*[ \t]*\/var\/mqm[ \t]/d' /etc/fstab
    - grep -q ^LABEL=wmq-data /etc/fstab || echo 'LABEL=wmq-data /var/mqm ext4 defaults' >> /etc/fstab
    - grep -q "^$(readlink -f /dev/xvdd) /var/mqm " /proc/mounts || mount /var/mqm

It has to be hard-coded, as the write-files module of cloud-config is run after the bootcmd.

Conclusion

Mounting a persistent EBS volume at boot time is no trivial task, and with the Nitro based images it has become even more difficult. By using the readlink command, this script will work both on NVMe based instances types and on older instance types. I do hope that AWS will provide a simpler and more robust way of mounting EBS volumes at boot time in the near future.