Learnings about Systemd for mounting EBS volumes

Recently I needed to learn more about systemd. The man-pages are great, but here are a few semi-surprising points.

Mounting disks

The entries in /etc/fstab are dynamically converted to systemd mount units, and mounting is managed by systemd. After changing fstab, you can invoke systemctl daemon-reload to regenerate the dynamic config, and systemctl list-units --type=mount to list the mount units. Try the latter now.

This means other units can depend on mounts being ready, using the RequiresMountsFor unit directive.

Even though an fstab entry and a hand-written mount unit is supposed to be equal, fstab is more equal - there are certain magic fstab options that don't (yet) have a hand-written mount unit counterpart. Specifically, x-systemd.device-timeout= lets you specify how long the mount unit should wait for the device to appear. Also, x-systemd.makefs will format the disk if it doesn't yet contain a filesystem, which is handy when creating nodes and disks automatically. Therefore using fstab is superior to hand-written mount units.

Sidenote: cloud-init's mounts module is not a good way to write fstab entries, if the devices attach to the instance later - since cloud-init will check if the device exists, and if not, omits adding the fstab entry. Rather, use bootcmd to write the fstab entries manually.

Which device name to use? When attaching multiple EBS volumes, the device name (/dev/nvme...) doesn't hint about the identity of the device. But you can use the /dev/disk/by-id/....volXXX stable identifier based on the EBS volume id.

Chained loading

There are orthogonal but interrelated concepts of startup ordering between units (Before/After) and dependencies between units (Wants/Requires/...). You can inspect unit dependencies by issuing systemd-analyze dot the-unit, which will output the ordering and dependency relations in graphviz format.

Dependencies control what other units a given unit will pull in to the activation set (transaction), if the given unit is to be activated itself. But it doesn't impose ordering.

Ordering relations take effect only if both units are activated in the same transaction. They are attempted to be activated in the same transaction, if they are both transitively pulled in by a top-level unit to be activated. But most often, they will be direct dependencies.

What about restarts and dependencies? I would have naively assumed that a restart happens in response to a failure, but there was no mention in the systemd docs that after a successful restart of a failed unit, its reverse dependencies would be restarted as well. As far I understand now, I was wrong. In systemd terminology, restarts don't yet cause a unit to fail. Only exhausting the restart limit will transition the unit to a failed state. There are no automatic attempts to recover from the failed state.

What is multi-user.target anyway?

So what determines the set of units to start up during boot? The pseudo-target multi-user.target's dependencies. This is why you often see a reverse-dependency WantedBy=multi-user.target in the Install section.

What is the Install section, by the way? The reverse dependencies listed there only get added, if you enable a unit using systemctl enable the-unit (enabling actually just adds a symlink). So a unit can declare a reverse-dependency on multi-user.target - that is, the potential to be auto-started during boot -, but users can actually control if they want the auto-start to happen.