Suppose, not entirely hypothetically, that you have an Ubuntu 18.04 system (which has systemd v237) and you want to add a bind mount for a ZFS filesystem; for example, you want to bind mount /cs/mail on to /var/mail . You're managing ZFS filesystems themselves through ZFS's usual boot time mechanisms instead of /etc/fstab , but putting the bind mount itself in /etc/fstab is the simplest and shortest way. So you put the following in /etc/fstab :

/cs/mail /var/mail none bind

If you boot the system and do a ' df /var/mail ', everything will look good. But this is probably an illusion that will be exposed if you do ' grep /var/mail /proc/mounts ', because you most likely really have two bind mounts, one on top of the other:

/dev/sda1 /var/mail ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
tank/cs/mail /var/mail zfs rw,nosuid,nodev,noatime,xattr,noacl 0 0

The ultimate cause of this is the same as it was in 2014, when I wrote an entry on bind mounts with systemd and non- fstab filesystems . Since there is no /etc/fstab line or .mount unit for /cs/mail , systemd doesn't know that it's a separate filesystem until ZFS's commands run and the filesystem magically appears. So apparently systemd does the bind mount twice, once when just the root filesystem is mounted (which looks like everything needed) and then a second time when a /cs/mail filesystem appears and it knows more. If another program (such as your IMAP server) looks at /var/mail before the second mount appears, it will see (and possibly take a reference to) the bad, empty version on the root filesystem.

The automatically created systemd mount unit for /cs/mail will include a ' RequiresMountsFor =/var /cs/mail ', but this doesn't mean that systemd will require a /cs/mail mount. When systemd starts, as far as it knows /cs/mail only requires the root filesystem.

Suppose that you then decide to get clever and use brute force. If systemd is bind mounting the root filesystem's empty /cs/mail directory, let's give it a mount source that doesn't exist on the root filesystem by changing fstab to be:

/cs/mail/mail /var/mail none bind

Your ZFS filesystem remains /cs/mail ; you will just put everything in a directory in it, /cs/mail/mail . So you make this change, reboot your system, and now you find that /cs/mail is not mounted as a ZFS filesystem, and you still have a bind mount from the root filesystem. As far as I can tell, systemd v237 will automatically mkdir the source of a bind mount if it doesn't exist . So systemd created a /cs/mail/mail on the root filesystem and bind mounted it to /var/mail , and then ZFS found that its /cs/mail mount point wasn't empty and refused to mount the ZFS filesystem. My clever idea made me worse off than before.

(This behavior doesn't seem to be documented in the current systemd.mountd(5) manual page, but systemd v237 does date from 2018. On the other hand, it's not documented in the v237 version of the manual page either.)

These days, systemd allows you to express some but not all systemd unit conditions in /etc/fstab (see [systemd.mountd(5)]]). However, I still think that a real .mount unit file is your most reliable bet. This also lets you use ' ConditionPathExists ' to check for the presence of some known file or subdirectory in your original filesystem in a way that hopefully won't cause systemd to create it on the root filesystem if it's missing.

(In a modern systemd you can also use ConditionPathIsMountPoint , but that's not in v237.)

PS: I haven't tested this on anything more recent than the Ubuntu 18.04 systemd, since Ubuntu 18.04 is what we're going to be using for the particular server where this is an issue (because it's what all our other ZFS fileservers use ). That is also part of why this is not currently any sort of bug report to anyone. For that matter, I don't know if this is a bug, other than perhaps a documentation bug.