zfs recv doesn't always respect -x mountpoint

[DigitalBrains1]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	12/bookworm/stable
Kernel Version	6.1.0-32-amd64 (Debian linux-image-6.1.0-32-amd64)
Architecture	x86_64
OpenZFS Version	zfs-2.3.1-1~bpo12+1 zfs-kmod-2.3.1-1~bpo12+1 (Debian bookworm-backports)

Describe the problem you're observing

Under some specific circumstances (I managed to suss out two), zfs recv does not respect the -x mountpoint flag, instead transferring the mount point from the source and mounting it at an unintended path.

Describe how to reproduce the problem

If you send a filesystem without properties, and then later send an incremental update with properties, a -x mountpoint is not respected:

# zfs create rpool/test
# zfs create -o mountpoint=/foo rpool/test/source
# zfs snap rpool/test/source@snap1
# zfs send rpool/test/source@snap1 | zfs receive rpool/test/dest
# zfs list -o name,mountpoint,mounted -r rpool/test
NAME               MOUNTPOINT  MOUNTED
rpool/test         /test       yes
rpool/test/dest    /test/dest  yes
rpool/test/source  /foo        yes
# zfs snap rpool/test/source@snap2
# zfs send -p -i @snap1 rpool/test/source@snap2 | zfs receive -x mountpoint rpool/test/dest
# zfs list -o name,mountpoint,mounted -r rpool/test
NAME               MOUNTPOINT  MOUNTED
rpool/test         /test       yes
rpool/test/dest    /foo        yes
rpool/test/source  /foo        yes

Second problematic case. If there was an unrelated filesystem at the destination that you overwrite with -F, -x mountpoint is ignored:

# zfs create rpool/test
# zfs create -o mountpoint=/foo rpool/test/source
# zfs create rpool/test/dest
# zfs list -o name,mountpoint,mounted -r rpool/test
NAME               MOUNTPOINT  MOUNTED
rpool/test         /test       yes
rpool/test/dest    /test/dest  yes
rpool/test/source  /foo        yes
# zfs snap rpool/test/source@snap1
# zfs send -p rpool/test/source@snap1 | zfs receive -F -x mountpoint rpool/test/dest
# zfs list -o name,mountpoint,mounted -r rpool/test
NAME               MOUNTPOINT  MOUNTED
rpool/test         /test       yes
rpool/test/dest    /foo        yes
rpool/test/source  /foo        yes

This made me suspect maybe the point is whether or not a set mountpoint existed was what caused the problem, but the following shows that it also happens if mountpoint was already set:

# zfs create rpool/test
# zfs create -o mountpoint=/foo rpool/test/source
# zfs create -o mountpoint=/bar rpool/test/dest
# zfs list -o name,mountpoint,mounted -r rpool/test
NAME               MOUNTPOINT  MOUNTED
rpool/test         /test       yes
rpool/test/dest    /bar        yes
rpool/test/source  /foo        yes
# zfs snap rpool/test/source@snap1
# zfs send -p rpool/test/source@snap1 | zfs receive -F -x mountpoint rpool/test/dest
# zfs list -o name,mountpoint,mounted -r rpool/test
NAME               MOUNTPOINT  MOUNTED
rpool/test         /test       yes
rpool/test/dest    /foo        yes
rpool/test/source  /foo        yes

so the problem is still there if the original unrelated filesystem already had a mountpoint set.

Doing just the most obvious thing does not lead to issues:

# zfs create rpool/test
# zfs create -o mountpoint=/foo rpool/test/source
# zfs snap rpool/test/source@snap1
# zfs send -p rpool/test/source@snap1 | zfs receive -x mountpoint rpool/test/dest
# zfs list -o name,mountpoint,mounted -r rpool/test
NAME               MOUNTPOINT  MOUNTED
rpool/test         /test       yes
rpool/test/dest    /test/dest  yes
rpool/test/source  /foo        yes
# zfs snap rpool/test/source@snap2
# zfs send -p -i @snap1 rpool/test/source@snap2 | zfs receive -x mountpoint rpool/test/dest
# zfs list -o name,mountpoint,mounted -r rpool/test
NAME               MOUNTPOINT  MOUNTED
rpool/test         /test       yes
rpool/test/dest    /test/dest  yes
rpool/test/source  /foo        yes

Here we see that -x mountpoint does actually work, and in the most common case of transferring your filesystems to a backup server, it all works swimmingly, until you do one of the things above. Obviously you do this with a filesystem with mountpoint=/ for maximum damage, as Murphy demands of you.

I managed to find two cases where this happened, but of course I don't know if there are more. I had similar problems before, with older versions of OpenZFS. I don't think it's a new bug.

Thanks for developing and maintaining OpenZFS!