ZFS on Linux and Ansible: order of dependencies

Installing ZFS on Linux (Debian Stretch) turned out to be more of a challenge than I thought. Combined with Ansible, order of tasks seems to be crucial.

I have been working on improving our backup infrastructure. This involves getting some new ZFS boxes up and running. We use Ansible for configuration management, ZFS on Linux and 'clean' Debian Stretch machines. I wrote a simple Ansible role that adds backports contrib repo, installs 'zfs-dkms', and creates zpools. This turned out to be somewhat more complex than I anticipated on, because I ended up in a loop:

• Installing 'zfs-dkms' from stretch-backports failed because I needed to modprobe zfs
• modprobe zfs failed because I was supposed to boot into a backports kernel
• Installing a backports kernel failed because I held broken packages ('zfs-dkms')

Some errors I got:

Setting up zfsutils-linux (0.7.12-2+deb10u1~bpo9+1) ...
Job for zfs-mount.service failed because the control process exited with error code.
See "systemctl status zfs-mount.service" and "journalctl -xe" for details.
invoke-rc.d: initscript zfs-mount, action "start" failed.
● zfs-mount.service - Mount ZFS filesystems
   Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2020-03-22 14:09:25 CET; 6ms ago
     Docs: man:zfs(8)
  Process: 32008 ExecStart=/sbin/zfs mount -a (code=exited, status=1/FAILURE)
 Main PID: 32008 (code=exited, status=1/FAILURE)

Mar 22 14:09:25 vlan5.backup.cyberfusion.space systemd[1]: Starting Mount ZFS filesystems...
Mar 22 14:09:25 vlan5.backup.cyberfusion.space zfs[32008]: The ZFS modules are not loaded.
Mar 22 14:09:25 vlan5.backup.cyberfusion.space zfs[32008]: Try running '/sbin/modprobe zfs' as root to load them.
Mar 22 14:09:25 vlan5.backup.cyberfusion.space systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Mar 22 14:09:25 vlan5.backup.cyberfusion.space systemd[1]: Failed to start Mount ZFS filesystems.
Mar 22 14:09:25 vlan5.backup.cyberfusion.space systemd[1]: zfs-mount.service: Unit entered failed state.
Mar 22 14:09:25 vlan5.backup.cyberfusion.space systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
dpkg: error processing package zfsutils-linux (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of zfs-zed:
 zfs-zed depends on zfsutils-linux (>= 0.7.12-2+deb10u1~bpo9+1); however:
  Package zfsutils-linux is not configured yet.

dpkg: error processing package zfs-zed (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 zfsutils-linux
 zfs-zed
E: Sub-process /usr/bin/dpkg returned an error code (1)

This meant that I had to change task order to first get a backports kernel, then install packages that depend on each other in another order:

• Installing and booting into backports kernel
• Install dkms instead of zfs-dkms
• Install zfs-dkms without recommends
• Then modprobe zfs

My Ansible code ended up looking like:

---
- name: Add repo for backports
  apt_repository:
    repo: 'deb http://debmirror.tuxis.nl/debian {{ ansible_distribution_release }}-backports main contrib non-free'
    state: present
    update_cache: yes

- name: Install kernel from backports
  apt:
    name: linux-image-amd64
    default_release: "{{ ansible_distribution_release }}-backports"
    state: present
  register: bpo_kernel_install

- name: Reboot for new kernel
  reboot:
  when: bpo_kernel_install.changed

- name: Install dkms
  apt: name=dkms state=present

- name: Install zfs-dkms
  apt: name=zfs-dkms state=present install_recommends=no default_release="{{ ansible_distribution_release }}-backports"

- name: Load kernel module
  modprobe: name=zfs state=present

- name: Install zfsutils-linux
  apt: name=zfsutils-linux state=present

I ran Ansible, and got ZFS on Linux running in a heartbeat!

root@vlan5:~# lsmod | grep zfs
zfs                  3522560  4
zunicode              331776  1 zfs
zavl                   16384  1 zfs
icp                   262144  1 zfs
zcommon                69632  1 zfs
znvpair                90112  2 zcommon,zfs
spl                   102400  4 znvpair,zcommon,zfs,icp
root@vlan5:~# zfs list
NAME       USED  AVAIL  REFER  MOUNTPOINT
urbackup  85.5K   984G    24K  /urbackup