Proxmox. ZFS-over-iSCSI network shared storage. PVE Cluster

Description

Proxmox. ZFS over iSCSI as a network shared storage.

In this example we will attach ZFS over iSCSI to PVE Cluster, but you can freely attach it to the only one PVE node.

Configuring

Server Linux iSCSI-target

https://www.server-world.info/en/note?os=CentOS_8&p=iscsi&f=1

For example, we will use Almalinux 8 as an iscsi-target server with ZFS pool

First off, install ZFS at Almalinux 8 server: ZFS. Linux. Almalinux 8. ZFS installation

Create ZFS-pool d1, which we will use as a storage for VM disks

zpool create -m /mnt/d1 d1 /dev/sdb

Install targetcli for iSCSI target management

dnf -y install targetcli

Run

targetcli

Create target (it's not an image, just target entity)

/> cd /iscsi

/> create iqn.2023-04.al8-test.iscsi:target01

LUNs (here you can create an iscsi-image, but now nothing you can do here, just sightseing)

cd /iscsi/iqn.2023-04.al8-test.iscsi:target01/tpg1/luns

Create rights for initiators (PVE nodes) to let them connect to the iscsi-target

/> cd /iscsi/iqn.2023-04.al8-test.iscsi:target01/tpg1/acls

/> create iqn.2023-04.al8-test.iscsi:pve01-test.initiator01

/> create iqn.2023-04.al8-test.iscsi:pve02-test.initiator02

/> exit

Service enable and start

systemctl enable --now target

netstat -luptn

Check if the service is running and listening on the port 3260

    ...
    LISTEN   0         256                 0.0.0.0:3260             0.0.0.0:*
    ...

Show iscsi target tree

targetcli ls

Client PVE iSCSI-initiators (PVE nodes)

192.168.199.173 - iSCSI-target server

We have to create RSA keys at PVE nodes (and add them to iSCSI-target) to let them connect to iSCSI-target (without a password) for creating ZVOL for each virtual disk in Proxmpx VM.

CLI

pve01

nano /etc/iscsi/initiatorname.iscsi

    ...
    iqn.2023-04.al8-test.iscsi:pve01-test.initiator01
    ...

systemctl restart iscsid.service open-iscsi.service

systemctl status iscsid.service open-iscsi.service

ssh-keygen -f /etc/pve/priv/zfs/192.168.199.173_id_rsa

ssh-copy-id -i /etc/pve/priv/zfs/192.168.199.173_id_rsa.pub root@192.168.199.173

pve02

nano /etc/iscsi/initiatorname.iscsi

    ...
    iqn.2023-04.al8-test.iscsi:pve02-test.initiator02
    ...    

systemctl restart iscsid.service open-iscsi.service

systemctl status iscsid.service open-iscsi.service

pve01 and pve02

nano ~/.ssh/config (if this file doesn't exist, just create it):

   -OR-

nano /etc/ssh/ssh_config

...
StrictHostKeyChecking no

WEB

ID: iscsi-target-linux

Portal: 192.168.199.173

Pool: d1

Block size: 8k

Target: iqn.2023-04.al8-test.iscsi:target01

iSCSI-Provider: LIO

Thin provision: V

Target portal group: tpg1

Now you can create VM with RAW disk at iscsi-target-linux storage

Server Linux iSCSI-target

After VM creating at PVE Cluster you will see ZFS-volumes at iSCSI-target server. They are VM disks.

zfs list

...
d1/vm-102-disk-0        16.3G  19.9G     8.02G  -
...

Show iscsi target tree

targetcli ls