Proxmox. Mount Ceph RBD image

Description

Mount Ceph RBD device (for VM disks and Containers)

! IMPORTANT !

Yes, Snapshots are supported.

BUT!

Snapshots work correctly ONLY if you create them WITH "Include RAM" CHECK. In this case you can create Snapshots and rollback to created Snapshot.

Snapshot WITH NO "Include RAM" CHECK rollback to previous Snapshot, that have RAM (or something like that, I don't understand how ot works, but it works wrong).

I have no idea why does it happen.

It happens because of "exclusive-lock enabled" maybe - https://ceph-users.ceph.narkive.com/roCNnlbG/snap-rollback-failed-with-exclusive-lock-enabled

Snapshots in Ceph don't rollback if VM is in running state - Error show up.

If VM is down - Snapshots in Ceph rollbacked correctly. But in Proxmox Snapshots still don't rollback correctly.

Do something with that!

Links

https://pve.proxmox.com/wiki/Storage:_RBD

Installation

Proxmox CLI

Get user admin auth Keyring from Ceph server

mkdir /etc/pve/priv/ceph

scp ceph01.local:/etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ceph-rbd01.keyring

And also copy files to /etc/ceph/. Otherwise RBD may not work correctly - you couldn't see RBD info at PVE serv in that case

scp ceph01.local:/etc/ceph/ceph.client.admin.keyring /etc/ceph/

scp ceph01.local:/etc/ceph/ceph.conf /etc/ceph/

Proxmox Web-interface

• Datacenter - Storage - Add RBD:
  • ID: rbd01 # This is only PVE's ID (just name). You don't need to use already created RBD Image at Ceph Cluster. You connect to only already created POOL (rbd01) at Ceph Cluster, but not to the already created RBD Image. You don't even should have already created RBD Images at Ceph Cluster at all.
  • Pool: rbd01 # Pool rbd01 should be already created at Ceph Cluster
  • Monitor(s): 192.168.199.121 192.168.199.122 192.168.199.123  # IPs of ceph nodes with MONITORs
  • User name: admin
  • Keyring: <key> # Get keyring on the Ceph Server: cat /etc/ceph/ceph.client.admin.keyring

Show RBD info. You can see it in the terminal of the Ceph Server AND in the PVE terminal

RBD info

RBD pools info

ceph osd lspools

1 .mgr
2 cephfs01_data
3 cephfs01_metadata
4 rbd01

rbd ls -p rbd01

...
vm-100-disk-0
...

Snapshots

RBD Snapshots with NO "Include RAM" Check. They DON'T rollback correctly!

rbd snap ls vm-100-disk-0 -p rbd01

SNAPID  NAME   SIZE   PROTECTED  TIMESTAMP
     4  clean  5 GiB             Mon Sep 12 12:03:18 2022

RBD Snapshots WITH "Include RAM" Check. They rollback correctly!

rbd ls -l rbd01

NAME                      SIZE     PARENT  FMT  PROT  LOCK
vm-100-disk-0               5 GiB            2        excl
vm-100-disk-0@clean         5 GiB            2
vm-100-disk-0@s456_ready    5 GiB            2
vm-100-state-clean        442 MiB            2
vm-100-state-s456_ready   443 MiB            2