Description

Mount Ceph RBD device (for VM disks and Containers)

Links

https://pve.proxmox.com/wiki/Storage:_RBD

Installation

Proxmox CLI

Get user admin auth Keyring from Ceph server

mkdir /etc/pve/priv/ceph

scp ceph01.local:/etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ceph-rbd01.keyring

And also copy files to /etc/ceph/. Otherwise RBD may not work correctly - you couldn't see RBD info at PVE serv in that case

scp ceph01.local:/etc/ceph/ceph.client.admin.keyring /etc/ceph/

scp ceph01.local:/etc/ceph/ceph.conf /etc/ceph/

Proxmox Web-interface

Datacenter - Storage - Add RBD:
- ID: rbd01 # This is only PVE's ID (just name). You don't need to use already created RBD Image at Ceph Cluster. You connect to only already created POOL (rbd01) at Ceph Cluster, but not to the already created RBD Image. You don't even should have already created RBD Images at Ceph Cluster at all.
- Pool: rbd01 # Pool rbd01 should be already created at Ceph Cluster
- Monitor(s): 192.168.199.121 192.168.199.122 192.168.199.123 # IPs of ceph nodes with MONITORs
- User name: admin
- Keyring: <key> # Get keyring (full content of the file /etc/ceph/ceph.client.admin.keyring) on the Ceph Server: cat /etc/ceph/ceph.client.admin.keyring

Show RBD info. You can see it in the terminal of the Ceph Server AND in the PVE terminal

RBD info

RBD pools info

ceph osd lspools

1 .mgr
2 cephfs01_data
3 cephfs01_metadata
4 rbd01

rbd ls -p rbd01

...
vm-100-disk-0
...

Snapshots

IMPORTANT ISSUE WITH PVE 6.x ! (PVE 7.3+ SEEMS TO WORK JUST FINE AND DOESN'T HAVE SUCH ISSUES)
Yes, Snapshots are supported.
BUT!
Snapshots work correctly ONLY if you create them WITH "Include RAM" CHECK. In this case you can create Snapshots and rollback to created Snapshot.
Snapshot WITH NO "Include RAM" CHECK rollback to previous Snapshot, that have RAM (or something like that, I don't understand how ot works, but it works wrong).
I have no idea why does it happen.
It happens because of "exclusive-lock enabled" maybe - https://ceph-users.ceph.narkive.com/roCNnlbG/snap-rollback-failed-with-exclusive-lock-enabled
Snapshots in Ceph don't rollback if VM is in running state - Error show up.
If VM is down - Snapshots in Ceph rollbacked correctly. But in Proxmox Snapshots still don't rollback correctly.
Do something with that!

Proxmox VE 7.3+ (Snapshots work just fine)

Create snapshots at the web-interface

Datacenter - pve01 - VMID - Snapshots - Take snapshot

Show snapshots

rbd ls -l rbd01

...
NAME                  SIZE     PARENT  FMT  PROT  LOCK
vm-100-disk-0           5 GiB            2        excl
vm-100-disk-0@s1        5 GiB            2
vm-100-disk-0@s2_ram    5 GiB            2
vm-100-disk-0@s3        5 GiB            2
vm-100-disk-0@s4        5 GiB            2
vm-100-state-s2_ram   510 MiB            2
...

rbd snap ls vm-100-disk-0 -p rbd01

...
SNAPID  NAME    SIZE   PROTECTED  TIMESTAMP
    50  s1      5 GiB             Mon Jan  9 19:56:46 2023
    51  s2_ram  5 GiB             Mon Jan  9 19:58:25 2023
    52  s3      5 GiB             Mon Jan  9 20:03:38 2023
    53  s4      5 GiB             Mon Jan  9 20:04:05 2023
...

Proxmox VE 6.x (Snapshots doesn't work correctly!)

RBD Snapshots with NO "Include RAM" Check. They DON'T rollback correctly!

rbd snap ls vm-100-disk-0 -p rbd01

SNAPID  NAME   SIZE   PROTECTED  TIMESTAMP
     4  clean  5 GiB             Mon Sep 12 12:03:18 2022

RBD Snapshots WITH "Include RAM" Check. They rollback correctly!