Proxmox. Mount Ceph RBD image: различия между версиями

Версия 10:56, 14 сентября 2022

Description

Mount Ceph RBD device (for VM disks and Containers)

! IMPORTANT !
Yes, Snapshots are supported.
BUT!
Snapshots work correctly ONLY if you create them WITH "Include RAM" CHECK. In this case you can create Snapshots and rollback to created Snapshot.
Snapshot WITH NO "Include RAM" CHECK rollback to previous Snapshot, that have RAM (or something like that, I don't understand how ot works, but it works wrong).
I have no idea why does it happen.
It happens because of "exclusive-lock enabled" maybe - https://ceph-users.ceph.narkive.com/roCNnlbG/snap-rollback-failed-with-exclusive-lock-enabled
Snapshots in Ceph don't rollback if VM is in running state - Error show up.
If VM is down - Snapshots in Ceph rollbacked correctly. But in Proxmox Snapshots still don't rollback correctly.
Do something with that!

Links

https://pve.proxmox.com/wiki/Storage:_RBD

Installation

Proxmox CLI

Get user admin auth Keyring from Ceph server

mkdir /etc/pve/priv/ceph

scp ceph01.local:/etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ceph-rbd01.keyring

And also copy files to /etc/ceph/. Otherwise RBD may not work correctly - you couldn't see RBD info at PVE serv in that case

scp ceph01.local:/etc/ceph/ceph.client.admin.keyring /etc/ceph/

scp ceph01.local:/etc/ceph/ceph.conf /etc/ceph/

Proxmox Web-interface

Datacenter - Storage - Add RBD:
- ID: ceph-rbd01
- Pool: rbd01
- Monitor(s): 192.168.199.121 192.168.199.122 192.168.199.123 # IPs of ceph nodes with MONITORs
- User name: admin
- Keyring: <key> # Get keyring on the Ceph Server: cat /etc/ceph/ceph.client.admin.keyring

Show RBD info. You can see it in the terminal of the Ceph Server AND in the PVE terminal

RBD info

RBD pools info

ceph osd lspools

1 .mgr
2 cephfs01_data
3 cephfs01_metadata
4 rbd01

rbd ls -p rbd01

ceph-rbd01
vm-100-disk-0

Snapshots

RBD Snapshots with NO "Include RAM" Check. They DON'T rollback correctly!

rbd snap ls vm-100-disk-0 -p rbd01

SNAPID  NAME   SIZE   PROTECTED  TIMESTAMP
     4  clean  5 GiB             Mon Sep 12 12:03:18 2022

RBD Snapshots WITH "Include RAM" Check. They rollback correctly!

rbd ls -l rbd01

NAME                      SIZE     PARENT  FMT  PROT  LOCK
vm-100-disk-0               5 GiB            2        excl
vm-100-disk-0@clean         5 GiB            2
vm-100-disk-0@s456_ready    5 GiB            2
vm-100-state-clean        442 MiB            2
vm-100-state-s456_ready   443 MiB            2

@@ Строка 6: / Строка 6: @@
 BUT!
-Snapshots work correctly ONLY if you create them '''WITH''' "'''Include RAM'''" CHECK. In this case you can create and rollback to created Snapshot.
+Snapshots work correctly ONLY if you create them '''WITH''' "'''Include RAM'''" CHECK. In this case you can create Snapshots and rollback to created Snapshot.
-Snapshot '''WITH NO''' "'''Include RAM'''" CHECK rollback to previous Snapshot, that have RAM.
+Snapshot '''WITH NO''' "'''Include RAM'''" CHECK rollback to previous Snapshot, that have RAM (or something like that, I don't understand how ot works, but it works wrong).
-I have no idea why does it happen. Maybe PVE or Ceph bug. Do something with that!</blockquote>
+I have no idea why does it happen.
+It happens because of "'''exclusive-lock enabled'''" maybe - https://ceph-users.ceph.narkive.com/roCNnlbG/snap-rollback-failed-with-exclusive-lock-enabled
+Snapshots in Ceph don't rollback if VM is in running state - Error show up.
+If VM is down - Snapshots in Ceph rollbacked correctly. But in Proxmox Snapshots still don't rollback correctly.
+Do something with that!</blockquote>
 == Links ==