Proxmox. Mount Ceph RBD image - История изменений

Cless в 20:10, 16 мая 2023

2023-05-16T20:10:54Z

← Предыдущая		Версия 20:10, 16 мая 2023
Строка 26:		Строка 26:
	** Monitor(s): '''192.168.199.121 192.168.199.122 192.168.199.123''' # IPs of ceph nodes with MONITORs		** Monitor(s): '''192.168.199.121 192.168.199.122 192.168.199.123''' # IPs of ceph nodes with MONITORs
	** User name: '''admin'''		** User name: '''admin'''
	**Keyring: '''<key>''' # Get keyring ~~(full content~~ of the file /etc/ceph/ceph.client.admin.keyring~~) on the Ceph Server:~~ <code>cat /etc/ceph/ceph.client.admin.keyring</code>		**Keyring: '''<key>''' # Get ceph admin keyring. On this '''PVE Server''' get '''FULL CONTENT''' of the file '''/etc/ceph/ceph.client.admin.keyring''' ('''<code>cat /etc/ceph/ceph.client.admin.keyring</code>''') and put it in this field.

	== Show RBD info. '''You can see it in the terminal of the Ceph Server AND in the PVE terminal''' ==		== Show RBD info. '''You can see it in the terminal of the Ceph Server AND in the PVE terminal''' ==

Cless в 12:56, 31 марта 2023

2023-03-31T12:56:33Z

← Предыдущая		Версия 12:56, 31 марта 2023
Строка 107:		Строка 107:

	= PVE Cluster and shared RBD storage for VMs =		= PVE Cluster and shared RBD storage for VMs =
	[[Proxmox. Кластер]]		Можно подключить RBD как Share хранилище внутри Ксастера PVE из нескольких нод.

			В этом случае будет работать миграция, аварийная миграция при выходе из строя одной из нод с ВМ, а также снэпшоты.

			Подробно: [[Proxmox. Кластер]]
	[[Категория:Proxmox]]		[[Категория:Proxmox]]
	[[Категория:Ceph]]		[[Категория:Ceph]]
	__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__		__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__

Cless в 11:26, 31 марта 2023

2023-03-31T11:26:22Z

← Предыдущая		Версия 11:26, 31 марта 2023
Строка 1:		Строка 1:
	== Description ==		= Description =
	Mount Ceph RBD device (for VM disks and Containers)		Mount Ceph RBD device (for VM disks and Containers)
	== Links ==		= Links =
	https://pve.proxmox.com/wiki/Storage:_RBD		https://pve.proxmox.com/wiki/Storage:_RBD

	== Installation ==		= Installation =

	=== Proxmox CLI ===		=== Proxmox CLI ===
Строка 105:		Строка 105:
	vm-100-state-clean 442 MiB 2		vm-100-state-clean 442 MiB 2
	vm-100-state-s456_ready 443 MiB 2		vm-100-state-s456_ready 443 MiB 2

			= PVE Cluster and shared RBD storage for VMs =
			[[Proxmox. Кластер]]
	[[Категория:Proxmox]]		[[Категория:Proxmox]]
	[[Категория:Ceph]]		[[Категория:Ceph]]
	__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__		__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__

Cless в 17:22, 9 января 2023

2023-01-09T17:22:51Z

← Предыдущая		Версия 17:22, 9 января 2023
Строка 1:		Строка 1:
	== Description ==		== Description ==
	Mount Ceph RBD device (for VM disks and Containers)~~<blockquote>'''! IMPORTANT !'''~~		Mount Ceph RBD device (for VM disks and Containers)

	~~Yes, Snapshots are supported.~~

	~~BUT!~~

	~~Snapshots work correctly ONLY if you create them '''WITH''' "'''Include RAM'''" CHECK. In this case you can create Snapshots and rollback to created Snapshot.~~

	~~Snapshot '''WITH NO''' "'''Include RAM'''" CHECK rollback to previous Snapshot, that have RAM (or something like that, I don't understand how ot works, but it works wrong).~~

	~~I have no idea why does it happen.~~

	~~It happens because of "'''exclusive-lock enabled'''" maybe - https://ceph-users.ceph.narkive.com/roCNnlbG/snap-rollback-failed-with-exclusive-lock-enabled~~

	~~Snapshots in Ceph don't rollback if VM is in running state - Error show up.~~

	~~If VM is down - Snapshots in Ceph rollbacked correctly. But in Proxmox Snapshots still don't rollback correctly.~~

	~~Do something with that!</blockquote>~~

	== Links ==		== Links ==
	https://pve.proxmox.com/wiki/Storage:_RBD		https://pve.proxmox.com/wiki/Storage:_RBD
Строка 63:		Строка 44:

	== Snapshots ==		== Snapshots ==
	=== RBD Snapshots '''with NO''' "'''Include RAM'''" Check. '''They DON'T rollback correctly'''! ===		<blockquote>'''IMPORTANT ISSUE WITH <u>PVE 6.x</u> ! (<u>PVE 7.3</u>+ SEEMS TO WORK JUST FINE AND DOESN'T HAVE SUCH ISSUES)'''

			Yes, Snapshots are supported.

			BUT!

			Snapshots work correctly ONLY if you create them '''WITH''' "'''Include RAM'''" CHECK. In this case you can create Snapshots and rollback to created Snapshot.

			Snapshot '''WITH NO''' "'''Include RAM'''" CHECK rollback to previous Snapshot, that have RAM (or something like that, I don't understand how ot works, but it works wrong).

			I have no idea why does it happen.

			It happens because of "'''exclusive-lock enabled'''" maybe - https://ceph-users.ceph.narkive.com/roCNnlbG/snap-rollback-failed-with-exclusive-lock-enabled

			Snapshots in Ceph don't rollback if VM is in running state - Error show up.

			If VM is down - Snapshots in Ceph rollbacked correctly. But in Proxmox Snapshots still don't rollback correctly.

			Do something with that!</blockquote>

			=== Proxmox VE 7.3+ (Snapshots work just fine) ===
			Create snapshots at the web-interface

			<code>Datacenter - pve01 - '''VMID''' - Snapshots - Take snapshot</code>

			Show snapshots

			<code>rbd ls -l rbd01</code>
			...
			NAME SIZE PARENT FMT PROT LOCK
			vm-100-disk-0 5 GiB 2 excl
			vm-100-disk-0@s1 5 GiB 2
			vm-100-disk-0@s2_ram 5 GiB 2
			vm-100-disk-0@s3 5 GiB 2
			vm-100-disk-0@s4 5 GiB 2
			vm-100-state-s2_ram 510 MiB 2
			...
			<code>rbd snap ls vm-100-disk-0 -p rbd01</code>
			...
			SNAPID NAME SIZE PROTECTED TIMESTAMP
			50 s1 5 GiB Mon Jan 9 19:56:46 2023
			51 s2_ram 5 GiB Mon Jan 9 19:58:25 2023
			52 s3 5 GiB Mon Jan 9 20:03:38 2023
			53 s4 5 GiB Mon Jan 9 20:04:05 2023
			...

			=== Proxmox VE 6.x (Snapshots doesn't work correctly!) ===

			==== RBD Snapshots '''with NO''' "'''Include RAM'''" Check. '''They DON'T rollback correctly'''! ====
	<code>rbd snap ls vm-100-disk-0 -p rbd01</code>		<code>rbd snap ls vm-100-disk-0 -p rbd01</code>
	SNAPID NAME SIZE PROTECTED TIMESTAMP		SNAPID NAME SIZE PROTECTED TIMESTAMP
	4 clean 5 GiB Mon Sep 12 12:03:18 2022		4 clean 5 GiB Mon Sep 12 12:03:18 2022

	=== RBD Snapshots '''WITH''' "'''Include RAM'''" Check. '''They rollback correctly'''! ===		==== RBD Snapshots '''WITH''' "'''Include RAM'''" Check. '''They rollback correctly'''! ====
	<code>rbd ls -l rbd01</code>		<code>rbd ls -l rbd01</code>
	NAME SIZE PARENT FMT PROT LOCK		NAME SIZE PARENT FMT PROT LOCK

Cless в 10:33, 14 сентября 2022

2022-09-14T10:33:23Z

← Предыдущая		Версия 10:33, 14 сентября 2022
Строка 45:		Строка 45:
	** Monitor(s): '''192.168.199.121 192.168.199.122 192.168.199.123''' # IPs of ceph nodes with MONITORs		** Monitor(s): '''192.168.199.121 192.168.199.122 192.168.199.123''' # IPs of ceph nodes with MONITORs
	** User name: '''admin'''		** User name: '''admin'''
	**Keyring: '''<key>''' # Get keyring on the Ceph Server: <code>cat /etc/ceph/ceph.client.admin.keyring</code>		**Keyring: '''<key>''' # Get keyring (full content of the file /etc/ceph/ceph.client.admin.keyring) on the Ceph Server: <code>cat /etc/ceph/ceph.client.admin.keyring</code>

	== Show RBD info. '''You can see it in the terminal of the Ceph Server AND in the PVE terminal''' ==		== Show RBD info. '''You can see it in the terminal of the Ceph Server AND in the PVE terminal''' ==

Cless в 10:32, 14 сентября 2022

2022-09-14T10:32:05Z

← Предыдущая		Версия 10:32, 14 сентября 2022
Строка 41:		Строка 41:

	* Datacenter - Storage - Add RBD:		* Datacenter - Storage - Add RBD:
	** ID: '''~~ceph-~~rbd01'''		** ID: '''rbd01''' # This is only PVE's ID (just name). You don't need to use already created RBD Image at Ceph Cluster. You connect to only already created POOL (rbd01) at Ceph Cluster, but not to the already created RBD Image. You don't even should have already created RBD Images at Ceph Cluster at all.
	** Pool: '''rbd01'''		** Pool: '''rbd01''' # Pool '''rbd01''' should be already created at Ceph Cluster
	** Monitor(s): '''192.168.199.121 192.168.199.122 192.168.199.123''' # IPs of ceph nodes with MONITORs		** Monitor(s): '''192.168.199.121 192.168.199.122 192.168.199.123''' # IPs of ceph nodes with MONITORs
	** User name: '''admin'''		** User name: '''admin'''
Строка 58:		Строка 58:
	4 rbd01		4 rbd01
	<code>rbd ls -p rbd01</code>		<code>rbd ls -p rbd01</code>
	~~ceph-rbd01~~		...
	vm-100-disk-0		vm-100-disk-0
			...

	== Snapshots ==		== Snapshots ==

Cless в 07:56, 14 сентября 2022

2022-09-14T07:56:48Z

← Предыдущая		Версия 07:56, 14 сентября 2022
Строка 6:		Строка 6:
	BUT!		BUT!

	Snapshots work correctly ONLY if you create them '''WITH''' "'''Include RAM'''" CHECK. In this case you can create and rollback to created Snapshot.		Snapshots work correctly ONLY if you create them '''WITH''' "'''Include RAM'''" CHECK. In this case you can create Snapshots and rollback to created Snapshot.

	Snapshot '''WITH NO''' "'''Include RAM'''" CHECK rollback to previous Snapshot, that have RAM.		Snapshot '''WITH NO''' "'''Include RAM'''" CHECK rollback to previous Snapshot, that have RAM (or something like that, I don't understand how ot works, but it works wrong).

	I have no idea why does it happen. ~~Maybe PVE or~~ Ceph ~~bug~~. Do something with that!</blockquote>		I have no idea why does it happen.

			It happens because of "'''exclusive-lock enabled'''" maybe - https://ceph-users.ceph.narkive.com/roCNnlbG/snap-rollback-failed-with-exclusive-lock-enabled

			Snapshots in Ceph don't rollback if VM is in running state - Error show up.

			If VM is down - Snapshots in Ceph rollbacked correctly. But in Proxmox Snapshots still don't rollback correctly.

			Do something with that!</blockquote>

	== Links ==		== Links ==

Cless в 05:28, 13 сентября 2022

2022-09-13T05:28:51Z

← Предыдущая		Версия 05:28, 13 сентября 2022
Строка 14:		Строка 14:
	== Links ==		== Links ==
	https://pve.proxmox.com/wiki/Storage:_RBD		https://pve.proxmox.com/wiki/Storage:_RBD

			== Installation ==

	=== Proxmox CLI ===		=== Proxmox CLI ===

Cless в 05:27, 13 сентября 2022

2022-09-13T05:27:11Z

← Предыдущая		Версия 05:27, 13 сентября 2022
Строка 1:		Строка 1:
	== Description ==		== Description ==
	Mount Ceph RBD device (for VM disks and Containers)		Mount Ceph RBD device (for VM disks and Containers)<blockquote>'''! IMPORTANT !'''

	Yes, Snapshots ~~snapshots~~ are supported		Yes, Snapshots are supported.

			BUT!

			Snapshots work correctly ONLY if you create them '''WITH''' "'''Include RAM'''" CHECK. In this case you can create and rollback to created Snapshot.

			Snapshot '''WITH NO''' "'''Include RAM'''" CHECK rollback to previous Snapshot, that have RAM.

			I have no idea why does it happen. Maybe PVE or Ceph bug. Do something with that!</blockquote>

	== Links ==		== Links ==
Строка 44:		Строка 52:

	== Snapshots ==		== Snapshots ==
	~~<blockquote>'''! IMPORTANT !'''~~		=== RBD Snapshots '''with NO''' "'''Include RAM'''" Check. '''They DON'T rollback correctly'''! ===

	Snapshots ~~work correctly ONLY if you create them WITH "~~'''~~Include RAM~~'''~~" CHECK. In this case you can create and rollback to created Snapshot.~~

	~~Snapshot WITH NO~~ "'''Include RAM'''" ~~CHECK during creation rollback to previous Snapshot, that have RAM.~~

	~~I have no idea why it happens. Maybe PVE or Ceph bug~~. ~~Do something with that!</blockquote>~~

	~~=== RBD Snapshots with NO "~~'''~~Include RAM'''" Check.~~ They DON'T rollback correctly ===
	<code>rbd snap ls vm-100-disk-0 -p rbd01</code>		<code>rbd snap ls vm-100-disk-0 -p rbd01</code>
	SNAPID NAME SIZE PROTECTED TIMESTAMP		SNAPID NAME SIZE PROTECTED TIMESTAMP
	4 clean 5 GiB Mon Sep 12 12:03:18 2022		4 clean 5 GiB Mon Sep 12 12:03:18 2022

	=== RBD Snapshots WITH "'''Include RAM'''" Check. They rollback correctly ===		=== RBD Snapshots '''WITH''' "'''Include RAM'''" Check. '''They rollback correctly'''! ===
	<code>rbd ls -l rbd01</code>		<code>rbd ls -l rbd01</code>
	NAME SIZE PARENT FMT PROT LOCK		NAME SIZE PARENT FMT PROT LOCK

Cless в 20:17, 12 сентября 2022

2022-09-12T20:17:10Z

← Предыдущая		Версия 20:17, 12 сентября 2022
Строка 13:		Строка 13:

	<code>scp ceph01.local:/etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ceph-rbd01.keyring</code>		<code>scp ceph01.local:/etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ceph-rbd01.keyring</code>

			And also copy files to '''/etc/ceph/'''. Otherwise RBD may not work correctly - you couldn't see RBD info at PVE serv in that case

			<code>scp ceph01.local:/etc/ceph/ceph.client.admin.keyring /etc/ceph/</code>

			<code>scp ceph01.local:/etc/ceph/ceph.conf /etc/ceph/</code>

	=== Proxmox Web-interface ===		=== Proxmox Web-interface ===
Строка 23:		Строка 29:
	**Keyring: '''<key>''' # Get keyring on the Ceph Server: <code>cat /etc/ceph/ceph.client.admin.keyring</code>		**Keyring: '''<key>''' # Get keyring on the Ceph Server: <code>cat /etc/ceph/ceph.client.admin.keyring</code>

	== Show RBD ~~and Snapshots Info~~. '''You can see it ~~only~~ in the terminal of the Ceph Server~~, not~~ in the PVE terminal''' ==		== Show RBD info. '''You can see it in the terminal of the Ceph Server AND in the PVE terminal''' ==

	=== RBD info ===		=== RBD info ===
Строка 37:		Строка 43:
	vm-100-disk-0		vm-100-disk-0

	=== ~~Optionally~~. RBD ~~Snap info~~ ===		== Snapshots ==
			<blockquote>'''! IMPORTANT !'''

			Snapshots work correctly ONLY if you create them WITH "'''Include RAM'''" CHECK. In this case you can create and rollback to created Snapshot.

			Snapshot WITH NO "'''Include RAM'''" CHECK during creation rollback to previous Snapshot, that have RAM.

			I have no idea why it happens. Maybe PVE or Ceph bug. Do something with that!</blockquote>

			=== RBD Snapshots with NO "'''Include RAM'''" Check. They DON'T rollback correctly ===
	<code>rbd snap ls vm-100-disk-0 -p rbd01</code>		<code>rbd snap ls vm-100-disk-0 -p rbd01</code>
	SNAPID NAME SIZE PROTECTED TIMESTAMP		SNAPID NAME SIZE PROTECTED TIMESTAMP
	4 clean 5 GiB Mon Sep 12 12:03:18 2022		4 clean 5 GiB Mon Sep 12 12:03:18 2022

			=== RBD Snapshots WITH "'''Include RAM'''" Check. They rollback correctly ===
			<code>rbd ls -l rbd01</code>
			NAME SIZE PARENT FMT PROT LOCK
			vm-100-disk-0 5 GiB 2 excl
			vm-100-disk-0@clean 5 GiB 2
			vm-100-disk-0@s456_ready 5 GiB 2
			vm-100-state-clean 442 MiB 2
			vm-100-state-s456_ready 443 MiB 2
	[[Категория:Proxmox]]		[[Категория:Proxmox]]
	[[Категория:Ceph]]		[[Категория:Ceph]]
	__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__		__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__