Links

https://vmik.net/2022/07/22/howto-ceph-2022/

https://kifarunix.com/install-and-setup-ceph-storage-cluster-on-ubuntu-2/

Ceph server

Pools create

Set neccessary flag

ceph fs flag set enable_multiple true

Create pools for cephfs

ceph osd pool create cephfs01_data

ceph osd pool create cephfs01_metadata

Change Replica Object quantity in pool

It is important that the number of replica objects does not exceed the number of hosts or you will have a WARNING

ceph osd pool set cephfs01_data size 3

ceph osd pool set cephfs01_metadata size 3

Optionally. Remove pools

Set the mon_allow_pool_delete config option to true before you can destroy a pool

ceph tell mon.\* injectargs '--mon-allow-pool-delete=true'

Destroy pools (yes, point pool's name twice in command)

ceph osd pool delete cephfs01_data cephfs01_data --yes-i-really-really-mean-it

ceph osd pool delete cephfs01_metadata cephfs01_metadata --yes-i-really-really-mean-it

Create custom cephfs "cephfs01"

Cephfs size will take all free space after creating (it's OK, size will be decreased after it will taken by other entities in this cluster (RDB, for example))

ceph fs new cephfs01 cephfs01_metadata cephfs01_data

Ceph fs list

ceph fs ls

    ...
    name: cephfs01, metadata pool: cephfs01_metadata, data pools: [cephfs01_data ]
    ...

Optionally .Remove cephfs (mds services at all nodes will automatically standby)

Down cephfs

ceph fs set cephfs01 down true

Status

ceph fs status

Remove the Ceph File System

ceph fs rm cephfs01 --yes-i-really-mean-it

Deploy mds services (at least 2, they are neccessary for cephfs)

ceph orch apply mds cephfs01 --placement="3 ceph01 ceph02 ceph03"

Ceph fs status

ceph fs status

...
data02 - 0 clients
======
RANK  STATE           MDS              ACTIVITY     DNS    INOS   DIRS   CAPS
0    active  data02.ceph01.pzqzpo  Reqs:    0 /s    10     13     12      0
    POOL         TYPE     USED  AVAIL
data02_metadata  metadata  96.0k  7488M
data02_data      data       0   7488M
    STANDBY MDS
data02.ceph03.lkjueq
data02.ceph02.izkpri
...

Optionally. Remove mds services

Show MDS services

ceph orch ls

        ...
        mds.cephfs01                      3/3  9m ago     21m  ceph01;ceph02;ceph03;count:3
        ...

Remove MDS services

ceph orch rm mds.cephfs01

Mount NOT ROOT cephfs, BUT subdirectory folder01 BY OWNER-USER user01

Create user01 key

ceph-authtool --create-keyring /etc/ceph/ceph.client.user01.keyring --gen-key -n client.user01

Add permissions to ceph.client.user01.keyring before import

nano /etc/ceph/ceph.client.user01.keyring

[client.user01]
        key = AQBhgxxjrFvTHBAAy1/aw12V8UAp2av4b2Ao8Q==
        caps mds = "allow rw path=/folder01,allow r path=/"
        caps mon = "allow r"
        caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=cephfs01_metadata,allow rwx pool=cephfs01_data"

Import key

ceph auth import -i /etc/ceph/ceph.client.user01.keyring

After import you can see users' perms by this way

ceph auth list

Optionally, after import you can modify MDS-services and OSD-pools permissions for user01 (to allow user01 at client RW mount)

The same permissions that was added by original ceph.client.user01.keyring. RW access to custom subfoder in root /, RO access to root / (mds 'allow rw path=/folder01,allow r path=/')

ceph auth caps client.user01 mon 'allow r' mds 'allow rw path=/folder01,allow r path=/' osd 'allow rwx pool=cephfs01_metadata,allow rwx pool=cephfs01_data'

- OR -

Permissions with full access to MDSs (mds 'allow rw')

ceph auth caps client.user01 mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs01_metadata,allow rwx pool=cephfs01_data'

Check that user01 permissions applied

ceph auth list

    ...
    client.user01
            key: AQDUWRxjtv0uJRAADmlV40/XdpDBtsAiCank/A==
            caps: [mds] allow rw
            caps: [mon] allow r
            caps: [osd] allow rwx pool=cephfs01_metadata,allow rwx pool=cephfs01_data
    ...

Copy user01 keyring to the other hosts in the cluster

rsync -avPh /etc/ceph/ceph.client.user01.keyring ceph02:/etc/ceph/

rsync -avPh /etc/ceph/ceph.client.user01.keyring ceph03:/etc/ceph/

Install util to local mount cephfs

dnf install -y ceph-fuse

Create folder for ROOT cephfs - "cephfs01"

mkdir /mnt/cephfs01

Mount cephfs (it will be mounted DEFAULT cephfs - "cepfs01")

ceph-fuse /mnt/cephfs01

Create custom dir

mkdir /mnt/cephfs01/folder01

Size quota for subdirectory (NOT NECCESSARY)

107300000000 - 100GB

cd /mnt/cephfs01/

setfattr -n ceph.quota.max_bytes -v 107300000000 folder01

Ceph client Linux

Install ceph-fuse

AlmaLinux 8

dnf install -y python39

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/quincy/src/cephadm/cephadm

chmod +x cephadm

./cephadm add-repo --release quincy

./cephadm install

dnf install -y ceph-common

dnf install -y ceph-fuse

CentOS 7

!!! ПОСЛЕДНЯЯ НАЙДЕННАЯ ВЕРСИЯ CEPH-FUSE ДЛЯ CENTOS7 - 12.2.5 !!!
!!! http://rpm.pbone.net/info_idpl_51224733_distro_centos_7_com_ceph-fuse-12.2.5-0.el7.x86_64.rpm.html !!!
!!! ЕСЛИ В КЛАСТЕРЕ CEPH УСТАНОВЛЕНА ВЕРСИЯ CEPH >=13, ТО CEPH-FUSE НА КЛИЕНТЕ CENTOS 7 ИСПОЛЬЗОВАТЬ НЕ ПОЛУЧИТСЯ !!!
wget ftp://ftp.pbone.net/mirror/vault.centos.org/7.5.1804/storage/x86_64/ceph-luminous/ceph-fuse-12.2.5-0.el7.x86_64.rpm
yum localinstall ceph-fuse-12.2.5-0.el7.x86_64.rpm

Mount cephfs manualy by terminal command

Copy server config to this client

scp root@192.168.199.121:/etc/ceph/ceph.conf /etc/ceph/

Case 1. Mount default root fs cephfs of ceph cluster

cd ~

ssh root@ceph01 'sudo ceph-authtool -p /etc/ceph/ceph.client.admin.keyring' > /root/ceph.key

chmod 600 ceph.key

mkdir -p /mnt/cephfs01

By domain names

mount -t ceph ceph01:6789:/ /mnt/cephfs01/ -o name=admin,fs=cephfs01,secretfile=/root/ceph.key

-OR-

mount -t ceph ceph01:6789,ceph02:6789,ceph03:6789:/ /mnt/cephfs01/ -o name=admin,fs=cephfs01,secretfile=/root/ceph.key

by IPs

mount -t ceph 192.168.199.121:6789:/ /mnt/cephfs01/ -o name=admin,fs=cephfs01,secretfile=/root/ceph.key

-OR-

mount -t ceph 192.168.199.121:6789,192.168.199.122:6789,192.168.199.123:6789:/ /mnt/cephfs01/ -o name=admin,fs=cephfs01,secretfile=/root/ceph.key

Case 2. Mount subdirectory fs folder01 of default fs cephfs of ceph cluster by user01

cd ~

ssh root@192.168.199.121 "cat /etc/ceph/ceph.client.user01.keyring" | awk '/key =/{print $3}' > /etc/ceph/ceph.client.user01.keyring

chmod 600 /etc/ceph/ceph.client.user01.keyring

mkdir /mnt/folder01

mount -t ceph 192.168.199.121:6789:/folder01 /mnt/folder01/ -o name=user01,fs=cephfs01,secretfile=/etc/ceph/ceph.client.user01.keyring

- OR -

mount -t ceph 192.168.199.121:6789,192.168.199.122:6789,192.168.199.123:6789:/folder01 /mnt/folder01/ -o name=user01,fs=cephfs01,secretfile=/etc/ceph/ceph.client.user01.keyring

Mount cephfs automatically using fstab

In this step, we will permanently mount CephFS with fstab. We will edit /etc/fstab and add the configuration for mounting CephFS with the kernel driver, using the mount command.

Case 1. Mount root fs cephfs01

mkdir -p /mnt/cephfs

cat ceph.client.admin.keyring

    ...
    AQAvoRlj44oJDhAApP3iMHMRBCdvh5Xr7346Jg==
    ...

COPY CONTENT OF ceph.client.admin.keyring INTO /etc/fstab AFTER "secret=" LIKE THIS:

nano /etc/fstab

...
192.168.199.121:6789,192.168.199.122:6789,192.168.199.123:6789:/ /mnt/cephfs ceph name=admin,fs=cephfs01,secret=AQAvoRlj44oJDhAApP3iMHMRBCdvh5Xr7346Jg==,_netdev,noatime 0 0

Case 2. Mount subdirectory folder01 of root fs cephfs

mkdir -p /mnt/folder01

cat /etc/ceph/ceph.client.user01.keyring

    ...
    AQBhgxxjrFvTHBAAy1/aw12V8UAp2av4b2Ao8Q==
    ...

COPY CONTENT OF ceph.client.user01.keyring INTO /etc/fstab AFTER "secret=" LIKE THIS:

nano /etc/fstab

...
192.168.199.121:6789,192.168.199.122:6789,192.168.199.123:6789:/folder01 /mnt/folder01 ceph name=user01,fs=cephfs01,secret=AQBhgxxjrFvTHBAAy1/aw12V8UAp2av4b2Ao8Q==,_netdev,noatime 0 0

Mount all disks from the fstab with the mount -a command, then list all available disks.

sudo mount -a

sudo df -hT

Finally, reboot the system and check the available disks to ensure that the configuration is reboot-safe.

reboot

sudo df -hT

You will see that CephFS has been mounted automatically with fstab.

Snapshots CephFS

For creating snapshot you need to enter to a mounted BY ADMIN CephFS at any host

Create snapshots

You can create snapshot for any subfolder in mounted CephFS

For example:

Enter to the any subfolder in CephFS

cd /mnt/cephfs/folder01/test/

Create file

echo 123 > 1.txt

Create snapshot (yeah, exactly via mkdir. I know, it's weird, but it works exactly this way)

In this example we create snapshot with current time, bot you can use ANY name for snapshot, for example "snapshot1": mkdir snapshot1

mkdir `date +%Y-%m-%d_%H-%M-%S`

Now you can enter to new created snapshot folder, or just list its content

ls -lh .snap/2022-10-29_10-14-03/

total 512
-rw-r--r-- 1 root root 3 Oct 29 10:12 1.txt

You CAN'T list ".snap" directory from shell. CephFS clients present a hidden .snap directory that is used to access, create, delete, and rename snapshots. Although this directory is excluded from the readdir() system call, any process that tries to create a file or directory with the same name returns an error. The name of this hidden directory can be changed at mount time with the -o snapdirname=.<new_name> option or by using the client_snapdir configuration option.

Create another file

echo 123 > 2.txt

Create another snapshot

mkdir `date +%Y-%m-%d_%H-%M-%S`

And another file...

echo 123 > 3.txt

And another snapshot

mkdir `date +%Y-%m-%d_%H-%M-%S`

Now you can list all snapshots in .snap directory

ls -lh .snap/

...
total 0
drwxr-xr-x 2 root root 1 Oct 29 10:12 2022-10-29_10-14-03
drwxr-xr-x 2 root root 2 Oct 29 10:14 2022-10-29_10-14-31
drwxr-xr-x 2 root root 3 Oct 29 10:14 2022-10-29_10-14-43
...

Or you can list all files in the snapshot

ls -lh .snap/2022-10-29_10-14-03/

...
total 512
-rw-r--r-- 1 root root 3 Oct 29 10:12 1.txt
...

ls -lh .snap/2022-10-29_10-14-31/

...
total 1.0K
-rw-r--r-- 1 root root 3 Oct 29 10:12 1.txt
-rw-r--r-- 1 root root 4 Oct 29 10:14 2.txt
...

ls -lh .snap/2022-10-29_10-14-43/

...
total 1.5K
-rw-r--r-- 1 root root 3 Oct 29 10:12 1.txt
-rw-r--r-- 1 root root 4 Oct 29 10:14 2.txt
-rw-r--r-- 1 root root 4 Oct 29 10:14 3.txt
...

List all snapshots

You can use custom script to list all snapshots in the CephFS root directory of in the any subfolder

nano /tmp/ss.sh

#!/bin/bash
#$1 = CephFS directory

find $1 -type d | while read i; do
  ls --color=none $i/.snap | while read j; do
    echo $i/.snap/$j
  done
done

Run

/tmp/ss.sh "/mnt/cephfs/"

...
/mnt/cephfs/folder01/test01/.snap/2022-10-29_10-46-34
/mnt/cephfs/folder01/test01/.snap/2022-10-29_10-46-37
...

Delete snapshots

For deleting snapshot you need to remove snapshot directory "/mnt/cephfs/folder01/test/.snap/2022-10-29_10-14-43"

Restore snapshots

For restoring snapshot you just need to copy all contents from "/mnt/cephfs/folder01/test/.snap/2022-10-29_10-14-43/" to its original path " /mnt/cephfs/folder01/test/"

Ceph client Windows (BETA) ??? CAN'T MAKE IT WORK WITH CEPH 17. MAYBE LATER ???

Links

https://docs.ceph.com/en/latest/rbd/rbd-windows/

Download and install

###https://www.catalog.update.microsoft.com/search.aspx?q=kb4474419

Download and install

https://github.com/dokan-dev/dokany/releases/download/v2.0.5.1000/DokanSetup.exe

https://github.com/dokan-dev/dokany/releases/download/v2.0.5.1000/Dokan_x64.msi

Download "ceph_v16_0_0_beta.msi"

https://cloudbase.it/ceph-for-windows/

Install "ceph_v16_0_0_beta.msi"

INSERT CONTENT OF /etc/ceph/ceph.client.admin.keyring AT CEPH MONITOR IN FILE:

C:\ProgramData\Ceph\ceph.client.admin.keyring

[client.admin]
        key = AQA/74NgZc1oGxAAMJWuBRO7o22DFDOq+5omqg==
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"

C:\ProgramData\Ceph\ceph.conf

[global]
keyring = C:\\ProgramData\\Ceph\\ceph.client.admin.keyring
[mon]
[mon.ceph01]
    mon addr = 192.168.199.121:6789

Mount cephfs <cephfs_name> to disk X:\

C:\Program Files\Ceph\bin\ceph-dokan.exe -l x --client_fs <cephfs_name>

Ceph. AlmaLinux 8. Ceph 17 Quincy. CephFS Create and Mount at Client

Содержание

Links

Ceph server

Pools create

Optionally. Remove pools

Create custom cephfs "cephfs01"

Optionally .Remove cephfs (mds services at all nodes will automatically standby)

Deploy mds services (at least 2, they are neccessary for cephfs)

Optionally. Remove mds services

Mount NOT ROOT cephfs, BUT subdirectory folder01 BY OWNER-USER user01

Ceph client Linux

Install ceph-fuse

AlmaLinux 8

CentOS 7

Mount cephfs manualy by terminal command

Case 1. Mount default root fs cephfs of ceph cluster

Case 2. Mount subdirectory fs folder01 of default fs cephfs of ceph cluster by user01

Mount cephfs automatically using fstab

Case 1. Mount root fs cephfs01

Case 2. Mount subdirectory folder01 of root fs cephfs

Snapshots CephFS

Create snapshots

List all snapshots

Delete snapshots

Restore snapshots

Ceph client Windows (BETA) ??? CAN'T MAKE IT WORK WITH CEPH 17. MAYBE LATER ???

Links

Навигация

Поиск