How to create ceph on Centos 8 Stream via Ceph Ansible

I assume, that we have working Centos 8 Stream system. So, in this example, we will have three systems (nodes), with identical HW resources (4 GB ram, 4 vCPU, two NICs – one internal for ceph and one for world, and 10 TB spin-up hdd). In this article, every command must be run on all nodes. Public network is 192.168.1.0/24 and Ceph separate network is 192.168.2.0/24

Setting up time

As the first step, we must set up a time, I use chrony:

dnf install chrony -y
systemctl enable chronyd
timedatectl set-timezone Europe/Bratislava
timedatectl

Now, edit some variables in configurations file for chronyd. Add some servers from pool, and edit local subnets, where we delived time:

vim /etc/chrony.conf

pool 2.centos.pool.ntp.org iburst
pool 1.centos.pool.ntp.org iburst
pool 3.centos.pool.ntp.org iburst

Now start/restart our service, and check, if it is working:

systemctl restart chronyd
systemctl status chronyd.service
chronyc sources

Create hostnames, ssh rsa-keys, update and install som packages

Now, we must edit on all nodes our hostnames, set it permanent:

hostnamectl set-hostname ceph1

Now, add all hostnames, and IPs to file /etc/hosts:

tee -a /etc/hosts<<EOF
192.168.1.1    ceph1
192.168.1.2    ceph2
192.168.1.3    ceph3
192.168.2.1    ceph1-cluster
192.168.2.2    ceph2-cluster
192.168.3.3    ceph3-cluster

EOF

Now, create rsa-key pair, for password-less connect to and from each node:

ssh-keygen -t rsa -b 4096 -C "ceph1"

means:
-b bits. Number of bits in the key to create
-t type. Specify type of key to create
-C comment

And copy it to other nodes:

for host in ceph1 ceph2 ceph3; do
 ssh-copy-id root@$host
done

Now update and install packages:

dnf update -y
-reboot-
dnf install git vim bash-completion python3-pip

Preparing for ceph

Now, install epel repository and enable powertools:

dnf -y install dnf-plugins-core
dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
dnf config-manager --set-enabled powertools

dnf repolist
repo id            repo name
appstream          CentOS Stream 8 - AppStream
epel               Extra Packages for Enterprise Linux 8 - x86_64
epel-modular       Extra Packages for Enterprise Linux Modular 8 - x86_64
epel-next          Extra Packages for Enterprise Linux 8 - Next - x86_64
extras             CentOS Stream 8 - Extras
powertools         CentOS Stream 8 - PowerTools

Clone Ceph Ansible repository:

cd /root/
git clone https://github.com/ceph/ceph-ansible.git

Choose ceph-ansible branch you wish to use. The command Syntax is: git checkout $branch

I’ll switch to stable-5.0 which supports Ceph octopus version.

cd ceph-ansible
git checkout stable-5.0

pip3 install setuptools-rust
pip3 install wheel
export CRYPTOGRAPHY_DONT_BUILD_RUST=1
pip3 install --upgrade pip

pip3 install -r requirements.txt
echo "PATH=\$PATH:/usr/local/bin" >>~/.bashrc
source ~/.bashrc

Confirm Ansible version installed.

ansible --version
ansible 2.9.26
  config file = /root/ceph-ansible/ansible.cfg
  configured module search path = ['/root/ceph-ansible/library']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.8 (default, Sep 10 2021, 09:13:53) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]

Now, we find, which OSD (spin-up disks) are ready for us. In each my node, there is free disk /dev/sda. Look via lsblk:

lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                       8:0    0 10.7T  0 disk 
sr0                      11:0    1 1024M  0 rom  
vda                     252:0    0   32G  0 disk 
├─vda1                  252:1    0  512M  0 part /boot
└─vda2                  252:2    0 31.5G  0 part 
  ├─centos--vg0-root    253:0    0    3G  0 lvm  /
  ├─centos--vg0-swap    253:1    0    1G  0 lvm  [SWAP]
  ├─centos--vg0-tmp     253:2    0  512M  0 lvm  /tmp
  ├─centos--vg0-var_log 253:3    0  512M  0 lvm  /var/log
  ├─centos--vg0-var     253:4    0    3G  0 lvm  /var
  └─centos--vg0-home    253:5    0    2G  0 lvm  /home

Now, we are ready for installation of Ceph

Deploy Ceph Octopus (15) Cluster on CentOS 8 stream

Now, we are play some things 🙂 The first node (ceph1) I use as admin node for installation. Configure Ansible Inventory and Playbook files. Create Ceph Cluster group variables file on the admin node

cd /root/ceph-ansible
cp group_vars/all.yml.sample  group_vars/all.yml
vim group_vars/all.yml

And edit some variables of your Ceph cluster, as you fit:

#General
cluster: ceph

# Inventory host group variables
mon_group_name: mons
osd_group_name: osds
rgw_group_name: rgws
mds_group_name: mdss
nfs_group_name: nfss
rbdmirror_group_name: rbdmirrors
client_group_name: clients
iscsi_gw_group_name: iscsigws
mgr_group_name: mgrs
rgwloadbalancer_group_name: rgwloadbalancers
grafana_server_group_name: grafana-server

# Firewalld / NTP
configure_firewall: True
ntp_service_enabled: true
ntp_daemon_type: chronyd

# Ceph packages
ceph_origin: repository
ceph_repository: community
ceph_repository_type: cdn
ceph_stable_release: octopus

# Interface options
monitor_interface: ens18
radosgw_interface: ens18
public_network: 192.168.1.0/24
cluster_network: 192.168.2.0/24


# DASHBOARD
dashboard_enabled: True
dashboard_protocol: http
dashboard_admin_user: admin
dashboard_admin_password: strongpass

grafana_admin_user: admin
grafana_admin_password: strongpass

Now, set your OSDs. Create a new ceph nodes ansible inventory. Properly set your inventory file. Below is my inventory. Modify inventory groups the way you want services installed in your cluster nodes.

vim hosts

# Ceph admin user for SSH and Sudo
[all:vars]
ansible_ssh_user=root
ansible_become=true
ansible_become_method=sudo
ansible_become_user=root

# Ceph Monitor Nodes
[mons]
ceph1
ceph2
ceph3

# MDS Nodes
[mdss]
ceph1
ceph2
ceph3

# RGW
[rgws]
ceph1
ceph2
ceph3

# Manager Daemon Nodes
[mgrs]
ceph1
ceph2
ceph3

# set OSD (Object Storage Daemon) Node
[osds]
ceph1
ceph2
ceph3

# Grafana server
[grafana-server]
ceph1

Create Playbook file by copying a sample playbook at the root of the ceph-ansible project called site.yml.sample.

cp site.yml.sample site.yml 

Run Playbook.

ansible-playbook -i hosts site.yml 

If installation was successful, a health check should return OK or minimal WARN.

# ceph -s
  cluster:
    id:     dcfd26f5-49e9-4256-86c2-a5a0deac7b54
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim
 
  services:
    mon: 3 daemons, quorum eu-ceph1,eu-ceph2,eu-ceph3 (age 67m)
    mgr: ceph2(active, since 55m), standbys: ceph3, ceph1
    mds: cephfs:1 {0=ceph1=up:active} 2 up:standby
    osd: 3 osds: 3 up (since 60m), 3 in (since 60m)
    rgw: 3 daemons active (ceph1.rgw0, ceph2.rgw0, ceph3.rgw0)
 
  task status:
 
  data:
    pools:   7 pools, 169 pgs
    objects: 215 objects, 11 KiB
    usage:   3.1 GiB used, 32 TiB / 32 TiB avail
    pgs:     169 active+clean

This is a screenshot of my installation output once it has been completed.

As you see, I have warning: mons are allowing insecure global_id reclaim

So, silent it, as you fit it, or fix…

ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false
Total Page Visits: 767 - Today Page Visits: 4