{"id":802,"date":"2021-09-22T13:59:47","date_gmt":"2021-09-22T11:59:47","guid":{"rendered":"https:\/\/www.gonscak.sk\/?p=802"},"modified":"2022-04-19T10:20:14","modified_gmt":"2022-04-19T08:20:14","slug":"how-to-create-ceph-on-centos-8-stream","status":"publish","type":"post","link":"https:\/\/www.gonscak.sk\/?p=802","title":{"rendered":"How to create ceph on Centos 8 Stream via Ceph Ansible"},"content":{"rendered":"\n

I assume, that we have working Centos 8 Stream system. So, in this example, we will have three systems (nodes), with identical HW resources (4 GB ram, 4 vCPU, two NICs – one internal for ceph and one for world, and 10 TB spin-up hdd). In this article, every command must be run on all nodes. Public network is 192.168.1.0\/24 and Ceph separate network is 192.168.2.0\/24<\/p>\n\n\n\n

Setting up time<\/h2>\n\n\n\n

As the first step, we must set up a time, I use chrony:<\/p>\n\n\n\n

dnf install chrony -y\nsystemctl enable chronyd\ntimedatectl set-timezone Europe\/Bratislava\ntimedatectl<\/pre>\n\n\n\n
Now, edit some variables in configurations file for chronyd. Add some servers from pool, and edit local subnets, where we delived time:<\/p>\n\n\n\n
vim \/etc\/chrony.conf\n\npool 2.centos.pool.ntp.org iburst\npool 1.centos.pool.ntp.org iburst\npool 3.centos.pool.ntp.org iburst<\/pre>\n\n\n\n
Now start\/restart our service, and check, if it is working:<\/p>\n\n\n\n
systemctl restart chronyd\nsystemctl status chronyd.service\nchronyc sources<\/pre>\n\n\n\n
Create hostnames, ssh rsa-keys, update and install som packages<\/h2>\n\n\n\n
Now, we must edit on all nodes our hostnames, set it permanent:<\/p>\n\n\n\n
hostnamectl set-hostname ceph1<\/pre>\n\n\n\n
Now, add all hostnames, and IPs to file \/etc\/hosts:<\/p>\n\n\n\n
tee -a \/etc\/hosts<<EOF\n192.168.1.1    ceph1\n192.168.1.2    ceph2\n192.168.1.3    ceph3\n192.168.2.1    ceph1-cluster\n192.168.2.2    ceph2-cluster\n192.168.3.3    ceph3-cluster\n\nEOF<\/pre>\n\n\n\n
Now, create rsa-key pair, for password-less connect to and from each node:<\/p>\n\n\n\n
ssh-keygen -t rsa -b 4096 -C \"ceph1\"\n\nmeans:\n-b bits. Number of bits in the key to create\n-t type. Specify type of key to create\n-C comment<\/pre>\n\n\n\n
And copy it to other nodes:<\/p>\n\n\n\n
for host in ceph1 ceph2 ceph3; do\n ssh-copy-id root@$host\ndone<\/pre>\n\n\n\n
Now update and install packages:<\/p>\n\n\n\n
dnf update -y\n-reboot-\ndnf install git vim bash-completion python3-pip<\/pre>\n\n\n\n
Preparing for ceph<\/h2>\n\n\n\n
Now, install epel repository and enable powertools:<\/p>\n\n\n\n
dnf -y install dnf-plugins-core\ndnf -y install https:\/\/dl.fedoraproject.org\/pub\/epel\/epel-release-latest-8.noarch.rpm\ndnf config-manager --set-enabled powertools\n\ndnf repolist\nrepo id            repo name\nappstream          CentOS Stream 8 - AppStream\nepel               Extra Packages for Enterprise Linux 8 - x86_64\nepel-modular       Extra Packages for Enterprise Linux Modular 8 - x86_64\nepel-next          Extra Packages for Enterprise Linux 8 - Next - x86_64\nextras             CentOS Stream 8 - Extras\npowertools         CentOS Stream 8 - PowerTools<\/pre>\n\n\n\n
Clone Ceph Ansible repository:<\/p>\n\n\n\n
cd \/root\/\ngit clone https:\/\/github.com\/ceph\/ceph-ansible.git<\/pre>\n\n\n\n
Choose ceph-ansible branch you wish to use. The command Syntax is: git checkout $branch<\/p>\n\n\n\n
I\u2019ll switch to stable-5.0<\/em> which supports Ceph octopus<\/strong> version.<\/p>\n\n\n\n
cd ceph-ansible\ngit checkout stable-5.0\n\npip3 install setuptools-rust\npip3 install wheel\nexport CRYPTOGRAPHY_DONT_BUILD_RUST=1\npip3 install --upgrade pip\n\npip3 install -r requirements.txt\necho \"PATH=\\$PATH:\/usr\/local\/bin\" >>~\/.bashrc\nsource ~\/.bashrc<\/pre>\n\n\n\n
Confirm Ansible version installed.<\/p>\n\n\n\n
ansible --version\nansible 2.9.26\n  config file = \/root\/ceph-ansible\/ansible.cfg\n  configured module search path = ['\/root\/ceph-ansible\/library']\n  ansible python module location = \/usr\/local\/lib\/python3.6\/site-packages\/ansible\n  executable location = \/usr\/local\/bin\/ansible\n  python version = 3.6.8 (default, Sep 10 2021, 09:13:53) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]<\/pre>\n\n\n\n
Now, we find, which OSD (spin-up disks) are ready for us. In each my node, there is free disk \/dev\/sda. Look via lsblk:<\/p>\n\n\n\n
lsblk\nNAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT\nsda<\/strong>                       8:0    0 10.7T  0 disk <\/strong>\nsr0                      11:0    1 1024M  0 rom  \nvda                     252:0    0   32G  0 disk \n\u251c\u2500vda1                  252:1    0  512M  0 part \/boot\n\u2514\u2500vda2                  252:2    0 31.5G  0 part \n  \u251c\u2500centos--vg0-root    253:0    0    3G  0 lvm  \/\n  \u251c\u2500centos--vg0-swap    253:1    0    1G  0 lvm  [SWAP]\n  \u251c\u2500centos--vg0-tmp     253:2    0  512M  0 lvm  \/tmp\n  \u251c\u2500centos--vg0-var_log 253:3    0  512M  0 lvm  \/var\/log\n  \u251c\u2500centos--vg0-var     253:4    0    3G  0 lvm  \/var\n  \u2514\u2500centos--vg0-home    253:5    0    2G  0 lvm  \/home\n<\/pre>\n\n\n\n
Now, we are ready for installation of Ceph<\/p>\n\n\n\n
Deploy Ceph Octopus (15) Cluster on CentOS 8 stream<\/h2>\n\n\n\n
Now, we are play some things \ud83d\ude42 The first node (ceph1) I use as admin node for installation. Configure Ansible Inventory and Playbook files. Create Ceph Cluster group variables file on the admin node<\/p>\n\n\n\n
cd \/root\/ceph-ansible\ncp group_vars\/all.yml.sample  group_vars\/all.yml\nvim group_vars\/all.yml<\/pre>\n\n\n\n
And edit some variables of your Ceph cluster, as you fit:<\/p>\n\n\n\n
#General\ncluster: ceph\n\n# Inventory host group variables\nmon_group_name: mons\nosd_group_name: osds\nrgw_group_name: rgws\nmds_group_name: mdss\nnfs_group_name: nfss\nrbdmirror_group_name: rbdmirrors\nclient_group_name: clients\niscsi_gw_group_name: iscsigws\nmgr_group_name: mgrs\nrgwloadbalancer_group_name: rgwloadbalancers\ngrafana_server_group_name: grafana-server\n\n# Firewalld \/ NTP\nconfigure_firewall: True\nntp_service_enabled: true\nntp_daemon_type: chronyd\n\n# Ceph packages\nceph_origin: repository\nceph_repository: community\nceph_repository_type: cdn\nceph_stable_release: octopus\n\n# Interface options\nmonitor_interface: ens18\nradosgw_interface: ens18\npublic_network: 192.168.1.0\/24\ncluster_network: 192.168.2.0\/24\n\n\n# DASHBOARD\ndashboard_enabled: True\ndashboard_protocol: http\ndashboard_admin_user: admin\ndashboard_admin_password: strongpass\n\ngrafana_admin_user: admin\ngrafana_admin_password: strongpass<\/pre>\n\n\n\n
Now, set your OSDs. Create a new ceph nodes ansible inventory. Properly set your inventory file. Below is my inventory. Modify inventory groups the way you want services installed in your cluster nodes.<\/p>\n\n\n\n
vim hosts\n\n# Ceph admin user for SSH and Sudo\n[all:vars]\nansible_ssh_user=root\nansible_become=true\nansible_become_method=sudo\nansible_become_user=root\n\n# Ceph Monitor Nodes\n[mons]\nceph1\nceph2\nceph3\n\n# MDS Nodes\n[mdss]\nceph1\nceph2\nceph3\n\n# RGW\n[rgws]\nceph1\nceph2\nceph3\n\n# Manager Daemon Nodes\n[mgrs]\nceph1\nceph2\nceph3\n\n# set OSD (Object Storage Daemon) Node\n[osds]\nceph1\nceph2\nceph3\n\n# Grafana server\n[grafana-server]\nceph1\n<\/pre>\n\n\n\n
Create Playbook file by copying a sample playbook at the root of the ceph-ansible<\/em> project called site.yml.sample<\/em>.<\/p>\n\n\n\n
cp site.yml.sample site.yml <\/pre>\n\n\n\n
Run Playbook.<\/p>\n\n\n\n
ansible-playbook -i hosts site.yml <\/pre>\n\n\n\n
If installation was successful, a health check should return OK or minimal WARN.<\/p>\n\n\n\n
# ceph -s\n  cluster:\n    id:     dcfd26f5-49e9-4256-86c2-a5a0deac7b54\n    health: HEALTH_WARN\n            mons are allowing insecure global_id reclaim\n \n  services:\n    mon: 3 daemons, quorum eu-ceph1,eu-ceph2,eu-ceph3 (age 67m)\n    mgr: ceph2(active, since 55m), standbys: ceph3, ceph1\n    mds: cephfs:1 {0=ceph1=up:active} 2 up:standby\n    osd: 3 osds: 3 up (since 60m), 3 in (since 60m)\n    rgw: 3 daemons active (ceph1.rgw0, ceph2.rgw0, ceph3.rgw0)\n \n  task status:\n \n  data:\n    pools:   7 pools, 169 pgs\n    objects: 215 objects, 11 KiB\n    usage:   3.1 GiB used, 32 TiB \/ 32 TiB avail\n    pgs:     169 active+clean\n<\/pre>\n\n\n\n
This is a screenshot of my installation output once it has been completed.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
As you see, I have warning: mons are allowing insecure global_id reclaim<\/p>\n\n\n\n
So, silent it, as you fit it, or fix…<\/p>\n\n\n\n
ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false<\/pre>\n ","protected":false},"excerpt":{"rendered":"
I assume, that we have working Centos 8 Stream system. So, in this example, we will have three systems (nodes), with identical HW resources (4 GB ram, 4 vCPU, two NICs – one internal for ceph and one for world, and 10 TB spin-up hdd). In this article, every command must be run on all … Continue reading How to create ceph on Centos 8 Stream via Ceph Ansible<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[116,139,8],"class_list":["post-802","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-centos-8","tag-ceph","tag-linux"],"_links":{"self":[{"href":"https:\/\/www.gonscak.sk\/index.php?rest_route=\/wp\/v2\/posts\/802","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.gonscak.sk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gonscak.sk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gonscak.sk\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gonscak.sk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=802"}],"version-history":[{"count":12,"href":"https:\/\/www.gonscak.sk\/index.php?rest_route=\/wp\/v2\/posts\/802\/revisions"}],"predecessor-version":[{"id":874,"href":"https:\/\/www.gonscak.sk\/index.php?rest_route=\/wp\/v2\/posts\/802\/revisions\/874"}],"wp:attachment":[{"href":"https:\/\/www.gonscak.sk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=802"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gonscak.sk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=802"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gonscak.sk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=802"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}