Configuring a remote logging solution – rsyslog

https://unixcop.com/how-to-install-syslog-server-and-client-centos8/

With so much complex information produced by multiple applications and systems, administrators need a way to review the details, so they can understand the cause of problems or plan appropriately for the future.

The Syslog (System Logging Protocol) system on the server can act as a central log monitoring point over a network where all servers, network devices, switches, routers and internal services that create logs, whether linked to the particular internal issue or just informative messages can send their logs.

To ensure that logs from various machines in your environment are recorded centrally on a logging server, you can configure the Rsyslog application to record logs that fit specific criteria from the client system to the server.

The Rsyslog application, in combination with the systemd-journald service, provides local and remote logging support.

In order to set up a centralized log server on a CentOS/RHEL 8 server, you need to check an confirm that the /var partition has enough space (a few GB minimum) to store all recorded log files on the system that send by other devices on the network. I recommend you to have a separate drive (LVM or RAID) to mount the /var/log/ directory.

Rsyslog service is installed and running automatically in CentOS/RHEL 8 server. In order to verify that the daemon is running in the system, run the following command

systemctl status rsyslog.service

If the service is not running by default, run the following command to start rsyslog daemon:

systemctl start rsyslog.service

Now, in the /etc/rsyslog.conf configuration file, find and uncomment the following lines to grant UDP transport reception to the Rsyslog server via 514 port. Rsyslog uses the standard UDP protocol for log transmission.

vim /etc/rsyslog.conf

module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

The UDP protocol doesn’t have the TCP overhead, and it makes data transmission faster than the TCP protocol. On the other hand, the UDP protocol doesn’t guarantee the reliability of transmitted data.

However, if you want to use TCP protocol for log reception you must find and uncomment the following lines in the /etc/rsyslog.conf the configuration file in order to configure Rsyslog daemon to bind and listen to a TCP socket on 514 port.

module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")

Now create a new template for receiving remote messages, as this template will guide the local Rsyslog server, where to save the received messages send by Syslog network clients:

$template RemoteLogs,"/var/log/%PROGRAMNAME%.log" 
*.* ?RemoteLogs

The $template RemoteLogs directive guides Rsyslog daemon to gather and write all of the transmitted log messages to distinct files, based on the client name and remote client application that created the messages based on the outlined properties added in the template configuration: %PROGRAMNAME%.

All received log files will be written to the local filesystem to an allocated file named after the client machine’s hostname and kept in /var/log/ directory.

The & ~ redirect rule directs the local Rsyslog server to stop processing the received log message further and remove the messages (not write them to internal log files).

The RemoteLogs is an arbitrary name given to this template directive. You can use whatever name you want that best suitable for your template.

To configure more complex Rsyslog templates, read the Rsyslog configuration file manual by running the man rsyslog.conf command or consult Rsyslog online documentation.

Now, we exit our configuration file and restart the daemon:

service rsyslog restart

systemctl status rsyslog.service
....
Feb 25 14:43:06 syslog-new rsyslogd[1668]: imjournal: journal files changed, reloading... 

Once you restarted the Rsyslog server, it should now act as a centralized log server and record messages from Syslog clients. To confirm the Rsyslog network sockets, run netstat command:

netstat -tulpn | grep rsyslog 

If we don’t have this command, run this, to find out, which packages have it:

dnf provides netstat

Last metadata expiration check: 2:06:49 ago on Fri 25 Feb 2022 12:38:49 PM CET.
net-tools-2.0-0.52.20160912git.el8.x86_64 : Basic networking tools
Repo        : baseos
Matched from:
Filename    : /usr/bin/netstat

So, install package and run it again:

dnf install net-tools -y
netstat -tulpn | grep rsyslog 
tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN      1668/rsyslogd       
tcp6       0      0 :::514                  :::*                    LISTEN      1668/rsyslogd       
udp        0      0 0.0.0.0:514             0.0.0.0:*                           1668/rsyslogd       
udp6       0      0 :::514                  :::*                                1668/rsyslogd      

To add firewall exceptions for this port, execute following:

firewall-cmd --permanent --add-service=syslog
firewall-cmd --permanent --add-port=514/tcp #if enabled tcp socket
firewall-cmd --reload

And now, we can set remote logging on others servers or hardware (switches)…

On mikrotiks set remote IP for logging, like this:

/system/logging/action
set remote remote=10.10.10.10
/system/logging/
add action=remote topics=info
add action=remote topics=critical
add action=remote topics=error
add action=remote topics=warning

On cisco switches SG350X:

configure
logging host 10.10.10.10 description syslog.example.sk 
logging origin-id hostname

Or we can set on webserver, to sent apache logs to our syslog. First, set apache httpd.conf and vhost conf:

vim /etc/httpd/conf/httpd.conf

ErrorLog "||/usr/bin/logger -t apache -i -p local4.info"
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

vim /etc/httpd/conf.d/0-vhost.conf
...
    LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhostform
    CustomLog "||/usr/bin/logger -t apache -i -p local4.info" vhostform

And now, set on webserver rsyslog, to sent logs to our syslog, :

vim /etc/rsyslog.conf

#### OWN RULES ####
local4.info                                             @@10.10.10.10

Now, we can see, that our logs are on syslog. There, we can set filter to separate logs into own directories and files based on hostname, IPs… This I explain later (we alter rsyslog to syslog-ng package.

Total Page Visits: 115207 - Today Page Visits: 56

How to create ceph on Centos 8 Stream via Ceph Ansible

I assume, that we have working Centos 8 Stream system. So, in this example, we will have three systems (nodes), with identical HW resources (4 GB ram, 4 vCPU, two NICs – one internal for ceph and one for world, and 10 TB spin-up hdd). In this article, every command must be run on all nodes. Public network is 192.168.1.0/24 and Ceph separate network is 192.168.2.0/24

Setting up time

As the first step, we must set up a time, I use chrony:

dnf install chrony -y
systemctl enable chronyd
timedatectl set-timezone Europe/Bratislava
timedatectl

Now, edit some variables in configurations file for chronyd. Add some servers from pool, and edit local subnets, where we delived time:

vim /etc/chrony.conf

pool 2.centos.pool.ntp.org iburst
pool 1.centos.pool.ntp.org iburst
pool 3.centos.pool.ntp.org iburst

Now start/restart our service, and check, if it is working:

systemctl restart chronyd
systemctl status chronyd.service
chronyc sources

Create hostnames, ssh rsa-keys, update and install som packages

Now, we must edit on all nodes our hostnames, set it permanent:

hostnamectl set-hostname ceph1

Now, add all hostnames, and IPs to file /etc/hosts:

tee -a /etc/hosts<<EOF
192.168.1.1    ceph1
192.168.1.2    ceph2
192.168.1.3    ceph3
192.168.2.1    ceph1-cluster
192.168.2.2    ceph2-cluster
192.168.3.3    ceph3-cluster

EOF

Now, create rsa-key pair, for password-less connect to and from each node:

ssh-keygen -t rsa -b 4096 -C "ceph1"

means:
-b bits. Number of bits in the key to create
-t type. Specify type of key to create
-C comment

And copy it to other nodes:

for host in ceph1 ceph2 ceph3; do
 ssh-copy-id root@$host
done

Now update and install packages:

dnf update -y
-reboot-
dnf install git vim bash-completion python3-pip

Preparing for ceph

Now, install epel repository and enable powertools:

dnf -y install dnf-plugins-core
dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
dnf config-manager --set-enabled powertools

dnf repolist
repo id            repo name
appstream          CentOS Stream 8 - AppStream
epel               Extra Packages for Enterprise Linux 8 - x86_64
epel-modular       Extra Packages for Enterprise Linux Modular 8 - x86_64
epel-next          Extra Packages for Enterprise Linux 8 - Next - x86_64
extras             CentOS Stream 8 - Extras
powertools         CentOS Stream 8 - PowerTools

Clone Ceph Ansible repository:

cd /root/
git clone https://github.com/ceph/ceph-ansible.git

Choose ceph-ansible branch you wish to use. The command Syntax is: git checkout $branch

I’ll switch to stable-5.0 which supports Ceph octopus version.

cd ceph-ansible
git checkout stable-5.0

pip3 install setuptools-rust
pip3 install wheel
export CRYPTOGRAPHY_DONT_BUILD_RUST=1
pip3 install --upgrade pip

pip3 install -r requirements.txt
echo "PATH=\$PATH:/usr/local/bin" >>~/.bashrc
source ~/.bashrc

Confirm Ansible version installed.

ansible --version
ansible 2.9.26
  config file = /root/ceph-ansible/ansible.cfg
  configured module search path = ['/root/ceph-ansible/library']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.8 (default, Sep 10 2021, 09:13:53) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]

Now, we find, which OSD (spin-up disks) are ready for us. In each my node, there is free disk /dev/sda. Look via lsblk:

lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                       8:0    0 10.7T  0 disk 
sr0                      11:0    1 1024M  0 rom  
vda                     252:0    0   32G  0 disk 
├─vda1                  252:1    0  512M  0 part /boot
└─vda2                  252:2    0 31.5G  0 part 
  ├─centos--vg0-root    253:0    0    3G  0 lvm  /
  ├─centos--vg0-swap    253:1    0    1G  0 lvm  [SWAP]
  ├─centos--vg0-tmp     253:2    0  512M  0 lvm  /tmp
  ├─centos--vg0-var_log 253:3    0  512M  0 lvm  /var/log
  ├─centos--vg0-var     253:4    0    3G  0 lvm  /var
  └─centos--vg0-home    253:5    0    2G  0 lvm  /home

Now, we are ready for installation of Ceph

Deploy Ceph Octopus (15) Cluster on CentOS 8 stream

Now, we are play some things 🙂 The first node (ceph1) I use as admin node for installation. Configure Ansible Inventory and Playbook files. Create Ceph Cluster group variables file on the admin node

cd /root/ceph-ansible
cp group_vars/all.yml.sample  group_vars/all.yml
vim group_vars/all.yml

And edit some variables of your Ceph cluster, as you fit:

#General
cluster: ceph

# Inventory host group variables
mon_group_name: mons
osd_group_name: osds
rgw_group_name: rgws
mds_group_name: mdss
nfs_group_name: nfss
rbdmirror_group_name: rbdmirrors
client_group_name: clients
iscsi_gw_group_name: iscsigws
mgr_group_name: mgrs
rgwloadbalancer_group_name: rgwloadbalancers
grafana_server_group_name: grafana-server

# Firewalld / NTP
configure_firewall: True
ntp_service_enabled: true
ntp_daemon_type: chronyd

# Ceph packages
ceph_origin: repository
ceph_repository: community
ceph_repository_type: cdn
ceph_stable_release: octopus

# Interface options
monitor_interface: ens18
radosgw_interface: ens18
public_network: 192.168.1.0/24
cluster_network: 192.168.2.0/24


# DASHBOARD
dashboard_enabled: True
dashboard_protocol: http
dashboard_admin_user: admin
dashboard_admin_password: strongpass

grafana_admin_user: admin
grafana_admin_password: strongpass

Now, set your OSDs. Create a new ceph nodes ansible inventory. Properly set your inventory file. Below is my inventory. Modify inventory groups the way you want services installed in your cluster nodes.

vim hosts

# Ceph admin user for SSH and Sudo
[all:vars]
ansible_ssh_user=root
ansible_become=true
ansible_become_method=sudo
ansible_become_user=root

# Ceph Monitor Nodes
[mons]
ceph1
ceph2
ceph3

# MDS Nodes
[mdss]
ceph1
ceph2
ceph3

# RGW
[rgws]
ceph1
ceph2
ceph3

# Manager Daemon Nodes
[mgrs]
ceph1
ceph2
ceph3

# set OSD (Object Storage Daemon) Node
[osds]
ceph1
ceph2
ceph3

# Grafana server
[grafana-server]
ceph1

Create Playbook file by copying a sample playbook at the root of the ceph-ansible project called site.yml.sample.

cp site.yml.sample site.yml 

Run Playbook.

ansible-playbook -i hosts site.yml 

If installation was successful, a health check should return OK or minimal WARN.

# ceph -s
  cluster:
    id:     dcfd26f5-49e9-4256-86c2-a5a0deac7b54
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim
 
  services:
    mon: 3 daemons, quorum eu-ceph1,eu-ceph2,eu-ceph3 (age 67m)
    mgr: ceph2(active, since 55m), standbys: ceph3, ceph1
    mds: cephfs:1 {0=ceph1=up:active} 2 up:standby
    osd: 3 osds: 3 up (since 60m), 3 in (since 60m)
    rgw: 3 daemons active (ceph1.rgw0, ceph2.rgw0, ceph3.rgw0)
 
  task status:
 
  data:
    pools:   7 pools, 169 pgs
    objects: 215 objects, 11 KiB
    usage:   3.1 GiB used, 32 TiB / 32 TiB avail
    pgs:     169 active+clean

This is a screenshot of my installation output once it has been completed.

As you see, I have warning: mons are allowing insecure global_id reclaim

So, silent it, as you fit it, or fix…

ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false
Total Page Visits: 115207 - Today Page Visits: 56

Rsync review and some examples

rsync — a fast, versatile, remote (and local) file-copying tool
-a        archive mode
-r        recursive – recurse into directories
-v         verbose – increase verbosity
-z        compress – With this option, rsync compresses the file data as it is sent to the destination machine, which  reduces the amount of data being transmitted something that is useful over a slow connection Note  that  this  option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection
-P        is equivalent to –partial –progress.  Its purpose is to make it  much  easier     to  specify these two options for a long transfer that may be interrupted
-n        perform a trial run with no changes made
-u        skip files that are newer on the receiver
-t        preserve modification times
–bwlimit=KBPS    limit I/O bandwidth; KBytes per second
This option allows you to specify a maximum transfer rate in kilobytes per second.  This  option  is  most effective  when using rsync with large files (several megabytes and up). Due to the nature of rsync transfers, blocks of data are sent, then if rsync determines the transfer was too fast,  it  will  wait  before
sending  the next data block. The result is an average transfer rate equaling the specified limit. A value of zero specifies no limit.
(25Mb = 3200 KB)
(10Mb = 1250 KB)
(7.5 Mb = 960 KB)
(5Mb = 640 KB)
(2.5Mb = 320 KB)
(3Mb = 384 KB)
(1Mb = 128 KB)
–append              append data onto shorter files
–append-verify        append w/old data in file checksum

rsync -avz foo:src/bar /data/tmp

This  would  recursively  transfer all files from the directory src/bar on the machine foo into the /data/tmp/bar directory on the local machine. The files are transferred in “archive” mode, which ensures that  symbolic  links,
devices, attributes, permissions, ownerships, etc. are preserved in the transfer.  Additionally, compression will be used to reduce the size of data portions of the transfer.
– Trailing slash on the source avoid to create directory on the destinations. So without trailing slash at the end, this will
create this directory at the destination. This is the same

 rsync -av /src/foo /dest
 rsync -av /src/foo/ /dest/foo

This will synchronize and copy left folder to to right. It preserve unfinished files. With next commenad, it will resume
and append data to unfinished files.

rsync -avP /mnt/nfs /media/adm-nfs/
rsync -avP --append /mnt/nfs /media/adm-nfs/
Total Page Visits: 115207 - Today Page Visits: 56

hdparm useful commands

Some useful commands:
This check current IDE power mode status of the disk:
– unknown (drive does not support this command),
– active/idle (normal operation),
– standby (low power mode, drive has spun down),
– sleeping (lowest power mode, drive is completely shut down)
The operators: -S, -y, -Y, -Z can be used to manipulate the IDE power modes

hdparm -C /dev/sda

Force an IDE drive to immediately enter the low power consumption STANDBY mode, usually causing it to spin down:

hdparm -y /dev/sda

Force  an  IDE  drive to immediately enter the lowest power consumption sleep mode, causing it to shut down completely. A hard or soft reset is required before the drive can be accessed again:

hdparm -Y /dev/sda

Put the drive into idle  (low-power)  mode,  and  also  set  the standby (spindown) timeout for the drive.  This timeout value is used by the drive to determine how long to wait  (with  no  disk activity)  before  turning  off the spindle motor to save power:

hdparm -S /dev/sda

Disable  the  automatic power-saving function of certain Seagate drives (ST3xxx models?), to prevent them  from  idling/spinning down at inconvenient times.
An example:

hdparm -y /dev/sdd
/dev/sdd
      issuing standby command
hdparm -C /dev/sdd
/dev/sdd
      drive state is: standby

 

And this is the READ test of the disk:

 hdparm -Tt /dev/sdc
/dev/sdc:
Timing cached reads: 21590 MB in 1.99 seconds = 10828.31 MB/sec
Timing buffered disk reads: 310 MB in 3.00 seconds = 103.20 MB/sec

Total Page Visits: 115207 - Today Page Visits: 56

Check disks for bad blocks with complete erase and smart-self test

This is another useful script, which do a complete erase and test disk in linux.
The first, we must have a tools “smartmontools”. We install it:

yum install smartmontools

Maybe, we nee som usefull software:

yum install epel-release.noarch
yum install htop dstat lm_sensors.x86_64 hddtemp
#!/bin/bash
#path of disk
test = /dev/sda
#find a serial number of disk
disk = `smartctl -a /dev/sda | grep Serial | awk '{print $3}'`
log = /home/vasil/$disk.log
#next, we destroy any of the partition table (mbr or gpt)
sgdisk --zap-all $test > $log
sleep 5
#notice a temperature of disk to log
hddtemp $test >> $log
#now we write zeros to every block of disk - secure erase contents
dd if=/dev/zero of=$test bs=4M
#notice a temperature of disk
hddtemp $test >> $log
sleep 5
#begin an internal self test - short
smartctl --test=short $test
sleep 150
#begin a long smart self test
hddtemp $test >> $log
smartctl --test=long $test
sleep 300
hddtemp $test >> $log
#print the output of tests
smartctl -l selftest $test >> $log
sleep 5
#start check of disk for bad-blocks a log bad blocks to log
`badblocks -v /dev/sda > /home/vasil/sda.txt` >> $log
sleep 5
hddtemp $test >> $log
smartctl --test=long $test
exit 0

This is fully automatized. We can start this script with modification of variable $test for more disks. And next day, we can examine logs. For bad blocks, for smart self-test and other.

Total Page Visits: 115207 - Today Page Visits: 56

Periodically check temp of hdd

This is a little how to. Its periodically in cron checking temperature of internal hard-disk and in some condition make an alert by mail.
So, as always, install some useful packages:

yum install epel-release.noarch
yum install htop dstat lm_sensors.x86_64 hddtemp

First, we must check, which disk we have inside. We use only paragraph without numbers:

ls -lah /dev/sd*
----------------
brw-rw---- 1 root disk 8,  0 Dec  1 07:20 /dev/sda
brw-rw---- 1 root disk 8,  1 Dec  1 07:20 /dev/sda1
brw-rw---- 1 root disk 8, 16 Dec  1 07:20 /dev/sdb
brw-rw---- 1 root disk 8, 17 Dec  1 07:20 /dev/sdb1
brw-rw---- 1 root disk 8, 32 Dec  1 07:20 /dev/sdc
brw-rw---- 1 root disk 8, 33 Dec  1 07:20 /dev/sdc1
brw-rw---- 1 root disk 8, 48 Dec  1 07:20 /dev/sdd
brw-rw---- 1 root disk 8, 49 Dec  1 07:20 /dev/sdd1
brw-rw---- 1 root disk 8, 64 Dec  1 07:20 /dev/sde
brw-rw---- 1 root disk 8, 65 Dec  1 07:20 /dev/sde1
brw-rw---- 1 root disk 8, 80 Dec  1 07:20 /dev/sdf
brw-rw---- 1 root disk 8, 81 Dec  1 07:20 /dev/sdf1
-----------------
we user only:
/dev/sda, /dev/sdb, /dev/sdc, /dev/sdd, /dev/sde, /dev/sdf

Create a folder, maybe like this:

mkdir /root/hddtemp

Create a script to check this temperatures, end some conditions:

#! /bin/bash
#written by vasil
HDDS="/dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf"
HDT=/usr/sbin/hddtemp
mail="/root/hddtemp/messagebody.txt"  #file for messasege by mail
LOG=/root/hddtemp/hddtemp.log         #log file
ALERT_LEVEL=48                        #temperature treshold
date >> $LOG
echo "Subject: "WARNING - name-of-server - temperature of hdds is high"" > $mail
echo " " >> $mail
date >> $mail
m=0
for disk in $HDDS
do
                        HDTEMP=$($HDT $disk | awk -F ":" '{ print $3}' | awk -F "°"  '{print $1}')
                        if [ $HDTEMP -ge $ALERT_LEVEL ];
                        then
                                echo "Temperature of disk $disk is higher then limit $ALERT_LEVEL celsius" >> $LOG
                                echo "Temperature of disk $disk is higher then limit $ALERT_LEVEL celsius" >> $mail
                                $HDT $disk >> $LOG
                                $HDT $disk >> $mail
                                echo " " >> $LOG
                                echo " " >> $mail
                                m=1
                        else
                                echo "Temperature of disk $disk Is normal: $HDTEMP celsius" >> $LOG                             
                                echo "Temperature of disk $disk Is normal: $HDTEMP celsius" >> $mail
                        fi
done
if [ "$m" -ne "0" ];
then
       sh /root/hddtemp/send_mail_script.sh
else
        echo
fi
echo "end of script...." >> $LOG
echo " " >> $LOG
exit 0

Next, we must create a script, to send mail to us, if we have a mail server in local network with allowed port 25:

vim /root/hddtemp/send_mail_script.sh
#!/bin/bash
#written by vasil
#
#if there is an error in syntax "sendmail", configure /etc/ssmtp/ssmtp.conf
#
# subject of email
SUBJECT="WARNING - name-of-server - temperature of hdds is high"
# destination
EMAIL="vasil@gonscak.sk"
# Email body
EMAILMESSAGE=/root/hddtemp/messagebody.txt
# send message using /bin/mail
#sendmail $EMAIL < $EMAILMESSAGE
/usr/bin/mail -s "$SUBJECT" $EMAIL < $EMAILMESSAGE

Now, we can test the above scripts. Maybe we must add +x permissions:

sh -x /root/hddtemp/hddtemp.sh
------------------------------
+ HDDS=/dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
+ HDT=/usr/sbin/hddtemp
+ mail=/root/hddtemp/messagebody.txt
+ LOG=/root/hddtemp/hddtemp.log
+ ALERT_LEVEL=28
+ date
+ echo Subject: WARNING - name-of-server - temperature of hdds is high
+ echo
+ date
+ m=0
+ /usr/sbin/hddtemp /dev/sda
+ awk -F : { print $3}
+ awk -F ° {print $1}
+ HDTEMP= 38
+ [ 38 -ge 28 ]
+ echo Temperature of disk /dev/sda is higher then limit 28 celsius
+ echo Temperature of disk /dev/sda is higher then limit 28 celsius
+ /usr/sbin/hddtemp /dev/sda
+ /usr/sbin/hddtemp /dev/sda
+ echo
+ echo
+ m=1
+ /usr/sbin/hddtemp /dev/sdb
+ awk -F : { print $3}
+ awk -F ° {print $1}
+ HDTEMP= 35
+ [ 35 -ge 28 ]
+ echo Temperature of disk /dev/sdb is higher then limit 28 celsius
+ echo Temperature of disk /dev/sdb is higher then limit 28 celsius
+ /usr/sbin/hddtemp /dev/sdb
...
...
...
+ m=1
+ [ 1 -ne 0 ]
+ sh /root/hddtemp/send_mail_script.sh
+ echo end of script....
+ echo
+ exit 0

And then, we have a mail:

Subject: WARNING - name-of-server - temperature of hdds is high
Thu Jan 26 11:52:20 CET 2017
Temperature of disk /dev/sda is higher then limit 28 celsius
/dev/sda: TOSHIBA DT01ACA100: 37°C
Temperature of disk /dev/sdb is higher then limit 28 celsius
/dev/sdb: ST3500418AS: 35°C
....
Total Page Visits: 115207 - Today Page Visits: 56

How to create a site-to-site ipsec vpn connection

Install and enable the EPEL using Yum, with some useful software:

yum install epel-release.noarch
yum install htop dstat tcpdump

On Red Hat based Systems (CentOS, Fedora or RHEL):

yum install openswan

Now we disable VPN redirects, if any, in the server using these commands:

for vpn in /proc/sys/net/ipv4/conf/*;
do echo 0 > $vpn/accept_redirects;
echo 0 > $vpn/send_redirects;
echo 0 > $vpn/rp_filter;
done

Edit /etc/ipsec.conf to debug in pluto.log

    plutostderrlog=/var/log/pluto.log
    protostack=netkey
#if using NAT, use variable below
#    nat_traversal=yes
    virtual_private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12

Next, we modify the kernel parameters to allow IP forwarding and disable redirects permanently by:

 vim /etc/sysctl.conf
    net.ipv4.ip_forward = 1
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv4.conf.all.send_redirects = 0

Reload /etc/sysctl.conf:

 sysctl -p

Now, we customize firewall to allow ports for ipsec

firewall-cmd --zone=public --permanent --add-rich-rule='rule protocol value="esp" accept'
firewall-cmd --zone=public --permanent --add-rich-rule='rule protocol value="ah" accept'
firewall-cmd --zone=public --permanent --add-port=500/udp
firewall-cmd --zone=public --permanent --add-port=4500/udp
firewall-cmd --permanent --add-service="ipsec"
firewall-cmd --zone=public --permanent --add-port=4500/tcp
firewall-cmd --zone=public --add-port=50/udp --permanent
firewall-cmd --zone=public --add-port=51/udp --permanent

We don’t use masquerade, because ipsec tunnel parameters automatic enable routing in these situations. If  not working, we add masquerade, but first we must add rule for match packets for this tunnel. Like: src leftsubnet dst rightsubnet on both sides

#In some posts in world I found this code, but explanation above cancel this
#code and in my situation it not working with this
#firewall-cmd --zone=public --permanent --add-masquerade

We reload firewalld and check our rules:

firewall-cmd --reload
firewall-cmd --zone=public --list-all

Check if is ipsec OK for itself:

ipsec verify
------------
Verifying installed system and configuration files
Version check and ipsec on-path                         [OK]
Libreswan 3.15 (netkey) on 3.10.0-514.6.1.el7.x86_64
Checking for IPsec support in kernel                    [OK]
 NETKEY: Testing XFRM related proc values
         ICMP default/send_redirects                    [OK]
         ICMP default/accept_redirects                  [OK]
         XFRM larval drop                               [OK]
Pluto ipsec.conf syntax                                 [OK]
Hardware random device                                  [N/A]
Two or more interfaces found, checking IP forwarding    [OK]
Checking rp_filter                                      [OK]
Checking that pluto is running                          [OK]
 Pluto listening for IKE on udp 500                     [OK]
 Pluto listening for IKE/NAT-T on udp 4500              [OK]
 Pluto ipsec.secret syntax                              [OK]
Checking 'ip' command                                   [OK]
Checking 'iptables' command                             [OK]
Checking 'prelink' command does not interfere with FIPSChecking for obsolete ipsec.conf options                 [OK]
Opportunistic Encryption                                [DISABLED]

Now, create a configuration file for our one connection

vim /etc/ipsec.d/blava.conf
---------------------------
conn blava
    left=%defaultroute
    leftid=192.168.201.75
    leftsubnet=192.168.201.0/24
   
    right=#public IP other side#
    rightid=192.168.202.177
    rightsubnet=192.168.202.0/24
    type=tunnel
    authby=secret
    pfs=no
    auth=esp
    keyexchange=ike
    keyingtries=0
    ikelifetime=28800s
    salifetime=360000s
    esp=3des-sha1
    ike=aes256-sha1;modp1024
    auto=start
    compress=no

And configuration file for other connection:

vim /etc/ipsec.d/blava.conf
---------------------------
conn blava
    left=#public IP this side#
    leftid=192.168.202.177
    leftsubnet=192.168.202.0/24
    right=%any
    rightid=192.168.201.75
    rightsubnet=192.168.201.0/24
    type=tunnel
    authby=secret
    pfs=no
    auth=esp
    keyexchange=ike
    keyingtries=0
    ikelifetime=28800s
    salifetime=360000s
    esp=3des-sha1
    ike=aes256-sha1;modp1024
    auto=add
    compress=no
    keep_alive=30

Now create on both sides secrets file for PSK with your public IP:

vim /etc/ipsec.d/blava.secrets
------------------------------
%any 1.1.1.1: PSK "ahoj12345"
vim /etc/ipsec.d/blava.secrets
------------------------------
1.1.1.1 %any: PSK "ahoj12345"

Now, restart ipsec for apply configurations

systemctl restart ipsec.service

And if we are good, we must see some like this in pluto.log

 STATE_MAIN_R3: sent MR3, ISAKMP SA established
 STATE_QUICK_R2: IPsec SA established tunnel mode

Or check ipsec status:

ipsec auto --status
-------------------
Total IPsec connections: loaded 4, active 1
STATE_QUICK_R2 (IPsec SA established); EVENT_SA_REPLACE in 85318s
STATE_MAIN_R3 (sent MR3, ISAKMP SA established); EVENT_SA_REPLACE in 27718s;

Some usefull commands for work with ipsec…
When we update configuration file and if we must reload one ipsec tunnel, use these step rather then restart ipsec service itself:

ipsec auto --down blava
ipsec auto --replace blava
ipsec auto --up blava

If we change secrets file and PSK, we must use too, before –up:

ipsec auto --rereadall
Total Page Visits: 115207 - Today Page Visits: 56