Encrypted LVM partition on software raid-1 with mdadm

At another post https://www.gonscak.sk/?p=201 I posted how to create raid1 software raid with mdadm in linux. Now I tried to add a crypted filesystem to this.

First, check, that we have working software raid:

sudo mdadm --misc --detail /dev/md0

/dev/md0:
           Version : 1.2
     Creation Time : Wed Aug 22 09:34:23 2018
        Raid Level : raid1
        Array Size : 1953381440 (1862.89 GiB 2000.26 GB)
     Used Dev Size : 1953381440 (1862.89 GiB 2000.26 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent
     Intent Bitmap : Internal
       Update Time : Thu Aug 23 14:18:50 2018
             State : active 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0
Consistency Policy : bitmap
              Name : gw36:0  (local to host gw36)
              UUID : ded4f30e:1cfb20cb:c10b843e:df19a8ff
            Events : 3481
    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1

Now, we synced drives and clean. It is time to encrypt.  If we have not loaded modules for encryption, load it:q

modprobe dm-crypt

Now create the volume with passphrase:

sudo cryptsetup --cipher=aes-xts-plain --verify-passphrase --key-size=512 luksFormat /dev/md0

And we can open it:

sudo cryptsetup  luksOpen /dev/md0 cryptdisk

Now we can create as many times a physical volume, volume group and logical volume.

sudo pvcreate /dev/mapper/cryptdisk
sudo vgcreate raid1 /dev/mapper/cryptdisk
sudo lvcreate --size 500G --name lv-home raid1

sudo pvs
  PV                     VG        Fmt  Attr PSize    PFree
  /dev/mapper/cryptdisk  raid1     lvm2 a--    <1,82t 1,33t
sudo vgs
  VG        #PV #LV #SN Attr   VSize    VFree
  raid1       1   1   0 wz--n-   <1,82t 1,33t
sudo lvs
  LV      VG        Attr       LSize
  lv-home raid1     -wi-ao---- 500,00g            

Next, we create a filesystem on this logical volume:

sudo mkfs.ext4 /dev/mapper/raid1-lv--home

And we can mount it:

sudo mount /dev/mapper/raid1-lv--home crypt-home/

Now we have an encrypted partition (disk) for our home directory.

27219total visits,111visits today

How to resize Physical volume and shrink disk partition

I Installed proxmox environment on Intel 240GB SSD. Installation take the whole disk for lvm. So I need to reduce the used space and create a new partition for drbd.
This is my disk. You can see, that the disk is full allocated with 171G free.

root@pve1:/# gdisk -l /dev/sda
Disk /dev/sda: 468862128 sectors, 223.6 GiB
Number Start (sector) End (sector) Size Code Name
 1      34          2047        1007.0  KiB EF02
 2      2048        262143      127.0   MiB EF00
 3      262144      111411199   53.0    GiB 8E00 Linux LVM
root@pve1:/# pvs
 PV VG Fmt Attr PSize PFree
 /dev/sda3 pve lvm2 a-- 223.44g 171.44g
root@pve1:/# vgs
 VG #PV #LV #SN Attr VSize VFree
 pve 1 3 0 wz--n- 223.44g 171.44g
root@pve1:/# lvs
 LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
 data pve -wi-ao--- 40.00g
 root pve -wi-ao--- 10.00g
 swap pve -wi-ao--- 2.00g

So, we list our logical volumes with segments on physical volume /dev/sda3:

root@pve1:/# pvs -v --segments /dev/sda3
 Using physical volume(s) on command line
 PV VG Fmt Attr PSize PFree Start SSize LV Start Type PE Ranges
 /dev/sda3 pve lvm2 a-- 223.44g 171.44g 0 512 swap 0 linear /dev/sda3:0-511
 /dev/sda3 pve lvm2 a-- 223.44g 171.44g 512 2560 root 0 linear /dev/sda3:512-3071
 /dev/sda3 pve lvm2 a-- 223.44g 171.44g 3072 10240 data 0 linear /dev/sda3:3072-13311
 /dev/sda3 pve lvm2 a-- 223.44g 171.44g 13312 43889 0 free

We can see, the size of PV is 223,44G and we have free 171,44G. So, we must shrink this physical volume about 171,44G. So compute the space for size of this physical volume: 223,44 – 171,44 = 52G. So, our PV must have at least 52G. Next, we resize this pv:

root@pve1:/# pvresize --setphysicalvolumesize 52G /dev/sda3
 /dev/sda3: cannot resize to 13311 extents as 13312 are allocated.
 0 physical volume(s) resized / 1 physical volume(s) not resized
root@pve1:/# pvresize --setphysicalvolumesize 52.1G /dev/sda3
 Physical volume "/dev/sda3" changed
 1 physical volume(s) resized / 0 physical volume(s) not resized

As we can see, we cannost shrink exact to this space. So we add 100M and use the 52,1G size. Now we can see:

root@pve1:/# pvs -v --segments /dev/sda3
 Using physical volume(s) on command line
 PV VG Fmt Attr PSize PFree Start SSize LV Start Type PE Ranges
 /dev/sda3 pve lvm2 a-- 52.10g 100.00m 0 512 swap 0 linear /dev/sda3:0-511
 /dev/sda3 pve lvm2 a-- 52.10g 100.00m 512 2560 root 0 linear /dev/sda3:512-3071
 /dev/sda3 pve lvm2 a-- 52.10g 100.00m 3072 10240 data 0 linear /dev/sda3:3072-13311
 /dev/sda3 pve lvm2 a-- 52.10g 100.00m 13312 25 0 free

At this point, we must work on the lowest layer of disk, so we must delete this partition and create a new one. The new partition must start on the same sector as previous and the last sector must be after last segment of physical volume. I use gdisk, because my disk have GPT partition table:

root@pve1:/# gdisk /dev/sda
Command (? for help): p
Disk /dev/sda: 468862128 sectors, 223.6 GiB
Logical sector size: 512 bytes
First usable sector is 34, last usable sector is 468862094
Number Start (sector) End (sector) Size Code Name
 1 34 2047 1007.0 KiB EF02
 2 2048 262143 127.0 MiB EF00
 3 262144 468862094 223.4 GiB 8E00
Command (? for help): d
Partition number (1-3): 3
Command (? for help): n
Partition number (3-128, default 3):
First sector (262144-468862094, default = 262144) or {+-}size{KMGTP}:
Last sector (262144-468862094, default = 468862094) or {+-}size{KMGTP}: +53G
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 8E00
Changed type of partition to 'Linux LVM'
Command (? for help): p
Disk /dev/sda: 468862128 sectors, 223.6 GiB
Total free space is 357450895 sectors (170.4 GiB)
Number Start (sector) End (sector) Size         Code Name
 1     34             2047         1007.0 KiB   EF02
 2     2048           262143       127.0 MiB    EF00
 3     262144         111411199    53.0 GiB     8E00 Linux LVM
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.

Now, we must reboot our computer to use new partition table. And after reboot, use this command to resize physical volume on partition /dev/sda3

root@pve1:/# pvresize /dev/sda3
 Physical volume "/dev/sda3" changed
 1 physical volume(s) resized / 0 physical volume(s) not resized
root@pve1:/# pvs
 PV VG Fmt Attr PSize PFree
 /dev/sda3 pve lvm2 a-- 53.00g 1020.00m

Now, if we can use all of free space for the logical volume “data”, we can resize it to whole free space, like this:

root@pve1:/# lvresize /dev/pve/data -l +100%FREE
 Extending logical volume data to 41.00 GiB
 Logical volume data successfully resized
 root@pve1:/# lvs
 LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
 data pve -wi-ao--- 41.00g
 root pve -wi-ao--- 10.00g
 swap pve -wi-ao--- 2.00g
root@pve1:/# pvs
 PV VG Fmt Attr PSize PFree
 /dev/sda3 pve lvm2 a-- 53.00g 0

Now, we can create a new partition at the end of disk:

gdisk /dev/sda
Command (? for help): p
Disk /dev/sda: 468862128 sectors, 223.6 GiB
Logical sector size: 512 bytes
Number Start (sector) End (sector) Size Code Name
 1 34 2047 1007.0 KiB EF02
 2 2048 262143 127.0 MiB EF00
 3 262144 111411199 53.0 GiB 8E00 Linux LVM
Command (? for help): n
Partition number (4-128, default 4):
First sector (111411200-468862094, default = 111411200) or {+-}size{KMGTP}:
Last sector (111411200-468862094, default = 468862094) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
root@pve1:~# gdisk -l /dev/sda
Number Start (sector) End (sector) Size         Code Name
 1     34             2047         1007.0 KiB   EF02
 2     2048           262143       127.0 MiB    EF00
 3     262144         111411199    53.0 GiB     8E00 Linux LVM
 4     111411200      468862094    170.4 GiB    8300 Linux filesystem

And if we list details about physical volume, we can see, that there is no free space:

root@pve1:~# pvs -v --segments /dev/sda3
 Using physical volume(s) on command line
 PV VG Fmt Attr PSize PFree Start SSize LV Start Type PE Ranges
 /dev/sda3 pve lvm2 a-- 53.00g 0 0 512 swap 0 linear /dev/sda3:0-511
 /dev/sda3 pve lvm2 a-- 53.00g 0 512 2560 root 0 linear /dev/sda3:512-3071
 /dev/sda3 pve lvm2 a-- 53.00g 0 3072 10495 data 0 linear /dev/sda3:3072-13566

And what is drbd, you can see in another post on my page. Have a fun.

27219total visits,111visits today

how to set up drbd primary-primary mode on proxmox 4.x

Today, I met with an interesting problem. I tried to create a primary-primary (dual primary) DRBD cluster on proxmox.
The first we must have fully configured proxmox Two-node cluster. Like this:
https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster
We must have a good configuration of /etc/hosts to resolve names into IP:

root@cl3-amd-node1:/etc/drbd.d# cat /etc/hosts
cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.1.104 cl3-amd-node1 pvelocalhost
192.168.1.108 cl3-amd-node2
root@cl3-amd-node2:/etc/drbd.d# cat /etc/hosts
cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.1.104 cl3-amd-node1
192.168.1.108 cl3-amd-node2 pvelocalhost

One server was created on hardware raid PCI-E LSI 9240-4i (/dev/sdb) and second server was build on software raid via mdadm (/dev/md1) on debian jessie with installation with proxmox packages. So the backend for drbd devices was on one side – hardware raid and software raid on the other side.  We must create a two disks with the same size (in sectors):

root@cl3-amd-node1:
fdisk -l /dev/sdb
Disk /dev/sdb: 1.8 TiB, 1998998994944 bytes, 3904294912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
Device     Boot Start        End    Sectors   Size Id Type
/dev/sdb1        2048 1953260927 1953258880 931.4G 83 Linux
root@cl3-amd-node2:
fdisk -l /dev/md1
Disk /dev/md1: 931.4 GiB, 1000069595136 bytes, 1953260928 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
Device     Boot Start        End    Sectors   Size Id Type
/dev/md1p1       2048 1953260927 1953258880 931.4G 83 Linux

Now, we must have a direct network to each other of servers for drbd traffic, which will be very high. I use a bond of two gigabit network cards:

#cl3-amd-node1:
cat /etc/network/interfaces
auto bond0
iface bond0 inet static
        address  192.168.5.104
        netmask  255.255.255.0
        slaves eth2 eth1
        bond_miimon 100
        bond_mode balance-rr
#cl3-amd-node2:
cat /etc/network/interfaces
auto bond0
iface bond0 inet static
        address  192.168.5.108
        netmask  255.255.255.0
        slaves eth1 eth2
        bond_miimon 100
        bond_mode balance-rr

And we can test the speed of this network with package iperf:

apt-get install iperf

We start an iperf instance on one server by this command:

#cl3-amd-node2
iperf  -s -p 888

And from the other, we connect to this instance for 20 seconds:

#cl3-amd-node1
iperf -c 192.168.5.108 -p 888 -t 20
#and the conclusion
------------------------------------------------------------
Client connecting to 192.168.5.108, TCP port 888
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.5.104 port 49536 connected with 192.168.5.108 port 888
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.0 sec  4.39 GBytes  1.88 Gbits/sec

So we can see, that I have a bonded network from two network cards and the resulting speed is almost 2Gbps.
Now, we can continue with installing and setting up the drbd resource.

apt-get install drbd-utils drbdmanage

All aspects of DRBD are controlled in its configuration file, /etc/drbd.conf. Normally, this configuration file is just a skeleton with the following contents:
include “/etc/drbd.d/global_common.conf”;
include “/etc/drbd.d/*.res”;
The simplest configuration is:

cat /etc/drbd.d/global_common.conf
global {
        usage-count yes;
}
common {
        net {
        protocol C;
        }
}

And the configuration of resource itself. It must be the same on both nodes:

root@cl3-amd-node1:/etc/drbd.d# cat /etc/drbd.d/r0.res
resource r0 {
disk {
        c-plan-ahead 15;
        c-fill-target 24M;
        c-min-rate 90M;
        c-max-rate 150M;
}
net {
        protocol C;
        allow-two-primaries yes;
        data-integrity-alg md5;
        verify-alg md5;
}
on cl3-amd-node1 {
        device /dev/drbd0;
        disk /dev/sdb1;
        address 192.168.5.104:7789;
        meta-disk internal;
}
on cl3-amd-node2 {
        device /dev/drbd0;
        disk /dev/md1p1;
        address 192.168.5.108:7789;
        meta-disk internal;
}
}
root@cl3-amd-node2:/etc/drbd.d# cat /etc/drbd.d/r0.res
resource r0 {
disk {
        c-plan-ahead 15;
        c-fill-target 24M;
        c-min-rate 90M;
        c-max-rate 150M;
}
net {
        protocol C;
        allow-two-primaries yes;
        data-integrity-alg md5;
        verify-alg md5;
}
on cl3-amd-node1 {
        device /dev/drbd0;
        disk /dev/sdb1;
        address 192.168.5.104:7789;
        meta-disk internal;
}
on cl3-amd-node2 {
        device /dev/drbd0;
        disk /dev/md1p1;
        address 192.168.5.108:7789;
        meta-disk internal;
}
}

Now, we must create and initialize backend devices for drbd, on both nodes:

drbdadm create-md r0
#answer yes to destroy possible data on devices

Now, we can start the drbd service, on both nodes:

root@cl3-amd-node2:/etc/drbd.d# /etc/init.d/drbd start
[ ok ] Starting drbd (via systemctl): drbd.service.
root@cl3-amd-node1:/etc/drbd.d# /etc/init.d/drbd start
[ ok ] Starting drbd (via systemctl): drbd.service.

Or we can start it on both nodes:

drbdadm up r0

And we can see it as inconsistent and both of them are secondary:

root@cl3-amd-node1:~# drbdadm status
r0 role:Secondary
  disk:Inconsistent
  cl3-amd-node2 role:Secondary
    peer-disk:Inconsistent

Start the initial full synchronization. This step must be performed on only one  node, only on initial resource configuration, and only on the node you selected as the synchronization source. To perform this step, issue this command:

root@cl3-amd-node1:# drbdadm primary --force r0

And we can see the status of our drbd storage:

root@cl3-amd-node2:~# drbdadm status
r0 role:Secondary
  disk:Inconsistent
  cl3-amd-node1 role:Primary
    replication:SyncTarget peer-disk:UpToDate done:3.10

After synchronization successfully finish, we set up our secondary server to be primary:

root@cl3-amd-node2:~# drbdadm status
r0 role:Secondary
  disk:UpToDate
  cl3-amd-node1 role:Primary
    peer-disk:UpToDate
root@cl3-amd-node2:~# drbdadm primary r0

And we can see status of this dual-primary (primary-primary) drbd storage resource:

root@cl3-amd-node2:~# drbdadm status
r0 role:Primary
  disk:UpToDate
  cl3-amd-node1 role:Primary
    peer-disk:UpToDate

Now we have a new block device on both servers:

root@cl3-amd-node2:~# fdisk -l /dev/drbd0
Disk /dev/drbd0: 931.4 GiB, 1000037986304 bytes, 1953199192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

We can configure this drbd block device as physical volume for lvm. This lvm is on top of this drbd. So, we can continue as it is a physical disk. Do it only on one server. The change will reflect on second server, due to primary-primary disk of drbd:

pvcreate /dev/drbd0
  Physical volume "/dev/drbd0" successfully created

As we can see, we must adapt /etc/lvm/lvm.conf to our needs, because it scans all block devices and we can found duplicate entries:

root@cl3-amd-node2:~# pvs
  Found duplicate PV WXwDGteoexfmLxN6GQvt6Nd3jJxgvT2z: using /dev/drbd0 not /dev/md1p1
  Found duplicate PV WXwDGteoexfmLxN6GQvt6Nd3jJxgvT2z: using /dev/md1p1 not /dev/drbd0
  Found duplicate PV WXwDGteoexfmLxN6GQvt6Nd3jJxgvT2z: using /dev/drbd0 not /dev/md1p1
  PV         VG   Fmt  Attr PSize   PFree
  /dev/drbd0      lvm2 ---  931.36g 931.36g
  /dev/md0   pve  lvm2 a--  931.38g      0

So, we must edit filter option in this configuration.  Look at our resouce configuration r0.res. We must exlude our backend devices (/dev/sdb1 on one server and /dev/md1p1 on second server), or we can reject all devices and allow only specific. I prefer reject all and allow only what we want. So edit the filter variable.

root@cl3-amd-node1:~# cat /etc/lvm/lvm.conf | grep drbd
     filter =[ "a|/dev/drbd0|", "a|/dev/sda3|", "r|.*|" ]
root@cl3-amd-node2:~# cat /etc/lvm/lvm.conf | grep drbd
    filter =[ "a|/dev/drbd0|", "a|/dev/md0|", "r|.*|" ]

Now, we don’t see duplicates and  we can create a volume group. Only on one server:

root@cl3-amd-node2:~# vgcreate drbd0-vg /dev/drbd0
  Volume group "drbd0-vg" successfully created
...
root@cl3-amd-node2:~# pvs
  PV         VG       Fmt  Attr PSize   PFree
  /dev/drbd0 drbd0-vg lvm2 a--  931.36g 931.36g
  /dev/md0   pve      lvm2 a--  931.38g      0

And finally we add the LVM group to the proxmox. It can be done via web interface. So, go to proxmox web interface to Datacenter, click on storage and add (LVM).
Then create your ID (this is the name of your storage. It can not be changed later. Maybe: drbd0-vg),  next you will see the previously created volume group drbd0-vg. So select it and enable the sharing by click the ‘shared’ box.
Now, we can create virtual machine on this LVM and when we can migrate it without downtime from one server to another because of drbd. There is one shared storage. So when the migration starts, machine is started on another server and through ssh tunnel is migrate content of ram. And after few seconds, it is started.
Sometimes, after some circumstances with network disconnect and connect, there is split-brain detected. So if this happened, don’t panic. When this happened, both servers are marked as “standalone” and drbd storage started to diverge. From this time there happened different writes to both sides. We must one of this servers mark as victim, because one of these servers has the “right” data and the other has “wrong” data. So the only way is backup the running virtuals on the “victim” and then we must destroy/discard this data on drbd storage and synchronize it from other server, which has “right” data. So if this is happening, this is in logs:

root@cl3-amd-node1:~# dmesg | grep -i brain
[499210.096185] drbd r0/0 drbd0 cl3-amd-node1: helper command: /sbin/drbdadm initial-split-brain
[499210.097306] drbd r0/0 drbd0 cl3-amd-node1: helper command: /sbin/drbdadm initial-split-brain exit code 0 (0x0)
[499210.097313] drbd r0/0 drbd0: Split-Brain detected but unresolved, dropping connection!

We must manually solve this problem. So I choose as victim: cl3-amd-node1. We must set this node as secondary:

drbdadm secondary r0

And now, we must disconnect it and connect it back with marking data to be discarded.

root@cl3-amd-node1:~# drbdadm connect --discard-my-data r0

And after synchronization, mark it back to primary node:

root@cl3-amd-node1:~# drbdadm primary r0

And in log, we can see:

cl3-amd-node1 kernel: [246882.068518] drbd r0/0 drbd0: Split-Brain detected, manually solved. Sync from peer node

Have fun.
 

27219total visits,111visits today