Expand the LINSTOR store for Proxmox
LINBIT recently released their new solution for orchestrating and managing multiple DRBD arrays.
For example, you can have several nodes and each will have its own LVM or ZFS pool in which LINSTOR will automatically create new volumes and replicate them between nodes using the DRBD protocol.
LINSTOR supports thin-provisioning, snapshots and many other interesting things.
This solution is well suited for virtual machines and containers.
LINSTOR Satellites
I believe that you already have a Proxmox cluster, I have three nodes pve1
, pve2
and pve3
.
So the first step is to install drbd-utils and collect the core module is on each node.
Let's add the linstor repository and install kernel-headers , as well as all the necessary packages from it:
wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add -
echo "deb http://packages.linbit.com/proxmox/proxmox-5 drbd-9.0"
> /etc/apt/sources.list.d/linbit.list
apt-get update
apt-get -y install pve-headers
apt-get -y install drbd-dkms drbdtop
After installation, let's check your kernel module version:
modproble drbd
cat /proc /drbd
If you see version ? then something went wrong and you booted the in-tree kernel module, check dkms status
that would learn more about the assembly of the dkms module itself.
For LINSTOR, you must use the DRBD 9 version.
Next, on each node, set linstor-proxmox package:
apt-get -y install linstor-proxmox
systemctl start linstor-satellite.service
systemctl enable linstor-satellite.service
LINSTOR Controller
We will deploy the controller inside the LXC container.
Download the debian-template:
wget http://download.proxmox.com/images/system/debian-9.0-standard_9.3-1_amd64.tar.gz -P /var /lib /vz /template /cache /
Now create the container for the controller:
pct create 100 local: vztmpl /debian-9.0-standard_9.3-1_amd64.tar.gz
--hostname = linstor-controller
--net0 = name = eth? bridge = vmbr? gw = ???.? ip = ???.123 /16
Start the container and enter it:
pct start 100
pct exec 100 bash
Install the updates:
apt-get update
apt-get -y upgrade
Add the linstor repository and install linstor-controller and linstor-client :
wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add -
echo "deb http://packages.linbit.com/proxmox/proxmox-5 drbd-9.0"
> /etc/apt/sources.list.d/linbit.list
apt-get update && apt-get install -y linstor-server linstor-client
systemctl start linstor-controller.service
systemctl enable linstor-controller.service
Linstor requires a configured locale. Set up the locale:
sed -i '/en_US.UTF-8 UTF-8 /s /^ # //' /etc/locale.gen
locale-gen
Let's immediately configure the time zone:
dpkg-reconfigure tzdata
Configuring the store
Let's create the nodes:
linstor node create pve???.???r3r3468. linstor node create pve???.???r3r3468. linstor node create pve???.???r3r3461.
For each node, we describe the additional interface that will be used for DRBD replication:
linstor node interface create pve1 data ???.11
linstor node interface create pve2 data ???.12
linstor node interface create pve3 data ???.13
The output of the command is linstor node list
:
╭───────────────────────────────────────────────────────────────────────────────────────────────────
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ pve1 ┊ SATELLITE ┊ ???.1????.11: 3366 (PLAIN) ┊ Online ┊
┊ pve2 ┊ SATELLITE ┊ ???.1????.12: 3366 (PLAIN) ┊ Online ┊
┊ pve3 ┊ SATELLITE ┊ ???.1????.13: 3366 (PLAIN) ┊ Online ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────
Let's create the pools:
On each node you must prepare lvm thin-pool for drbd:
lvcreate -L 800G --thinpool drbdpool pve
Now you can add them to linstor:
linstor storage-pool create lvmthin pve1 drbdpool pve /drbdpool
linstor storage-pool create lvmthin pve2 drbdpool pve /drbdpool
linstor storage-pool create lvmthin pve3 drbdpool pve /drbdpool
We will configure the pools to use data
interface:
linstor storage-pool set-property pve1 drbdpool PrefNic data
linstor storage-pool set-property pve2 drbdpool PrefNic data
linstor storage-pool set-property pve3 drbdpool PrefNic data
The output of the command is linstor storage-pool list
:
╭─────────────────────────────────────────────────────────────────────────────────────────────────── ─────────────────────────────────────────────────────────────────────────────────────────────────── ┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ Support
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
┊ drbdpool ┊ pve1 ┊ LvmThinDriver ┊ hv1 /drbdpool ┊ 800 GiB ┊ 800 GiB ┊ true
┊ drbdpool ┊ pve1 ┊ LvmThinDriver ┊ hv1 /drbdpool ┊ 800 GiB ┊ 800 GiB ┊ true
┊ drbdpool ┊ pve3 ┊ LvmThinDriver ┊ hv3 /drbdpool ┊ 800 GiB ┊ 800 GiB ┊ true
╰─────────────────────────────────────────────────────────────────────────────────────────────────── ─────────────────────────────────────────────────────────────────────────────────────────────────
Now we can add our new repository to the Proxmox config:
cat /etc/pve/storage.cfg EOF
drbd: drbdpool
content rootdir, images
controller ???.123
controllervm 100
redundancy 3
EOF
HA setting for the
controller.Now we'll move our linstor container to linstor storage.
First we need to create a virtual disk for it:
pct exec 100 bash
linstor resource-definition create vm-100-disk-1
linstor volume-definition create vm-100-disk-1 4G
linstor resource create vm-100-disk-1 --auto-place 3 -s pve
The output of the command is linstor resource list
:
╭────────────────────────────────────────────────────────────────────────────────────────────────────╮ ┊ ResourceName ┊ Node ┊ Port ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ vm-100-disk-1 ┊ pve1 ┊ 7000 ┊ UpToDate ┊
┊ vm-100-disk-1 ┊ pve2 ┊ 7000 ┊ UpToDate ┊
┊ vm-100-disk-1 ┊ pve3 ┊ 7000 ┊ UpToDate ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────╯
Default is linstor-satellite deletes all resource configurations at the time the service was started, then it expects a new configuration from linstor-controller , but if we keep linstor-controller in the same place as the data, we must make an exception for its resource by adding the option --keep-res = vm-100
for linstor-satellite on all nodes:
echo -e "[Service]nExecStart = nExecStart = /usr /share /linstor-server /bin /Satellite -logs = /var /log /linstor-satellite --config-directory = /etc /linstor --keep-res = vm-100 "| SYSTEMD_EDITOR = tee systemctl edit linstor-satellite.service
This action should prevent the resource from being deleted. vm-100-disk-1
at each start.
Now we will stop linstor-controller container and transfer all data from the local disk to the drbd-disk.
pct shutdown 100
dd if = /var /lib /vz /images /100 /vm-100-disk-1.raw of = /dev /drbd /by-res /vm-100-disk-1/0 bs = 8M status = progress
. e2fsck -f /dev /drbd /by-res /vm-100-disk-1/0
resize2fs /dev /drbd /by-res /vm-100-disk-1/0
Update the container configuration:
sed -i '/^ rootfs: /d' /etc/pve/lxc/100.conf
echo 'rootfs: drbdpool: vm-100-disk-? size = 4G' /etc/pve/lxc/100.conf
Run the container, and if everything is ok, delete the old disk:
pct start 100
rm -f /var/lib/vz/images/100/vm-100-disk-1.raw
Now we just need to add our container to the proxmox ha-manager :
ha-manager add ct: 100 --max_relocate = 3 --max_restart = 3
PROFIT
Performance and tuning
To automatically resolve the situation with the split-brain, add the following options to the controller:
linstor controller drbd-options
--after-sb-0pri = discard-zero-changes
--after-sb-1pri = discard-secondary
--after-sb-2pri = disconnect
For my 10G network, I found the following settings the most optimal for fast synchronization:
linstor controller drbd-options
--max-buffers = 36864
--rcvbuf-size = 2097152
--sndbuf-size = 1048576
linstor controller drbd-options
--c-fill-target = 10240
- c-max-rate = 737280
-c-min-rate = 20480
--c-plan-ahead = 10
It may be interesting
weber
Author13-09-2018, 01:29
Publication DateData Warehousing / Data storage / IT Infrastructure / *nix
Category- Comments: 1
- Views: 1 303
entegrasyon programları
entegrasyon programları
Corvus Health provides medical training services as well as recruiting high quality health workers for you or placing our own best team in your facility. Check Out: Health Workforce Recruitment
I.T HATCH offers a wide range of IT services including remote access setup, small business servers, data storage solutions, IT strategy services, and more. Check Out: IT strategy services