• Guest
HabraHabr
  • Main
  • Users

  • Development
    • Programming
    • Information Security
    • Website development
    • JavaScript
    • Game development
    • Open source
    • Developed for Android
    • Machine learning
    • Abnormal programming
    • Java
    • Python
    • Development of mobile applications
    • Analysis and design of systems
    • .NET
    • Mathematics
    • Algorithms
    • C#
    • System Programming
    • C++
    • C
    • Go
    • PHP
    • Reverse engineering
    • Assembler
    • Development under Linux
    • Big Data
    • Rust
    • Cryptography
    • Entertaining problems
    • Testing of IT systems
    • Testing Web Services
    • HTML
    • Programming microcontrollers
    • API
    • High performance
    • Developed for iOS
    • CSS
    • Industrial Programming
    • Development under Windows
    • Image processing
    • Compilers
    • FPGA
    • Professional literature
    • OpenStreetMap
    • Google Chrome
    • Data Mining
    • PostgreSQL
    • Development of robotics
    • Visualization of data
    • Angular
    • ReactJS
    • Search technologies
    • Debugging
    • Test mobile applications
    • Browsers
    • Designing and refactoring
    • IT Standards
    • Solidity
    • Node.JS
    • Git
    • LaTeX
    • SQL
    • Haskell
    • Unreal Engine
    • Unity3D
    • Development for the Internet of things
    • Functional Programming
    • Amazon Web Services
    • Google Cloud Platform
    • Development under AR and VR
    • Assembly systems
    • Version control systems
    • Kotlin
    • R
    • CAD/CAM
    • Customer Optimization
    • Development of communication systems
    • Microsoft Azure
    • Perfect code
    • Atlassian
    • Visual Studio
    • NoSQL
    • Yii
    • Mono и Moonlight
    • Parallel Programming
    • Asterisk
    • Yandex API
    • WordPress
    • Sports programming
    • Lua
    • Microsoft SQL Server
    • Payment systems
    • TypeScript
    • Scala
    • Google API
    • Development of data transmission systems
    • XML
    • Regular expressions
    • Development under Tizen
    • Swift
    • MySQL
    • Geoinformation services
    • Global Positioning Systems
    • Qt
    • Dart
    • Django
    • Development for Office 365
    • Erlang/OTP
    • GPGPU
    • Eclipse
    • Maps API
    • Testing games
    • Browser Extensions
    • 1C-Bitrix
    • Development under e-commerce
    • Xamarin
    • Xcode
    • Development under Windows Phone
    • Semantics
    • CMS
    • VueJS
    • GitHub
    • Open data
    • Sphinx
    • Ruby on Rails
    • Ruby
    • Symfony
    • Drupal
    • Messaging Systems
    • CTF
    • SaaS / S+S
    • SharePoint
    • jQuery
    • Puppet
    • Firefox
    • Elm
    • MODX
    • Billing systems
    • Graphical shells
    • Kodobred
    • MongoDB
    • SCADA
    • Hadoop
    • Gradle
    • Clojure
    • F#
    • CoffeeScript
    • Matlab
    • Phalcon
    • Development under Sailfish OS
    • Magento
    • Elixir/Phoenix
    • Microsoft Edge
    • Layout of letters
    • Development for OS X
    • Forth
    • Smalltalk
    • Julia
    • Laravel
    • WebGL
    • Meteor.JS
    • Firebird/Interbase
    • SQLite
    • D
    • Mesh-networks
    • I2P
    • Derby.js
    • Emacs
    • Development under Bada
    • Mercurial
    • UML Design
    • Objective C
    • Fortran
    • Cocoa
    • Cobol
    • Apache Flex
    • Action Script
    • Joomla
    • IIS
    • Twitter API
    • Vkontakte API
    • Facebook API
    • Microsoft Access
    • PDF
    • Prolog
    • GTK+
    • LabVIEW
    • Brainfuck
    • Cubrid
    • Canvas
    • Doctrine ORM
    • Google App Engine
    • Twisted
    • XSLT
    • TDD
    • Small Basic
    • Kohana
    • Development for Java ME
    • LiveStreet
    • MooTools
    • Adobe Flash
    • GreaseMonkey
    • INFOLUST
    • Groovy & Grails
    • Lisp
    • Delphi
    • Zend Framework
    • ExtJS / Sencha Library
    • Internet Explorer
    • CodeIgniter
    • Silverlight
    • Google Web Toolkit
    • CakePHP
    • Safari
    • Opera
    • Microformats
    • Ajax
    • VIM
  • Administration
    • System administration
    • IT Infrastructure
    • *nix
    • Network technologies
    • DevOps
    • Server Administration
    • Cloud computing
    • Configuring Linux
    • Wireless technologies
    • Virtualization
    • Hosting
    • Data storage
    • Decentralized networks
    • Database Administration
    • Data Warehousing
    • Communication standards
    • PowerShell
    • Backup
    • Cisco
    • Nginx
    • Antivirus protection
    • DNS
    • Server Optimization
    • Data recovery
    • Apache
    • Spam and antispam
    • Data Compression
    • SAN
    • IPv6
    • Fidonet
    • IPTV
    • Shells
    • Administering domain names
  • Design
    • Interfaces
    • Web design
    • Working with sound
    • Usability
    • Graphic design
    • Design Games
    • Mobile App Design
    • Working with 3D-graphics
    • Typography
    • Working with video
    • Work with vector graphics
    • Accessibility
    • Prototyping
    • CGI (graphics)
    • Computer Animation
    • Working with icons
  • Control
    • Careers in the IT industry
    • Project management
    • Development Management
    • Personnel Management
    • Product Management
    • Start-up development
    • Managing the community
    • Service Desk
    • GTD
    • IT Terminology
    • Agile
    • Business Models
    • Legislation and IT-business
    • Sales management
    • CRM-systems
    • Product localization
    • ECM / EDS
    • Freelance
    • Venture investments
    • ERP-systems
    • Help Desk Software
    • Media management
    • Patenting
    • E-commerce management
    • Creative Commons
  • Marketing
    • Conferences
    • Promotion of games
    • Internet Marketing
    • Search Engine Optimization
    • Web Analytics
    • Monetize Web services
    • Content marketing
    • Monetization of IT systems
    • Monetize mobile apps
    • Mobile App Analytics
    • Growth Hacking
    • Branding
    • Monetize Games
    • Display ads
    • Contextual advertising
    • Increase Conversion Rate
  • Sundry
    • Reading room
    • Educational process in IT
    • Research and forecasts in IT
    • Finance in IT
    • Hakatonas
    • IT emigration
    • Education abroad
    • Lumber room
    • I'm on my way

Expand the LINSTOR store for Proxmox

Expand the LINSTOR store for Proxmox
 
LINBIT recently released their new solution for orchestrating and managing multiple DRBD arrays.
 
For example, you can have several nodes and each will have its own LVM or ZFS pool in which LINSTOR will automatically create new volumes and replicate them between nodes using the DRBD protocol.
 
LINSTOR supports thin-provisioning, snapshots and many other interesting things.
 
This solution is well suited for virtual machines and containers.
 
LINSTOR Satellites
 
I believe that you already have a Proxmox cluster, I have three nodes pve1 , pve2 and pve3 .
 

So the first step is to install drbd-utils and collect the core module is on each node.


 

Let's add the linstor repository and install kernel-headers , as well as all the necessary packages from it:


 
    wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add -
echo "deb http://packages.linbit.com/proxmox/proxmox-5 drbd-9.0"
> /etc/apt/sources.list.d/linbit.list
apt-get update
apt-get -y install pve-headers
apt-get -y install drbd-dkms drbdtop

 

After installation, let's check your kernel module version:


 
    modproble drbd
cat /proc /drbd

 

If you see version ? then something went wrong and you booted the in-tree kernel module, check dkms status that would learn more about the assembly of the dkms module itself.
 
For LINSTOR, you must use the DRBD 9 version.


 

Next, on each node, set linstor-proxmox package:


 
    apt-get -y install linstor-proxmox
systemctl start linstor-satellite.service
systemctl enable linstor-satellite.service

 

LINSTOR Controller


 

We will deploy the controller inside the LXC container.


 

Download the debian-template:


 
    wget http://download.proxmox.com/images/system/debian-9.0-standard_9.3-1_amd64.tar.gz -P /var /lib /vz /template /cache /  

 

Now create the container for the controller:


 
    pct create 100 local: vztmpl /debian-9.0-standard_9.3-1_amd64.tar.gz
--hostname = linstor-controller
--net0 = name = eth? bridge = vmbr? gw = ???.? ip = ???.123 /16

 

Start the container and enter it:


 
    pct start 100
pct exec 100 bash

 

Install the updates:


 
    apt-get update
apt-get -y upgrade

 

Add the linstor repository and install linstor-controller and linstor-client :


 
    wget -O- https://packages.linbit.com/package-signing-pubkey.asc | apt-key add -
echo "deb http://packages.linbit.com/proxmox/proxmox-5 drbd-9.0"
> /etc/apt/sources.list.d/linbit.list
apt-get update && apt-get install -y linstor-server linstor-client
systemctl start linstor-controller.service
systemctl enable linstor-controller.service

 

Linstor requires a configured locale. Set up the locale:


 
    sed -i '/en_US.UTF-8 UTF-8 /s /^ # //' /etc/locale.gen
locale-gen

 

Let's immediately configure the time zone:


 
    dpkg-reconfigure tzdata    

 

Configuring the store


 

Let's create the nodes:


 
    linstor node create pve???.???r3r3468. linstor node create pve???.???r3r3468. linstor node create pve???.???r3r3461.  

 

For each node, we describe the additional interface that will be used for DRBD replication:


 
    linstor node interface create pve1 data ???.11
linstor node interface create pve2 data ???.12
linstor node interface create pve3 data ???.13

 

The output of the command is linstor node list :


 
    ╭───────────────────────────────────────────────────────────────────────────────────────────────────
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ pve1 ┊ SATELLITE ┊ ???.1????.11: 3366 (PLAIN) ┊ Online ┊
┊ pve2 ┊ SATELLITE ┊ ???.1????.12: 3366 (PLAIN) ┊ Online ┊
┊ pve3 ┊ SATELLITE ┊ ???.1????.13: 3366 (PLAIN) ┊ Online ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────

 

Let's create the pools:


 

On each node you must prepare lvm thin-pool for drbd:


 
    lvcreate -L 800G --thinpool drbdpool pve    

 

Now you can add them to linstor:


 
    linstor storage-pool create lvmthin pve1 drbdpool pve /drbdpool
linstor storage-pool create lvmthin pve2 drbdpool pve /drbdpool
linstor storage-pool create lvmthin pve3 drbdpool pve /drbdpool

 

We will configure the pools to use data interface:


 
    linstor storage-pool set-property pve1 drbdpool PrefNic data
linstor storage-pool set-property pve2 drbdpool PrefNic data
linstor storage-pool set-property pve3 drbdpool PrefNic data

 

The output of the command is linstor storage-pool list :


 
    ╭─────────────────────────────────────────────────────────────────────────────────────────────────── ─────────────────────────────────────────────────────────────────────────────────────────────────── ┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ Support
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
┊ drbdpool ┊ pve1 ┊ LvmThinDriver ┊ hv1 /drbdpool ┊ 800 GiB ┊ 800 GiB ┊ true
┊ drbdpool ┊ pve1 ┊ LvmThinDriver ┊ hv1 /drbdpool ┊ 800 GiB ┊ 800 GiB ┊ true
┊ drbdpool ┊ pve3 ┊ LvmThinDriver ┊ hv3 /drbdpool ┊ 800 GiB ┊ 800 GiB ┊ true
╰─────────────────────────────────────────────────────────────────────────────────────────────────── ─────────────────────────────────────────────────────────────────────────────────────────────────

 

Now we can add our new repository to the Proxmox config:


 
    cat /etc/pve/storage.cfg EOF
drbd: drbdpool
content rootdir, images
controller ???.123
controllervm 100
redundancy 3
EOF

 

HA setting for the

controller.
 

Now we'll move our linstor container to linstor storage.


 

First we need to create a virtual disk for it:


 
    pct exec 100 bash
linstor resource-definition create vm-100-disk-1
linstor volume-definition create vm-100-disk-1 4G
linstor resource create vm-100-disk-1 --auto-place 3 -s pve

 

The output of the command is linstor resource list :


 
    ╭────────────────────────────────────────────────────────────────────────────────────────────────────╮ ┊ ResourceName ┊ Node ┊ Port ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ vm-100-disk-1 ┊ pve1 ┊ 7000 ┊ UpToDate ┊
┊ vm-100-disk-1 ┊ pve2 ┊ 7000 ┊ UpToDate ┊
┊ vm-100-disk-1 ┊ pve3 ┊ 7000 ┊ UpToDate ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────╯

 

Default is linstor-satellite deletes all resource configurations at the time the service was started, then it expects a new configuration from linstor-controller , but if we keep linstor-controller in the same place as the data, we must make an exception for its resource by adding the option --keep-res = vm-100 for linstor-satellite on all nodes:


 
    echo -e "[Service]nExecStart = nExecStart = /usr /share /linstor-server /bin /Satellite -logs = /var /log /linstor-satellite --config-directory = /etc /linstor --keep-res = vm-100 "| SYSTEMD_EDITOR = tee systemctl edit linstor-satellite.service    

 

This action should prevent the resource from being deleted. vm-100-disk-1 at each start.


 

Now we will stop linstor-controller container and transfer all data from the local disk to the drbd-disk.


 
    pct shutdown 100
dd if = /var /lib /vz /images /100 /vm-100-disk-1.raw of = /dev /drbd /by-res /vm-100-disk-1/0 bs = 8M status = progress
. e2fsck -f /dev /drbd /by-res /vm-100-disk-1/0
resize2fs /dev /drbd /by-res /vm-100-disk-1/0

 

Update the container configuration:


 
    sed -i '/^ rootfs: /d' /etc/pve/lxc/100.conf
echo 'rootfs: drbdpool: vm-100-disk-? size = 4G' /etc/pve/lxc/100.conf

 

Run the container, and if everything is ok, delete the old disk:


 
    pct start 100
rm -f /var/lib/vz/images/100/vm-100-disk-1.raw

 

Now we just need to add our container to the proxmox ha-manager :


 
    ha-manager add ct: 100 --max_relocate = 3 --max_restart = 3    

 

PROFIT


 

Performance and tuning


 

To automatically resolve the situation with the split-brain, add the following options to the controller:


 
    linstor controller drbd-options
--after-sb-0pri = discard-zero-changes
--after-sb-1pri = discard-secondary
--after-sb-2pri = disconnect

 

For my 10G network, I found the following settings the most optimal for fast synchronization:


 
    linstor controller drbd-options
--max-buffers = 36864
--rcvbuf-size = 2097152
--sndbuf-size = 1048576
linstor controller drbd-options
--c-fill-target = 10240
- c-max-rate = 737280
-c-min-rate = 20480
--c-plan-ahead = 10

It may be interesting

  • Comments
  • About article
  • Similar news
Karl 3 January 2020 21:35
Hello,

and first thx for the great docu! I want run a home lap and have 2 physical HP DL G7 Server. So i want to use a Proxmox HA Cluster with this 2 Server and as 3rd Server a disk-less quorum device (Raspberry PI). Do you think that your configuration runs in this setup as well? Or did you have a better idea how i can run a HA Cluster with 2 server? The DRBD as in the "older" version´s is not supported anymore from Proxmox and the new Setup (Linstor) is not so easy (for me) to handle at this time.

Thx for Feedback!

Karl

weber

Author

13-09-2018, 01:29

Publication Date

Data Warehousing / Data storage / IT Infrastructure / *nix

Category
  • Comments: 1
  • Views: 1 303
Hypervisor for Dummies
We deploy the Kubernetes HA cluster on
Build a web-based server farm for
We collect InnoDB cluster from mysql
Reliable storage with DRBD9 and Proxmox
Illustrated guide to the device network
Write a comment
Name:*
E-Mail:


Comments
this is really nice to read..informative post is very good to read..thanks a lot! How is the cost of house cleaning calculated?
Yesterday, 17:14

Legend SEO

It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content.

entegrasyon programları
Yesterday, 17:09

taxiseo2

I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.

entegrasyon programları
Yesterday, 17:02

taxiseo2

I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work...먹튀

Yesterday, 16:50

raymond weber

Lose Weight Market provides the best fitness tips, workout guides, keto recipes and diet plans, yoga workout routine and plans, healthy recipes, and more! Check Out: Lose Weight Market


Corvus Health provides medical training services as well as recruiting high quality health workers for you or placing our own best team in your facility. Check Out: Health Workforce Recruitment




I.T HATCH offers a wide range of IT services including remote access setup, small business servers, data storage solutions, IT strategy services, and more. Check Out: IT strategy services
Yesterday, 22:33

noorseo

Adv
Website for web developers. New scripts, best ideas, programming tips. How to write a script for you here, we have a lot of information about various programming languages. You are a webmaster or a beginner programmer, it does not matter, useful articles will help to make your favorite business faster.

Login

Registration Forgot password