# Microservices: size matters, even if you have Kubernetes

3r? 3594. 3r3-31. September 19 in Moscow 3-3332. held 3r38080. The first thematic HUG (Highload ++ User Group) metap, which was dedicated to microservices. It included the report “Operation of microservices: size matters, even if you have Kubernetes”, in which we shared the extensive experience of the company “Flant” in the field of operating microservice architecture projects. First of all, it will be useful to all developers who are thinking about applying this approach in their present or future project. 3r33535. 3r? 3594. 3r33535. 3r? 3594.

3r? 3594. 3r33535. 3r? 3594. 3r? 3594. 3r33535. 3r? 3594. 3r? 3594. 3r33535. 3r? 3594. normal [/b] microservice architecture? It should bring you real benefits by increasing your work efficiency. If you go back to the schedule, here it is: 3r3353564. 3r? 3594. 3r33535. 3r? 3594. useful 3r37373. , then on the other side of the chart will be

very specific answer already mentioned Martin Fowler:

3r? 3594. 3r33535. 3r? 3594. 3r3124. 3r? 3594. 3r33578. clear limits of modularity; 3r33581. 3r? 3594. 3r33578. independent deployment; 3r33581. 3r? 3594. 3r33578. freedom of choice of technology. 3r33581. 3r? 3594.

3r33535. 3r? 3594. I talked a lot with architects and software developers and asked why they need microservices. And he made his list of their expectations. Here's what happened:

3r? 3594. 3r33535. 3r? 3594. 3r3142. 3r33535. 3r? 3594. 3r33535. 3r? 3594. If we describe “in sensations” some of the points, then: 3r3353564. 3r? 3594. 3r33535. 3r? 3594. 3r3566. 3r? 3594. 3r33578. clear borders of the modules: here we have a terrible monolith, and now everything will be neatly laid out in Git-repositories, in which everything is “on the shelves”, not warm and soft; 3r33581. 3r? 3594. 3r33578. Deployment independence: we will be able to roll out services independently, so that development goes faster (in parallel, publish new features); 3r33581. 3r? 3594. 3r33578. development independence: we can give this microservice to that team /developer, and that one is different, thanks to which we can develop it faster; 3r33581. 3r? 3594. 3r33578. b

3r33535. 3r? 3594. Video from the performance (~ 50 minutes; unfortunately, it does not convey the numerous emotions of visitors, which is largelydetermined the mood of the report, but as it is): 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r3502.

3r33590. 3r33590. 3r33590. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Presentation of the report: 3-333564. 3r? 3594. 3r33535. 3r? 3594. 3r3502.

3r3504. 3r33590. 3r33590. 3r33590. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r???. P.S. 3r33515. 3r33535. 3r? 3594. Other reports in our blog: 3r3353564. 3r? 3594. 3r33535. 3r? 3594. 3r3566. 3r? 3594. 3r33578. "3r33525. Monitoring and Kubernetes »3r33555. (Dmitry Stolyarov; May 2? 2018 on the RootConf) 3-353556. ; 3r33581. 3r? 3594. 3r33578. "3r33532. Best CI /CD practices with Kubernetes and GitLab

»3r33555. (Dmitry Stolyarov; November ? 2017 at HighLoad ++) 3r33556. ; 3r33581. 3r? 3594. 3r33578. “3r3-3539. Our experience with Kubernetes in small projects

»3r33555. (Dmitry Stolyarov; June ? 2017 at RootConf) 3r33556. ; 3r33581. 3r? 3594. 3r33578. "3r33546. We build Docker images for CI /CD quickly and conveniently along with dapp

»3r33555. (Dmitry Stolyarov; November ? 2016 at HighLoad ++) 3r33556. ; 3r33581. 3r? 3594. 3r33578. "3r33553. Practice Continuous Delivery with Docker

»3r33555. (Dmitry Stolyarov; May 3? 2016 at RootConf) 3r33556. . 3r33581. 3r? 3594. 3r?383. 3r33535. 3r? 3594. You will probably also be interested in the following publications: 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r3566. 3r? 3594. 3r33578. “3r3-3569. The death of microservice madness in 2018 3r33580. "; 3r33581. 3r? 3594. 3r33578. “3r3-3574. 7 best practices for the operation of containers according to Google

"; 3r33581. 3r? 3594. 3r33578. “3r3-3579. Statistics The New Stack on the difficulties of implementing Kubernetes

". 3r33581. 3r? 3594. 3r?383. 3r33590. 3r? 3594. 3r? 3594. 3r33587. ! function (e) {function t (t, n) {if (! (n in e)) {for (var r, a = e.document, i = a.scripts, o = i.length; o-- ;) if (-1! == i[o].src.indexOf (t)) {r = i[o]; break} if (! r) {r = a.createElement ("script"), r.type = "text /jаvascript", r.async =! ? r.defer =! ? r.src = t, r.charset = "UTF-8"; var d = function () {var e = a.getElementsByTagName ("script")[0]; e.parentNode.insertBefore (r, e)}; "[object Opera]" == e.opera? a.addEventListener? a.addEventListener ("DOMContentLoaded", d,! 1): e.attachEvent ("onload", d ): d ()}}} t ("//mediator.mail.ru/script/2820404/"""_mediator") () (); 3r33588. 3r? 3594. 3r33590. 3r? 3594. 3r? 3594. 3r? 3594. 3r? 3594.

**video with the report**3r? 3580. (50 minutes, much more informative than the article), as well as the main squeeze out of it in text form. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33555. NB: Videos and presentations are also available at the end of this publication. 3r33556. became 3r3805. Martin Fowler:3r? 3594. 3r33535. 3r? 3594. 3r? 3594. 3r33535. 3r? 3594. 3r? 3594. 3r33535. 3r? 3594. normal [/b] microservice architecture? It should bring you real benefits by increasing your work efficiency. If you go back to the schedule, here it is: 3r3353564. 3r? 3594. 3r33535. 3r? 3594. useful 3r37373. , then on the other side of the chart will be

**unhealthy**microservice (interferes with work): 3r33564. 3r? 3594. 3r33535. 3r? 3594. 85 projects 3r3473. . Not all of them were microservice (approximately one third to one half of them had such architecture), but this is still a large number. We (the company "Flant") as outsourcers manage to see a wide variety of applications developed both in small companies (with 5 developers) and large ones (~ 500 developers). An additional advantage is that we see how these applications live and develop over the years. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r???. Why microservices? 3r33515. 3r33535. 3r? 3594. When asked about the benefits of microservices, there isvery specific answer already mentioned Martin Fowler:

3r? 3594. 3r33535. 3r? 3594. 3r3124. 3r? 3594. 3r33578. clear limits of modularity; 3r33581. 3r? 3594. 3r33578. independent deployment; 3r33581. 3r? 3594. 3r33578. freedom of choice of technology. 3r33581. 3r? 3594.

3r33535. 3r? 3594. I talked a lot with architects and software developers and asked why they need microservices. And he made his list of their expectations. Here's what happened:

3r? 3594. 3r33535. 3r? 3594. 3r3142. 3r33535. 3r? 3594. 3r33535. 3r? 3594. If we describe “in sensations” some of the points, then: 3r3353564. 3r? 3594. 3r33535. 3r? 3594. 3r3566. 3r? 3594. 3r33578. clear borders of the modules: here we have a terrible monolith, and now everything will be neatly laid out in Git-repositories, in which everything is “on the shelves”, not warm and soft; 3r33581. 3r? 3594. 3r33578. Deployment independence: we will be able to roll out services independently, so that development goes faster (in parallel, publish new features); 3r33581. 3r? 3594. 3r33578. development independence: we can give this microservice to that team /developer, and that one is different, thanks to which we can develop it faster; 3r33581. 3r? 3594. 3r33578. b

*about 3r?556. Greater reliability: if partial degradation happens (one microservice of 20 drops), then only one button will stop working, and the system as a whole will continue to function. 3r33581. 3r? 3594. 3r?383. 3r33535. 3r? 3594. 3r???. Typical (harmful) microservice architecture 3r33535. 3r33535. 3r? 3594. To explain why, in reality, things are not as we expect, I will introduce 3r-3555. collectively 3r33556. image of microservice architecture, based on the experience of many different projects. 3r33535. 3r? 3594. 3r33535. 3r? 3594. An example would be an abstract online store that is going to compete with Amazon, or at least OZON. Its microservice architecture looks like this:*

3r? 3594. 3r33535. 3r? 3594. 3r3184. 3r33535. 3r? 3594. 3r33535. 3r? 3594. For a combination of reasons, these microservices are written on different platforms: 3r-3564. 3r? 3594. 3r33535. 3r? 3594. 3r3193. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Since each microservice should have autonomy, many of them need their own database and cache. The final architecture is obtained as follows: 3r33564. 3r? 3594. 3r33535. 3r? 3594. 3r3202. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r???. What are its consequences? 3r33515. 3r33535. 3r? 3594. At Fowler'a and on this account There is an article 3r38080. - about “payback” for using microservices: 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33535. 3r? 3594. But

3r? 3594. 3r33535. 3r? 3594. 3r3402. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Combine all microservices responsible for front-end generation: 3r-3564. 3r? 3594. 3r33535. 3r? 3594. 3r33411. 3r33535. 3r? 3594. 3r33535. 3r? 3594. into one microservice, written in one (modern and normal, as you yourself think) language /framework:

3r? 3594. 3r33535. 3r? 3594. 3r33420. 3r33535. 3r? 3594. 3r33535. 3r? 3594. It will have one ORM (one DBMS) and first a couple of applications: 3r-3564. 3r? 3594. 3r33535. 3r? 3594. 3r33434. 3r33535. 3r? 3594. 3r33535. 3r? 3594. but in general there can be transferred much more, having received such a result: 3r33535. 3r? 3594. 3r33535. 3r? 3594. In the word “microservices”, the part “micro” is superfluous 3r-3473. . They are “micro” only for the reason that they are smaller than a huge monolith. But do not think of them as something small. 3r33535. 3r? 3594. 3r33535. 3r? 3594. And for the final thought let us return to the original schedule: 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3rr3465. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Written note for him [i] (right above)comes down to the fact that3r? 3594. 3r33535. 3r? 3594. 3r3184. 3r33535. 3r? 3594. 3r33535. 3r? 3594. For a combination of reasons, these microservices are written on different platforms: 3r-3564. 3r? 3594. 3r33535. 3r? 3594. 3r3193. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Since each microservice should have autonomy, many of them need their own database and cache. The final architecture is obtained as follows: 3r33564. 3r? 3594. 3r33535. 3r? 3594. 3r3202. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r???. What are its consequences? 3r33515. 3r33535. 3r? 3594. At Fowler'a and on this account There is an article 3r38080. - about “payback” for using microservices: 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33535. 3r? 3594. But

**how many microservices we really need to fix 3r33473. to roll out the change? Can we even figure out how everything works without a distributed tracer (after all, any request is processed by half of microservices)? 3r33535. 3r? 3594. 3r33535. 3r? 3594. There is a pattern “**

a large clump of dirta large clump of dirt

**", And here and did get a distributed clump of dirt. In confirmation of this - here is a rough illustration of how requests go:**

3r? 3594. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Technically, it has been achieved: we can roll each microservice separately. But in practice, it should be borne in mind that

3r33535. 3r? 3594. She is. Only it is worth remembering that often freedom borders on lawlessness. It is very important here not to choose technologies only in order to “play” with them. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33333. Independence of development

3r33535. 3r? 3594. How to make a test loop for the entire application (from so many components)? But still need to keep it up to date. All this leads to the fact that

3r33535. 3r? 3594. Yes, but it is limited in the field of the used DBMS. In the example architecture, Cassandra will have no problems, but MySQL and PostgreSQL will. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33333. Used 3r33555. about 3r?556. more reliability

3r33535. 3r? 3594. Not only that, in fact, the disabling of one microservice often breaks the correct functioning of the entire system, so there is also a new problem:

3r33535. 3r? 3594. This is really all good. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33333. “Lightness” of microservices 3r33333. 3r33535. 3r? 3594. Not only have we got a huge! 3r33535. 3r? 3594. 3r33535. 3r? 3594. For example, for the collective image considered above 3r33535. 3r? 3594. 3r33535. 3r? 3594. Get rid of the most dubious microservices:3r? 3594. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Technically, it has been achieved: we can roll each microservice separately. But in practice, it should be borne in mind that

**always rolls out. a lot of microservices**, and we need to consider**order them rolled out**. In an amicable way, we generally need to test in a separate circuit, whether we roll out the release in the correct order. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33333. Freedom of choice of technology3r33535. 3r? 3594. She is. Only it is worth remembering that often freedom borders on lawlessness. It is very important here not to choose technologies only in order to “play” with them. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33333. Independence of development

3r33535. 3r? 3594. How to make a test loop for the entire application (from so many components)? But still need to keep it up to date. All this leads to the fact that

**The real number of test circuits is**which in principle we can contain,**turns out to be minimal**. 3r33535. 3r? 3594. 3r33535. 3r? 3594. And to deploy all this locally? It turns out that often the developer does his work independently, but “at random”, because he has to wait for the circuit to be released for testing. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33333. Separate scaling3r33535. 3r? 3594. Yes, but it is limited in the field of the used DBMS. In the example architecture, Cassandra will have no problems, but MySQL and PostgreSQL will. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33333. Used 3r33555. about 3r?556. more reliability

3r33535. 3r? 3594. Not only that, in fact, the disabling of one microservice often breaks the correct functioning of the entire system, so there is also a new problem:

**it is very difficult to make fault-tolerant every microservice**. Because microservices use different technologies (memcache, Redis, etc.), everyone needs to think over and implement everything, which, of course, is possible, but requires huge resources. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33333. Load measurability3r33535. 3r? 3594. This is really all good. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r33333. “Lightness” of microservices 3r33333. 3r33535. 3r? 3594. Not only have we got a huge

**network overhead**(multiplies queries for DNS, etc.), but also because of the many subqueries we started**Replicate data**(store caches), which resulted in a significant amount of storage. 3r33535. 3r? 3594. 3r33535. 3r? 3594. And here is the result of meeting our expectations: 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r? 3594. 3r33535. 3r? 3594. 3r3566. 3r? 3594. 3r33578. Most likely we need a message bus. 3r33581. 3r? 3594. 3r33578. How to make a consistent backup at the right time? The only 3r35555. real 3r33556. option - turn off traffic for this. But how to do it in production? 3r33581. 3r? 3594. 3r33578. If we are talking about supporting several regions, then organizing sustainability in each of them is a very time consuming task. 3r33581. 3r? 3594. 3r33578. There is a problem of making centralized changes. For example, if we need to update the version of PHP, then we need to make a commit to each repository (there are dozens of them). 3r33581. 3r? 3594. 3r33578. The growth of operational complexity offhand turns out to be exponential. 3r33581. 3r? 3594. 3r?383. 3r33535. 3r? 3594. 3r???. What to do with all this? 3r33515. 3r33535. 3r? 3594.**Start with a monolithic application**. Experience Fowler'a says 3r38080. that almost all successful microservice applications began with a monolith, which became too large, after which it was broken. At the same time, almost all systems built as microservices from the very beginning, sooner or later experienced serious problems. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Another valuable thought is that for a microservice architecture project to be successful, you should know very well**and the subject area, and how to do microservices**. And the best way to know the subject area is to make a monolith. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r???. But what if we are already in this situation? 3r33515. 3r33535. 3r? 3594. The first step to solving any problem is to agree with it and understand that this is a problem, that we no longer want to suffer. 3r33535. 3r? 3594. 3r33535. 3r? 3594. If in the case of an overgrown monolith (when we’ve run out of the opportunity to re-buy resources for it), we cut it, then in this case we get the opposite story: when excessive microservice doesn’t help, but hinders - 3r3343472. cut off excess and enlarge3r? 3594. 3r33535. 3r? 3594. 3r3402. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Combine all microservices responsible for front-end generation: 3r-3564. 3r? 3594. 3r33535. 3r? 3594. 3r33411. 3r33535. 3r? 3594. 3r33535. 3r? 3594. into one microservice, written in one (modern and normal, as you yourself think) language /framework:

3r? 3594. 3r33535. 3r? 3594. 3r33420. 3r33535. 3r? 3594. 3r33535. 3r? 3594. It will have one ORM (one DBMS) and first a couple of applications: 3r-3564. 3r? 3594. 3r33535. 3r? 3594. 3r33434. 3r33535. 3r? 3594. 3r33535. 3r? 3594. but in general there can be transferred much more, having received such a result: 3r33535. 3r? 3594. 3r33535. 3r? 3594. In the word “microservices”, the part “micro” is superfluous 3r-3473. . They are “micro” only for the reason that they are smaller than a huge monolith. But do not think of them as something small. 3r33535. 3r? 3594. 3r33535. 3r? 3594. And for the final thought let us return to the original schedule: 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3rr3465. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Written note for him [i] (right above)

**the skills of the team that makes your project are always primary**- they will play a key role in your choice between microservices and monolith. If a team does not have enough skills, but it starts doing microservices, the story will definitely be fatal. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r???. Videos and slides3r33535. 3r? 3594. Video from the performance (~ 50 minutes; unfortunately, it does not convey the numerous emotions of visitors, which is largelydetermined the mood of the report, but as it is): 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r3502.

3r33590. 3r33590. 3r33590. 3r33535. 3r? 3594. 3r33535. 3r? 3594. Presentation of the report: 3-333564. 3r? 3594. 3r33535. 3r? 3594. 3r3502.

3r3504. 3r33590. 3r33590. 3r33590. 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r???. P.S. 3r33515. 3r33535. 3r? 3594. Other reports in our blog: 3r3353564. 3r? 3594. 3r33535. 3r? 3594. 3r3566. 3r? 3594. 3r33578. "3r33525. Monitoring and Kubernetes »3r33555. (Dmitry Stolyarov; May 2? 2018 on the RootConf) 3-353556. ; 3r33581. 3r? 3594. 3r33578. "3r33532. Best CI /CD practices with Kubernetes and GitLab

»3r33555. (Dmitry Stolyarov; November ? 2017 at HighLoad ++) 3r33556. ; 3r33581. 3r? 3594. 3r33578. “3r3-3539. Our experience with Kubernetes in small projects

»3r33555. (Dmitry Stolyarov; June ? 2017 at RootConf) 3r33556. ; 3r33581. 3r? 3594. 3r33578. "3r33546. We build Docker images for CI /CD quickly and conveniently along with dapp

»3r33555. (Dmitry Stolyarov; November ? 2016 at HighLoad ++) 3r33556. ; 3r33581. 3r? 3594. 3r33578. "3r33553. Practice Continuous Delivery with Docker

»3r33555. (Dmitry Stolyarov; May 3? 2016 at RootConf) 3r33556. . 3r33581. 3r? 3594. 3r?383. 3r33535. 3r? 3594. You will probably also be interested in the following publications: 3r33535. 3r? 3594. 3r33535. 3r? 3594. 3r3566. 3r? 3594. 3r33578. “3r3-3569. The death of microservice madness in 2018 3r33580. "; 3r33581. 3r? 3594. 3r33578. “3r3-3574. 7 best practices for the operation of containers according to Google

"; 3r33581. 3r? 3594. 3r33578. “3r3-3579. Statistics The New Stack on the difficulties of implementing Kubernetes

". 3r33581. 3r? 3594. 3r?383. 3r33590. 3r? 3594. 3r? 3594. 3r33587. ! function (e) {function t (t, n) {if (! (n in e)) {for (var r, a = e.document, i = a.scripts, o = i.length; o-- ;) if (-1! == i[o].src.indexOf (t)) {r = i[o]; break} if (! r) {r = a.createElement ("script"), r.type = "text /jаvascript", r.async =! ? r.defer =! ? r.src = t, r.charset = "UTF-8"; var d = function () {var e = a.getElementsByTagName ("script")[0]; e.parentNode.insertBefore (r, e)}; "[object Opera]" == e.opera? a.addEventListener? a.addEventListener ("DOMContentLoaded", d,! 1): e.attachEvent ("onload", d ): d ()}}} t ("//mediator.mail.ru/script/2820404/"""_mediator") () (); 3r33588. 3r? 3594. 3r33590. 3r? 3594. 3r? 3594. 3r? 3594. 3r? 3594.