3r3-31. 3r3688.  
3r3688. Velocity is a conference dedicated to distributed systems. It is organized by O'Reilly, and it takes place three times a year: once in California, once in New York and once in Europe (and the city changes every year). 3r3690.
 
3r3688. In 201? the conference was in London from October 30 to November 2. The main office of Badoo is in the same place, so my colleagues and I had just two reasons to go to Velocity. 3r3690.
 
3r3688. Her device turned out to be somewhat more complicated than the one I encountered at Russian conferences. In addition to the rather familiar two days of reports, there were two more days of training, which can be taken in full, in part or not at all. All together it turns into a serious quest for choosing the type of ticket you need. 3r3690.
 
3r3688. In this review I will tell about those reports and master classes that I remember. I attach references to additional materials to some reports. Partly these are materials the authors referred to, and partly materials for further study that I found myself. 3r3690. official twitter account during the conference there were a lot of operational retweets with reports. This allowed us to see what was happening in other halls. 3r3690.
 
Workshops
 
3r3688. October 31 was the day when there were no reports, but six or eight master classes took place, three hours of pure time each, of which two had to be chosen. 3r3690.
 
3r3688. P.S. In the original they are called the tutorial, but it seems to me correct to translate them as a "master class". 3r3690.
 
Chaos Engineering Bootcamp
 
3r3688. Host: 3r3358. Ana Medina
, engineer in the company 3r360. Gremlin
| 3r362. Description
3r3690.
 
3r3688. The workshop was dedicated to introducing chaos engineering. Ana spoke fluently about what it is, what benefits it brings, demonstrated how it can be used, which software can help and how to start using it in the company. 3r3690.
 
3r3688. Overall, it was a good introduction for beginners, but I didn’t really like the practical part, which was the deployment of a demo web application in a cluster of several machines using Kubernetes and screwing monitoring on it from 3r3722. DataDog 3r3676. . The main problem was that we spent almost half the time of the master class on this and this was only needed in order to play around for 5-10 minutes with scripts that emulate various problems in the cluster and look at the changes in the graphs. 3r3690.
 
3r3688. It seems to me that for the same effect it was enough to give access to a pre-configured DataDog and /or demonstrate it all from the scene, and this time to spend, for example, on a more detailed review and examples of using the same Chaos Monkey, about which it was just literally told a couple of phrases. 3r3690.
 
3r3688.  
3r3688.  
3r3602. Interesting: at this conference, speakers often mentioned the term "blast radius", which I had not met before. They were designated a part of the system that turns out to be off when a specific problem occurs.
3r3688. Additional materials: 3r39090.
 
 
3r3634. 3r3-300. Chaos Engineering: The History, Principles, and Practice
3r3637.  
3r3634. 3r3105. Chaos Monkey Guide for Engineers
3r3637.  
3r3634. 3r31-10. A repository with scripts to emulate problems in the system 3r3676. (Scripts were used in the master class and there are also links to the presentation from the same master class)
 
3r3634. 3r3115. Chaos Engineering Monitoring & Metrics Guide
3r3637.  
3r3634. Planning Your Own Chaos Day 3r3637.  
 
Building evolutionary infrastructure
 
3r3688. Host: Kief Morris , infrastructure consultant and author of the book 3r3133. "Infrastructure as a code"
| Description 3r3690.
 
3r3688. The main theses of the master class can be reduced to two things: 3r-33690.
 
3r3145.  
3r3634. Systems change all the time, so it’s normal that the infrastructure also needs to change; 3r3637.  
3r3634. Once the infrastructure is changing, you need to ensure that it is simple and safe, and this can be achieved only by automation. 3r3637.  
 
3r3688. The main part of his story was devoted precisely to automating changes in infrastructure, possible solutions to this problem, and testing changes. I am not an expert in this topic, but it seemed to me that he spoke very confidently and in detail (and very quickly). 3r3690.
 
3r3688. The main point that I remember from this master class is the recommendation to make the most distinction between environments (production, steaming, etc.) from the code into environment variables. This will reduce the likelihood of errors in the infrastructure when changing environments and make it more testable. 3r3690.
 
3r3688. 3r3165. 3r3690.
 
3r3688. 3r33170. 3r3690.
 
3r3688. 3r33175. 3r3690.
 
3r3179. Reports 3–3–380.
 
3r3688. November 1 and 2 were the days of reports. They were divided into two principal blocks: a series of three or four short keynote reports, which went in the morning in one stream (and a large hall of two smaller ones was set up for them) and longer thematic reports in five streams, which went the rest of the day . During the day there were several big pauses between the reports, when you could walk around the expo with the stands of the conference partners. 3r3690.
 
3r3187. Evolution of Runtastic Backend
 
3r3688. Simon Lasselsberger (Runtastic GmbH) | 3r3192. Description and slides
3r3690.
 
3r3688. One of the few reports in which the author did not just tell how something needs to be done, but showed the details of a specific project and what happened with it. 3r3690.
 
3r3688. In the beginning, Runtastic had a common Percona Server database and a monolith with code serving mobile applications and the site. Then they began to write in Cassandra (I do not remember why it was in it) a part of the data for which the key-value of the repository was sufficient. Gradually, the base was plump, and they added MongoDB, to which they began to write data from most services. Over time, they made a general level that serves requests from both the web and mobile applications (something like our Apification, r3r3676. As I understand it). 3r3690.
 
3r3688. 3r3208. 3r3690.
 
3r3688. Most of the report was devoted to moving between data centers. At first they kept the server in Hetzner, which after some time was considered not stable enough and the data was migrated to T-Systems. And a few years later they were faced with a shortage of space already there and moved again to Linz AG. The most interesting part here is data migration. They started copying data, which lasted several months. They could not wait so long, because they ran out of space and could not add it, so they made a fallback in the code that tried to read data from the old data center if it was not in the new one. 3r3690.
 
3r3688. In the future, they plan to split the data into several separate data centers (Simon said several times that this is necessary for Russia and China) and to strictly separate the databases for individual services (the common pool is now used for all services). 3r3690.
 
3r3688. A curious approach to the design of modules in the system, about which Simon casually mentioned:
hexagonal architecture
. 3r3690.
 
3r3602. It is a program that will allow you to use it.
 
 
Alistair Cockburn
3r3688. Additional materials: 3r39090.
 
 
3r3634.
The presentation of the same report from another conference 3-33676. 3r3637.  
3r3634.
Hexagonal Architecture 3r3637.  
 
Monitoring custom metrics; or how I learned later on
 
3r3688. Maxime Petazzoni (SignalFx) | Description and presentation of 3r3690.
 
3r3688. The story was devoted to collecting metrics necessary for understanding the operation of the application. The main message was that the usual RED metrics (Rate, Errors, and Duration) are absolutely not enough, and besides them, you need to immediately assemble others that will help you understand what is happening inside the application. 3r3690.
 
3r3688. Thesis, the author proposed to collect counters and timers for some important actions in the system (and failures counters), build graphs and distribution histograms on them, define a meta model for custom metrics (so that different metrics have the same set of required parameters and the same meanings are called the same everywhere). 3r3690.
 
3r3688.  
3r3688. Words to retell the details is hard enough, it will be easier to see the details and examples in the presentation, the link to which is on the report page on the conference website. 3r3690.
 
3r3688. Additional materials: 3r39090.
 
 
3r3634. Monitoring and Observability with USE and RED 3r3637.  
 
How serverless changes the IT department
 
3r3688. Paul Johnston (Roundabout Labs) | Description and presentation of 3r3690.
 
3r3688. The author introduced himself as a CTO and environmentalist, and said that serverless is not a technological, but a business decision ("You pay nothing if it's unused"). Then he described the best practices for working with serverless, what competencies are needed to work with him and how it affects the choice of new employees and work with existing ones. 3r3690.
 
3r3688. 3r3307. 3r3690.
 
3r3688. The key point of the "influence on the IT department" that I remembered was the shift of the necessary competencies from simply writing code to working with the infrastructure and its automation ("More" engineering "than" developing "). Everything else was pretty trite (you need to constantly code review, to document the data streams and events available for use in the system, to communicate more and learn quickly), but for some reason the author attributed them to the features of serverless. 3r36903. 3r3686.  
3r33333.
 
3r3688. Overall, the report seemed a bit ambiguous. Many things that the speaker was talking about can be attributed to any complex system that does not fit entirely in the head. 3r3690.
 
3r3688. Additional materials: 3r39090.
 
 
3r3634. Serverless Best Practices - article of the author with disclosure of best practices
 
 
3r33336. Don't panic! How to cope now
 
3r3688. Euan Finlay (Financial Times) | 3r33333. Description and presentation of
3r3690.
 
3r3688. A report on how to deal with production incidents if something goes wrong right now. The main theses were divided into parts by time. 3r3690.
 
3r3688. Before the incident:
 
 
3r3634. delimit alerts on criticality - perhaps some may wait, and they do not need to be urgently dealt with; 3r3637.  
3r3634. Prepare a plan for incident analysis in advance and keep the documentation up to date; 3r3637.  
3r3634. conduct exercises - break something and see what happens (aka chaos engineering); 3r3637.  
3r3634. get a single place where all information about changes and problems flows. 3r3637.  
 
3r3688. During the incident:
 
 
3r3634. it’s normal that you don’t know everything — get other people involved if necessary; 3r3637.  
3r3634. create a single place for communication between people working on solving the incident; 3r3637.  
3r3634. Look for the simplest solution that will bring production back to working condition, rather than trying to completely solve the problem. 3r3637.  
 
3r3688. After the incident:
 
 
3r3634. Understand why the problem arose and what it taught you; 3r3637.  
3r3634. it is important to write a report about this ("incident report"); 3r3637.  
3r3634. Determine what can be improved and plan specific actions. 3r3637.  
 
3r3688. In the end, Yuen told the funny story of the incident in the Financial Times, which arose from the fact that the production base (which was called 3r31313) was mistakenly modified prod
) instead of pre-production (3r3363613. pprod 3r3-3614.), and advised to avoid such similar names. 3r3690.
 
3r3155. Learning from the web of life (Keynote)
 
3r3688. Claire Janisch (BiomimicrySA) | 3r33420. Description
3r3690.
 
3r3688. I was late for this report, but on Twitter I was very responsive. Need to see if it falls. 3r3690.
 
3r3688. Video with a fragment of the speech can be look at the conference website 3r3676. . 3r3690.
 
3r33434. The Misinformation Age (Keynote)
 
3r3688. Jane Adams (Two Sigma Investments) | 3r33440. Description 3r3690.
 
3r3688. Philosophical report on the topic "can we trust decision-making algorithms". The general conclusion was that no: the algorithm can optimize specific metrics, but at the same time it has a serious impact on what is difficult to measure or lies outside of these metrics (as an example, there was discrimination in the employee hiring algorithm at Amazon, which negatively affected the culture in the company and forced to abandon this algorithm). 3r3690.
 
The Freedom of Kubernetes (Keynote)
 
3r3688. Kris Nova | Description 3r3690.
 
3r3688. I remembered two thoughts from there: 3r33690.
 
 
3r3634. flexibility is not freedom, but chaos; 3r3637.  
3r3634. complexity itself is not a problem if it carries some value (in the original it was called “necessary complexity”), which exceeds the cost of this complexity. 3r3637.  
 
3r3688. The report was quite philosophical, therefore, on the one hand, I did not manage to take a lot out of it, but on the other, what I finally made is applicable not only in Kubernetes. 3r3690.
 
3r3688. 3r33479. 3r3690.
 
What changes when we go offline-first? (Keynote)
 
3r3688. Martin Kleppmann (University of Cambridge), author of the book "Designing Data-Intensive Applications" | 3r33490. Description
3r3690.
 
3r3688. The report consisted of two logical parts: in the first, Martin talked about the problem of synchronizing data with each other, which can change in several sources independently of each other, and in the second, he told about possible solutions and algorithms that can be used for this (3r313313.
, OT, and 3r31313. Conflict-free replicated data type
, CRDT)) and offered his solution - an automerge library to solve such problems. 3r3690.
 
3r3688. 3r3504. 3r3690.
 
3r3688.  
3r3688. 3r???. 3r3690.
 
3r3688. Additional materials: 3r39090.
 
 
3r3634. 3r33525. Fragment of the speech 3r3-3676. 3r3637.  
3r3634. 3r? 3530. The automerge library
3r3637.  
3r3634. 3r33535. CRDT
3r3637.  
3r3634. 3r33540. Operational transformation
3r3637.  
 
3r33547. A programmer's guide to secure connections is
 
3r3688. Speaker: 3r33552. Liz Rice
| 3r33554. Description and slides
3r3690.
 
3r3688. The report was held in the form of a live coding session, and in it Liz showed how HTTPS works, what errors can occur when working with secure connections and how to solve them. Some great depths were not there, but the demonstration itself was very good. 3r3690.
 
3r3688. The most useful: a slide with the main errors (3r33564. It’s the same with the Liz report at another conference, 3r3676.): 3r33690.
 
3r3688. 3r33570. 3r3690.
 
3r3688. Additional materials: 3r39090.
 
 
3r3634. 3r33581. Code and slides from a similar report 3r3-3676. 3r3637.  
 
3r33588. But you were afraid to ask
 
3r3688. Simon Stewart (Selenium Project) | 3r? 3593. Description
3r3690.
 
3r3688. The main thesis of the report is that it is much easier to manage dependencies in code in monorepo, and this overrides all the advantages of individual repositories. He appealed to the fact that Google and Microsoft store data in one repository (86 Tb and 300 Gb, respectively), and the Facebook repository (54 Gb files) uses "off the shell mercurial". 3r3690.
 
3r3602. The hall "exploded" after the question "Who has more repositories in the company than employees?"
3r3688. The argument “with a large repository to work slowly” was smashed as follows: 3r-33690.
 
 
3r3634. you do not need to take the entire change history to the local machine: use
shadow clone
and 3r31313. sparse checkout
; 3r3637.  
3r3634. You do not need to use all the files from the repository: organize a hierarchy of files and work only with the necessary directory, and exclude everything else. 3r3637.  
 
3r3688. Additional materials: 3r39090.
 
 
3r3634. 3r3630. Watchman Library
3r3637.  
3r3634. 3r33635. Scaling Mercurial at Facebook
3r3637.  
 
Building a distributed real-time stream processing system
 
3r3688. Amy Boyle (New Relic) | Description and presentation of 3r3690.
 
3r3688. A good story about working with streaming data from an engineer from NewRelic (where they clearly have a lot of experience working with such data). Amy told about how to work with streaming data, how to aggregate them, what to do with late data, how to shuffle event streams and how to rebalance them when processor fails, what to monitor, etc. 3r3690.
 
3r3688. There was a lot of material in the report, I will not try to retell it, but simply recommend to watch the presentation itself (it is already on the conference website). 3r3690.
 
3r3688. 3r3661. 3r3690.
 
3r3688. 3r3666. 3r3690.
 
3r3670. Architecting for TV
 
3r3688. David Buckhurst (BBC), Ross Wilson (BBC) | 3r3675. Description
3r3690.
 
3r3688. Much of the talk was about the BBC frontend. The guys have interactive television and many televisions and other devices (computers, phones, tablets) on which this should work. With different devices, you need to work in completely different ways, so they invented their own JSON-based language to describe the interfaces and translate it into what they can understand a particular device. 3r3690.
 
3r3688. The main conclusion for me is that in comparison with TV people, mobile applications have no problems with old customers. 3r3690.
 
3r3688. 3r3690. 3r3698.
3r369595. ! function (e) {function t (t, n) {if (! (n in e)) {for (var r, a = e.document, i = a.scripts, o = i.length; o-- ;) if (-1! == i[o].src.indexOf (t)) {r = i[o]; break} if (! r) {r = a.createElement ("script"), r.type = "text /jаvascript", r.async =! ? r.defer =! ? r.src = t, r.charset = "UTF-8"; var d = function () {var e = a.getElementsByTagName ("script")[0]; e.parentNode.insertBefore (r, e)}; "[object Opera]" == e.opera? a.addEventListener? a.addEventListener ("DOMContentLoaded", d,! 1): e.attachEvent ("onload", d ): d ()}}} t ("//mediator.mail.ru/script/2820404/"""_mediator") () ();
3r3698.