3r31717. 3r3-31. Habr, this is a report by software engineer Alexey Starkov at the Moscow Python Conf ++ 2018 conference in Moscow. Video at the end of the post.
Hello! My name is Alexey Starkov - I, in my best years, work at the plant. 3r? 3596.  3r31717. Now I work in Qrator Labs. Basically, all my life, I have been working in C and C ++ - I love Alexandrescu, “Gang of Four”, the principles of SOLID - that’s all. Which makes me an architectural astronaut. The last couple of years I've been writing in Python, because I like it. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Actually, who are the "architectural astronauts"? The first time I met this term was Joel Spolsky, you probably read it. He describes "astronauts" as people who want to build an ideal architecture, who hang an abstraction, above an abstraction, above an abstraction that is becoming more and more common. In the end, these levels go so high that they describe all possible programs, but they do not solve any practical problems. At this point, the "astronaut" (this is the last time this term is surrounded by quotes) ends the air and he dies. 3r? 3596.  3r31717. 3r? 3596.  3r31717. I also have a tendency towards architectural cosmonautics, but in this report I will talk a little about how it bit me and did not allow me to build a system with the necessary performance. The main thing - how I overcame it. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The summary of my report: was /became. 3r? 3596.  3r31717.
3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Where could I have done so much? If you do not want to mess up just like me - read on. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. What does it briefly consist of? We have a database that stores a replica of our configuration for the entire network, and there is a server that processes the commands that come to it and in some way changes the configuration. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Our technical administrators and clients come to this server and use the console, through endpoint API endpoints, REST API, JSON RPC and other issues commands to the server, as a result of which it changes our configuration. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Teams can be very simple and more difficult. Then, we have a certain set of receivers that make up our SDN and the server pushes the configuration to these receivers. Sounds pretty simple. Basically, I will talk about this part here. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. What was the problem after we designed this system? The execution of one command took from one to thirty seconds, depending on the complexity of the command. Accordingly, the delay of execution reached five minutes. One team came - 30 seconds, the second and so on, a pile of accumulated - a delay of 5 minutes. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The delay in applying the configuration is up to ten minutes. It was decided that this is not enough for us and it is necessary to carry out optimization. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r31-10. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The first is that before carrying out any optimization, it is necessary to conduct an investigation and find out what the matter is. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r3119. 3r? 3596.  3r31717. 3r? 3596.  3r31717. As it turned out, we lacked the most important component to investigate - we did not have telemetry. Therefore, if you are designing a system, first, at the design stage, lay telemetry into it. Even if the system is initially small, then a little more, then even more - in the end, everyone comes to a situation where you need to watch the traces, but there is no telemetry. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r3128. 3r? 3596.  3r31717. 3r? 3596.  3r31717. What can be done next if you do not have telemetry? You can analyze the logs. Here, quite simple scripts go through our logs and turn them into such a table, illustrating the fastest, slowest and average command execution time. Starting from here, we already see in which places we have gaps: which commands are executed longer, which ones are faster. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The main problem is immediately visible - this is mapping. At first we do it once from the Python object to the mapper, then the mapper mappit to the base. Additional mapping is one or two calls, which may not be so bad. The main problem was the manual synchronization. We have two objects of our “clean” interface and one of them has an attribute change - how do we see that the attribute has changed in the other? No It is necessary to merge the changes into the database and get the attribute in another object. Of course, if we know that objects are present in the same context, we can somehow track this. But if we have two sessions in different places - only through the base, or to base the base in memory, which we did not do. 3r? 3596.  3r31717. 3r? 3596.  3r31717. This load /save /delete is another mapper that completely duplicates the guts of alchemy, which is well written, tested. This tool has been around for many years, a lot of help is available on the Internet and duplicating it is also not very good. 3r? 3596.  3r31717. 3r? 3596.  3r31717. See the icon in the upper right corner? So I will mark the slides on which something is done for "purity", to increase the level of abstraction, for architectural cosmonautics. That is, slides without this icon are pragmatic and boring, uninteresting and can be not read. 3r? 3596.  3r31717. 3r? 3596.  3r31717. What to do if a lot of requests are slow. How many? Actually a lot. Imagine a chain of inheritance: one object, it has one parent, and that one has another parent. We synchronize the child object - in order to do this, you first need to synchronize the parents. In order to synchronize the parent, you need to synchronize and his parent. Well, all synchronized. Actually, depending on how the graph is constructed here, we could walk and synchronize all these objects a hundred times - hence a large number of requests. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33232. 3r? 3596.  3r31717. 3r? 3596.  3r31717. What have we done? We took all our business logic and put it in the mapper. All other objects we have also merged with mappers and all of our API, the entire data abstraction layer, turned out to be "dirty." 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Of course, from the point of view of any astronaut, a dirty API is a disadvantage. Business logic in the declarative description of the database. Schemes mixed with business logic. Fu Ugly. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The description of the scheme is cluttered. This is actually a problem - if we do not have two lines of business logic, but a larger volume, then we have to scroll or search for a long time in this class in order to get to specific descriptions. Before that, everything was beautiful: in one place the description of the base, declarative, description of the schemes, in another place the business logic. And then the scheme is cluttered. 3r? 3596.  3r31717. 3r? 3596.  3r31717. But, on the other hand, we immediately get the mechanisms of alchemy: the unit of work, which allows us to track which objects are dirty and which relays need to be updated; we receive the relationship, allowing to get rid of additional questions in a database, without watching that the corresponding collections were filled; and the identity map that helped us the most. The Identity map ensures that two Python objects are the same Python object if they have a primary key. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Accordingly, we immediately dropped the complexity to linear. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Mixins We take business logic, again we take it out of our mapper, but so that again there is no mapping, we will inherit our mapper inside alchemy from our mixin. Why not in the opposite direction? In alchemy, this will not work, she will curse and say: "You have two different classes referring to the same tablet, there is no polyformism - go here." Is that allowed. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Thus, we have a declarative description in the mapper, which is inherited from the mixin and gets all the business logic. Very comfortably. And the rest of the classes are exactly the same. It would seem - cool, everything is clean. But there is one nuance - connections and relays remain inside alchemy, and when we have, say, join through an intermediate table secondary table, then the mapper of this label will somehow be present in the client code, which is not very nice. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Alchemy would not be such a good, well-known, framework if it did not give an opportunity to fight this. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. We have a schema file that contains all of our declarative classes - let's call it schema.py. And we have entities in business logic, separately. And, these entities are inherited inside the schema file - we write a separate class for each entity, and inherit it in the schema. Thus, the business logic lies in one pile, the scheme is in the other, and they can be independently changed. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33333. 3r? 3596.  3r31717. 3r? 3596.  3r31717. As an example of improvement, we will consider a simple scheme of two plates: receivers (Receiver table) and configuration slices (ReceiverPlanes table). The configuration slices with the many-to-one relationship are associated with the receivers label. There is nothing especially complicated. 3r? 3596.  3r31717. 3r? 3596.  3r31717. In order to hide relationships inside the “dirty” interface of alchemy, we use relationships and collections. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33333. 3r? 3596.  3r31717. 3r? 3596.  3r31717. They allow us to hide our mappers from client code. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33333. 3r? 3596.  3r31717. 3r? 3596.  3r31717. In particular, two very useful collections are association_proxy and attribute_mapped_collection. We use them together. How a classic relationship works in alchemy: we have a relationship - this is a kind of collection, list, mappers. Mappers objects are the far end of the relationship. Attribute_mapped_collection allows you to replace this list with a dict, the keys in which are some of the attributes of the mappers, and the values ​​are the mappers themselves. 3r? 3596.  3r31717. 3r? 3596.  3r31717. This is the first step. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33354. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The second step, we are doing association_proxy over this relationship. It allows us not to transfer the mapper to the collection, but to transfer some value that will later be used to initialize our mapper, ReceiverPlanes. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Here we have a lambda, in which we give the key and value. The key becomes the name of the configuration slice, and the value becomes the value of the configuration slice. As a result, in the client code, everything looks like this. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33333. 3r? 3596.  3r31717. 3r? 3596.  3r31717. We just put a dictation in some dictionary. Everything works: no mappers, no alchemy, no databases. 3r? 3596.  3r31717. 3r? 3596.  3r31717. True, there are pitfalls. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33380. 3r? 3596.  3r31717. 3r? 3596.  3r31717. If we assign two different keys to the same key, or even one, two values ​​— for each such set item lambda is called, an object is created - a mapper. And, depending on how the scheme is arranged, this can lead to different consequences, from “just violation of constraints” to unpredictable consequences. For example, you sort of deleted the object from the collection, but it remained there anyway: you deleted only one. When I first started, I killed a lot of time for such things. 3r? 3596.  3r31717. 3r? 3596.  3r31717. And a bit implicit sync. Association_proxy and attribute_mapped_collection may be slightly delayed: when we create a mapper object, it is added to the database, but it is not yet present in the collection attribute. It will appear there only when the attribute expires in this session. When it is expired, a new synchronization with the database will occur and it will get there. 3r? 3596.  3r31717. 3r? 3596.  3r31717. To fight this, we used our own, self-written, collections. This is not even alchemy - you can simply create your own collection to overcome all this. 3r? 3596.  3r31717. 3r? 3596.  3r31717.  3r31717. In __setitem__, we put these objects in our collection, in a relationship. The only thing is that we assign the value at the very end. Thus, we implement the same mechanism as association_proxy - we pass there the value, the dict, and it is assigned to the corresponding attribute. 3r? 3596.  3r31717. 3r? 3596.  3r31717. __getitem__ does reverse manipulation. It receives a key from an object from a relay and returns its attribute. There is also a small underwater stone here - if you cache the collection inside our mapping, it is possible to slightly out of sync. Because when alchemy has a collection attribute, then the collection is replaced with another one after it expires. Therefore, we can save the reference to the old collection and not know that the old has expired and a new one has already appeared. Therefore, in the last part we go straight to the alchemy instance, again we get the collection through __getattr__ and we do __getitem__ from it. That is, we cannot cache the Planes collection here. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33420. 3r? 3596.  3r31717. 3r? 3596.  3r31717. How does this collection spike on our mixins? As usual - we get a collection attribute. The only interesting place is that when we load an instance from the database, the __init__ method is not called. All attributes are substituted after the fact. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Alchemy gives the standard decorator reconstructor, which allows you to mark a method as callee after loading an object from the base. And just at boot time we have to initialize our collection. Self - just this instance. The use is exactly the same as in the previous example. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33333. 3r? 3596.  3r31717. 3r? 3596.  3r31717. But we still have the database ears visible in the schema - this is the configuration. What type of configuration? Is it varchar or is it a blob? In fact, the client is not interested. He has to work with abstract entities of his level. For this, alchemy provides type decoration. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r3442. 3r? 3596.  3r31717. 3r? 3596.  3r31717. A simple example. Our database stores IPAddress in the form of a varchar. We use the TypeDecorator class, which is included in alchemy, which allows, first, to specify which underlying database type will be used for this type and, second, to define two parameters: process_bind_param that converts the value to the database type and process_result_value, when we value from the type of database we translate into a Python object. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The attribute from the address acquires the Python type IPAddress. And we can both call methods of this type and assign objects of this type to it and everything works for us. And the database is stored I do not know what is stored, varchar (45), but we can replace that line and the blob will be stored. Or if some native type supports IP addresses, then you can use it. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The client code does not depend on it, it does not need to be rewritten. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Here is an example. Here three events are used:
 3r31717. on_before_flush - before the sql-code is emitted to the database, we go through all the objects that alchemy has marked as dirty in this session and check if this object is modified or not. Why is this necessary if alchemy has already marked everything? Alchemy marks an object as dirty as soon as an attribute has changed. If we assign this attribute the same value that it had - it will be marked as dirty. For this there is a session method is_modified - it is used internally, I did not draw it. Further, from the point of view of our semantics, from the point of view of our business logic, even if the attribute has changed - the object can still remain unmodified. For example, there is a certain list, in which two elements were swapped - from the alchemy point of view, the attribute has changed, but for business logic it does not matter if, for example, a set is stored in this list. 3r? 3596.  3r31717. 3r? 3596.  3r31717. And, as a result, we call another method specific to each object in order to understand whether the object is actually modified or not. And we add them to a certain variable associated with the session that we created ourselves - this is our dirty_instances variable to which we add this object. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The next event occurs before the commit - before_commit. Here, too, there is a small caveat: if for the entire transaction we did not have a single flush, then flush will be called before the commit — in my case, the handler before the commit was called before the flush. 3r? 3596.  3r31717. 3r? 3596.  3r31717. As you can see, what we have done in the previous paragraph may not help us and session.dirty_instances will be empty. Therefore, inside the handler, we once again do the flush, so that all the handlers before the flush come forward and simply increment the version by one. 3r? 3596.  3r31717. 3r? 3596.  3r31717. after_commit, after_soft_rollback - after a commit just clean, so that next time there are no excesses. 3r? 3596.  3r31717. 3r? 3596.  3r31717. So you see, this install_handler method installs handlers for three events at once. As a class, we pass the session here, as this is an event of its level. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33512. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Here you go. I will remind you what we have achieved - the speed of 30-40 seconds for complex and large teams. Not at all, some were performed in a second, others in 200 milliseconds, as you can see from the RPS. Requests in the database began to number in the hundreds. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33521. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The result was a fairly balanced system. There was, however, one nuance. Some requests come from us in batches, emissions. That is, 30 requests arrive, and each of them is such! (the speaker shows the thumb)
 3r31717. 3r? 3596.  3r31717. If we process them in one second, then the last request in the queue will work for 30 seconds. The first is one, the second is two, and so on. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r? 3534. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Therefore, we need to accelerate. What do we do? 3r? 3596.  3r31717. 3r? 3596.  3r31717. In fact, alchemy consists of two parts. The first is an abstraction over the sql database called SQLAlchemy Core. The second is the ORM, the actual mapping between the relational database and the object representation. Accordingly, the alchemy core approximately one to one coincides with the sql - if you know the latter, then you will not have problems with the core. If you do not know sql - learn sql. 3r? 3596.  3r31717. In addition, the core represents the smallest overhead. There is practically no pumping - requests are generated using a request generator, and then executed. Overhead over dbapi minimum. 3r? 3596.  3r31717. 3r? 3596.  3r31717. We can build queries of any complexity, of any type, we can optimize them for the task. That is, if in the general case ORM doesn’t care how we build the database schema, there is a certain description of the tables, it generates some queries, not knowing what in this case will, for example, optimally remove from here, in the other - from there, here apply a filter, and there it is another, then we can make requests for the task. 3r? 3596.  3r31717. 3r? 3596.  3r31717. The disadvantage is that we again came to manual synchronization. All events, relays - all this in the core does not work. We made a select, objects came to us, we did something with them, then an update, an insert you need to increment the version, check the constraints yourself. Core does not allow all this to be done conveniently, at a high level. 3r? 3596.  3r31717. 3r? 3596.  3r31717. Well, we are not living the first day. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33561. 3r? 3596.  3r31717. 3r? 3596.  3r31717. A simple example of use. Each mapper inside contains an __table__ object that is used in the core. Next, you see - we take the usual select, we list the columns, join two plates, indicate the left and right, indicate by what condition we join it, well, for order, add order bye. Next we feed this generated query into the session and it returns us an iterable, in which the tap-like objects are indexed both by the column name and by number. The number corresponds to the order in which they are listed in the select. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r33570. 3r? 3596.  3r31717. 3r? 3596.  3r31717. It became much better. The performance in the worst case dropped to 2-4 seconds, the most complex and longest query contained 14 teams and RPS 10-15. Solid. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r???. 3r? 3596.  3r31717. 3r? 3596.  3r31717. What I would like to say in conclusion. 3r? 3596.  3r31717. Do not procreate entities where they are not needed - do not wind up your own where there is a ready. 3r? 3596.  3r31717. Use SQLA ORM is a very convenient tool that allows you to track events at a high level, respond to various events associated with the database, hide all the ears of alchemy. 3r? 3596.  3r31717. If all else fails, the speed is not enough - use SQLA Core. This is still better than using pure raw SQL, because it provides a relational abstraction over the database. Automatically extract parameters, correctly does the binding, it doesn’t matter what database is under it - it can be changed and Core supports different dialects. It is very convenient. 3r? 3596.  3r31717. 3r? 3596.  3r31717. That's all I wanted to tell you today. 3r? 3596.  3r31717. 3r? 3596.  3r31717. 3r3-3598. 3r3599. 3r3600. 3r3601. 3r3602. 3r33613. 3r33613. 3r33613. 3r33613. 3r31717. 3r31717. 3r31717. 3r3610. ! function (e) {function t (t, n) {if (! (n in e)) {for (var r, a = e.document, i = a.scripts, o = i.length; o-- ;) if (-1! == i[o].src.indexOf (t)) {r = i[o]; break} if (! r) {r = a.createElement ("script"), r.type = "text /jаvascript", r.async =! ? r.defer =! ? r.src = t, r.charset = "UTF-8"; var d = function () {var e = a.getElementsByTagName ("script")[0]; e.parentNode.insertBefore (r, e)}; "[object Opera]" == e.opera? a.addEventListener? a.addEventListener ("DOMContentLoaded", d! ): d ()}}} t ("//mediator.mail.ru/script/2820404/","_mediator") ((indow); 3r-3611. 3r-?617.