3r33565. 3r3-31. On the Pixonic DevGAMM Talks, our DTO Anton Grigoriev also performed. We in the company have already said that we are working on a new PvP-shooter and Anton shared some of the nuances of the architecture of this project. He told how to build development so that changes in the client's game logic appear on the server automatically (and vice versa), and whether it is possible not to write code, but at the same time minimize traffic. Below - recording and decoding of the report. 3r33515.  3r33565. 3r33515.  3r33565.
3r33561. 3r33561. 3r33561. 3r314. 3r? 3551. 3r33515.  3r33565. I will not teach how to do something, I will tell you how we did it. So that you do not step on the same rake and can use our experience. A year and a half ago, we in the company did not know how to make shooters on mobile phones. You say how, you have War Robots, 100 million downloads, 1.5 million DAU. But in this game the robots are very slow, and we wanted to make a quick shooter and the architecture of War Robots did not allow it. 3r33515.  3r33565. 3r33515.  3r33565. We knew how and what to do, but we had no experience. Then we hired a person who had this experience and said: do the same thing that you have already done a hundred times, only better. Then they sat down and began to think about architecture. 3r33515.  3r33565. 3r33515.  3r33565. 3r33515.  3r33565. 3r33515.  3r33565. The second feature is the separation of logic and data, about which I have already spoken. What does this give us? We can store the state of the world and its history in batches, we can serialize it, we can send this data over the network and change it in real-time. This is just data in memory - we can change any value at any time. Thus it is very convenient to change the logic of the game (or for debug). 3r33515.  3r33565. 3r33515.  3r33565. It is also very important to follow the order of calling systems. All systems follow each other, are called by the Execute () method and, ideally, should be independent. In practice, this does not happen. One system changes something in the world, another system then uses it. And if we break this order - the game will go differently. Probably not much, but definitely not the way it used to be. 3r33515.  3r33565. 3r33515.  3r33565. Finally, one of the main and most important feature for us is that we can execute the same code, both on the client and on the server. 3r33515.  3r33565. 3r33515.  3r33565. Give the developer an opportunity, and he will find 99 ways and reasons to make his decision, and not to use the existing ones. I think many did. We were at that time looking for the ECS Framework. They considered Entitas, Artemis C #, Ash.net and their own solution, which they could write from the experience of the specialist who came to us. 3r33515.  3r33565. 3r33515.  3r33565. 3r360. 3r33515.  3r33565. 3r33515.  3r33565. Do not try to read what is written on the slide, it is not so important. What matters is how much green and red are in the columns. Green means that the solution supports the requirements, red does not support, yellow supports, but not quite. 3r33515.  3r33565. 3r33515.  3r33565. In the ECS column - potentially our solution. As you can see, it is cooler - we could support a lot more requirements. As a result, we did not support some of them (mainly because they were not needed), and some, without which we could not work further, had to be done. We chose the architecture, worked for a long time, did the minimally playable version and fakap. 3r33515.  3r33565. 3r33515.  3r33565. 3r373. 3r33515.  3r33565. 3r33515.  3r33565. It turned out the most non-playable version. The player was constantly rolling back, brakes, the server was hanging in the middle of the match. It was impossible to play it. What were the reasons for the failures? 3r33515.  3r33565. 3r33515.  3r33565. Reason # 1 and the most important is inexperience. But how so? We hired an experienced person who had to make everything beautiful. Yes, but in fact we gave him only part of the work. We said: "Here's your game server, work on it." And in our architecture (more on that later), the client plays a very important role. And it is this part that we gave to a person who did not have the necessary experience. No, he is a good programmer, señor - just had no experience. Those. he could not even imagine what rake there might be. 3r33515.  3r33565. 3r33515.  3r33565. Reason number 2 - unreallocation. 80 Kb /frame. Is it a lot or not? If we consider that we have 30 frames per second, then in a second we get 2.5 MB, and for a 5-minute match, it is already more than 600 MB. In short, a lot. Garbage collector begins to try hard to free up all this memory (when we demand more and more from it), which leads to spikes. Given that we wanted 30 frames per second, these spikes prevented us very much. Moreover, both on the client and on the server. 3r33515.  3r33565. 3r33515.  3r33565. The main reason for the allocation was that we constantly allocated data arrays. Every time almost every frame. Used LINQ, lambda expressions and Photon. Photon is an online library with which we are familiar and use in War Robots. And everything seems to be fine, but it allocates memory each time it sends data or receives it. 3r33515.  3r33565. 3r33515.  3r33565. If we dealt with the first problems (copied to our custom collections, did caching), then practically nothing could be done with Photon, because it is a third-party library. It was only possible to reduce the size of the package, and we had it 5 KB. Lot? Yes. There is an MTU — this is the minimum actual packet size that is sent via UDP without breaking the packet into small parts. It is about 1.5 Kbytes, and we had 5 (this was, on average, more). 3r33515.  3r33565. 3r33515.  3r33565. Accordingly, Photon cut our package into small ones and sent each piece as reliable, i.e. with guaranteed delivery. Every time a part did not reach, he sent it again and again. We got even more latency and the network worked poorly. 3r33515.  3r33565. 3r33515.  3r33565. All these allocations led to the fact that we got a frame for about 100 milliseconds, when 33 was needed. And there, rendering, simulation and other actions - all this takes up the CPU. All these problems are complex, i.e. it was impossible to decide there was any one, and everything would be fine. It was necessary to solve them all at once. 3r33515.  3r33565. 3r33515.  3r33565. And another small problem that was during development - a large number of repositories. The slide says ? but it seems to me that there were even more of them. All these repositories (for the client, the game server, the common code, the settings, and something else) were connected by sub modules to the two main repositories on the client and the game server. It was hard to work with. Programmers know how to work with Git, SVN, but there are still artists, designers, etc. I think many have tried to teach the artist or designer to work with the version control system. This is really hard, so if your designer knows how to do it - take care of him, he is a valuable employee. In our case, even programmers freaked out, and in the end we reduced every single repository. 3r33515.  3r33565. 3r33515.  3r33565. This was a great solution. We have a folder with the server and a folder with the client there. The server consists of a game server project, a code generator and auxiliary tools. 3r33515.  3r33565. 3r33515.  3r33565. 3r3114. 3r33515.  3r33565. 3r33515.  3r33565. A client is a Unity client and a common code. The common code is the data structure of the world, i.e. Entities, components and system simulation. This code is mainly generated by the server generator. It also uses the server. Those. This is a common part for the client and server. 3r33515.  3r33565. 3r33515.  3r33565. Lifetime We take TeamCity, we set on our repository, we collect and deploy the server. Every time when a client changes the common logic, we immediately have a game server going in - now you don’t need a server programmer. Usually there is a server, a client and some kind of feature. The client is cutting it at home, the server at home, and once they have it all will work. In our case, not so - the client can write this feature and everything works on the server. 3r33515.  3r33565. 3r33515.  3r33565. A match consists of a common part (designated as ECS) and views (these are unitary MonoBehaviour classes, GameObjects, models, effects — everything that the world is represented in). They are not related. 3r33515.  3r33565. 3r33515.  3r33565.  3r33565. 3r33515.  3r33565. 3r3179. 3r33515.  3r33565. 3r33515.  3r33565. This is when you send one full state, and then only change it. There is no such thing that the whole world changes completely at once. Constantly changing only some parts and these changes in size are much smaller than the state itself. We received an average of 300 bytes, i.e. 17 times less than it was originally. 3r33515.  3r33565. 3r33515.  3r33565. Why is it necessary, if so and got into the MTU? The game is constantly growing, new features appear, and with it appear objects, an entity, new components. Data size is growing. If we stopped at 1 KB, we would very soon return to the same problem. Now, having rewritten to delta compression, we will not reach this very soon. 3r33515.  3r33565. 3r33515.  3r33565. Now is the sweetest part. Synchronization. If you play shooters, you know what Input Lag is - when you press a button, and the character starts to move after some time, for example, half a second. For any games in the mob genre, this is normal. But in the shooter you want the hero to shoot and inflict damage right away. 3r33515.  3r33565. 3r33515.  3r33565. 3r33394. 3r33515.  3r33565. 3r33515.  3r33565. Why is input lag happening? The client collects player input (input) and sends it to the game server (sending takes time). Next, the game server processes it (again, time) and sends the result back (again, time). This is the delay. How to remove it? There is a thing called prediction - the client does not wait for a response from the server and immediately starts trying to do the same thing that the game server does, i.e. simulates. Takes the player input and starts the simulation. We simulate only a local client, because we do not know the input of other players - they do not come to us. Therefore, we run the simulation system only on our player. 3r33515.  3r33565. 3r33515.  3r33565. Firstly, it allows to reduce the simulation time. The client starts the simulation as soon as it receives input and is a few steps ahead, relative to the game server. For example, in this picture it simulates tick number 20. At this point, the game server simulates tick number 15 in the past. The client sees the rest of the world, again, in the past, himself - in the future. Until he sends the 20th tick to the server, until this input reaches, the game server will already start to simulate the 18th tick or already the 20th. If the 18th, then he will put it in the buffer, reach the 20th, process and return the result back. 3r33515.  3r33565. 3r33515.  3r33565. Suppose now he simulates tick number 15. Processed, returns the result to the client. The client has some simulated 15th tick, 15th game state and game world that he predicted. Begins comparison with the server. In fact, he does not compare the whole world, but only his client, because we are not responsible for the rest of the world. We are responsible only for ourselves. If the player matches, everything is good, then we correctly simulated, the physics worked correctly and no collisions arose. Further we continue to simulate the 20th tick, the 21st and so on. 3r33515.  3r33565. 3r33515.  3r33565. If the client /player does not match, then we are mistaken somewhere. Example: since physics is not deterministic, it considered our position to be wrong or something happened. Maybe just a bug. Then the client takes the state from the game server, because the game server has already confirmed it (he trusts the server - if he didn’t trust, the players would cheat), and resimulate everyone else from 15th to 20th. Because this branch of time is now erroneous. 3r33515.  3r33565. 3r33515.  3r33565. Create a new timeline, i.e. Parallel Worlds. We restimulate these five ticks in one tick. Once our simulation took 5 milliseconds, but if we need to restimulate 10 ticks, this is already 50 milliseconds and we do not fall into our 30 milliseconds. Optimized and received one millisecond - now 10 ticks are processed in 10 milliseconds. Because there is still rendering. 3r33515.  3r33565. 3r33515.  3r33565. All these things work on the client, and we gave it to a person without the right experience. The minus is that we had a facac, and a plus is that the programmer now knows how to do it right. 3r33515.  
3r33515.  3r33565. 3r33515.  3r33565. 3r33515.  3r33565. So, we have reached 30 ticks per second, 30 frames per second. Now on our server plays about 600 players at a time. In a match of 6 players, i.e. about 100 matches. We do not have a server programmer, we do not need it. All logic clients write in Unity editor, Rider’e, on C # and it works on a game server. Almost always. We reduced the packet size by 17 times and reduced memory allocations by 80 times — now even less than a kilobyte on the client and server. The average ping was 200-250 ms, now it is 150. 200 is the standard for mobile network games, unlike PCs, where everything happens much faster, especially over a local network. 3r33515.  3r33565. 3r33515.  3r33565. - For code generation, what do you use? Your own decision? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - Everything is simple - hands. We thought to make something ready, but it turned out to be faster just to write with our own hands. Let's go this way, it worked well then and now. 3r33515.  3r33565. 3r33515.  3r33565. [b] - You abandoned the server developer, but you didn’t just reduce development time by reusing the same code. Unity does not support the latest version of C #, it has its own engine under the hood. You can not use .NET Core, you can not use the latest features, certain structures and so on. Doesn't the performance suffer by about a third? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. “When we started doing all this, we thought that in order to use not classes, but structures, it had to work much faster. We wrote a prototype of how it will look in the code, how programmers will use these structures to write logic. And it turned out to be terribly uncomfortable. We stopped at the classes and the performance that we have now is enough for us. 3r33515.  3r33565. 3r33515.  3r33565. [b] - How do you live now without interpolation? And how do you pretend a player if the snapshot does not come to the right frame? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - We have interpolation, only it is not visual, but on those packets that come over the network. Suppose we have the 18th, 19th and 20th state. The 18th came, the 20th came, and the 19th was either lost or not yet reached - that’s what we interpolate. Just use code generation in order not to write interpolation code. 3r33515.  3r33565. 3r33515.  3r33565. [b] - Are there lifehacks to compress quaternions more strongly? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - I talked about 2D - there is just an alpha angle, and on new projects, quaternions have their problems there. From life hacking, I can say this: since we use UDP so that the input is not lost, we send it in batches: from zero to fifth input, then from first to sixth, and so on. It is cheaper and delivery is better. 3r33515.  3r33565. 3r33515.  3r33565. [b] - But after all, the order of reproduction of input on the server plays a role? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - Yes of course. If any input is suddenly lost (although this is unlikely, you either have a very bad network, or it has completely fallen off), then we have 2 options: either copy the previous input, or take the zero input. 3r33515.  3r33565. 3r33515.  3r33565. [b] - You do not pretend on the client of other players. And how does it look visually? They do not teleport? If ping is more than 1000 in the region of a second, what will happen? Just move, when the next thread comes? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - Now there is no interpolation, respectively, sometimes jerking. But we now have an interval per second, if the client has gone beyond the game server for a second, then we just break the connection, it tries to reconnect. It is cheaper than restimulating all 30 ticks. 3r33515.  3r33565. 3r33515.  3r33565. [b] - Do you have a discrepancy between steits, how do you find them and how do you fight? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - Of course. Initially, when we compared so many things (now we compare only the local player), we constantly had missions. This generates over-reductions — restimulations, and this is the processing load, the processor. At some point, we realized that some things do not need to simulate, cut them and now everything has become much easier. And if, for example, to take shots, then we actually predict them only in order to visually display. There are some tricks. 3r33515.  3r33565. 3r33515.  3r33565. [b] - How much of the game do you have on ECS, how fat are the systems, how loaded are entities and components? How is your balance organized? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - 30 systems on the client spin, but we only simulate a local player. 80 on the server at the moment, maybe more. Essentially now I don’t remember exactly. 3r33515.  3r33565. 3r33515.  3r33565. [b] - Question on discrepancy in prediction. When we predicted something on the 20th frame, and the state came to us from the server that we could not shoot and we have some kind of command pool - how do you roll, how do you drag? The feeling that there is a specific solution for each individual case. Or are there any common solutions? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - The general solution is simple: you take the state from the server and substitute it as the last filed. And on its basis (say, the 15th) you restimulate the 16th, 17th, 18th, and so on. 3r33515.  3r33565. 3r33515.  3r33565. [b] - Based on the teams in your team pool? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - Yes, we keep the history of the client's input and store the history of the state of the world. But there are some things like shots. The shots generate a lot of Entity entities (for every shot practically) that we display there. And since this generation of objects occurs on the client, there can be more such shots on the server, from other players as well. ID may diverge, we have our own hacks on this account. 3r33515.  3r33565. 3r33515.  3r33565. [b] - If there is a shot from the bazooka - we want to draw this rocket, and then it turns out that we could not shoot, should we delete it? I understand that in each case a separate solution? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - Yes, for example, there is a grenade. In 3D, when you throw a grenade, what kind of hacks are there - you start throwing it, it is not visible yet on the screen and only when it is created on the server will some time pass. But you will not see her, then she will appear, and everything seems to be fine. We have a top-down, so it is more difficult - there the grenade can be seen immediately. We also wrote with their tricks. On the client, we create it right away, we launch it there, but when it comes from the server really, we interpolate its path. In the end, everything looks fine. 3r33515.  3r33565. 3r33515.  3r33565. [b] - And if it was impossible to launch this grenade? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - She just disappears. This also happens. 3r33515.  3r33565. 3r33515.  3r33565. [b] - A question about the moment when all clients send their input to the server, it is going to a buffer. At some point, the server takes the input from all clients and simulates. But if a client lags behind, but does not lag behind by a full second, but simply jumps, 500 milliseconds, there is a delay, and the server does not have any client's input at any particular moment. How is this resolved? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - I'll show you now. 3r33515.  3r33565. 3r33515.  3r33565. 3r33394. 3r33515.  3r33565. 3r33515.  3r33565. The client throws input into the future, i.e. he simulates the 20th tick and takes the 20th input, then throws it into the future server. As far as the future is a separate complex thing, the number varies depending on the ping. He is trying to predict: if I send the 20th tick now, will the server at this moment reach this tick or not? There is again an input buffer on the server where these inputs are added. Accordingly, if he sent a little further - someday the server will simply reach him. If he sent for this buffer, then the input will be lost. The client will then receive a certificate from the server that “I have now processed your such input, not the 21st, but the 18th.” Server: "Yeah, so I need to squeeze a little." Such a flexible system. 3r33515.  3r33565. 3r33515.  3r33565. [b] - Ie they can sometimes disappear, but then he drives the client and the client sends a more fresh input? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - Yes, the client is trying to adapt in this situation. 3r33515.  3r33565. 3r33515.  3r33565. [b] - You briefly mentioned reliable UDP - do you have any implementation of your own? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - This is Photon, Photon has reliable UDP, unreliable, with guaranteed and without guaranteed delivery. 3r33515.  3r33565. 3r33515.  3r33565. [b] - guaranteed all packages are delivered? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - We are not guaranteed, just sent somewhere. Reach they or not reach, for the network game is not so important. For example, the client sends input. So that they are not lost, we send them in batches. If we send packs of five, then at least five times it will be sent there. If the package is lost by 100%, then naturally nothing will come; if by 80%, then most likely it will. 3r33515.  3r33565. 3r33515.  3r33565. [b] - And again? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - No, do not send again, Photon does it himself, if the packet size is larger than MTU. 3r33515.  3r33565. 3r33515.  3r33565. [b] - Is the use of code generation more true? How to support him? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - When the person who wrote it, just started, we had discussions with him. Whether we need it or not. As it turned out, really necessary. Then it was inefficient, but now the opposite is true, because there is practically nothing to do, you just slap the attributes. 3r33515.  3r33565. 3r33515.  3r33565. [b] - And an example where you can gain time, where you can get controllability? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - You saw a parameter that you do not need to send. If you didn't have a code generator, you would go write code. I forgot, everything is sent. Then someday you'll find it. And here you hung the attribute, everything works automatically. Or a parameter that can be sent with a smaller size. 3r33515.  3r33565. 3r33515.  3r33565. [b] - Just from my practice to write code reading /reading from the stream - this is once a month for about five minutes. And if I forget to write the code, then I will forget to put the attribute. 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - The game is constantly evolving, respectively, such components appear quite a lot and constantly have to spank the case. By the way, in addition to serialization (we also generate serialization), we have a certain viewer on the game server, which allows us to see the world as it is on the game server. All objects, all their fields. Fields can be changed - this part, it is also generated. 3r33515.  3r33565. 3r33515.  3r33565. [b] - Your physics is not deterministic, respectively, some problems arise. You did not think to make it deterministic? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - We're not going to make deterministic, although there were such ideas. But at the beginning of the project we had a choice of three points: ECS, physics, and the network library. We have taken physics and network library ready, ECS have written. Now we are very glad that we wrote ECS ourselves. It is very painful that they did not write the network library and physics. As it turned out, the student that we took the physics we wrote (we looked at it then, it seemed normal, and when we started working, digging deeper, it turned out to be not exactly what we needed). Plus, we have a 2D game, but there are things that need to be done in 3D - we write them ourselves. In fact, 3D physics for these things, objects, we write ourselves. And determinism in such an architecture is not needed. There will be some discrepancies, but they will be minimal. If there is a determinism, then there is no need to send a game state from the game server, just input from all players is enough. 3r33515.  3r33565. 3r33515.  3r33565. [b] - The report was the thesis that ECS allows you to run the code on both the client and the server. As I understand it, the fact that C # is everywhere allows? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - First of all - yes. 3r33515.  3r33565. 3r33515.  3r33565. [b] - Ie In principle, this does not apply to the ECS? As I understand it, ECS is just a way to write a game model, as a result, it spits out the state of the world and no matter how I wrote it, if I get the same output. Those. ECS is the right way to write a model so you don’t get confused. 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - I will not say what is right, this is just one of the ways. He has both pros and cons. The main disadvantage is that a person should be taught how to do it correctly, how to cook it. Because those who used OOP or some other approach before, it will be hard for them to understand how and what to do. 3r33515.  3r33565. 3r33515.  3r33565. [b] - So you used the ECS approach as a paradigm? 3r3504. 3r33515.  3r33565. 3r33515.  3r33565. - If in a good way, ECS is not only the paradigm that we used, it is also (if we write on structures) a pretty strong optimization - both in memory and in many other things. We didn’t do optimization, we didn’t write on structures - we wrote on classes. The main thing is the approach and the fact that we have data that are separate from the logic. We can work with them, send over the network, receive, compare, etc. 3r33515.  3r33565. 3r33515.  3r33565.

More reports from Pixonic DevGAMM Talks

3r33515.  3r33565. 3r? 3517.  3r33565. 3r3-3549. 3r33520. Use Consul to scale stateful services 3r33551. (Ivan Bubnov, DevOps in the company BIT.GAMES); 3r???.  3r33565. 3r3-3549. 3r33525. CICD: Seamless Deploy on Distributed Cluster Systems without Downtime 3r3-33551. (Egor Panov, Pixonic system administrator); 3r???.  3r33565. 3r3-3549. 3r? 3530. The practice of using the model of actors in the backend platform of the game Quake Champions
(Roman Rogozin, backend developer of Saber Interactive); 3r???.  3r33565. 3r3-3549. 3r33535. The architecture of the meta-server mobile online shooter Tacticool
(Pavel Platto, Lead Software Engineer in PanzerDog); 3r???.  3r33565. 3r3-3549. 3r33540. As ECS, C # Job System and SRP change the approach to the
architecture. (Valentin Simonov, Field Engineer in Unity); 3r???.  3r33565. 3r3-3549. 3r33545. The principle of KISS in the development of 3r33551. (Konstantin Gladyshev, Lead Game Programmer at 1C Game Studios); 3r???.  3r33565. 3r3-3549. 3r33550. Cucumber in the cloud: using BDD scripts for load testing the product 3r33551. (Anton Kosyakin, Technical Product Manager, ALICE Platform). 3r???.  3r33565. 3r33554. 3r33561. 3r33565. 3r33565. 3r33558. ! function (e) {function t (t, n) {if (! (n in e)) {for (var r, a = e.document, i = a.scripts, o = i.length; o-- ;) if (-1! == i[o].src.indexOf (t)) {r = i[o]; break} if (! r) {r = a.createElement ("script"), r.type = "text /jаvascript", r.async =! ? r.defer =! ? r.src = t, r.charset = "UTF-8"; var d = function () {var e = a.getElementsByTagName ("script")[0]; e.parentNode.insertBefore (r, e)}; "[object Opera]" == e.opera? a.addEventListener? a.addEventListener ("DOMContentLoaded", d,! 1): e.attachEvent ("onload", d ): d ()}}} t ("//mediator.mail.ru/script/2820404/"""_mediator") () (); 3r3-3559. 3r33565. 3r33561. 3r33565. 3r33565. 3r33565. 3r33565.