Inside kitchen Veeam: how the R & D process is arranged
Evening. The next R & D-interview is coming to an end, and our interviewers are tuned to unexpected questions from a future colleague. But no surprises: the ratio, deduced Wilfredo Pareto, works here. In 80% of cases we hear four questions - about 20% of the total number of received. How is your process? What will I do? How to become a senor /timlid for a year? What about the relocation to Europe?
In this post we will answer the first question and tell you about the development process in Veeam - from team to team this response changes least.
So, the process. This is a repeated sequence of actions leading to success from day to day, well, or at least sometimes. You have learned how to cook borsch and each time you get the same delicious - the process. Do not park yourself for knocking - you learned the process. The process allows the brain not to think every time over the routine, turning it into a mechanical work. The brain is released for creativity.
The development process is a sequence of actions that transform the desires of users into tangible products. These desires are formulated by analysts and product managers, implemented by developers, critically evaluated by the testers, described by the technicians.
We in Veeam make mass products for backup and replication of data centers - so that nothing is lost. A classic boxed product without a specific customer. At first glance, the thing is simple, but there are nuances, so we are doing it for the second decade already.
Each release is the result of the work of several groups:
Product managers, or analysts . They set priorities in their work, communicating with the outside world - customers and partners. Partnership can be technological as well. For example, the distributor can tell you what it lacks to increase sales, and the hypervisor manufacturer tells you about plans for the future. For this team, it is important to have the "ability to speak", the ability to catch and prioritize trends in a turbulent flow of opinions. And then defend the chosen position, say no, explain why something is done just like that, and not otherwise. It does not matter in press releases, at conferences or privately. These people are training the world of sales.
Technical support service . Our customers trust telephone. The most important indicators of the team - the response time to the problem and the time of its resolution (SLA). During the month, about a thousand applications are processed. The team is multilayered, it includes both groups of interaction with clients, and groups of analysts of appeals, development of workarounds, etc. Based on the information received by the support, a list of improvements and urgency is formulated - whether to implement it in a private fix, the next update, or postpone for a major release.
R & D-developers . People who materialize requests in code.
Testers, or QA . First-timers, a tank test site and a vibration table at the same time. They not only check what has already been implemented - but also connect to work at the design stage. Repeating the tasks of administrators in the infrastructure, close to combat, it is easier to understand how convenient the created interface or the selected algorithms are. When technical support comes to the conclusion that there is a defect in the product, QA reproduces problematic scenarios and monitors the fixes.
A team of technical writers. They create documentation for end users, as well as specific documents such as "How it works" and "Deployment guide". The material for the work they receive from developers, testers and analysts.
Some teams prefer open space, but more often - cabinets
Teams are contacted through the requirements registration system. We implemented it on the basis of Microsoft Team Foundation System, because we historically used it as a source version storage system. The system stores requirements, defects (bugs) and support calls (issues) that require the participation of QA and developers. Each issue involves hundreds of participants who work on thousands of tasks, requirements and defects. The system helps to keep all this and, more importantly, evenly distribute the load, set priorities for developers.
Annual rings: development cycles
The development of our product is cyclical, each cycle ends with the release of the next version - a release. Here is what is reflected in the releases:
Long-term trends on the market . For example, virtualization and the emergence of cloud infrastructures. Changing the paradigms of IT work takes years - at this time, users shift from suspicion and denial ("what is this crap") to mass recognition ("yes everyone does that"). The virtualization of data centers in its time gave birth to Veeam, because in the virtualization environment, old products for backing up machines were inefficient.
Support for new platforms . Once upon a time, Veeam was intended only for virtualized data centers on the VMWare platform. With the growth in the number and size of customers, there was a need to support new platforms. In addition to VMWare appeared other hypervisors (Hyper-V), physical servers, cloud platforms (AWS, Azure), etc.
Tactical changes on the market . The next versions of operating systems and hypervisors are released. The experience of using previous versions of our product is accumulating. For example, this is how we got support for item-level - selective recovery from popular application servers, such as Microsoft Exchange, Microsoft SQL Server, Oracle Databases, etc.
Defects . Despite all our efforts, life is no-no, and presents surprises. Of course, we try to keep them to a minimum.
Every quarter we have updates (updates), about once a year - major (major) releases. They are good in that they minimize the overhead of creating large-scale functionality related to the support of new platforms and the paradigm shift. Based on the specifics of budgeting, IT departments of clients are most active at the end of the year, so we also roll out big releases at this time.
Quarterly updates mainly pursue two purposes: support for new versions of protected servers and elimination of defects. In updates, we collect all the bug fixes that were discovered by customers since the release of the major version. Also with the help of updates we quickly react to changes in supported platforms. Under SLA, Veeam should add support for the new version of the hypervisor not more than three months .
And what if the product does not work for a particular client? In this case, we issue a private correction - in other words, a crutch. Such fixes are issued every week and are not always associated with defects in the product. For example, the client's security settings may be incompatible with the product. Issue a private fix, we analyze the frequency of the problem and decide whether to include the fix in the subsequent quarterly update.
From dawn to dusk: a chronicle of the release
Each release cycle begins with planning - at the product level in general and at the level of a single requirement. In the first case, the question of business priorities and the distribution of requirements between the teams is decided. First of all, the most urgent requirements or epic are discussed. In an amicable way, it is worthwhile to allocate not more than 60% of the total volume of work on the release to the epic, so that there is a cushion of time. Product planning is carried out at the level of heads of departments - products, testers, developers.
Developers and testers are divided into teams. The optimal number of people in the team is five. Teams can be both functional and universal. In the latter case, the team is self-sufficient, contains developers with expertise in several areas. Functional commands are more focused - they work on the user interface, system components, etc. People from different functional teams form virtual teams that begin implementing requirements. Here, at a minimum, representatives of the PM group, development and testing teams gather. Responsible for the requirement is assigned timlid one of the functional commands.
Work begins on the demand. A virtual team meets weekly. Participants tell about the successes over the past week and plan to work on the next.
Responsible for the requirement timlid moderates meetings and records the results. He also solves issues that can not be solved within a virtual team. For example, if you need to move the deadlines or postpone part of the tasks.
Inside the functional development or testing commands, the control points are more densely packed. As a rule, the weekly plan is divided into tasks of no more than two to three days. In some teams, spam practices have taken root with daily flying, somewhere the point-to-point interaction between the team and the team is more effective.
A typical conversation to discuss the current status of the project
The entire development is divided into weekly or bi-weekly iterations. During the first iterations, you need to create a minimally working functional that will later be fouled - for example, an extended user interface, API for clients, etc. And most importantly, the presence of the skeleton already allows the testers to get the feature.
The release cycle includes alpha and beta releases. With their help, we show new functions to the outside world and collect feedback in advance. If necessary, decisions on architecture or functionality are changed. Prior to the state of alpha and beta scenarios are not brought immediately, and packs. As a rule, there are about two bet in the release cycle.
After the beta stage, the teams go into regression testing mode. At this stage, the product as a whole already works, the user interface and the work scenarios have already hardened and change with less intensity. In the case comes a team of technical writers. At the same time, technical support teams are trained.
Regression testing is carried out in two-week cycles. The cycle time is determined by the time it takes to view all product scenarios. The larger the product, the more scenarios and, correspondingly, the cycles. Before the last cycle, a code is announced. This means that the product will only be brought to the critical changes - and only after numerous code reviews. Such draconian methods are needed in order to not accidentally introduce new defects into the product.
The closer the time of release, the more free time for developers and less - for everyone else. Product managers need to prepare press releases, coordinate the work of marketing services. Testers should check the fixes and perform the final regression testing. Technical writers add user documentation. At this fertile time, developers are deploying research activities for the requirements of the next version.
And here it is an exciting moment, called RTM - Ready To Market. This means that the product is already ready for transfer to consumers. Journalists, service providers will torment him for two weeks. It will be shown on presentations. Two weeks later, the product will be available to all. At this time, there are internal changes: new branches are being prepared, code is being deposited. And, of course, the build infrastructure is being upgraded to the next version. After the public release (GA), there is a hot time for technical support. And the others are already working on the next version.
On the priorities
And finally a bit of materiel. As you know, in the trinity "fast, high-quality, inexpensive" you can choose a maximum of two options. Quality, timing and functionality are constantly fighting among themselves. At us in a boxed product quality costs on the first place. Hmm, and that there is some area where quality does not matter? Of course not. The whole issue in determining quality.
For us, the quality is:
Preservation of reliability and performance in the zoo of configurations . One client has a modest data center of two Borodino battle servers, while the other has a high-end infrastructure in a nearby hangar with Amazon. The product should work adequately in both cases.
Easy to use . The user must tense to the minimum and certainly cope without any instructions. But behind the external simplicity is not always hidden the same simple code - try to cross the horror with the hedgehog seamlessly.
Heritability . Investments from enterprises are multi-year, and financial directors will not be wasted on IT without a good reason. So you need to maintain compatibility with previous versions, and with related products. Often, when the data centers are rebuilt, the postal servers of the 1980s are walled in. And they all buzz and do not think about dying.
With this set of priorities for quality preservation, you must always combine something, both to developers and testers. Small changes in the behavior of functions can lead to a forced integration re-testing of the lion's share of the product. Try adding support for Asian locales to the product and understand what it's about. Therefore, the issue of priorities is a matter of joint discussion between PMs, testers and developers.
The second, almost inviolable priority is timing. In the case of updates, the release dates are set by the SLA, in the case of large releases, the business calendar. According to statistics in each release cycle, almost 50% of the time is spent on development, 50% - to bring the product to the mind (the stage of bugfix).
What can change is the functionality of the next release. Here, a prioritized list of requirements, or backlog, helps. Theoretically, everything is simple: choose from the backlog the next highest priority function, look for the remaining time. When the time is close to the end, stop and release the next version of the product. The devil is hidden in nuances:
Uncertainty of the requirements . For example, the requirement "to support the backup of physical machines on Linux OS" may be further refined. Which kernels should be supported? What are the distributions? What file systems? Oneand the same high-level requirement can be implemented both for the month and for the year. The question is in fullness.
Teams have specializations . Not every requirement can be taken by any team. The C # developer will not write the drivers, the developer of the system components will not always cope with the web development.
Requirements depend on each other . This is not always visible at the level of user scripts, but there are such connections inside. From the perspective of the outside world, backup support from NTFS and ExtFS file systems can be requirements with different priorities, but inside at first you will need to write a common engine.
The requirements are divided into delayed and non-postponed . If the market is waiting for another version of a function, and it was announced, it will not be postponed.
Part of the requirements assumes the research work . Without the results of research, it is impossible to plan the complexity of the task (maybe it is not feasible at all), and it is difficult to predict these results.
Flexible development comes into play here. For us, flexible development means the need for periodic rescheduling. New circumstances became known - they changed plans. Added new priority requirements in backlog - changed plans. We do not have time to postpone the requirement - we cut some tasks or change the requirement. In the theory of technical management, this is called a feedback system. Remember how autopilot works.
Any planning in the conditions above is based on peer review. The expert evaluation of the person responsible for the task of timlide is the element that makes the whole subsequent process clear and structured. Another name for the expert evaluation is "the method of Lenin's squint", as Alexander Orlov of Stratoplan likes to repeat. This is when you make a decision based on previous experience and intuition. Despite the possible criticism, it works. Works, like all our processes described above. If you have any questions left on them, we invite you to comment.
The current technical process is comfortable and cozy as home slippers. The only problem is that in Veeam some invisible awl always urges: faster, more, more, more.
We have recently built pilot offices in Finland and the Czech Republic, and this year we are preparing for the large opening of the Prague R & D center for several hundred people.
The lobby of our Prague office is
Recently, a development office has been established in Israel, teams in Canada and Germany are growing. The number of joint development projects with HP, NetApp, Nutanix, EMC is increasing.
Manufacture is transformed into a geographically distributed conveyor, and at the same time new processes are crystallized. However, this is a topic for a separate article.
It may be interesting
Corvus Health provides medical training services as well as recruiting high quality health workers for you or placing our own best team in your facility. Check Out: Health Workforce Recruitment