3r3666. 3r3-31. 3r3654. Our team is responsible for the operation and development of a large corporate product.
 3r3666. In early 201? having rested from a major introduction and re-reading "lessons learned", we firmly decided to revise the process of developing and delivering our application. We were worried about the low speed and quality of delivery, not allowing us to provide the level of service that our customers expect from us. 3r33655.
 3r3666. 3r3654. It was time to move from words to deeds - to change processes. 3r33655.
 3r3666. 3r3654. This article will briefly explain what we started with, what we did, what the situation is now, what difficulties we have encountered, what we had to leave behind brackets, what we are planning to do. 3r33655. 3r314. 3r376.
 3r3666. 3r318. Start 3r3-33534.
 3r3666.
A little about the system
 3r3666. 3r3654. The application is a classic example of a monolithic enterprise application of the "architectural spill of the 2000s": 3r3655.
 3r3666. 3r33541.  3r3666. 3r? 3567. Operated and developed over 15 years. 3r3-3568.  3r3666. 3r? 3567. It is a set of one and a half dozen WinForms, windows services and ASP .Net applications tied to a single MS SQL database. 3r3-3568.  3r3666. 3r? 3567. Codebase size: ~ 1MLOC for C #, ~ 9000 database objects. Much of the business logic is executed on the database side. 3r3-3568.  3r3666. 3r? 3567. The application consists of ~ 250 + solutions for creating a win /web client (one solution per group of related forms). This is the legacy of the previous development process and client architecture. 3r3-3568.  3r3666. 3r? 3567. The application supports several types of processes (clients) by changing the internal configuration: setting up processes, authorities, flexible fields, etc., in the system database configuration tables. At the same time, the application code base is the same for all clients. 3r3-3568.  3r3666. 3r? 3567. The application is deployed and supported on 25+ sites (each site is an independent copy of the system) and serves a total of several thousand end users in different time zones. 3r3-3568.  3r3666. 3r33570.
 3r3666.
The delivery process before transformation
 3r3666. 3r33333.  3r3666. 3r? 3567. development and assembly of the finished application and its components is carried out by the contracting organization. 3r3-3568.  3r3666. 3r? 3567. the code was stored on the side of the contracting organization (local version of MS TFS). The code is transmitted to the customer on a monthly basis in the form of an archive of the current version of the main repository branch. 3r3-3568.  3r3666. 3r? 3567. Delivery was carried out by delivery of "delta updates": for the application (set of dll, exe, etc.) and database components (set of sql scripts create /alter). The application was assembled and the delta packages were prepared by the contractor. 3r3-3568.  3r3666. 3r? 3567. the deployment process was supported by the transport system, the changes were applied automatically. 3r3-3568.  3r3666.
 3r3666. 3r3654. Delivery is carried out in the framework of monthly releases (as it was arranged with us, I told before 3r375. Here 3r3766.). 3r33655.
 3r3666. 3r380. Existing problems
 3r3666. 3r3654.
Lack of control 3r3493. 3r33655.
 3r3666. 3r33541.  3r3666. 3r? 3567. despite the formal possession of the code, the actual assembly of the application by the customer was impossible. 3r3-3568.  3r3666. 3r? 3567. as a result, it is impossible to verify the operation of the code transmitted to the customer. 3r3-3568.  3r3666. 3r? 3567. changes in the code - are not transparent to the customer. It is not possible to compare the requested and actual changes in the product. 3r3-3568.  3r3666. 3r? 3567. code analysis is difficult for SQL, and is not possible for C # components
 3r3666. 3r33570.
 3r3666. 3r3654.
The complexity and mistakes
3r33655.
 3r3666. 3r33541.  3r3666. 3r? 3567. Delta package preparation is a time-consuming procedure to develop, a source of errors and certain project costs. 3r3-3568.  3r3666. 3r? 3567. Deploying an application from the Delta Packages package requires order tracking. Error in breaking package order is a major deployment problem and the source of a significant portion of incidents. 3r3-3568.  3r3666. 3r? 3567. Regressions regularly occur: errors that seem to have already repaired and rolled out corrections to the product appeared again. 3r3-3568.  3r3666. 3r33570.
 3r3666. 3r3654.
Limitations 3r3493. 3r33655.
 3r3666. 3r33541.  3r3666. 3r? 3567. the ability to restore the state of the system to the moment in the past (roll back the changes) is actually absent 3r3-3568.  3r3666. 3r? 3567. the ability to effectively scale development resources and early testing by attracting employees of the customer is practically absent. 3r3-3568.  3r3666. 3r33570.
 3r3666. 3r3144. Expected results
 3r3666. 3r3654. At the beginning of the project, we set clear goals to solve the problems outlined above. 3r33655.
 3r3666. 3r33541.  3r3666. 3r? 3567. Move the code repository under the control of the customer 3r3-3568.  3r3666. 3r? 3567. Move the application building process to the customer side 3r33535.  3r3666. 3r? 3567. Modify the change propagation process by abandoning the "delta of change" in favor of a full update 3r-3568.  3r3666. 3r33570.
 3r3666. 3r3654. Additionally, using the solutions obtained when achieving the first two goals, we calculated: 3r3-3655.
 3r3666. 3r33541.  3r3666. 3r? 3567. Improve the technical quality of the solutions obtained by monitoring the code 3r33568.  3r3666. 3r? 3567. Increase the involvement and convenience of testing by providing self-service deployment. 3r3-3568.  3r3666. 3r33570.
 3r3666. 3r3181. Stages of the big road
 3r3666. 3r3185. Analysis of the current state of development processes
 3r3666. 3r3654. First step: analyze the existing contractor development process. This helped plan changes so that, if possible, do not interrupt work. 3r33655.
 3r3666. 3r3654. Unfortunately, familiarity with the development process showed that in the understanding of the IT industry of the present time - the process was absent. 3r33655.
 3r3666. 3r33333.  3r3666. 3r? 3567. The database code and business logic to it was not maintained in the repository up to date. The main reason: the lack of tools that implement the assembly of the code in the repository and the deployment of the result. So, the code in the repository is just documentation. 3r3-3568.  3r3666. 3r? 3567. The "real" version of the database code is in a common "development database", which has dozens of developers. 3r3-3568.  3r3666. 3r? 3567. Client application code (C #, ASP.NET) was maintained in the repository, but the quality and timeliness of the committees was not guaranteed. 3r3-3568.  3r3666. 3r? 3567. The assembly of components (not the entire application) was carried out at developer stations. It is not entirely clear how the code was updated before assembly. The assembled component was laid out on a shared shared folder. From there, the "delta package" was formed for the customer. 3r3-3568.  3r3666. 3r? 3567. The complete lack of practice of branches development. By indirect evidence, we suspected this a long time ago - but after immersion in the process everything became obvious. 3r3-3568.  3r3666.
 3r3666.
Transition to a new repository and version control system
 3r3666. 3r3654. Dependence on MS platforms and corporate standards predetermined the choice of development environment - Team Foundation Server.
 3r3666. However, by the time we started the project directly (April 2017), a version of Visual Studio Team Services had just been released. The product seemed very interesting, was designated as a strategic direction for MS, offered git repositories, build and deployment for on-prem and cloud. 3r33655.
 3r3666. 3r3654. Corporate on-prem TFS was lagging behind the version and functionality of VSTS, migration to the new version was only in the process of discussion. We did not want to wait. We decided to go straight to VSTS, as it reduced our overhead for supporting the platform and gave us full control over how and what we were doing. 3r33655.
 3r3666. 3r3654. At the time of the change, the development team had experience with TFSVC, the application code was stored in such a repository. On the other hand, GIT has actually become a standard for the IT community long ago — the customer and third-party consultants recommended switching to this system.
 3r3666. We wanted the development team to be involved in making a decision on a new version control system, and made an informed choice. 3r33655.
 3r3666. 3r3654. We deployed two projects in VSTS with different repositories - TFSVC and GIT. A set of scenarios were identified that were proposed to test and evaluate the usability of each system. 3r33655.
 3r3666. 3r3654. Among the estimated scenarios were: 3r???.
 3r3666. 3r33541.  3r3666. 3r? 3567. Create a merge of
branches.  3r3666. 3r? 3567. Organizing collaboration (on the same or different branches) 3r33535.  3r3666. 3r? 3567. Operations on chains of changes (commit, cancel) 3r3-3568.  3r3666. 3r? 3567. Integration of third-party changes 3r33535.  3r3666. 3r? 3567. The ability to continue to work when the server is unavailable. 3r3-3568.  3r3666. 3r33570.
 3r3666. 3r3654. As a result, as expected, GIT was chosen, and so far no one regretted it. 3r33655.
 3r3666. 3r3654. We started using GitFlow as a process. This process provided enough control over the changes and allowed delivery of releases, as we have become accustomed to. 3r33655.
 3r3666. 3r33333.  3r3666. 3r? 3567. We defended the develop branch with a policy that required all changes to go through pull requests. 3r3-3568.  3r3666. 3r? 3567. We try to adhere to the practice of "one ticket - one pull-requack". Changes from different tickets are never combined within a single change. We try to do our best testing on the feature branch to avoid the situation with corrections in subsequent pull-requests. 3r3-3568.  3r3666. 3r? 3567. When you merge into develop, all changes are merged into one commit (squash). 3r3-3568.  3r3666. 3r? 3567. Release branches are created from develop. 3r3-3568.  3r3666. 3r? 3567. If necessary, the latest changes can be added to the release branch selectively (cherry-pick) or all (rebase). We do not fix the fix directly in the release branch. 3r3-3568.  3r3666. 3r? 3567. After deploying the last release to the product, he goes to the master via push force (only a few people have this right)  3r3666.
 3r3666.
Automate product assembly
 3r3666. 3r3654. The application consisted of a large number of assemblies, hundreds of solutions. As it turned out during the audit process, all this was collected separately and "manually."
 3r3666. In the first stage, we decided not to redo everything “from scratch” (so as not to stop the existing delivery), but to “wrap” the assembly into the msbuild set of scripts — one script per component.
 3r3666. Thus, we quickly obtained scenarios that carried out all the necessary intermediate artifacts, and in the end - the finished product. 3r33655.
 3r3666. 3r3654. A separate story is a database project. Unfortunately, the system contains several CLR components that were not well structured. Dependencies do not allow a simple database deployment. At the moment, this is solved by the pre-deployment script.
 3r3666. In addition, due to the uneven system landscape (SQL Server versions 2008 and 2014 were installed at different points), the project base assembly for .Net versions 2.0 and 4.0 had to be organized. 3r33655.
 3r3666. 3r3654. After all the scripts were ready and tested, they were used in the build VSTS scripts. 3r33655.
 3r3666. 3r3654. Immediately before the start of the assembly, the versions of all products were updated to a common standard number, including the pass-through build number. The same number was saved in the post-deployment script. Thus, all the components: the database and all client applications — came out consistent and equally numbered. 3r33655.
 3r3666.
Deploying to test stand
 3r3666. 3r3654. Once the primary version of the build process has been completed, we proceed to the preparation of the deployment script. 3r33655.
 3r3666. 3r3654. As expected, the database was the most troublesome. 3r33655.
 3r3666. 3r3654. Deploying on top of a copy of the real database revealed many conflicts between the build and the state of real systems: 3r3655.
 3r3666. 3r33541.  3r3666. 3r? 3567. Inconsistent versions in GIT and in the real system 3r356868.  3r3666. 3r? 3567. DB schemas owned by users that were planned to be deleted. 3r3-3568.  3r3666. 3r33570.
 3r3666.
Stabilization of the development process
 3r3666. 3r3654. This, of course, is strange to talk about, and even more so to write here, but the most serious change for developers was the introduction of the principle “if this is not in git, this does not exist”. Previously, the code was committed "for reporting to the customer". Now - without this, it is impossible to deliver anything. 3r33655.
 3r3666. 3r3654. The hardest thing was with the database code. After moving to the deployment of the database from the repository, through the assembly and deployment using sqlpackage, the "delta" approach was replaced by the "desired state" approach. Packages were a thing of the past, everything had to be deployed automatically. 3r33655.
 3r3666. 3r3654. But! Until the full transition to the new deployment process, it was still necessary to deliver the changes. And it was necessary to do it in the old manner - "delta updates." 3r33655.
 3r3666. 3r3654. We were faced with the task of ensuring full and constant consistency of the state of the system upon delivery of delta packages, and the contents of the repository. 3r33655.
 3r3666. 3r3654. To do this, we organized the following process:
 3r3666. 3r33333.  3r3666. 3r? 3567. Regularly, the code from the repository was collected and deployed to an empty “model” database. 3r3-3568.  3r3666. 3r? 3567. Based on the “model” base, a special autotest was prepared. For each object of the "model" database, checksums were calculated. The autotest contains all these checksums and, at startup, calculates the checksums of the corresponding objects of the "checked" database. Any discrepancy in the composition of the objects or their checksums leads to a drop in the test. 3r3-3568.  3r3666. 3r? 3567. The “falling” test automatically prohibited the transfer of packages from the test environment further along the landscape. Such integration has already been implemented in the previous transport system. 3r3-3568.  
 3r3666. 3r3654. Thus, with the help of automatic control, it was possible to relatively quickly bring the product database code to git in its current state and maintain it without additional efforts from the project team. At the same time, the developers began to get used to the need to correctly and promptly commit the code to the repository. 3r33655.
 3r3666. 3r3333391. Deploying the product to the integration test environments
 3r3666. 3r3654. After the previous stage was completed, we proceeded directly to deploying the application on a test environment. We completely stopped using delta packages for test systems and switched to automatic deployment using VSTS. 3r33655.
 3r3666. 3r3654. From that moment, the whole team began to receive the first fruits from the efforts expended earlier: the deployment took place without any additional efforts. The custom code is automatically built, deployed, and tested. 3r33655.
 3r3666. 3r3654. Unfortunately, as we understood later, the "alignment of the repository" carried out led to the fact that we had a version of the stably supported version of "develop", but the version of "production" was also not available. And therefore, beyond the test environment - there was nothing to go to QAS and PRD with. 3r33655.
 3r3666. 3r3654. The application code on the database side could be compared with the productive one and understand the differences. There was nothing to compare client applications with - there was only the actual productive version in the form of a set of executable files, and from which they were assembled reliably it was impossible to say. 3r33655.
 3r3666. 3r33411. Product testing as a result of automatic build
 3r3666. 3r3654. After changing the approach to assembly, the product had to be subjected to a large regression test. It was necessary to make sure that the application is running and nothing is lost.
 3r3666. When testing just got easier with the functionality placed on the side of the database. By the way, we had a considerable set of autotests, covering critical areas. 3r33655.
 3r3666. 3r3654. But there were no tests for C # - so everything was checked by hand. It was a significant amount of work, and the check took some time. 3r33655.
 3r3666. 3r33434. "Leap of faith" - a pilot deployment to produce
 3r3666. 3r3654. Despite the testing, deploying to production was the first time scary. 3r33655.
 3r3666. 3r3654. We were lucky - we had just scheduled the next deployment of the system at the new site. And we decided to use this chance for a pilot deployment.
 3r3666. Users did not see, it was easy to fix possible errors of the assembly, the real productive work had not yet begun. 3r33655.
 3r3666. 3r3654. We deployed the system, and for several weeks it was in the mode of pre-productive use (low load, a certain pattern of use, which can be skipped in production). During this time, several defects were revealed during testing. They were corrected as they were found, and the new version was immediately rolled out for checking. 3r33655.
 3r3666. 3r3654. After the official launch and the week of post-launch support, we announced that this is the first copy assembled and delivered "in a new way". 3r33655.
 3r3666. 3r3654. This version of the assembly became the first stable version of the master branch, it was hung with holiday tags "fisrt_deployment" (we didn’t order the badges with the commit hash, however). 3r33655.
 3r3666.
Scaling the deployment to the entire productive landscape
 3r3666. 3r3654. As James Bond said: "the second time is much easier." After the success of the pilot deployment, we quickly connected the remaining instances of systems of a similar type. 3r33655.
 3r3666. 3r3654. But the system has several types of use - one functionality can be used for one type, and not used in other cases. Accordingly, the functionality tested on the implementation of the first type did not necessarily guarantee success for other cases. 3r33655.
 3r3666. 3r3654. To test the functionality of the remaining types of use, we began to use active projects that were under development. The idea was similar and the first deployment - we began to use automatic assemblies, "slipping" them to users along with the project functionality. Thus, users, working with the "project" version of the product at the same time tested and the old functionality. 3r33655.
 3r3666. 3r3654. Scaling in itself revealed unexpected technical problems: 3r33655.
 3r3666. 3r3654.
Non-uniform system landscape
 3r3666. In addition to directly deploying the application, we first had to take care that everything was the same everywhere - the .Net versions, Powershell, and modules. It all took a fair amount of time. 3r33655.
 3r3666. 3r3654.
Network connection
 3r3666. At some sites, the network connection simply did not allow all the components of the assembly to be pumped. There were timeouts, damage in the process of transfer. A lot of things checked and tried - not very successfully. 3r33655.
 3r3666. 3r3654. I had to dwell on the following solution: the build script was finalized so that all the results were packed into one large archive, which was then cut into small fragments (2 MB each). We finalized the deployment scenario to eliminate concurrency when downloading artifacts, took all 2 megabyte fragments and restored from them what is already possible to deploy. 3r33655.
 3r3666. 3r3654.
Conflict with antivirus
 3r3666. Another strange problem we are facing is antivirus software conflict and one of the deployment steps: when any “suspicious” files, such as .js, .dll, are extracted from artifact archives, the antivirus starts to look at them closely. And the strangest thing is that the antivirus starts to rush to the file even before the end of the unpacking and the unpacking process drops with the message "the file is busy by another process". While we are struggling with this, excluding the local folder with artifacts from scanning is not very good, but nothing else has been invented. 3r33655.
 3r3666. 3r3499. Process Improvement
 3r3666. 3r3654. After stabilization of the assembly and deployment processes, we switched to “sewing shoes for shoemakers” - improving internal processes. 3r33655.
 3r3666. 3r33541.  3r3666. 3r? 3567. built the basic integration of the corporate IT support system (service-now.com) with VSTS for transferring tickets in the form of Work Items. We have corrected the merge policy in develop - now we will definitely require the presence of a connected ticket. 3r3-3568.  3r3666. 3r? 3567. CI connected to all feature branches. As soon as the code change is published, the version is automatically compiled and ready for testing 3r3-3568.  3r3666. 3r? 3567. on the side of the contracting organization, they deployed a large set of test benches and organized a “self-service” deployment of the assemblies for internal testing 3r-3568.  3r3666. 3r? 3567. On the customer side, several test benches were also deployed - for each type of system. Any developer or employee of the department can initiate the deployment, the deployment permission is given by the employee responsible for the type of system and coordinating testing. 3r3-3568.  3r3666. 3r? 3567. organized the infrastructure for working on projects: special naming of project branches, CI /CD for dedicated instances of the system with which developers, analysts and end-business users work 3—3–3568.  3r3666. 3r? 3567. integrated storage of auto-tests into a common repository (previously tests were stored separately) and build and deploy processes. Now the assembly of the product contains code and tests - both for development branches and for the product as a whole. 3r3-3568.  3r3666. 3r? 3567. as an experiment, they installed VSTS agents on the local computers of analysts who are engaged in testing and deploying the client part there, which needs to be tested for assembly. 3r3-3568.  3r3666. 3r33570.
 3r3666. 3r33333. Results
 3r3666. 3r33537. Current situation
 3r3666. 3r33541.  3r3666. 3r? 3567. All application code is stored and maintained in MS VisualStudio Team Services (more recently, Azure Devops) managed by the client. Version Control Systems - GIT
 3r3666. 3r? 3567. All changes are completely transparent and tied to service tickets (incidents /changes) 3r356868.  3r3666. 3r? 3567. Due to the transition to git /GitFlow, parallel development has been greatly simplified. 3r3-3568.  3r3666. 3r? 3567. A code review procedure was introduced with the participation of a customer representative. 3r3-3568.  3r3666. 3r? 3567. The application is built by the CI system. Full assembly is carried out for both the main and feature branches, which allows you to organize early testing. 3r3-3568.  3r3666. 3r? 3567. Deploying the application to all target nodes is based on scripts. Gradually, the configuration steps for the base components and applications are added to the application deployment steps. 3r3-3568.  3r3666. 3r? 3567. Employees of the customer’s IT department can initiate the deployment of assemblies (final or intermediate) to test environments - for testing or early familiarization. Business users also have access to test environments. 3r3-3568.  3r3666. 3r? 3567. we still adhere to the main release cycles of 1 month. Although in the last month we are rolling out something practical weekly. 3r3-3568.  3r3666. 3r? 3567. The process of an "experimental" rollout of a version to some sites appeared 3r3-3568.  3r3666. 3r33570.
 3r3666. 3r33573. Time on stages
 3r3666. 3r? 3577.  3r3666. 3r???.  3r3666. 3r32424.  3r3666. 3r3-3589. No. 3r33590.  3r3666. 3r3-3589. Description stage 3r33590.  3r3666. 3r3-3589. Duration 3r33590.  3r3666. 3r33635.  3r3666. 3r? 3594.  3r3666. 3r? 3596.  3r3666. 3r32424.  3r3666. 3r33232. 1 3r33333.  3r3666. 3r33232. From the start of the project, to full control over the code, the build process and delivery to the test environment 3r363333.  3r3666. 3r33232. 6 months 3r3333633.  3r3666. 3r33635.  3r3666. 3r32424.  3r3666. 3r33232. 2
 3r3666. 3r33232. From the first deployment to the test environment - to the first pilot release on the product 3r-33333.  3r3666. 3r33232. 3 months 3r3333633.  3r3666. 3r33635.  3r3666. 3r32424.  3r3666. 3r33232. 3 3r33333.  3r3666. 3r33232. From pilot deployment to production — until the first release on all instances of 3r-33633.  3r3666. 3r33232. 5 months 3r3333633.  3r3666. 3r33635.  3r3666. 3r3637.  3r3666.
 3r3666. 3r3654. Total duration - 14 months
 3r3666. 3r3654. The duration, especially at the final stage, was largely determined by the coordination and coordinated system maintenance calendar. 3r33655.
 3r3666. 3r3650. Labor costs
 3r3666. 3r3654. The total cost of the involved employees of the customer and the contractor for all work related to the change is approximately 250 people * days. 3r33655. 3r3662. 3r3666. 3r3666. 3r3659. ! function (e) {function t (t, n) {if (! (n in e)) {for (var r, a = e.document, i = a.scripts, o = i.length; o-- ;) if (-1! == i[o].src.indexOf (t)) {r = i[o]; break} if (! r) {r = a.createElement ("script"), r.type = "text /jаvascript", r.async =! ? r.defer =! ? r.src = t, r.charset = "UTF-8"; var d = function () {var e = a.getElementsByTagName ("script")[0]; e.parentNode.insertBefore (r, e)}; "[object Opera]" == e.opera? a.addEventListener? a.addEventListener ("DOMContentLoaded", d! ): d ()}}} t ("//mediator.mail.ru/script/2820404/"""_mediator") () (); 3r3660. 3r3666. 3r3662. 3r3666. 3r3666. 3r3666. 3r3666.