The transition to the Google Cloud platform (Google Cloud Platform - GCP)
[Часть 1 из 2]
3r314. Hike blog appeared December 1? 201? and then there were very few readers. By 201? we reached 100 million registered users and 40 billion messages per month. But such growth marked the problem associated with the scaling of our infrastructure. To eliminate it, we needed a high-performance platform at an affordable price. In 2016 and 201? we were faced with numerous interruptions in work, with this it was necessary to do something urgently, so we began to consider various options.
We needed a cloud platform that would quickly build, test, and deploy applications in a scalable and reliable cloud environment. At first glance, it may seem that all large cloud platforms are similar in many respects, but they have several fundamental differences.
We will divide this publication into 2 parts: 3r33258.
The reason for choosing GCP is
Switch to GCP without downtime
3r3342. Confirmation of the concept
We started with a proof of concept, within which we considered the compatibility of the existing infrastructure with the services offered by the Google Cloud cloud platform, and also planned elements for future development.
3r350. Key areas under the conceptual confirmation:
⊹ Networking and firewalls
⊹ Cloud availability
⊹ Big data
Confirmation of the concept included testing and checking the bandwidth of the virtual machines /network /load balancer, as well as stability, scalability, security, monitoring, billing, big data and machine learning services. In June 201? we made an important decision to transfer the entire infrastructure to the cloud platform Google Cloud.
We wanted to choose a cloud platform that can handle the innumerable problems we faced:
Нагрузки Load Balancer:
We had a lot of problems related to managing local HAProxy clusters to handle several tens of millions of active users daily connections. The global load balancer (GLB) has solved many of our problems.
Using global GCP load balancing, a single anycast IP address can forward up to 1 million requests per second to various GCP servers, such as Managed Instance Groups (MIG), and this does not require “preheating”. Our overall response time improved 1.7–2 times, because GLB uses a pool implementation that allows traffic to be distributed across multiple sources.
3r3-300. ⊹ Computer:
There were no major problems in the computers themselves, but we needed a high-performance platform at an affordable price. The overall throughput of Google virtual machines increased 1.3-1.5 times, which reduced the total number of running virtual machine instances.
Redis tests were conducted with a cluster of 6 copies (8 cores, 30 GB each). Based on the results below, we conclude that GCP provides performance improvements of up to 48% (on average) for most REDIS operations and up to 77% for specific REDIS operations.
redis-benchmark -h -p 6379 -d 2048 -r 15 -q -n 10?00?000 -c 100
The Google Compute Engine (GCE) cloud computing service has provided additional benefits in managing our infrastructure through the use of the following: 3r33258.
● Managed Instance Group (MIG): MIG helps us keep application services running in a reliable environment with multi-zone features instead of allocating resources for each zone. MIG automatically identifies and corrects inoperable instances in a group to ensure optimal performance of all instances.
● Dynamic migration: Live migration helps keep virtual machine instances running even in the event of a host system crash, for example, when updating software or hardware. Working with our previous cloud partner, we received a notification about a planned maintenance event and had to stop and start the virtual machine in order to switch to a workable virtual machine.
● Custom virtual machines: As part of the GCP, we can create our own virtual machines with the necessary processing power and memory for specific workloads.
3r3143. ⊹ Networking and firewalls:
Managing multiple firewall networks and rules is not easy and can lead to risk. The default network VPC GCP is global and provides inter-regional communication without additional configuration and without changes in network bandwidth. Firewall rules provide flexibility within the VPC for projects that use the name of a tag rule.
For a network with a low latency and a higher bandwidth, we were forced to select expensive instances with a bandwidth of 10 Gbit /s and activated the extended networks on these instances.
3r3-3160. ⊹ Security:
Security is the most important aspect for any cloud service provider. In the past, security was either not available for most services, or was only an additional option.
Google cloud services are encrypted by default. To protect data, GCP uses several levels of encryption. Using multiple levels of encryption protects the backed up data and allows you to choose the best approach based on the requirements of the application, for example, using the Identity-Aware Proxy service and encrypting inactive data by default.
In addition, GCP closes recent catastrophic vulnerabilities based on speculative execution in the vast majority of modern processors (Meltdown, Specter). Google developed a new binary modification method called Retpoline , which allows to bypass this problem and transparently make changes to the entire working infrastructure imperceptibly to users.
3r3178. ⊹ Cloud availability:
The availability of GCP resources is different from that of other cloud solutions providers, since most GCP resources, including the control panel, are either zonal or regional. We had to manage several VPCs for individual projects from separate accounts that needed a VPC peering or a VPN connection for a private connection. We also had to keep a copy of the image in a separate account.
In the Google Cloud, most resources are either global or regional. Such resources include the control panel (where we can see all the virtual machines of our project on one screen), disk images, data storage containers (several regions within the continent), VPC (but separate subnets are regional), global load balancing, publishing and subscription, etc.
3r3195. ⊹ Big dаta:
We moved from a monolithic, hard-to-manage analytical configuration to a fully manageable system from 3r to 3200. BQ [/u] , which led to improvements in three areas: 3r33258.
● Increase query processing speed up to 50 times.
● Fully managed data processing systems with automatic scaling.
● Data processing time was reduced from hours to 15 minutes.
It was difficult to compare different cloud providers because many services were not similar or comparable, differed for different usage scenarios and depended on unique usage scenarios.
Advantages of GCP:
● Long term use discounts: They are used for increasing the use of virtual machines when certain threshold values are reached. We can automatically receive a discount of up to 30% on workloads that are performed during most of the reference month.
● Per-minute billing: When allocating a virtual machine in GCE, a fee is charged for a minimum period of 10 minutes, after which per-minute billing begins for the actual use of the virtual machine. This provides a significant reduction in costs, because we do not have to pay a full hour, even if the copy of the machine is running for less than an hour.
● Excellent equipment, fewer copies: We found that for almost all levels and applications, using GCP, you can perform the same workload with the same performance, but fewer instances.
● Commitment, not reservation: Another factor is the approach of GCP to lplaet virtual machine instances. In AWS, the primary way to reduce the cost of a virtual machine copy is to purchase reserved copies for a period of 1-3 years. If the workload required a virtual machine configuration change, or we did not need this instance, we had to sell it on the market for reserved instances at a lower price. The GCP has a “Discount for Obligation to Use”, which is valid when reserving processor and memory resources, and it does not matter which virtual machine instances we use.
Based on this detailed analysis, we decided to switch to GCP and started working on the transition map and checklists. In the next article we will talk about what we learned during the implementation of this project.
It may be interesting
Wonder Woman Diana Prince Jacket
Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming.R_Tech
I feel really happy to have seen your webpage and look forward to so many more entertaining times reading here. Thanks once more for all the details.PBN
I have a mission that I’m just now working on, and I have been at the look out for such informationWeb designing