Google Cloud Platform Blog: Google Compute Engine uses Live Migration technology to service infrastructure without application downtime

Google Cloud Platform Blog

Product updates, customer stories, and tips and tricks on Google Cloud Platform

Google Compute Engine uses Live Migration technology to service infrastructure without application downtime

2015年3月3日星期二

IntroductionGoogle Compute Enginetransparent maintenance The Benefits of Transparent Maintenance

Regular infrastructure maintenance and upgrades

Network and power grid maintenance in the data centers

Bricked memory, disk drives, and machines

Host OS and BIOS upgrades

Security-related updates, with the need to respond quickly

System configuration changes, including changing the size of the host root partition, for storage of the host image and packages

unexpected

Flapping network cards — Network cards were intermittently failing. We were able to repeatedly try the VM migrations and successfully migrate them. This even worked with partially-failing NICs.

Cascading battery/power supply issues — Some overheating batteries were overheating the neighboring machines. We were able to migrate the VMs away before bringing down the machines to swap out their batteries.

A buggy update was pushed to production — We halted the rollout, but not before it reached some of our production machines (it didn't manifest in our canary environment). The buggy software would’ve crashed VMs within a week. Instead, we migrated the VMs on the affected machines to other hosts that didn’t have the buggy software.

Unexpected host memory consumption — One of our backend components consumed more memory than we had allocated and threatened to OOM (out of memory) the VMs. We migrated some VMs away from the over-loaded machines and avoided the OOM failures while patching the backend system to ensure it could not overrun its allocation.

Transparent Maintenance in Action “We took a look at our log files and all the data in the database and we saw…nothing unusual. In other words, if Google hadn’t told us that our instances had been migrated, we would have never known. All our logs and data looked normal, and we saw no changes in the RightScale Cloud Management dashboard to any of our resources, including the zone, instance sizes, and IP addresses.” “Just tested @googlecloud live migration of a @MongoDB replica set - no impact. None of the nodes noticed the primary was moved!” How it works

imminent

During pre-migration brownout, the VM is still executing on the source, while most state is sent from the source to the target. For instance, we copy all the guest memory to the target, while tracking the pages that have been re-dirtied on the source. The time spent in pre-migration brownout is a function of the size of the guest memory and the rate at which pages are being dirtied.

During blackout, which is a very brief moment when the VM is not running anywhere, it is paused, and all the remaining state required to begin running the VM on the target is sent. We go into blackout when sending state during pre-migration brownout reaches a point of diminishing returns. We use an algorithm that balances numbers of bytes of memory being sent against the rate at which the guest VM is dirtying pages, amongst other things.

During post-migration brownout, the VM executes on the target. The source VM is present, and may be providing supporting functionality for the target. For instance, until the network fabric has caught up with the new location of the VM, the source VM provides forwarding services for packets to and from the target VM.

logs Conclusionimmortality

Free Trial

GCP Blogs

Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Firebase Blog
Apigee Blog

Popular Posts

World's largest event dataset now publicly available in BigQuery
Understanding Cloud Pricing
A look inside Google’s Data Center Networks
Enter the Andromeda zone - Google Cloud Platform’s latest networking stack
Getting your data on, and off, of Google App Engine

标签

Announcements 193
Big Data & Machine Learning 134
Compute 271
Containers & Kubernetes 92
CRE 27
Customers 107
Developer Tools & Insights 151
Events 38
Infrastructure 44
Management Tools 87
Networking 43
Open 1
Open Source 135
Partners 102
Pricing 28
Security & Identity 85
Solutions 24
Stackdriver 24
Storage & Databases 164
Weekly Roundups 20

Feed

Subscribe by email

Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.

Technical questions? Check us out on Stack Overflow.
Subscribe to our monthly newsletter.

Company-wide

Official Google Blog
Enterprise Blog
Student Blog

Products

Official Android Blog
Chrome Blog
Lat Long Blog

Developers

Ads Developer Blog
Android Developers Blog
Developers Blog

Google
Privacy
Terms