Google Cloud Platform Blog: CRE

Last month today: July on GCP

2018年8月3日星期五

What caught your attention this month: Creating the open cloud

One of the most-read posts this month covered the launch of our Cloud Services Platform, which allows you to build a true hybrid cloud infrastructure. Some of the key components of Cloud Services Platform include the managed Istio service mesh, Google Kubernetes Engine (GKE) On-Prem and GKE Policy Management, Cloud Build for fully managed CI/CD, and several serverless offerings (more on that below). Combined, these technologies can help you gain consistency, security, speed and flexibility of the cloud in your local data center, along with the freedom of workload portability to the environment of your choice.
Another popular read was a rundown of Google Cloud’s new serverless offerings. These include core serverless compute announcements such as new App Engine runtimes, Cloud Functions general availability and more. It also included serverless containers, so you can run serverless workloads in a fully managed container environment; GKE Serverless add-on to easily run serverless workloads on Kubernetes Engine; and Knative, the open-source project on which that add-on is built. There are even more features included in this post, too, like Cloud Build, Stackdriver monitoring and Cloud Firestore integration with GCP.

Bringing detailed metrics and Kubernetes apps to the forefront

Another must-read post this month for many of you was Transparent SLIs: See Google Cloud the way your application experiences it, announcing the availability of detailed data insights on GCP services that your workloads use—helping you see like a Google site reliability engineer (SRE). These new service-level indicators (SLIs) go way beyond basic uptime and downtime to delve into response codes, latency and more. You can then separate out metrics by GCP service to see things like API version, location and protocol. The result is that you can filter and sort to get extremely fine-grained information on your software and the GCP services you use, which helps cut resolution times and improve the support experience. Transparent SLIs are available now through the Stackdriver monitoring console. Learn more here about the basics of using SLIs and other SRE tools to measure and manage availability.
It’s also now faster and easier to find production-ready commercial Kubernetes apps in the GCP Marketplace. These apps are prepackaged and configured to get up and running easily, whether on Kubernetes Engine or other Kubernetes clusters, and run the gamut from security, data analytics and developer tools to storage, machine learning and monitoring.

There was obviously a lot to talk about at the show, and you can get even more detail on what happened at Next ‘18 here.

Building the cloud back-end

For all of you developing cloud apps with Java, the availability of Jib was an exciting announcement last month. This open-source container image builder, available as Gradle and Maven plugins, cuts out several steps from the Docker build flow. Jib does all the work required to package your app into a container image—you don’t need to write a Dockerfile or even have Docker installed. You end up with faster builds and reproducible container images.
And on that topic, this best practices for building containers post was a hit, too, giving you tips that will set you up to run your environment more smoothly. The tips in this blog post cover graceful application shutdowns, how to simplify containers and how to choose and tag the container images you’ll use.

It’s been a busy month at GCP, and we’re glad to share lots of new tools with you. Till next time, build away!

SRE fundamentals: SLIs, SLAs and SLOs

2018年7月19日星期四

By Jay Judkowitz, Senior Product Manager and Mark Carter, Group Product ManagerGoogle Cloud Next ‘18Forrester report on infrastructure transformationBen TreynorNext ‘18 Spotlight sessionRegister now Defining the terms of site reliability engineeringSLOs, SLIs, SLAs, oh my - CRE life lessons 1. Service-Level Objective (SLO)
SRE begins with the idea that a prerequisite to success is availability. A system that is unavailable cannot perform its function and will fail by default. Availability, in SRE terms, defines whether a system is able to fulfill its intended function at a point in time. In addition to being used as a reporting tool, the historical availability measurement can also describe the probability that your system will perform as expected in the future.

When we set out to define the terms of SRE, we wanted to set a precise numerical target for system availability. We term this target the availability Service-Level Objective (SLO) of our system. Any discussion we have in the future about whether the system is running sufficiently reliably and what design or architectural changes we should make to it must be framed in terms of our system continuing to meet this SLO. Keep in mind that the more reliable the service, the more it costs to operate. Define the lowest level of reliability that you can get away with for each service, and state that as your SLO. Every service should have an availability SLO—without it, your team and your stakeholders cannot make principled judgments about whether your service needs to be made more reliable (increasing cost and slowing development) or less reliable (allowing greater velocity of development). Excessive availability can become a problem because now it’s the expectation. Don’t make your system overly reliable if you don’t intend to commit to it to being that reliable. Within Google, we implement periodic downtime in some services to prevent a service from being overly available. You might also try experimenting with planned-downtime exercises with front-end servers occasionally, as we did with one of our internal systems. We found that these exercises can uncover services that are using those servers inappropriately. With that information, you can then move workloads to somewhere more suitable and keep servers at the right availability level. 2. Service-Level Agreement (SLA)
At Google, we distinguish between an SLO and a Service-Level Agreement (SLA). An SLA normally involves a promise to someone using your service that its availability SLO should meet a certain level over a certain period, and if it fails to do so then some kind of penalty will be paid. This might be a partial refund of the service subscription fee paid by customers for that period, or additional subscription time added for free. The concept is that going out of SLO is going to hurt the service team, so they will push hard to stay within SLO. If you’re charging your customers money, you will probably need an SLA.

Because of this, and because of the principle that availability shouldn’t be much better than the SLO, the availability SLO in the SLA is normally a looser objective than the internal availability SLO. This might be expressed in availability numbers: for instance, an availability SLO of 99.9% over one month, with an internal availability SLO of 99.95%. Alternatively, the SLA might only specify a subset of the metrics that make up the internal SLO. If you have an SLO in your SLA that is different from your internal SLO, as it almost always is, it’s important for your monitoring to measure SLO compliance explicitly. You want to be able to view your system’s availability over the SLA calendar period, and easily see if it appears to be in danger of going out of SLO. You will also need a precise measurement of compliance, usually from logs analysis. Since we have an extra set of obligations (described in the SLA) to paying customers, we need to measure queries received from them separately from other queries. That’s another benefit of establishing an SLA—it’s an unambiguous way to prioritize traffic.

When you define your SLA’s availability SLO, you need to be extra-careful about which queries you count as legitimate. For example, if a customer goes over quota because they released a buggy version of their mobile client, you may consider excluding all “out of quota” response codes from your SLA accounting. 3. Service-Level Indicator (SLI)
We also have a direct measurement of a service’s behavior: the frequency of successful probes of our system. This is a Service-Level Indicator (SLI). When we evaluate whether our system has been running within SLO for the past week, we look at the SLI to get the service availability percentage. If it goes below the specified SLO, we have a problem and may need to make the system more available in some way, such as running a second instance of the service in a different city and load-balancing between the two. If you want to know how reliable your service is, you must be able to measure the rates of successful and unsuccessful queries as your SLIs.Google Cloud PlatformSpotlight session with Ben Treynor and Snapchat

Automatic dashboards in Stackdriver for GCP services enable you to group several ways: per service, per method and per response code any of the 50th, 95th and 99th percentile charts. You can also see latency charts on log scale to quickly find outliers.

See related content:

We wrote the book on SRE—check it out

Learn how SRE differs from DevOps

Read about SLOs for services with dependencies

Understanding error budget overspend - part one - CRE life lessons

2018年6月28日星期四

By Adrian Hilton, Alec Warner and Alex BramleyCRE Life LessonsCustomer Reliability Engineering (CRE) Where are you spending your error budget?

Do most of your errors happen when you’re doing binary releases? That implies that you’re not going to be able to keep within budget unless you do something to make releases less frequent, less error-prone or lower-impact when there is an error.

Are you seeing steady error spend coming from intermittent application failure, which adds up to the majority of your budget? That’s telling you that you’ve got a fundamental failure in your application. It’s a strong signal that you need to drill down in your logs to find the troublesome queries, and that you should expect to dedicate some of your engineers to identify the root causes and either address them directly or plan to fix them in your next project planning cycle.

Are large chunks of your error budget getting spent by major application failures, where most of your service goes down for many minutes due to configuration pushes, excessive load or queries-of-death? You need to run effective postmortems to identify the root causes and mitigate them. You will need to redirect some of your development engineering effort to address the top action items from those postmortems—so feature development and releases will naturally happen more slowly. (More on this in another post.)

Is the bulk of your spend coming from a dependency outside your control, such as a critical backend or your compute platform? You’ll need to address the dependency or platform owner directly, showing them your SLI metrics and negotiating about how they can make their service more reliable—or how you can be more resilient to the expected failure modes.

Are you measuring the right signal? Can your users tolerate a slightly worse experience? When your users’ experience isn’t definitivepart twohow to apply SRE principles to deploy and manage services at Next ‘18 Related content:

Building good SLOs - CRE life lessons: Make sure your error budget relates to the right thing

Consequences of SLO violations — CRE life lessons: See how an SRE team might respond to a developer team overspend

Good housekeeping for error budgets - part two - CRE life lessons

2018年6月28日星期四

By Adrian Hilton, Alec Warner and Alex BramleyCRE Life Lessonserror budget for a serviceyou’ve done some digging Paying off your error budget debt Looking at downstream spendDefining SLOs for services with dependencies When your releases are the problem

Make an explicitly agreed-upon adjustment to the feature vs. reliability work balance. For example, your company normally does two-week sprints, where 95% of the work is feature-driven, and 5% is postmortem action items and other reliability work. You agree that while your service is out of SLO, the sprints will be instead be 50/50.

Overprovision your service. For instance, pay more money to replicate to another cloud zone and/or region, or run more replicas of your service to handle higher traffic loads. This is only effective if you have determined that this approach will help mitigate error budget spend.

Declare a reliability incident. Appoint a specific person to analyze the postmortem and error budget spend and come up with recommendations. It’s important that the business has pre-committed to prioritizing those recommendations.

Winter is cominganalyzed an example escalation policy The long-term costs of freezessilver bullet Using the error budget to your (and your users’) advantagehow to apply SRE principles to deploy and manage services at Next ‘18 Related content:

An example escalation policy — CRE life lessons: See how an SRE team might agree to respond to an overspend

Available . . . or not? That is the question - CRE life lessons: Here’s how to know if your budget

Defining SLOs for services with dependencies - CRE life lessons

2018年5月7日星期一

By Robert van Gent, Customer Reliability Engineer and Cody Smith, Site Reliability Engineerprevious episodeCRE Life Lessonschapter in the SRE bookGoogle Cloud Platform service discovery.

Some of your dependencies may not even have SLOs, or their SLOs may not capture how you're using them.

The effect of a dependency's SLO on your service isn't always straightforward. In addition to the "hard" vs "soft" vs "degraded" impact discussed above, your code may complicate the effect of a dependency's SLOs on your service. For example, you have a 10s timeout on an RPC, but its SLO is based on serving a response within 30s. Or, your code does retries, and its impact on your service depends on the effectiveness of those retries (e.g., if the dependency fails 0.1% of all requests, does your retry have a 0.1% chance of failing or is there something about your request that means it is more than 0.1% likely to fail again?).

How to combine SLOs of multiple dependencies depends on the correlation between them. At the extremes, if all of your dependencies are always unavailable at the same time, then theoretically your unavailability is based on the max(), i.e., the dependency with the longest unavailability. If they are unavailable at distinct times, then theoretically your unavailability is the sum() of the unavailability of each dependency. The reality is likely somewhere in between.

Services usually do better than their SLOs (and usually much better than their service level agreements), so using them to estimate your downtime is often too conservative.

Risky businessSRECon18 Americasslidesrisk analysis spreadsheeta previous CRE Life Lessons blog postspreadsheet1

Trying to remove it from your critical path

Making it more reliable; e.g., running multiple copies and failing over between them

Automating manual failover processes

Replacing it with a more reliable alternative

Sharding it so that the scope of failure is reduced

Adding retries

Increasing (or decreasing, sometimes it is better to fail fast and retry!) RPC timeouts

Adding caching and using stale data instead of live data

Adding graceful degradation using partial responses

Asking for an SLO that better meets your needs

123balance velocity and risk1 If you're interested, The Calculus of Service Availability has more in-depth discussion about modeling risks from dependencies, and strategies for mitigating them.

Applying the Escalation Policy — CRE life lessons

2018年2月8日星期四

By Will Tipton, Site Reliability Engineer and Alex Bramley, Customer Reliability Engineer how to escalate SLO violationsreal-world example

Threshold 1: Automated alerts notify SRE of an at-risk SLO

Threshold 2: SREs conclude they need help to defend SLO and escalate to devs

Threshold 3: The 30-day error budget is exhausted and the root cause has not been found; SRE blocks releases and asks for more support from the dev team

Threshold 4: The 90-day error budget is exhausted and the root cause has not been found; SRE escalates to executive leadership to commandeer more engineering time for reliability work

Scenario 1: A short but severe outage is quickly root-caused to a dependency problemScenarioError budget
Escalation Scenario 2: A short but severe outage has no obvious root causeScenarioError budgetEscalation Scenario 3: A slow burn of error budget with a badly attributed root causeScenarioError budgetEscalation Scenario 4: Recurring, transient SLI excursionsScenarioError budgetEscalation Summary—

An example escalation policy — CRE life lessons

2018年1月19日星期五

By Alex Bramley, Customer Reliability Engineer; Will Tipton, Site Reliability Engineerearlier blog post Escalation policy preamble

finding the root cause and fixing the relevant class of issue, or

automating remediation such that ongoing manual intervention is no longer necessary, or

simply waiting one week, if the class of issue is extremely unlikely to recur with frequency and severity sufficient to threaten the SLO in the future

Escalation policy thresholdsThreshold 1 - wherein SRE are notified that an SLO is potentially impactedThreshold 2 - wherein SRE escalates to the developers

If,

SRE have concluded they cannot bring the service into SLO without help, and
SRE and dev agree that the SLO represents desired user experience

Then,

SRE and dev on-calls prioritize fixing the root cause and update the bug daily
SRE escalates to dev leads for visibility and additional assistance if necessary
Alerting thresholds may be relaxed to avoid continually paging for the known issue, while continuing to provide protection against further regressions

When the service is brought back into SLO,

SRE will revert any alerting changes
SRE may create a postmortem
Or, if the SLO does not accurately represent desired user experience, the SRE, dev and product teams will agree to change or retire the SLO

Threshold 3 - wherein SRE pauses feature releases and focuses on reliability

If,

Conditions for the previous threshold are met for at least one week, and
The service has not been brought back into SLO, and
The 30-day error budget is exhausted

Then during the following week,

Only cherry-picked fixes for diagnosed root causes may be pushed to production
SRE may escalate to their leadership and dev management to request that members of the dev team prioritize finding and fixing the root cause over any non-emergency work
Daily updates may be made to an "escalations" mailing list (used to broadcast information about outages to a wide audience, including executive leadership).

When the service is brought back into SLO,

Normal binary releases resume
SRE creates a postmortem
Team members may re-prioritize normal project work

Threshold 4 - wherein SRE may escalate or revoke support

If,

Conditions for the previous threshold are met for at least one week, and
The service has not been brought back into SLO, and
The 90-day error budget is exhausted or the dev team is unwilling to pause feature work to improve reliability

Then,

SRE may escalate to executive leadership to commandeer more people dedicated to fixing the problem
SRE may revoke support for the SLO or the service, and re-direct or disable relevant alerting

On escalation and incident response On blocking releases

Commonly, the largest source of burnt error budget at steady state is the release push. If you’ve already burned all your budget, not pushing new releases lowers the steady-state burn rate, bringing the service back into SLO more quickly

It eliminates the risk of further unexpected SLO violations due to bugs in new code. This is also why any fixes for diagnosed root causes must be patched into the current release, rather than rolling forward to a new release

While blocking releases is not intended as a punitive measure, it does directly impact release velocity, which the dev org cares about deeply. As such, tying SLO violations to reduced velocity aligns the incentives of both organizations. SRE wants the service to stay within SLO, the dev org wants to build new features quickly. This way, either both happen or neither do.

Summary

Consequences of SLO violations — CRE life lessons

2018年1月3日星期三

By Alex Bramley, Customer Reliability EngineerCRE life lessonsquantifying a service's availabilityusing SLOsGood SLOs Features or reliability?spherical SREs Inflection pointsblameless postmortemsnot enough error budget burned to notify anyone immediatelysomeone needs to investigate this right now before the service is out of its long-term SLO. ConsequencesNotify someone of potential or actual SLO violationEscalate the violation to the relevant dev teamMitigate risk of service changes causing further impact to SLOsRevoke support for the service Summary

What a year! Google Cloud Platform in 2017

2017年12月26日星期二

By Alex Barrett and Barrett Williams, Google Cloud blog editorsGoogle Cloud Platform

You love to hear about advanced infrastructure: CPUs, GPUs, TPUs, better network plumbing and more regions.

How we harden our infrastructure is endlessly interesting to you, as are tips about how to use our security services.

Open source is always a crowd-pleaser, particularly if it presents a cloud-native solution to an age-old problem.

You’re inspired by Google innovation — unique technologies that we developed to address internal, Google-scale problems. So, without further ado, we present to you the most-read stories of 2017.

Cutting-edge infrastructurefirst cloud provider to offer Intel Skylake architectureGPUs for Compute Engine and Cloud Machine Learningwhy cloud GPUs made senseperformance of TPUsthen-largest cloud-based compute clusternew GPU modelsmachine types with a whopping 96 vCPUs and 624GBEspresso, our peering-edge architectureTCP BBR congestion controlAndromeda 2.1Dedicated InterconnectNetwork Service TiersTransfer ApplianceNorthern VirginiaSingaporeSydneyLondonGermanySao PaoloMumbaiexpanded to Mars Security above allsecure our infrastructureSecurity in plaintext series7 ways we harden our KVM hypervisorFuzzing PCI ExpressTitan in depthCloud Key Managementmanaged SSL certificates for App Engine applicationsBeyondCorp as a more secure alternative to VPNGDPR data protection laws Open, hybrid developmentKubernetes 1.8advanced management featuresGoogle Kubernetes EnginePivotal Container Servicenew partnership with CiscoCloud Container BuilderSpinnaker 1.0IstioGrafeasPartnering on open sourcerelease canariesAPI design: Choosing between names and identifiers in URLs Google innovation Cloud SpannerCAP TheoremandCloud IoT Corean incidenthow to use Cloud Dataprep, Dataflow, and BigQuerydon’t need a PhD to learn to use TensorFlowTensorFlow Playgrounda game of rock-paper-scissorslighthearted and relatable exampleVideo Intelligence APItraining a custom object detectoradd machine learning to your IoT projectsNext

Getting the most out of shared postmortems — CRE life lessons

2017年12月8日星期五

By Adrian Hilton and Gwendolyn Stockman, Customer Reliability Engineers, and Dave Rensin, Director of Customer Reliability Engineeringour previous post
Postmortems should fix your customer tooHow to get outages to benefit everyoneMTTD and MTTRquite a lotWhat went well

The shared monitoring implemented with CustomerName showed a clear single-region latency hit which resulted in a quick escalation to storage oncall.

What went wrong

A new release of the storage frontend suffered from a performance regression for uncached reads that was not detected during testing or rollout.

Where we got lucky

Only reads of objects between 4KB and 32KB in size were materially affected.

Action items

Add explicit read/write latency testing in testing for both cached and uncached objects in buckets of 1KB, 4KB, 32KB, …

Have paging alerts for latency over SLO limits, aggregated by Cloud region, for both cached and uncached objects, in buckets of 1KB, 4KB, 32KB, ...

What went well

We had anticipated a generic single-region platform failure and had the capability to fail over out of an affected region.

What went wrong

Although the latency increase was detected quickly, we didn’t have accessible thru-stack monitoring that could show us that it was coming from platform storage-service rather than our own backends.

Our decision to fail out of the affected region took nearly 30 minutes to complete because we had not practiced it for one year and our playbook instructions were out of date.

Where we got lucky

This happened during business hours so our development team was on hand to help diagnose the cause.

Action items

Add explicit dashboard monitoring for aggregate read and write latency to and from platform storage-service.

Run periodic (at least once per quarter) test failovers out of a region to validate that the failover instructions still work and increase ops team confidence with the process.

Prioritize and track your action items A postmortem isn’t complete until the root causes have been fixed Other benefits Opening up is an opportunity Joint postmortemsIf you want joint operations, you need joint postmortems
Summary

Gain a better understanding of the impact and consequences of your outages

Increase the reliability of your customers’ service

Give customers confidence in continuing to run on your platform even after an outage.

an example of an external postmortemSRE Book postmortem template

Fearless shared postmortems — CRE life lessons

2017年11月27日星期一

By Adrian Hilton and Gwendolyn Stockman, Customer Reliability Engineers postmortemCRE Life Lessons Well-known external postmortems There is prior art, and you should read it. public incident report

Gitlab accidentally deleted its Postgres database and found out its backups were missing

CloudFlare edge servers had a memory leak

Honeycomb.io was bitten by a Kafka bug

schadenfreudeDiRT exerciseGoogle Cloud PlatformCustomer Reliability EngineeringSLO

Define a comprehensive set of business-relevant SLOs

Get the customer to measure compliance to those SLOs in their monitoring platform (how much of the service error budget has been consumed)

Share that live SLO information with Google support and product SRE teams (which we term shared monitoring)

Jointly monitor and react to SLO breaches with the customer (shared operational fate)

you too should practice SRE with your customers

How much of the error budget did this consume in total?

Why did the incident happen?

What can / should be done to stop it from happening again?

platform Foundations of an external postmortem the standard practicespostmortem template Selecting an audience for your external postmortem Deciding how much to share, and why?

How important are the details to understanding how to defend against a future re-occurrence of the event?

How badly did the event damage their service, i.e., how much error budget did it consume?

What to say in a postmortem

Why did this happen?

Could it have been worse?

How can we make sure it won’t happen again?

What not to say

Names of humans - Rather than “John Smith accidentally kicked over a server”, say “a network engineer accidentally kicked over a server,” Internally, we try to express the role of humans in terms of role rather than name. This helps us keep a blameless postmortem culture.

Names of internal systems - The names of your internal systems are not clarifying for your users and creates a burden on them to discover how these things fit together. For example, even though we’ve discussed Chubby externally, we still refer to it in postmortems we make external as “our globally distributed lock system.”

Customer-specific information - The internal version of your postmortem will likely say things like “on XX:XX, Acme Corp filed a Support ticket alerting us to a problem.” It’s not your place to share this kind of detail externally as it may create an undue burden for the reporting company (in this case Acme Corp.). Rather, simply say “on XX:XX, a customer filed…”. If you’re going to reference more than one customer, then just label them Customer A, Customer B, etc..

Other things to watch out forleast detailed explanationvery useful guidance as to when this is not true) A side note on the role of luckWhat’s luck got to do, got to do with it? What’s luck but a source of future failures?

“the right person was on-call” implies tribal knowledge that needs to be fed into a playbook and exercised in a DiRT test

“this other thing (e.g., a batch process or user action) wasn’t happening at the same time” implies that your system may not have sufficient surplus capacity to handle a peak load, and you should consider adding resources

“the incident happened during business hours” implies a need for automated alerting and 24-hour pager coverage by an on-call

“we were already watching monitoring” implies a need to tune alerting rules to pick up the leading edge of a similar incident if it isn’t being actively inspected.

an outage occurred on your busiest day of the year

you had a fix for the problem that hadn't been rolled out for other reasons

a weather event caused a power loss.

Where we got unlucky

There were various production inconsistencies caused by past outages and experiments. These weren’t cleaned up properly and made it difficult to reason about the state of production.always Summary