Google Cloud Platform Blog

Oregon region (us-west1) adds third zone, Cloud SQL and Regional Managed Instance Groups

Tuesday, May 30, 2017

By Dave Stiver, Product Manager, Google Cloud PlatformlaunchedGoogle Cloud Platform

Regional Managed Instance GroupsCloud SQL

Compute Engine

Container Engine

Dataflow

Dataproc

Datalab

locations pageshout

RailsConf 2017: a round-up

Thursday, May 25, 2017

By Aja Hammerly, Developer AdvocateRailsConfGoogle Cloud Platformcrap data jokeDHH's"What’s my app *really* doing in production?""Google Cloud <3 Ruby""Syntax Isn't Everything: NLP for Rubyists”Google’s Cloud Natural Language APIcodelabs

Cloud Source Repositories: now GA and free for up to five users and 50GB of storage

Wednesday, May 24, 2017

By Chris Sells, Product ManagerApp EngineCompute EngineCloud Source Repositories

Up to five project-users accessing repositories

Source repos consume less than 50GB in storage

Access to repos uses less than 50GB of network egress bandwidth

Cloud Source Repositories pricing page Getting started with Cloud Source Repositorieshttps://console.cloud.google.com/code/

Creating a CSR repo

from the Source Repositories section of the Cloud Console

Cloud Tools for IntelliJ

can mirror your existing repo into your GCP project

access the CSR API interactively in the Cloud API Explorerdocumentation for the CSR API Where are we?Cloud Shell and it’s new code editorStackOverflowGoogle GroupsSlack

Istio: a modern approach to developing and managing microservices

Wednesday, May 24, 2017

By Varun Talwar, Product Manager, Cloud Service Platform
IstioKuberneteswithout requiring any changes to application codeMoving from monolithic apps to microservicesHow Istio helpsa single command

Automatic load balancing for HTTP, gRPC, and TCP traffic

Fine-grained control of traffic behavior with rich routing rules

Traffic encryption, service-to-service authentication and strong identity assertions

Fleet-wide policy enforcement

In-depth telemetry and reporting

“Google's experience is that having a uniform substrate for developing and operating microservices is critical to our ability to scale while maintaining both feature velocity and reliability” — Eric Brewer, Vice President, Google Cloud An open communityIstio launch blog postKubernetesgRPCEnvoy Beyond IstioOpen Service Broker Looking aheadGoogle Container EngineCloud EndpointsApigee suite of products Get started todayheresampleistio-users@googlegroups.com mailing-list issueform

Know thy enemy: how to prioritize and communicate risks - CRE life lessons

Tuesday, May 23, 2017

By Matt Brown, Customer Reliability EngineerEditor’s note: We’ve spent a lot of time in CRE Life Lessons talking about how to identify and mitigate risks in your system. In this post, we’re going to talk about how to effectively communicate and stack-rank those risks. notoriously unreliableTake one: the risk matrix

The likelihood of the risk occurring in a given time period.

The impact that would be felt if the risk materializes.

	Catastrophic	Damaging	Minimal
Frequent	Overload results in slow or dropped requests during the peak hour each day.	The wrong server is turned off and requests are dropped.	Restarts for weekly upgrades drop in-progress requests (i.e., no lame ducking).
Common	A bad release takes the entire service down. Rollback is not tested.	Users report an outage before monitoring and alerting notifies the operator.	A daylight savings bug drops requests.
Rare	There is a physical failure in the hosting location that requires complete restoration from a backup or disaster recovery plan.	Overload results in a cascading failure. Manual intervention is required to halt or fix the issue.	A leap year bug causes all servers to restart and drop requests.

risk matrixrisk assessment field

The distribution of risks across the cells was extremely uneven. Most risks ended up in the “common, damaging” cell, which doesn’t help to explain relative importance of the items within each cell.

Assigning a risk to a cell (and its subsequent position in the list of risks) is subjective and depends on the reliability target of the application. For example, the “frequent, catastrophic” example of dropping traffic for a few minutes during a release is catastrophic at four nines, but less so at two nines.

Ordering the cells into a ranking is not straightforward. Is it more important to handle a “rare, catastrophic” risk, or a “frequent, minimal” risk? The answer is not clear from the names or definitions of the categories alone. Further, the desired order can change from matrix to matrix depending on the number of items in each cell.

Risk expressed as expected lossesimpactlikelihoodavailabilitySLOs

Red: This risk is unacceptable, as it falls above the acceptable error budget for a single risk (we typically use 25%), and therefore, can have a major impact on your reliability in a single event.

Amber: This risk should not be acceptable, as it’s a major consumer of your error budget and therefore, needs to be addressed. You may be able to accept some amber risks by addressing some less urgent (green) risks to buy back budget.

Green: This is an acceptable risk. It's not a major consumer of your error budget, and in aggregate, does not cause your application to exceed the error budget. You don't have to address green risks, but may wish to do so to give yourself more budget to cover unexpected risks, or to accept amber risks that are hard to mitigate or eliminate.

Risk	Bad minutes/year
Overload results in slow or dropped requests during the peak hour each day.	3559
A bad release takes the entire service down. Rollback is not tested.	507
Users report an outage before monitoring and alerting notifies the operator.	395
There is a physical failure in the hosting location that requires complete restoration from a backup or disaster recovery plan.	242
The wrong server is turned off and requests are dropped.	213
Overload results in a cascading failure. Manual intervention is required to halt or fix the issue.	150
Operator accidentally deletes database; restore from backup is required	129
Unnoticed growth in usage triggers overload; service collapses.	125
A configuration mishap reduces capacity; causing overload and dropped requests	122
A new release breaks a small set of requests; not detected for a day.	119
Operator is slow to debug and root cause bug due to noisy alerting	76
A daylight savings bug drops requests.	71
Restarts for weekly upgrades drop in-progress requests (i.e., no lame ducking).	52
A leap year bug causes all servers to restart and drop requests.	16

Other Considerations

Mean Time To Detect (MTTD): The time between when the risk first manifests and when the issue is brought to the attention of someone (or something) capable of remediating it.

Mean Time To Repair (MTTR): Redefined to mean the time between when the issue is brought to the attention of someone capable of remediating it and when it is actually remediated.

(MTTD + MTTR) * (365.25 / MTBF) * percent of affected usersSummary

How to do serverless pixel tracking with GCP

Thursday, May 18, 2017

By Matthieu Mayran, Cloud Solutions Architect

You need to set up, manage and monitor your ad servers

Users are usually global, which means that you need ad servers around the world

User visits are spiky, so pixel servers must scale up to sustain the load and scale down to limit the spend.

Google Cloud PlatformContainer Enginemanaged autoscaled instance groupsDoiT Internationalserverless pixel tracking solution

Google Cloud Storage: A global or regional object store that offers different options such as Standard, Nearline, Cold with various prices and SLAs depending on your needs. In our case, we used Standard, which offers low millisecond latency

Google HTTP(s) Load Balancer: A global anycast IP load balancer service that can scale to millions of QPS with integrated logging. It also can be leveraged by Cloud CDN to prevent useless access to Google Cloud Storage by caching pixels closer to the user in Google edges

BigQuery: Google's fully managed, petabyte-scale, low-cost enterprise data warehouse for analytics

Stackdriver Logging: A logging system that allows you to store, search, analyze, monitor and alert on log data and events from GCP and Amazon Web Services (AWS). It supports Google load balancers and can export data to Cloud Storage, BigQuery or Pub/Sub

A client calls a pixel URL that's served directly by Cloud Storage.

A Google Cloud Load Balancer in front of Cloud Storage records the request to Stackdriver Logging, whether there was a cache hit or not.

Stackdriver Logging exports every request to BigQuery as they come in, which acts as a storage and querying engine for ad-hoc analytics that can help business analysts better understand their users.

Reading the solution paper: Serverless Pixel Tracking Architecture

Following the tutorial: How to Do Serverless Pixel Tracking

Load-testing it: How to load test a serverless pixel tracking setup

Introducing Google Cloud IoT Core: for securely connecting and managing IoT devices at scale

Tuesday, May 16, 2017

By Indranil Chakraborty, Product Manager, Google CloudGoogle Cloud PlatformGoogle Cloud IoT Core

Utilities can monitor, analyze and predict consumer energy usage in real time

Transportation and logistics firms can proactively stage the right vehicles/vessels/aircraft in the right places at the right times

Oil and gas and manufacturing companies can enable intelligent scheduling of equipment maintenance to maximize production and minimize downtime

About all the things— —

Operational cost and complexity: The overhead of managing the deployment, maintenance and upgrades for exponentially more devices is stifling. And even with a custom solution in place, the resource investments required for necessary IT infrastructure are significant.

Patchwork security: Ensuring world-class, end-to-end security for globally distributed devices is out of reach — or at least not a core competency — for most organizations.

Data fragmentation: Despite the fact that machine-generated data is now an important data source for making good business decisions, the massive amount of data generated by these devices is often stored in silos with a short expiration date, and hence never reaches downstream analytic systems (nor decision makers).

—

Key features and benefits

Fast and easy setup and management: Cloud IoT Core lets you connect up to millions of globally dispersed devices into a single system with smooth and even data ingestion ensured under any condition. Devices are registered to your service quickly and easily via the industry-standard MQTT protocol. For Android Things-based devices, firmware updates can be automatic.

Security out-of-the-box: Secure all device data via industry-standard security protocols. (Combine Cloud IoT Core with Android Things for device operating-system security, as well.) Apply Google Cloud IAM roles to devices to control user access in a fine-grained way.

Native integration with analytic services: Ingest all your IoT data so you can manage it as a single system and then easily connect it to our native analytic services (including Google Cloud Dataflow, Google BigQuery and Google Cloud Machine Learning Engine) and partner BI solutions (such as Looker, Qlik, Tableau and Zoomdata). Pinpoint potential problems and uncover solutions using interactive data visualizations, or build rich machine-learning models that reflect how your business works.

Auto-managed infrastructure: All this in the form of a fully-managed, pay-as-you-go GCP service, with no infrastructure for you to deploy, scale or manage.

"With Google Cloud IoT Core, we have been able to connect large fleets of bicycles to the cloud and quickly build a smart transportation fleet management tool that provides operators with a real-time view of bicycle utilization, distribution and performance metrics, and it forecasts demand for our customers."
— Jose L. Ugia, VP Engineering, Noa Technologies Next stepspartners