Google Cloud Platform Blog: May 2017

Compute Engine updates bring Skylake GA, extended memory and more VM flexibility

Wednesday, May 31, 2017

By Hanan Youssef, Sami Iqram and Scott Van Woudenberg, Product Managers, Google Compute Engine
Google Compute Engine

Increased total amount of memory per instance by removing memory caps

Increased variety of machine shapes

Simple process to select a baseline processor type

Availability of 64 vCPUs processors in all regions

Broadwell CPUs available in all regions

Skylake is generally available we first launched Skylake—priced CPU platform selectorGoogle Cloud PlatformAVX-512

instances create--min-cpu-platformgcloud beta compute instances create example-instance --machine-type n1-standard-1 --min-cpu-platform “Intel Broadwell”Available Regions and Zonesdocumentation Extended memory, where you want itCustom Machine Typespricing page

gcloud beta compute instances create example-instance --custom-cpu 2 --custom-memory 15 --custom-extensionsdocumentation Get started today try all of this out when you sign up for $300 free trial

Oregon region (us-west1) adds third zone, Cloud SQL and Regional Managed Instance Groups

Tuesday, May 30, 2017

By Dave Stiver, Product Manager, Google Cloud PlatformlaunchedGoogle Cloud Platform

Regional Managed Instance GroupsCloud SQL

Compute Engine

Container Engine

Dataflow

Dataproc

Datalab

locations pageshout

RailsConf 2017: a round-up

Thursday, May 25, 2017

By Aja Hammerly, Developer AdvocateRailsConfGoogle Cloud Platformcrap data jokeDHH's"What’s my app *really* doing in production?""Google Cloud <3 Ruby""Syntax Isn't Everything: NLP for Rubyists”Google’s Cloud Natural Language APIcodelabs

Cloud Source Repositories: now GA and free for up to five users and 50GB of storage

Wednesday, May 24, 2017

By Chris Sells, Product ManagerApp EngineCompute EngineCloud Source Repositories

Up to five project-users accessing repositories

Source repos consume less than 50GB in storage

Access to repos uses less than 50GB of network egress bandwidth

Cloud Source Repositories pricing page Getting started with Cloud Source Repositorieshttps://console.cloud.google.com/code/

Creating a CSR repo

from the Source Repositories section of the Cloud Console

Cloud Tools for IntelliJ

can mirror your existing repo into your GCP project

access the CSR API interactively in the Cloud API Explorerdocumentation for the CSR API Where are we?Cloud Shell and it’s new code editorStackOverflowGoogle GroupsSlack

Istio: a modern approach to developing and managing microservices

Wednesday, May 24, 2017

By Varun Talwar, Product Manager, Cloud Service Platform
IstioKuberneteswithout requiring any changes to application codeMoving from monolithic apps to microservicesHow Istio helpsa single command

Automatic load balancing for HTTP, gRPC, and TCP traffic

Fine-grained control of traffic behavior with rich routing rules

Traffic encryption, service-to-service authentication and strong identity assertions

Fleet-wide policy enforcement

In-depth telemetry and reporting

“Google's experience is that having a uniform substrate for developing and operating microservices is critical to our ability to scale while maintaining both feature velocity and reliability” — Eric Brewer, Vice President, Google Cloud An open communityIstio launch blog postKubernetesgRPCEnvoy Beyond IstioOpen Service Broker Looking aheadGoogle Container EngineCloud EndpointsApigee suite of products Get started todayheresampleistio-users@googlegroups.com mailing-list issueform

Know thy enemy: how to prioritize and communicate risks - CRE life lessons

Tuesday, May 23, 2017

By Matt Brown, Customer Reliability EngineerEditor’s note: We’ve spent a lot of time in CRE Life Lessons talking about how to identify and mitigate risks in your system. In this post, we’re going to talk about how to effectively communicate and stack-rank those risks. notoriously unreliableTake one: the risk matrix

The likelihood of the risk occurring in a given time period.

The impact that would be felt if the risk materializes.

	Catastrophic	Damaging	Minimal
Frequent	Overload results in slow or dropped requests during the peak hour each day.	The wrong server is turned off and requests are dropped.	Restarts for weekly upgrades drop in-progress requests (i.e., no lame ducking).
Common	A bad release takes the entire service down. Rollback is not tested.	Users report an outage before monitoring and alerting notifies the operator.	A daylight savings bug drops requests.
Rare	There is a physical failure in the hosting location that requires complete restoration from a backup or disaster recovery plan.	Overload results in a cascading failure. Manual intervention is required to halt or fix the issue.	A leap year bug causes all servers to restart and drop requests.

risk matrixrisk assessment field

The distribution of risks across the cells was extremely uneven. Most risks ended up in the “common, damaging” cell, which doesn’t help to explain relative importance of the items within each cell.

Assigning a risk to a cell (and its subsequent position in the list of risks) is subjective and depends on the reliability target of the application. For example, the “frequent, catastrophic” example of dropping traffic for a few minutes during a release is catastrophic at four nines, but less so at two nines.

Ordering the cells into a ranking is not straightforward. Is it more important to handle a “rare, catastrophic” risk, or a “frequent, minimal” risk? The answer is not clear from the names or definitions of the categories alone. Further, the desired order can change from matrix to matrix depending on the number of items in each cell.

Risk expressed as expected lossesimpactlikelihoodavailabilitySLOs

Red: This risk is unacceptable, as it falls above the acceptable error budget for a single risk (we typically use 25%), and therefore, can have a major impact on your reliability in a single event.

Amber: This risk should not be acceptable, as it’s a major consumer of your error budget and therefore, needs to be addressed. You may be able to accept some amber risks by addressing some less urgent (green) risks to buy back budget.

Green: This is an acceptable risk. It's not a major consumer of your error budget, and in aggregate, does not cause your application to exceed the error budget. You don't have to address green risks, but may wish to do so to give yourself more budget to cover unexpected risks, or to accept amber risks that are hard to mitigate or eliminate.

Risk	Bad minutes/year
Overload results in slow or dropped requests during the peak hour each day.	3559
A bad release takes the entire service down. Rollback is not tested.	507
Users report an outage before monitoring and alerting notifies the operator.	395
There is a physical failure in the hosting location that requires complete restoration from a backup or disaster recovery plan.	242
The wrong server is turned off and requests are dropped.	213
Overload results in a cascading failure. Manual intervention is required to halt or fix the issue.	150
Operator accidentally deletes database; restore from backup is required	129
Unnoticed growth in usage triggers overload; service collapses.	125
A configuration mishap reduces capacity; causing overload and dropped requests	122
A new release breaks a small set of requests; not detected for a day.	119
Operator is slow to debug and root cause bug due to noisy alerting	76
A daylight savings bug drops requests.	71
Restarts for weekly upgrades drop in-progress requests (i.e., no lame ducking).	52
A leap year bug causes all servers to restart and drop requests.	16

Other Considerations

Mean Time To Detect (MTTD): The time between when the risk first manifests and when the issue is brought to the attention of someone (or something) capable of remediating it.

Mean Time To Repair (MTTR): Redefined to mean the time between when the issue is brought to the attention of someone capable of remediating it and when it is actually remediated.

(MTTD + MTTR) * (365.25 / MTBF) * percent of affected usersSummary

How to do serverless pixel tracking with GCP

Thursday, May 18, 2017

By Matthieu Mayran, Cloud Solutions Architect

You need to set up, manage and monitor your ad servers

Users are usually global, which means that you need ad servers around the world

User visits are spiky, so pixel servers must scale up to sustain the load and scale down to limit the spend.

Google Cloud PlatformContainer Enginemanaged autoscaled instance groupsDoiT Internationalserverless pixel tracking solution

Google Cloud Storage: A global or regional object store that offers different options such as Standard, Nearline, Cold with various prices and SLAs depending on your needs. In our case, we used Standard, which offers low millisecond latency

Google HTTP(s) Load Balancer: A global anycast IP load balancer service that can scale to millions of QPS with integrated logging. It also can be leveraged by Cloud CDN to prevent useless access to Google Cloud Storage by caching pixels closer to the user in Google edges

BigQuery: Google's fully managed, petabyte-scale, low-cost enterprise data warehouse for analytics

Stackdriver Logging: A logging system that allows you to store, search, analyze, monitor and alert on log data and events from GCP and Amazon Web Services (AWS). It supports Google load balancers and can export data to Cloud Storage, BigQuery or Pub/Sub

A client calls a pixel URL that's served directly by Cloud Storage.

A Google Cloud Load Balancer in front of Cloud Storage records the request to Stackdriver Logging, whether there was a cache hit or not.

Stackdriver Logging exports every request to BigQuery as they come in, which acts as a storage and querying engine for ad-hoc analytics that can help business analysts better understand their users.

Reading the solution paper: Serverless Pixel Tracking Architecture

Following the tutorial: How to Do Serverless Pixel Tracking

Load-testing it: How to load test a serverless pixel tracking setup

Introducing Google Cloud IoT Core: for securely connecting and managing IoT devices at scale

Tuesday, May 16, 2017

By Indranil Chakraborty, Product Manager, Google CloudGoogle Cloud PlatformGoogle Cloud IoT Core

Utilities can monitor, analyze and predict consumer energy usage in real time

Transportation and logistics firms can proactively stage the right vehicles/vessels/aircraft in the right places at the right times

Oil and gas and manufacturing companies can enable intelligent scheduling of equipment maintenance to maximize production and minimize downtime

About all the things— —

Operational cost and complexity: The overhead of managing the deployment, maintenance and upgrades for exponentially more devices is stifling. And even with a custom solution in place, the resource investments required for necessary IT infrastructure are significant.

Patchwork security: Ensuring world-class, end-to-end security for globally distributed devices is out of reach — or at least not a core competency — for most organizations.

Data fragmentation: Despite the fact that machine-generated data is now an important data source for making good business decisions, the massive amount of data generated by these devices is often stored in silos with a short expiration date, and hence never reaches downstream analytic systems (nor decision makers).

—

Key features and benefits

Fast and easy setup and management: Cloud IoT Core lets you connect up to millions of globally dispersed devices into a single system with smooth and even data ingestion ensured under any condition. Devices are registered to your service quickly and easily via the industry-standard MQTT protocol. For Android Things-based devices, firmware updates can be automatic.

Security out-of-the-box: Secure all device data via industry-standard security protocols. (Combine Cloud IoT Core with Android Things for device operating-system security, as well.) Apply Google Cloud IAM roles to devices to control user access in a fine-grained way.

Native integration with analytic services: Ingest all your IoT data so you can manage it as a single system and then easily connect it to our native analytic services (including Google Cloud Dataflow, Google BigQuery and Google Cloud Machine Learning Engine) and partner BI solutions (such as Looker, Qlik, Tableau and Zoomdata). Pinpoint potential problems and uncover solutions using interactive data visualizations, or build rich machine-learning models that reflect how your business works.

Auto-managed infrastructure: All this in the form of a fully-managed, pay-as-you-go GCP service, with no infrastructure for you to deploy, scale or manage.

"With Google Cloud IoT Core, we have been able to connect large fleets of bicycles to the cloud and quickly build a smart transportation fleet management tool that provides operators with a real-time view of bicycle utilization, distribution and performance metrics, and it forecasts demand for our customers."
— Jose L. Ugia, VP Engineering, Noa Technologies Next stepspartners

Actions Semiconductor

Cloud Spanner is now production-ready; let the migrations begin!

Tuesday, May 16, 2017

By Dominic Preuss, Product ManagerCloud Spanner Looking toward migrationAloomaInformaticaXplentyFrom Alooma “Cloud Spanner is a game-changer because it offers horizontally scalable, strongly consistent, highly available OLTP infrastructure in the cloud for the first time. To accelerate migrations, we recommend that customers replicate their data continuously between the source OLTP database and Cloud Spanner, thereby maintaining both infrastructures in the same state — this allows them to migrate their workloads gradually in a predictable manner.”From Informatica: “Informatica customers are stretching the limits of latency and data volumes, and need innovative enterprise-scale capabilities to help them outperform their competition. We are excited about Cloud Spanner because it provides a completely new way for our mutual customers to disrupt their markets. For integration, migration and other use cases, we are partnering with Google to help them ingest data into Cloud Spanner and integrate a variety of heterogeneous batch, real-time, and streaming data in a highly scalable, performant and secure way.” From Xplenty: "Cloud Spanner is one of those cloud-based technologies for which businesses have been waiting: With its horizontal scalability and ACID compliance, it’s ideal for those who seek the lower TCO of a fully managed cloud-based service without sacrificing the features of a legacy, on-premises database. In our experience with customers migrating to Cloud Spanner, important considerations include accounting for data types, embedded code and schema definitions, as well as understanding Cloud Spanner’s security model to efficiently migrate your current security and access-control implementation." Next stepstrialcontact usnew papergrowing partner ecosystem

Mapping your organization with the Google Cloud Platform resource hierarchy

Thursday, May 11, 2017

By Marco Cavalli, Product ManagerGoogle Cloud PlatformCloud Identity & Access ManagementCloud Resource ManagerOrganization policiesOrganization resourceresource hierarchyFolders "As our work with GCP scaled, we started looking for ways to streamline our projects, Thanks to Cloud Resource Manager, we now centrally control and monitor how resources are created and billed in our domain. We use IAM and Folders to provide our departments with the autonomy and velocity they need, without losing visibility into resource access and usage. This has significantly reduced our management overhead, and had a direct positive effect on our ability to support our customers at scale.” — Marcin Kołda, Senior Software Engineer at Ocado Technology. The Google Cloud resource hierarchyownershipgroupingscaffolding

What to consider when mapping your organization onto GCPIsolationIAM policiesDelegationleast privilegeInheritancebob@myorganization.comShared resources Managing the GCP resource hierarchy

bob@myorganization.com

# Find your Organization ID me@cloudshell:~$ gcloud organizations list DISPLAY_NAME ID DIRECTORY_CUSTOMER_ID myorganization.com 358981462196 C03ryezon # Create first level folder “Engineering” under the Organization node me@cloudshell:~$ gcloud alpha resource-manager folders create --display-name=Engineering --organization=358981462196 Waiting for [operations/fc.2201898884439886347] to finish...done. Created [<Folder createTime: u'2017-04-16T22:49:10.144Z' displayName: u'Engineering' lifecycleState: LifecycleStateValueValuesEnum(ACTIVE, 1) name: u'folders/1000107035726' parent: u'organizations/358981462196'>]. # Add a Folder Admin role to the “Engineering” folder me@cloudshell:~$ gcloud alpha resource-manager folders add-iam-policy-binding 1000107035726 --member=user:bob@myorganization.com --role=roles/resourcemanager.folderAdmin bindings: - members: - user:bob@myorganization.com - user:admin@myorganization.com role: roles/resourcemanager.folderAdmin - members: - user:alice@myorganization.com role: roles/resourcemanager.folderEditor etag: BwVNX61mPnc= # Check the IAM policy set on the “Engineering” folder me@cloudshell:~$ gcloud alpha resource-manager folders get-iam-policy 1000107035726 bindings: - members: - user:bob@myorganization.com - user:admin@myorganization.com role: roles/resourcemanager.folderAdmin - members: - user:alice@myorganization.com role: roles/resourcemanager.folderEditor etag: BwVNX61mPnc= # Create second level folder “Product_A” under folder “Engineering” me@cloudshell:~$ gcloud alpha resource-manager folders create --display-name=Product_A --folder=1000107035726 Waiting for [operations/fc.2194220672620579778] to finish...done. Created []. # Crate third level folder “Development” under folder “Product_A” me@cloudshell:~$ gcloud alpha resource-manager folders create --display-name=Development --folder=1000107035726 Waiting for [operations/fc.3497651884412259206] to finish...done. Created []. # List all the folders under the Organization me@cloudshell:~$ gcloud alpha resource-manager folders list --organization=358981462196 DISPLAY_NAME PARENT_NAME ID IT organizations/358981462196 575615098945 Engineering organizations/358981462196 661646869517 Operations organizations/358981462196 895951706304 # List all the folders under the “Engineering” folder me@cloudshell:~$ gcloud alpha resource-manager folders list --folder=1000107035726 DISPLAY_NAME PARENT_NAME ID Product_A folders/1000107035726 732853632103 Product_B folders/1000107035726 941564020040 # Create a new project in folder “Product_A” me@cloudshell:~$ gcloud alpha projects create my-awesome-service-2 --folder 732853632103 Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/my-awesome-service-3].Waiting for [operations/pc.2821699584791562398] to finish...done. # List projects under folder “Production” me@cloudshell:~$ gcloud alpha projects list --filter 'parent.id=725271112613' PROJECT_ID NAME PROJECT_NUMBER my-awesome-service-1 my-awesome-service-1 869942226409 my-awesome-service-2 my-awesome-service-2 177629658252Quickstarts