Google Cloud Platform Blog: June 2017

Solution guide: Building connected vehicle apps with Cloud IoT Core

Friday, June 30, 2017

By Charles Baer, Solutions Architectsolution guideGoogle Cloud PlatformA data deluge

Device management. Connecting devices to any platform requires authentication, authorization, the ability to push update software, configuration and monitoring. These services must be able to scale to millions of devices and constant availability.

Data ingestion. Messages must be reliably received, processed and stored.

Data analytics. Complex analysis of time-series data generated from devices must be used to gain insights into event, tolerances, trends and possible failures.

Applications. Business-level application logic must be developed and integrated with existing data sources that may come from a third party or exist in on-premise data centers.

Predictive models. In order to predict business-level outcomes, predictive models based on current and historical data must be developed.

Cloud IoT Coreend-to-end security model

(click to enlarge)

Device ManagementData IngestionData AnalyticsApplicationsPredictive ModelsTensorFlowCloud Machine Learning Engine—SummaryDesigning a Connected Vehicle Platform on Cloud IoT Core

How SREs find the landmines in a service - CRE life lessons

Thursday, June 29, 2017

By Adrian Hilton, Customer Reliability Engineer
Part 1 Onboarding reviewQ: We have a new application that would make sense for SRE to support. Do I just throw it over the wall and tell the SRE team “Here you are; you’re on call for this now, best of luck”?————Customer Reliability Engineering

SRE entrance review: where an SRE team assess whether a developer-supported service should be onboarded by SRE, and what the onboarding preconditions should be.

SRE onboarding/takeover: where a dev and SRE team agree in principle that the SRE team should take on primary operational responsibility for a service, and start negotiating the exact conditions for takeover (how and when the SREs will onboard the service).

Developers want someone else to pick up support for the service, and make it run as well as possible. They want users to feel that the service is working properly, otherwise they'll move to a service run by someone else.

The SRE team wants to be sure that they're not being “sold a pup” with a hard-to-support service, and have a vision for making the production service lower in toil and more robust.

Meanwhile the company management wants to reduce the number of embarrassing service outages, as long as it doesn’t cost them too much in engineer time.

The SRE entrance review

Assess how the service would benefit from SRE ownership

Identify service design, implementation and operational deficiencies that could be a barrier to SRE takeover

And if SRE ownership is determined to be beneficial, identify the bug fixes, process changes and necessary service behavior needed before onboarding the service

Service Level Objectives

what the service does

day-to-day service operation (traffic variation, releases, experiment management, config pushes)

how the service tends to break and how this manifests in alerts

rough edges in monitoring and alerting

where the service configuration diverges from the SRE team’s practices

major operational risks for the service

whether the service follows SRE team best practices, and if not, how to retrofit it

how to integrate the service with the SRE team’s existing tools and processes

the desired engagement model and separation of responsibilities between the SRE team and the SWE team. When debugging a critical production problem, at what point should the SRE on-call page the developer on-call?

The SRE takeoverExtant bugsReliabilityAutomationMonitoring/alerting—Treynor

Paging on something that’s not intrinsically important e.g., task restart or hitting 80% capacity of disk. Instead, downgrade the page to a bug (if it’s not urgent) or eliminate it entirely. Moving to symptom-based monitoring (“users are actually seeing problems”) can help improve this situation.

Page storms where one small incident/outage generates many pages. Try to group related pages for an incident into a single outage, to get a clearer picture of the system’s outage metrics.

A system that’s having too many genuine problems. In this case SRE takeover in the near future is unlikely, but SREs may be able to help diagnose and resolve the root causes of the problems.

integrating the service with standard SRE tools and practices e.g., load shedding, release processes and configuration pushes

extending and improving playbook entries to rely less on the developer team’s tribal knowledge

aligning the service’s configurations with the SRE team’s common languages and infrastructure

Smoothing the pathSee part 3 of this series, "Making the most of an SRE service takeover - CRE life lessons."

Google App Engine standard now supports Java 8

Wednesday, June 28, 2017

By Amir Rouzrokh, Product Manager

top requestsJava 8 on App Engine standard environmentJetty 9gRPCGoogle Cloud Java Library supportclass whitelistfully supports Spring BootKotlinApache GroovyApp Engine APIsStackdriverCloud SDKMavenGradleIntelliJEclipseGoogle Cloud Next 2017Power your Java Workloads on Google Cloud Platform—how to choose between App Engine flexible and standard environmentsdeploy your first application to App Enginequickstartjava8herethis repositoryJava 8 on App Engine flexiblePython upgradesMavenGradleIntelliJEclipseGoogle Cloud Java Libraries

Enterprise identity made easy in Google Cloud Platform with Cloud Identity

Tuesday, June 27, 2017

By Zack Ontiveros, Product Manager, Google Cloud IdentityG SuiteGoogle Cloud’s identity servicesGoogle Cloud Platform Introducing Cloud Identity support in GCPCloud Resource Manager

Try it today Getting Started Guide

Versioning APIs at Google

Monday, June 26, 2017

By Dan Ciruli, Product Manager
semantic versioningcoolcloudapi.googleapis.com/v1/coolthings/12301221312132coolcloudapi.googleapis.com/v1/coolthings/12301221312132Cloud Endpoints—10-minute-quickstartin-depth tutorials—

Why should your app get SRE support? - CRE life lessons

Friday, June 23, 2017

By Adrian Hilton, Customer Reliability EngineerEditor’s note: When you start to run many applications or services in your company, then you'll start to bump up against the limit of what your primary SRE (or Ops) team can support. In this installment of CRE Life Lessons we're going to look at how you can make good, principled and defensible decisions about which of your company’s applications and services you should give to your SREs to support, and how to decide when that subset needs to change.Q: How will I know when my company’s SRE team is at its limit? How do I choose the best subset of applications to support? When should the SRE team drop support for an application? Practical limitations on SRE supportmostly
Q: Surely 12 to 16 engineers can handle support for all the applications your development team can feasibly write? differentteam mitosis

the regular operational tasks needed to keep the services running well, for example releases, bug fixes, non-urgent alerts/bugs. These can be reduced (but not eliminated) by automation;

“interrupts” -- unscheduled non-critical human requests. We’ve found these awkwardly resistant to efforts to reduce them; the most effective strategy has been self-service tools that address the 50-70% of repeated queries;

emergency alert response, incident management and follow-up. The best way to spend less time on these is to make the service more reliable, and to have better-tuned alerts (i.e., that are actionable and which, if they fire, strongly indicate real problems with the service).

Q: What about the four weeks out of six during which an SRE isn’t doing operational work — could we use that time to increase our SRE team’s supported service capacity? You’re out of SRE support capacity - now what?Q: What about the next application we want the developers to write? Won’t they be too busy supporting the current application?—still— Choosing which applications to supportQ: Sounds good to me; surely prioritizing by business impact is always the right choice?

Monitoring and metrics. For example, detecting response latency, error or unanswered query rate, and peak utilization of resources

Emergency response. Running on-call rotations, traffic-dip detection, primary/secondary/escalation, writing playbooks, running Wheels of Misfortune

Capacity planning. Doing quarterly projections, handling a sudden sustained load spike, running utilization-improvement projects

Service turn-up and turn-down. For services which run in many locations (e.g., to reduce end-user latency), planning location turn-up/down schedules and automating the process to reduce risks and operational load

Change management. Canarying, 1% experiments, rolling upgrades, quick-fail rollbacks, and measuring error budgets

Performance. Stress and load testing, resource-usage efficiency monitoring and optimization.

Data Integrity. Ensuring that non-reconstructible data is stored resiliently and highly available for reads, including the ability to rapidly restore it from backups

——See parts 2, "How SREs find the landmines in a service - CRE life lessons," and 3, "Making the most of an SRE service takeover - CRE life lessons," of this series.

Google Compute Engine ranked #1 in price-performance by Cloud Spectator

Thursday, June 22, 2017

By Paul Nash, Group Product Manager, Compute Enginecomparative benchmarking studyestimates

Insight #1: VM performance varies across cloud providers. In testing, Cloud Spectator observed differences of up to 1.4X in VM performance and 6.1X in block storage performance.

Insight #2: You don’t always get what you pay for. Cloud Spectator’s study found no correlation between price and performance.

Insight #3: Resource contention (the “Noisy Neighbor Effect”) can affect performance — but CSPs can limit those effects. Cloud Spectator points out that noisy neighbors are a real problem with some cloud vendors. To try and handle the problem, some vendors throttle down their customers access to resources (like disks) in an attempt to compensate for other VMs (so called Noisy Neighbors) on the same host machine.

here Key finding: Google leads for overall price-performance

2017 Best Hyperscale Cloud Providers: AWS vs. Azure vs. Google vs. SoftLayer Methodology

Google Cloud Platform expands to Australia with new Sydney region - open now

Tuesday, June 20, 2017

By Dave Stiver, Product Manager, Google Cloud PlatformEditor's Note: This post was updated on July 18, 2017.Google Cloud Platformrecently launched Singapore

Sydney GCP region

in MumbaiSJC cableIndigo cableWhat customers are saying

"The regional expansion of Google Cloud Platform to Australia will help enable PwC's rapidly growing need to experiment and innovate and will further extend our work with Google Cloud.

It not only provides a reliable and resilient platform that can support our firm's core technology needs, it also makes available to us, GCP's market leading technologies and capabilities to support the unprecedented demand of our diverse and evolving business."

—Hilda Clune, Chief Information Officer, PwC Australia

"Monash University has one of the most ambitious digital transformation agendas in tertiary education. We're executing our strategy at pace and needed a platform which would give us the scale, flexibility and functionality to respond rapidly to our development and processing needs. Google Cloud Platform (GCP) and in particular App Engine have been a great combination for us, and we're very excited at the results we're getting. Having Google Cloud Platform hosted now in Australia is a big bonus." —Trevor Woods, Chief Information Officer, Monash University

Modern geophysical technologies place a huge demand on supercomputing resources. Woodside utilises Google Cloud as an on-demand solution for our large computing requirements. This has allowed us to push technological boundaries and dramatically reduce turnaround time.— Sean Salter, VP Technology,Woodside Energy Ltd.Next stepsShine SolutionsServian3WKSAxalonOnigroupPwCDeloitteGlintechFrondeDialogMegaportSydney region pageshoutUpdate: July 18, 2017App EngineCloud DatastoreGCP Solutionsbuild mobile apps using Firebase and App Enginestorage options

New Singapore GCP region – open now

Wednesday, June 14, 2017

By Dave Stiver, Product Manager, Google Cloud Platform
Google Cloud Platform

"We are excited to be able to deploy into the GCP Singapore region, as it will allow us to offer our services closer to BBM Messenger key markets. Coupled with Google's global load balancers and extensive global network, we expect to be able to provide a low latency, high-speed experience for our users globally. During our POCs, we found that GCP outperformed most vendors on key metrics such as disk I/O and network performance on like-for-like benchmarks. With sustained usage discounts and continuous support from Google's PSO and account team, we are excited to make GCP the foundation for the next generation of BBM consumer services." — Matthew Talbot, CEO of Creative Media Works, the company that runs BBM Messenger Consumer globally.

"As one of the largest and fastest growing mobile classifieds marketplaces in the world, Carousell needed a platform that was agile enough for a startup, but could scale quickly as we expand. We found all these qualities in the Google Cloud Platform (GCP), which gives us a level of control over our systems and environment that we didn't find elsewhere, along with access to cutting edge technologies. We're thrilled that GCP is launching in Singapore, and look forward to being inspired by the way Google does things at scale." — Jordan Dea-Mattson, Vice President Engineering, Carousell

"We are extremely pleased with the performance of GCP, and we are excited about the opportunities opening in Indonesia and other markets, and making use of the Singapore Cloud Region. The outcomes we’ve achieved in scaling, stability and other areas have proven how fantastic it is to have Google and GCP among our key service partners." — Ajey Gore, CTO, Go-Jek

DLPSpannerBigQuery Singapore Multi-Tier Cloud Security certification Next stepsSakura SkyCloudCoverCloud ComradePowerupcloudSingapore region portallocations pageshou

Best practices for App Engine startup time: Google Cloud Performance Atlas

Tuesday, June 13, 2017

By Colt McAnlis, Developer Advocate[Editor’s note: In the past couple of months, Colt McAnlis of Android Developers fame joined the Google Cloud developer advocate team. He jumped right in and started blogging — and vlogging — for the new Google Cloud Performance Atlas series, focused on extracting the best performance from your GCP assets. Check out this synopsis of his first video, where he tackles the problem of cold boot performance in App Engine standard environment. Vroom vroom!]App Engine standard environmentApp Engine startup time is really fast

Javascript file from most CDNsApp Engine manages instancesloading requests Leverage resident instancesResident instanceseven when you’ve scaled to zero, these instances will still be alive

Resident instances helped a developer reduce their startup time Be careful with initializing global variables during parallel requests

Global variables caused one developer a lot of headaches Be careful with dependencies

a platypus-based calculator tracked down a dependency problem Every millisecond countsGoogle Cloud Performance Atlas blog postsvideos

Add log statements to your application on the fly with Stackdriver Debugger Logpoints

Monday, June 12, 2017

By Morgan McLean, Product Manager, Stackdriver Trace and DebuggerStackdriver DebuggerStackdriver Debugger Logpoints

(click to enlarge)

{variableName}hereJavaGoNode.jsPythonRubyfor some time

Partnering on open source: Google and Ansible engineers on managing GCP infrastructure

Friday, June 9, 2017

By Tom Melendez, Senior Software Engineer, GooglePartnering on open source seriesAnsibleGoogle Cloud Platform

Apply a camera-settings hotfix to a satellite orbiting Earth by spinning up a Google Compute Engine instance, testing the latest satellite image build and pushing the settings to the satellite.

Provision and manage GCP's advanced networking features like globally available load-balancers with L7 routing to serve satellite ground images on a public website.

Create a set of networks, routes and firewall rules with security rules to help isolate and protect the various systems involved in the imagery processing pipeline. The raw images may contain sensitive data that must be appropriately screened and scrubbed before being added to the public image repository and network security is critical.

Google Cloud on YouTube@GoogleCloud on Twitter

App Engine users, now you can configure custom domains from the API or CLI

Thursday, June 8, 2017

By Lorne Kligerman, Product Manager

Managing App Engine custom domains from the CLI Google Cloud SDKgcloud components install betagcloud components updategcloud beta domains verify <DOMAIN> gcloud beta domains list-verifiedgcloud beta app domain-mappings create <DOMAIN>gcloud beta app ssl-certificates create --display-name <CERT_DISPLAY_NAME> --certificate <CERT_DIRECTORY_PATH> --private-key <KEY_DIRECTORY_PATH>gcloud beta app domain-mappings update <DOMAIN> --certificate-id <CERT_ID>gcloud beta app ssl-certificates update <CERT_ID> --certificate <CERT_DIRECTORY_PATH> --private-key <KEY_DIRECTORY_PATH>full set of instructionsAPI referenceGoogle App Engine forumpublic issueApp Engine slack channel (#app-engine)

Solutions guide: Preparing Container Engine environments for production

Thursday, June 8, 2017

By Vic Iglesias, Cloud Solutions ArchitectGoogle Cloud PlatformContainer Enginecreated a guide

full solution guidedocumentation

Getting started with Shared VPC

Wednesday, June 7, 2017

By Neha Pattan, Staff Software Engineer
Shared VPCOrganizationGoogle Cloud Platform How does Shared VPC work?host projectservice projectssame network throughput caps

Example topology of a Shared VPC setup.

Shared VPC administrator roleXPN administrator roleInstanceAdmin role Subnetworks USE permissionNetworkUser role Subnetwork IP rangesexpand an existing subnetwork Shared VPC and foldersfolders
Control external accessset up an organization policy that disables external IP accessdocumentation