Google Cloud Platform Blog: March 2017

Google Cloud Platform expands to Mars

Friday, March 31, 2017

By Google Cloud Storage and Google Geo teamsGoogle Cloud Platform

Visit Mars with Google Street View

Mars and other bodies in space

Explore more of Mars in Google Maps on Chrome

Google Cloud StorageOuter Space Treaty

Walk inside our new data center in Google Street View

faster-than-light data coordinationP.S. Check out Curiosity’s journey across the Red Planet on Mars Street View.

How release canaries can save your bacon - CRE life lessons

Friday, March 31, 2017

By Adrian Hilton, Customer Reliability EngineerReliable releases and rollbacks

Photo taken by David Carroll

John Scott Haldane—canary processreceives live production traffic—rollbacks are safe Client canarying

How will you deploy the new version to only a small percentage of users?

How will you detect if the new version is crash-looping, dropping traffic or showing users errors? (“What's the monitoring sound of no queries happening?”)

— —application package file Web clients—— Android APK releasesstaged rollouts

You have no control over when eligible-for-update users will actually check in; normally it’ll be within 24 hours, assuming they have adequate connectivity, but this may not be true for users in countries where cellular and Wi-Fi data services are slow and expensive per-byte.

You have no control over whether users will accept your update on their mobile device, which can be a particular issue if the new release requires additional permissions.

Release percentage steps

The first (canary) step should generate enough traffic for any problems to be clear in your monitoring or logging; normally somewhere between 1% and 10% depending on the size of your user base.

Each step involves significant manual work and delays the overall release. If you step by 3% per day, it will take you a month to do a complete release.

Going up by a single large increment (say, 10% to 100%) can reveal dramatic traffic problems that weren’t apparent at much smaller traffic levels: try not to increase your upgraded user base by more than 2x per step if this is a risk.

If a new version is good, you generally want most of your users to pick it up quickly. If you're doing a rollback, you want to ramp up to 100% much faster than for a new release.

Traffic patterns are often diurnal — typically, highest during the daytime — so you may need at least 24 hours to see the peak traffic load after a release.

In the case of mobile apps, you'll also need to allow time for the users to pick up and start using the new release after they’ve been enabled for it.

staged update———

Summary

“Rollback early, rollback often.” Try to move your service towards this philosophy, and you’ll reduce the Mean Time To Recover of your service.

“Canary your rollouts.” No matter how good your testing and QA, you'll find that your binary releases occasionally have problems with live traffic. An effective canarying strategy and good monitoring can reduce the Mean Time To Detect these problems, and dramatically reduce the number of affected users.

Google App Engine flexible environment now available from europe-west region

Thursday, March 30, 2017

By Justin Beckwith, Product Managerbig newsGoogle App Engine flexible environment

Cloud SDKgcloud app create --region europe-westCloud LocationsGeography and Regions pages

Solution guide: Archive your cold data to Google Cloud Storage with Komprise

Thursday, March 30, 2017

By Manvinder Singh, Strategic Technology Partnerships, Google Cloud56% of enterprisesGoogle Cloud StorageKomprisefull tutorial guide

NearlineColdlineself-service tutorialwebinarreach out to us

Enterprise Slack apps on Google Cloud--now easier than ever

Wednesday, March 29, 2017

By Tim Swast, Developer Programs Engineer, Google Cloudstreamlined path to building appsmessage buttonsEvents API

Google App EngineCloud FunctionsOAuth2 flow
Google Cloud Platform samples for SlackGoogle Cloud Platform (GCP)Google Cloud Next '17best practices for building bots for the enterprise#bots channelGoogle Cloud Platform Slack community

Reliable releases and rollbacks - CRE life lessons

Friday, March 24, 2017

By Adrian Hilton, Customer Reliability EngineerEditor’s note: One of the most common causes of service outages is releasing a new version of the service binaries; no matter how good your testing and QA might be, some bugs only surface when the affected code is running in production. Over the years, Google Site Reliability Engineering has seen many outages caused by releases, and now assumes that every new release may contain one or more bugs.

Detecting when a new release is actually broken;

Moving users safely from a bad release to a “hopefully” fixed release; and

Preventing too many clients from suffering through a bad release in the first place (“canarying”).

nginxHierarchy of ReliabilitySRE Book Detectiongoodbad Rollbacks

easy to perform; and

trusted to be low-risk.

Testing rollbacks Incompatible changes

Release binary v+1

Upgrade database schema

Release binary v+2

Summary

Solution guide: backing up Windows files using CloudBerry Backup with Google Cloud Storage

Friday, March 24, 2017

By Brad Svee, Head of Solutions, Google Cloud PlatformGoogle Cloud Platform

Creating incremental data copies with low impact on production workloads

Data encryption on all transferring paths

Flexible retention policy, allowing you to balance the volume of data stored and storage space used

Ability to carry out hybrid restores with the use of local and cloud storage resources

Google Cloud Storagehow to use CloudBerry Backup with Google Cloud Storage and easily restore any files

Cloud SQL for PostgreSQL: Managed PostgreSQL for your mobile and geospatial applications in Google Cloud

Thursday, March 23, 2017

By Brett Hesterberg, Product Manager, Google Cloud PlatformGoogle Cloud Next ‘17Google Cloud SQL

Feature highlights

Flexible backups: Schedule automatic daily backups or run them on-demand.

Automatic storage increase: Enable automatic storage increase and Cloud SQL will add storage capacity whenever you approach your limit.

Open standards: We embrace the PostgreSQL wire protocol (the standard connection protocol for PostgreSQL databases) and SSL, so you can access your database from nearly any application, running anywhere.

Security features: Our Cloud SQL Proxy creates a local socket and uses OAuth to help establish a secure connection with your application or PostgreSQL tool. It automatically creates the SSL certificate and makes more secure connections easier for both dynamic and static IP addresses.

Geospatial support: Easily enable the popular PostGIS extension for geospatial objects in Postgres.

Custom instance sizes: Create your Postgres instances with the optimal amount of CPU and memory for your workloads

Create Cloud SQL for PostgreSQL instances customized to your needs.

More features coming soon

Automatic failover for high availability

Read replicas

Additional extensions

Precise restores with point-in-time recovery

Compliance certification as part of Google’s Cloud Platform BAA

Case study: Descartes Labs delves into Earth’s resources with Cloud SQL for PostgreSQLGoogle Earth EngineDescartes Labs

click to enlarge

Google Cloud Platformvirtual supercomputer on demand

click to enlarge

GeoVisual Searchautomatic storage increases

click to enlarge

See what GCP can do for youSign up100% sustained use discount during betaAloomaInformaticaSegmentStitchTalendXplentyChartIOiChartsLookerMetabaseZoomdata "PostgreSQL is one of Segment’s most popular database targets for our Warehouses product. Analysts and administrators appreciate its rich set of OLAP features and the portability they’re ensured by it being open source. In an increasingly “serverless” world, Google’s Cloud SQL for PostgreSQL offering allows our customers to eschew costly management and operations of their PostgreSQL instance in favor of effortless setup, and the NoOps cost and scaling model that GCP is known for across their product line." — Chris Sperandio, Product Lead, Segment "At Xplenty, we see steady growth of prospects and customers seeking to establish their data and analytics infrastructure on Google Cloud Platform. Data integration is always a key challenge, and we're excited to support both Google Cloud Spanner and Cloud SQL for PostgreSQL both as data sources as well as targets, to continue helping companies integrate and prepare their data for analytics. With the robustness of Cloud Spanner and the popularity of PostgreSQL, Google continues to innovate and prove it is a world leader in cloud computing." — Saggi Neumann, CTO, Xplenty

Crash exploitability analysis on Google Cloud Platform: security in plaintext

Wednesday, March 22, 2017

By Ben Scarlato, Software Engineer Security vulnerabilities in crashesGoogle Cloud Platformuse-after-freeSeptember 2016 Analysis tacticsGoogle Compute EngineGoogle App Enginegdb Automating analysis Always evolving

Digging deep on PHP 7.1 for Google App Engine

Friday, March 17, 2017

By Brent Shaffer, Developer Programs Engineer, Google Cloud PlatformGoogle Cloud Next ‘17Google App EngineGetting started
getting started guidessamplescodelabsinteractive tutorialsLaravelSymfonyWordpressMongoDBMySQLCloud DatastoreGoogle Container EngineGoogle Compute EngineDeploying to App Engine on PHP 7.1Google Cloud SDKecho "<?php echo 'Hello, World';"> index.php gcloud app deployenv: flex runtime: php runtime_config: document_root: .Cloud ConsoleInstalling dependenciesComposercomposer require "php:7.1.*" --ignore-platform-reqsUsing Google’s APIs and servicesGoogle Cloud client libraryGoogle Cloud DatastoreGoogle Cloud Pub/SubGoogle BigQuerycomposer require google/cloud0.24.0{ "require": { "google/cloud": "^0.24.0" } }index.php<?php require_once __DIR__ . '/vendor/autoload.php'; $client = new Google\Cloud\BigQuery\BigQueryClient(); $query = 'SELECT TOP(corpus, 10) as title, COUNT(*) as unique_words ' . 'FROM [publicdata:samples.shakespeare]'; $queryResults = $client->runQuery($query); foreach ($queryResults->rows() as $result) { print($result['title'] . ': ' . $result['unique_words'] . PHP_EOL); }gcloud app deploy gcloud app browseUse your favorite frameworkLaravelSymfonyDrupalWordpressSilexWordpress pluginGoogle Cloud StorageCommitment to PHP and open source

https://github.com/GoogleCloudPlatform/php-docker

https://github.com/GoogleCloudPlatform/google-cloud-php

https://github.com/google/google-api-php-client

https://github.com/GoogleCloudPlatform/wordpress-plugins

@googlecloudrequest an invite#PHP

Discover and redact sensitive data with the Data Loss Prevention API

Thursday, March 16, 2017

By Scott Ellis, Product Managerwe introduced a number of security enhancementsData Loss Prevention APIGoogle Cloud PlatformDLP on Gmail and Drive

Be smart with your data

Your data is your most critical asset Google BigQueryGoogle Cloud Natural Language APICloud Machine Learning Engine—demo

DLP API product page

Cloud KMS GA, new partners expand encryption options

Wednesday, March 15, 2017

By Maya Kaczorowski, Product ManagerGoogle Cloud Next ‘17Cloud Key Management Service (KMS) is now generally available—Cloud KMS is now generally available.Google Cloud PlatformCloud Identity Access ManagementCloud Audit Loggingavailability SLARavelin “Encryption is absolutely critical to any company managing their own systems, transmitting data over a network or storing sensitive data, including sensitive system configurations. Cloud KMS makes it easy to implement best practices for secret management, and its low latency allows us to use it for protecting frequently retrieved secrets. Cloud KMS gives us the cryptographic tools necessary to protect our secrets, and the features to keep encryption practical.” — Leonard Austin, CTO at Ravelin. Managing your secrets in Google Cloudrecommendations on how to manage your secrets in Google CloudGoogle Cloud Storage Customer-Supplied Encryption Key partnersCloud StorageCompute Engine Gemalto “Gemalto is dedicated to multi-cloud enterprise key management by ensuring customers have the best choices to maintain high assurance key ownership and control as they migrate operations, workloads and data to the cloud. Gemalto KeySecure has supported Client-Side Encryption with Google Cloud Storage for years, and is now extending support for Customer Supplied Encryption Keys (CSEK)." — Todd Moore SVP of Encryption Products at Gemalto Ionic "We are excited to announce the first of many powerful capabilities leveraging Google's Customer Supplied Encryption Keys (CSEK). Our new Ionic Protect for Cloud Storage solution enables developers to simply and seamlessly use their own encryption keys with the full capabilities of the Ionic platform while natively leveraging Google Cloud Storage.” — Adam Ghetti, Founder and CEO of Ionic KeyNexus "KeyNexus helps customers supply their own keys to encrypt their most sensitive data across Google Cloud Platform as well as hundreds of other bring-your-own-key (BYOK) use cases spanning SaaS, IaaS, mobile and on-premise, via secure REST APIs. Customers choose KeyNexus as a centralized, platform-agnostic, key management solution which they can deploy in numerous highly available, scalable and low latency cloud or on-premise configurations. Using KeyNexus, customers are able to supply keys to encrypt data server-side using Customer-Supplied Encryption Keys (CSEKs) in Google Cloud Storage and Google Compute Engine" — Jeff MacMillan, CEO of KeyNexus Thales “Protected by FIPS 140-2 Level 3 certified hardware, the Thales nShield HSM uses strong methods to generate encryption keys based on its high-entropy random number generator. Following generation, nShield exports customer keys into the cloud for one-time use via Google’s Customer-Supplied Encryption Key functionality. Customers using Thales nShield HSMs and leveraging Google Cloud Platform can manage their encryption keys from their own environments for use in the cloud, giving them greater control over key material” — Sol Cates, Vice President Technical Strategy at Thales e-Security Virtru “Virtru offers business privacy, encryption and data protection for Google Cloud. Virtru lets you choose where your keys are hosted and how your content is encrypted. Whether for Google Cloud Storage, Compute Engine or G Suite, you can upload Virtru-generated keys to Google’s CSEK or use Virtru’s client-side encryption to protect content before upload. Keys may be stored on premise or in any public or private cloud." — John Ackerly, Founder and CEO of Virtru Encryption by default, and more key management optionsencrypts customer content stored at rest, without any action required from the customer, using one or more encryption mechanisms

ASP.NET Core containers run great on GCP

Tuesday, March 14, 2017

By Chris Sells, Product Managerrun Docker containers on Google App Enginemanage containerized ASP.NET Core apps with Kubernetesbeta support for ASP.NET Core on App Engine flexible environment ASP.NET Core on App Engine Flexible Environmentruntime: aspnetcore env: flexGoogle Container Enginedotnet restore dotnet publish -c Release copy app.yaml .\bin\Release\netcoreapp1.0\publish\app.yaml gcloud beta app deploy .\bin\Release\netcoreapp1.0\publish\app.yaml gcloud app browse— ASP.NET Core on Container Enginegcloud container clusters create cluster-1 gcloud container clusters get-credentials cluster-1Google Cloud Container Builder,steps: - name: 'gcr.io/gcp-runtimes/aspnetcorebuild-1.0:latest' - name: gcr.io/cloud-builders/docker:latest args: [ 'build', '-t', 'gcr.io/&ltprojectid&gt/app:0.0.1', '--no-cache', '--pull', '.' ] images: ['gcr.io/&ltprojectid&gt/app:0.0.1'] with the project ID where you want to run your app. To build the Docker image for your published ASP.NET Core app, run the following commands:

dotnet restore dotnet publish -c Release gcloud container builds submit --config=cloudbuild.yaml .\bin\release\netcoreapp1.0\publish\kubectl run --image=gcr.io//app:latest --replicas=2 --port=8080 kubectl expose deployment --port=80 --target-port=8080 --type=LoadBalancer kubectl get services

GCP ASP.NET Core runtime in Visual Studio

Being able to deploy from the command line is great for automated CI/CD processes. For more interactive usage, we’ve also built full support for deploying to both App Engine and Container Engine from Visual Studio via the Cloud Tools for Visual Studio extension. Once it’s installed, simply right-click on your ASP.NET Core project in the Solution Explorer, choose Publish to Google Cloud and choose where to run your code:

If you deploy to App Engine, you can choose App Engine-specific options without an app.yaml file:

Likewise, if you choose Container Engine, you receive Kubernetes-specific options that also don’t require any configuration files:

The same underlying commands are executed regardless of whether you deploy from the command line or from within Visual Studio (not counting differences between App Engine and Container Engine, of course). Choose the option that works best for you.

For more details about deploying from Visual Studio to App Engine and to Container Engine, check out the documentation. And if you’d like some help choosing between App Engine and Container Engine, the computing and hosting services section of the GCP overview provides some good guidance.

App Engine in Google Cloud Explorer

If you deploy to App Engine, the App Engine node in Cloud Explorer provides additional information about running services and versions inside Visual Studio.

The Google App Engine node lists all of the services running in your project. You can drill down into each service and see all of the versions deployed for that service, their traffic allocation and their serving status. You can perform most common operations directly from Visual Studio by right-clicking on the service, or version, including managing the service in the Cloud Console, browsing to the service or splitting traffic between versions of the service.

For more information about App Engine support for ASP.NET Core, I recommend the App Engine documentation for .NET.

Client Libraries for ASP.NET Core

There are more than 100 Google APIs available for .NET in NuGet, which means that it’s easy to get to them from the command line or from Visual Studio:

These same libraries work for both ASP.NET and ASP.NET Core, so feel free to use them from your container-based apps on GCP.

Stackdriver support for ASP.NET Core

Some of the most important libraries for you to use in your app are going to be those associated with what happens to your app once it’s running in production. As I already mentioned, simply using the ASP.NET Core runtime for GCP with your App Engine or Container Engine apps automatically routes the standard and error output to Stackdriver Logging. However, for more structured log statements, you can also use the Stackdriver logging API for ASP.NET Core directly:

using Google.Cloud.Diagnostics.AspNetCore;
...
public void Configure(ILoggerFactory loggerFactory) {
    loggerFactory.AddGoogle("<projectid>");
}
...
public void LogMessage(ILoggerFactory loggerFactory) {
    var logger = loggerFactory.CreateLogger("[My Logger Name]");
    logger.LogInformation("This is a log message.");
}

the Stackdriver Logging pagepublic void Configure(IApplicationBuilder app) { string projectId = ""; string serviceName = ""; string version = ""; app.UseGoogleExceptionLogging(projectId, serviceName, version); }public void ConfigureServices(IServiceCollection services) { services.AddGoogleTrace(""); } ... public void Configure(IApplicationBuilder app) { app.UseGoogleTrace(); }Stackdriver Trace

documentation for Google.Cloud.Diagnostics.AspNetCore

Where are we?

As containers become more central to app packaging and deployment, the GCP ASP.NET Core runtime lets you bring your ASP.NET skills, processes and assets to GCP. You get a Google-supported and maintained runtime and unstructured logging out of the box, as well as easy integration into Stackdriver Logging, Error Reporting and Trace. Further, you get all of the Google APIs in NuGet that support ASP.NET Core apps. And finally, you can choose between automated deployment processes from the command line, or interactive deployment and resource management from inside of Visual Studio.

Combine that with Google’s deep expertise in containers exposed via App Engine flexible environment and Google Container Engine (our hosted Kubernetes offering), and you get a great place to run your ASP.NET Core apps and services.