Google Cloud Platform Blog: September 2016

Google Container Engine now on Kubernetes 1.4

Friday, September 30, 2016

Posted by David Aronchick, Senior Product Manager, Container EngineGoogle Container EngineKubernetes 1.4Google Cloud Platform

One-click alpha clusters can be spun up as easily as a regular cluster, so testing Kubernetes’ alpha features like persistent application support is a one-click operation.

Support for AppArmor in the base image gives applications deployed to Container Engine multiple layers of defense-in-depth.

Integration with Kubernetes Cluster Federation allows you to add a Container Engine cluster to your existing federation, greatly simplifying cloud bursting and multi-cloud deployments.

Rich support for Google Cloud Identity & Access Management allows you to manage GKE clusters with the same multi-faceted roles you use across your GCP projects.

A new Google Container-VM Image makes upgrading a breeze and allows new upgrades to be automatically installed with a simple reboot.

Monitoring of all cluster add-ons ensures that all key functions for your cluster are available and ready to use — one less thing to think about when running a large distributed application.

Niantic - creators of the global phenomenon Pokémon GO, relies on Container Engine to power their extraordinary growth.

Philips - smart connected lighting system Hue, receives 200 million transactions a day that are easily handled by Container Engine.

Google Cloud ML - the new Cloud Machine Learning service from GCP is also running fully on Container Engine.

And many more companies, from Box to Pearson, are choosing Kubernetes to manage their production workloads.

kubernetes-users-mailing listkubernetes-usersfree trial here

Introducing Google Container-VM Image

Friday, September 30, 2016

Posted by Aditya Kali and Amey Deshpande, Software EngineersContainer-VM ImageGoogle Cloud PlatformGoogle Compute Engine Container-VM Image design philosophy Build environment
Designed for containers More secure by design Software updates Getting started$ gcloud compute images list --project google-containers --no-standard-images$ gcloud compute instances create \ --zone us-central1-a \ --image-family gci-stable --image-project google-containers$ gcloud compute ssh --zone us-central1-a$ gcloud compute instances create \ --zone us-central1-a \ --image-family gci-stable --image-project google-containers \ --metadata-from-file user-data=<cloud-config-file> What’s nextContainer-VM Image homepage

Google Cloud Platform sets a course for new horizons

Thursday, September 29, 2016

Posted by Brian Stevens, Vice President, Google CloudGoogle Cloud Delivering Google Cloud Regions for allGoogle Cloud PlatformGoogle Cloud Regions——recent expansion in Oregon

Embracing the multi-cloud worldKubernetesreached version 1.4 earlier this weekGoogle Container-VM image (GCI)Philips Lighting Accelerating the move to cloud data warehousing and machine learning—

Google BigQuery, our fully managed data warehouse, has been significantly upgraded to enable widespread adoption of cloud data analytics. BigQuery support for Standard SQL is now generally available, and we’ve added new features that improve compatibility with more data tools than ever and foster deeper collaboration across your organization with simplified query sharing. We also integrated Identity and Access Management (IAM) that allows businesses to fine-tune their security policies. And to make it accessible for any business to use BigQuery, we now offer unlimited flat-rate pricing that pairs unlimited queries with predictable data storage costs.

Cloud Machine Learning is now available to all businesses. Integrated with our data analytics and storage cloud services such as Google BigQuery, Google Cloud Dataflow, and Google Cloud Storage, it helps enable businesses to easily train quality machine learning models on their own data at a faster rate. “Seeing is believing” with machine learning, so we're rolling out dedicated educational and certification programs to help more customers learn about the benefits of machine learning for their organization and give them the tools to put it into use.

Data Lifecycle on GCP paper Introducing a new engagement model for customer support Niantic as they launched Pokémon GO

The Google Cloud GKE/Kubernetes team that supports many of our customers like Niantic

Bringing Pokémon GO to life on Google Cloud

Thursday, September 29, 2016

Posted by Luke Stone, Director of Customer Reliability EngineeringGoogle CloudNiantic

A shared fateHorizon Creating the Pokémon game worldCloud Datastore Pokémon powered by containersGoogle Container Engine (GKE)Kubernetes projectNetwork Load BalancerHTTP/S Load Balancertripled

The Google Cloud GKE/Kubernetes team that supports many of our customers like Niantic

The Pokémon GO game world was brought to life using over a dozen services across Google Cloud.

Pokémon GO was the largest Kubernetes deployment on Google Container Engine ever. Due to the scale of the cluster and accompanying throughput, a multitude of bugs were identified, fixed and merged into the open source project.

To support Pokémon GO’s massive player base, Google provisioned many tens of thousands of cores for Niantic’s Container Engine cluster.

Google’s global network helped reduce the overall latency for Pokémon Trainers inhabiting the game’s shared world. Game traffic travels Google’s private fiber network through most of its transit, delivering reliable, low-latency experiences for players worldwide. Even under the sea!

Using Google’s cloud networking products: a guide to all the guides

Wednesday, September 28, 2016

Posted by Mike Truty, Cloud Solutions Architect

Google Cloud PlatformGoogle Cloud Networking

Getting startedGoogle Cloud Platform OverviewCloud Platform ServicesKubernetesCloud Dataflow

(click to enlarge)

Networking CodelabsCodelabsCloud CodelabsNetworking 101Networking 102

Google Cloud Networking referencesCloud Networking ProductsCompute Engine networking

Compute Engine networking

measuring network throughputnetworking tutorialsGoogle Cloud Platform for AWS Professionals

Price and performanceCloud Platform PricingCompute Engine PricingGeneral network pricingIP address pricingcharges invoice

egress throughput capsPerfKitBenchMarkersourcesbenchmarks

Real world use casesSecurely Connecting to VM Instances

Seamlessly Migrating your Networks to GCPLoad Balancing, Autoscaling & Optimizing Your App Around the Globe

Regions and ZonesSoftware Defined Networking at ScaleGoogle infrastructurehere

geographic management of dataLoad Balancing and ScalingSetting Up Network Load Balancing

Google Edge Network/PeeringPeeringDBpeering infrastructure

Best practicesBest Practices for Networking and SecurityBest Practices for Enterprise OrganizationsDDoS Best Practices doc

Cloud Identity & Access Management (IAM)Understanding RolesUnderstanding Service AccountsUsing IAM SecurelyAccess Control for Organizations using IAM

A little history of Google Cloud NetworkingSeamlessly Migrating your Networks to GCP Cloud Networking fundamentals

Enter the Andromeda zone - Google Cloud Platform’s latest networking stack, a 2014 blog that details the fundamentals of network virtualization.

Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network. This 2015 paper provides an excellent description of the evolution of datacenter networking at Google. It even comes with a video.

Maglev: A Fast and Reliable Software Network Load Balancer, a 2016 paper that presents an overview of distributed load balancing.

Networking background

A Guided Tour of Data-Center Networking. This 2012 article provides a high-level system overview.

B4: Experience with a Globally Deployed Software Defined WAN. Read this 2013 paper for a detailed look at Google’s quest for simpler and more efficient WAN.

Software Defined Networking at Scale, slides from 2014 about SDN models.

A look inside Google’s Data Center Networks. “Jupiter fabrics...can deliver more than 1 Petabit/sec...enough for 100,000 servers to exchange information at 10Gb/s each, enough to read the entire...Library of Congress in less than 1/10th of a second.”

The Andromeda network architecture (source)

IoT is now easier with Particle and Google Cloud Platform

Tuesday, September 27, 2016

Posted by Preston Holmes, Head of IoT Solutions
ParticleGoogle Cloud PlatformGoogle Cloud DataflowGoogle BigQuery

You can now connect to GCP from the Particle platform developer console.

Google Cloud Pub/Substorage tutorial

Evaluating Cloud SQL Second Generation for your mobile game

Monday, September 26, 2016

Posted by Joseph Holley, Gaming Solutions Architect, Google Cloud PlatformACID-compliantnew solutions documentsource code for the frameworkdocumentationgaming solutions page

Using BigQuery and Firebase Analytics to understand your mobile app

Thursday, September 22, 2016

Posted by Sara Robinson, Developer Advocate
announcedGoogle BigQuery How does the BigQuery export work?linking your Firebase project to BigQuery

Diving into the dataone for iOSone for Android—SELECT user_dim.app_info.app_instance_id, user_dim.device_info.device_category, user_dim.device_info.user_default_language, user_dim.device_info.platform_version, user_dim.device_info.device_model, user_dim.geo_info.country, user_dim.geo_info.city, user_dim.app_info.app_version, user_dim.app_info.app_store, user_dim.app_info.app_platform FROM [firebase-analytics-sample-data:ios_dataset.app_events_20160601] User datauser_dim.app_info.app_instance_idunion featureUNION ALLEXACT_COUNT_DISTINCTSELECT user_dim.geo_info.country as country, EXACT_COUNT_DISTINCT( user_dim.app_info.app_instance_id ) as users FROM [firebase-analytics-sample-data:android_dataset.app_events_20160601], [firebase-analytics-sample-data:ios_dataset.app_events_20160601] GROUP BY country ORDER BY users DESCuser_propertiessome user propertiesSELECT user_dim.user_properties.value.value.string_value as language_code, EXACT_COUNT_DISTINCT(user_dim.app_info.app_instance_id) as users, FROM [firebase-analytics-sample-data:android_dataset.app_events_20160601], [firebase-analytics-sample-data:ios_dataset.app_events_20160601] WHERE user_dim.user_properties.key = "language" GROUP BY language_code ORDER BY users DESC Event dataSELECT event_dim.name, COUNT(event_dim.name) as event_count FROM [firebase-analytics-sample-data:android_dataset.app_events_20160601] GROUP BY event_dim.name ORDER BY event_count DESCSELECT event_dim.params.value.int_value as virtual_currency_amt, COUNT(*) as num_times_spent FROM [firebase-analytics-sample-data:android_dataset.app_events_20160601] WHERE event_dim.name = "spend_virtual_currency" AND event_dim.params.key = "value" GROUP BY 1 ORDER BY num_times_spent DESC Building complex queriesTABLE_DATE_RANGESELECT user_dim.geo_info.city, COUNT(user_dim.geo_info.city) as city_count FROM TABLE_DATE_RANGE([firebase-analytics-sample-data:android_dataset.app_events_], DATE_ADD('2016-06-07', -7, 'DAY'), CURRENT_TIMESTAMP()), TABLE_DATE_RANGE([firebase-analytics-sample-data:ios_dataset.app_events_], DATE_ADD('2016-06-07', -7, 'DAY'), CURRENT_TIMESTAMP()) GROUP BY user_dim.geo_info.city ORDER BY city_count DESCSELECT user_dim.app_info.app_platform as appPlatform, user_dim.device_info.device_category as deviceType, COUNT(user_dim.device_info.device_category) AS device_type_count FROM TABLE_DATE_RANGE([firebase-analytics-sample-data:android_dataset.app_events_], DATE_ADD('2016-06-07', -7, 'DAY'), CURRENT_TIMESTAMP()), TABLE_DATE_RANGE([firebase-analytics-sample-data:ios_dataset.app_events_], DATE_ADD('2016-06-07', -7, 'DAY'), CURRENT_TIMESTAMP()) GROUP BY 1,2 ORDER BY device_type_count DESCPARTITION BYEXACT_COUNT_DISTINCTuser_dim.user_idSELECT STRFTIME_UTC_USEC(eventTime,"%Y%m%d") as date, appPlatform, eventName, COUNT(*) totalEvents, EXACT_COUNT_DISTINCT(IF(userId IS NOT NULL, userId, fullVisitorid)) as users FROM ( SELECT fullVisitorid, openTimestamp, FORMAT_UTC_USEC(openTimestamp) firstOpenedTime, userIdSet, MAX(userIdSet) OVER(PARTITION BY fullVisitorid) userId, appPlatform, eventTimestamp, FORMAT_UTC_USEC(eventTimestamp) as eventTime, eventName FROM FLATTEN( ( SELECT user_dim.app_info.app_instance_id as fullVisitorid, user_dim.first_open_timestamp_micros as openTimestamp, user_dim.user_properties.value.value.string_value, IF(user_dim.user_properties.key = 'user_id',user_dim.user_properties.value.value.string_value, null) as userIdSet, user_dim.app_info.app_platform as appPlatform, event_dim.timestamp_micros as eventTimestamp, event_dim.name AS eventName, event_dim.params.key, event_dim.params.value.string_value FROM TABLE_DATE_RANGE([firebase-analytics-sample-data:android_dataset.app_events_], DATE_ADD('2016-06-07', -7, 'DAY'), CURRENT_TIMESTAMP()), TABLE_DATE_RANGE([firebase-analytics-sample-data:ios_dataset.app_events_], DATE_ADD('2016-06-07', -7, 'DAY'), CURRENT_TIMESTAMP()) ), user_dim.user_properties) ) GROUP BY date, appPlatform, eventNameexport your Google Analytics data
Visualizing analytics dataGoogle Data Studio

this post What’s next?herethis detailed guideBigQuery reference docsfirebase-analyticsgoogle-bigquerylet me know

Global Historical Daily Weather Data now available in BigQuery

Thursday, September 22, 2016

Posted by Lak Lakshmanan, Google Cloud Platform Professional ServicesGoogle BigQuerytemporally and spatially consistentdirectly affect 3.4% of the US Gross Domestic Product—National Oceanic and Atmospheric Association (NOAA) Where are the GHCN weather stations?SELECT name, value/10 AS min_temperature, latitude, longitude FROM [bigquery-public-data:ghcn_d.ghcnd_stations] AS stn JOIN [bigquery-public-data:ghcn_d.ghcnd_2016] AS wx ON wx.id = stn.id WHERE wx.element = 'TMIN' AND wx.qflag IS NULL AND STRING(wx.date) = '2016-08-15'

Google Cloud Datalab— full Datalab notebook on github

Blue dots represent GHCN weather stations around the world.

Using GHCN weather data in your applicationsDatalab notebook

SELECT wx.date, wx.value/10.0 AS prcp FROM [bigquery-public-data:ghcn_d.ghcnd_2015] AS wx WHERE id = 'USW00094846' AND qflag IS NULL AND element = 'PRCP' ORDER BY wx.datewx.valueSELECT wx.date, MAX(prcp) AS prcp, MAX(tmin) AS tmin, MAX(tmax) AS tmax, IF(MAX(haswx) = 'True', 'True', 'False') AS haswx FROM ( SELECT wx.date, IF (wx.element = 'PRCP', wx.value/10, NULL) AS prcp, IF (wx.element = 'TMIN', wx.value/10, NULL) AS tmin, IF (wx.element = 'TMAX', wx.value/10, NULL) AS tmax, IF (SUBSTR(wx.element, 0, 2) = 'WT', 'True', NULL) AS haswx FROM [bigquery-public-data:ghcn_d.ghcnd_2015] AS wx WHERE id = 'USW00094846' AND qflag IS NULL ) GROUP BY wx.date ORDER BY wx.date

Pandas DataFramesee notebook in github for queries and plotting code

BigQuery Views and Data Studio 360 dashboardsSELECT REPLACE(date,"-","") AS date, MAX(prcp) AS prcp, MAX(tmin) AS tmin, MAX(tmax) AS tmax FROM ( SELECT STRING(wx.date) AS date, IF (wx.element = 'PRCP', wx.value/10, NULL) AS prcp, IF (wx.element = 'TMIN', wx.value/10, NULL) AS tmin, IF (wx.element = 'TMAX', wx.value/10, NULL) AS tmax FROM [bigquery-public-data:ghcn_d.ghcnd_2016] AS wx WHERE id = 'USW00094846' AND qflag IS NULL AND value IS NOT NULL AND DATEDIFF(CURRENT_DATE(), date) < 15 ) GROUP BY date ORDER BY date ASCData Studio

Mashing datasets in BigQuerySELECT wx.date, wx.prcp, f.departure_delay, f.arrival_airport FROM ( SELECT STRING(date) AS date, value/10 AS prcp FROM [bigquery-public-data:ghcn_d.ghcnd_2005] WHERE id = 'USW00094846' AND qflag IS NULL AND element = 'PRCP') AS wx JOIN [bigquery-samples:airline_ontime_data.flights] AS f ON f.date = wx.date WHERE f.departure_airport = 'ORD' LIMIT 100

Digging in on Cloud SQL automatic storage increases

Wednesday, September 21, 2016

Posted by Greg Wilson, Head of Developer AdvocacyCloud SQL

Having a database that grows as needed can reduce application downtime by reducing the risk of running out of database space. You can take the guesswork out of capacity sizing without incurring any downtime or performing database maintenance.

If you're managing a growing database, automatic storage increases can save a considerable amount of money. That’s because allocated database storage grows as needed rather than you having to provision a lot of space upfront. In other words, you pay for only what you use plus a small margin.

the documentationMath.min((Math.floor(currentCapacity/25) + 5),25)

Current capacity	Threshold	Amount auto-added
50GB	7GB	7GB
100GB	9GB	9GB
250GB	15GB	15GB
500GB	25GB	25GB
1000GB	25GB	25GB
5000GB	25GB	25GB