Google Cloud Platform Blog: April 2016

While doves were crying: this week on Google Cloud Platform

Friday, April 29, 2016

Posted by Alex Barrett, Editor, Google Cloud Platform Blog
Google App Engine—Why Google App Engine rocks: A Google engineer’s takeGoogle BigQueryGCP PodcastGoogle Cloud StorageCheck it outCapacitor—Google Compute Engine—one-minute video tutorialOpenStack Summit—discuss the benefits

Now playing: New ISO security and privacy certifications for Google Cloud Platform

Thursday, April 28, 2016

Posted by Matthew O’Connor, Product Manager, Google Cloud Platform
additionGoogle Cloud PlatformISO27017ISO27018ISO27001Cloud DataflowCloud BigtableContainer EngineCloud DataprocContainer RegistryCompute EngineApp EngineCloud SQLCloud StorageCloud DatastoreBigQueryGenomicshere

Taming the herd: using Zookeeper and Exhibitor on Google Container Engine

Wednesday, April 27, 2016

Posted by Vic Iglesias, Cloud Solutions ArchitectZookeeperGoogle Container Engine

Exhibitor

Backup and restore

A lightweight GUI for Zookeeper nodes

A REST API for getting the state of the Zookeeper ensemble

Rolling updates of configuration changes

shared file serverhost the shared configuration fileGoogle Cloud PlatformSingle Node File ServerGoogle Cloud LauncherRed Hat Gluster StorageAvere

Provision your file server

Select the Cloud Platform project in which you’d like to launch it

Choose the following parameters for the dimensions of your file server

Zone — must match the zone of your Container Engine cluster (to be provisioned later)
Machine type — for this tutorial we chose a n1-standard-1 as throughput will not be very high for our hosted files
Storage disk size — for this tutorial, we chose a 100GB disk as we will not need high throughput nor IOPS

Click Deploy

Container Engine

Create a new Kubernetes cluster using Google Container Engine

Ensure that the zone setting is the same as you use to deploy your file server (for the purposes of this tutorial, leave the defaults for the other settings)

Click Create

Cloud ShellCloud Console clicking the button in the top right of the interface. It looks like:

Now set up your Zookeeper ensemble in Kubernetes.

Set the default zone for the gcloud CLI to the zone you created for your file server and Kubernetes cluster:

gcloud config set compute/zone us-east1-d

Download your Kubernetes credentials via the Cloud SDK:

gcloud container clusters get-credentials cluster-1

Create a file named exhibitor.yaml with the following content. This will define our Exhibitor deployment as well as the service that other applications can use to communicate with it:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: exhibitor
spec:
 replicas: 3
 template:
   metadata:
     labels:
       name: exhibitor
   spec:
    containers:
    - image: mbabineau/zookeeper-exhibitor
      imagePullPolicy: Always
      name: exhibitor
      volumeMounts:
      - name: nfs
        mountPath: "/opt/zookeeper/local_configs"
      livenessProbe:
        tcpSocket:
          port: 8181
        initialDelaySeconds: 60
        timeoutSeconds: 1
      readinessProbe:
        httpGet:
          path: /exhibitor/v1/cluster/4ltr/ruok
          port: 8181
        initialDelaySeconds: 120
        timeoutSeconds: 1
      env:
       - name: HOSTNAME
         valueFrom:
           fieldRef:
             fieldPath: status.podIP
    volumes:
    - name: nfs
      nfs:
        server: singlefs-1-vm
        path: /data
---
apiVersion: v1
kind: Service
metadata:
  name: exhibitor
  labels:
    name: exhibitor
spec:
  ports:
  - port: 2181
    protocol: TCP
    name: zk
  - port: 8181
    protocol: TCP
    name: api
  selector:
    name: exhibitor

In this manifest we’re configuring the NFS volume to be attached on each of the pods and mounted in the folder where Exhibitor expects to find its shared configuration.

Create the artifacts in that manifest with the kubectl CLI

kubectl apply -f exhibitor.yaml

Monitor the pods until they all enter the RUNNING state

kubectl get pods -w

Run the kubectl proxy command so that you can access the Exhibitor REST API for the Zookeeper state:

kubectl proxy &

Query the Exhibitor API using curl, then use jq to format the JSON response to be more human-readable

export PROXY_URL=http://localhost:8001/api/v1/proxy/
export EXHIBITOR_URL=${PROXY_URL}namespaces/default/services/exhibitor:8181
export STATUS_URL=${EXHIBITOR_URL}/exhibitor/v1/cluster/status
export STATUS=`curl -s $STATUS_URL`
echo $STATUS | jq '.[] | {hostname: .hostname, leader: .isLeader}'

After a few minutes, the cluster state will have settled and elected a stable leader.

DNS nameenvironment variablesthis pageexhibitor.yaml

Why Google App Engine rocks: A Google engineer’s take

Tuesday, April 26, 2016

Posted by Luke Stone, Director of Technical Support—PM EngPMGoogle App Engine——Google Cloud DatastoreBigQueryGoogle Cloud DatalabGoogle Cloud Platform—

Lessons learned from a year of using live migration in production on Google Cloud

Monday, April 25, 2016

Posted by Scott Van Woudenberg, Product Manager, Google Cloud Platform
Google Compute Engine Down and to the rightnote the log scale

Lessons learned60 second pre-migration signalZoneOperationsCloud Logs What's next?metadata maintenance event URIInstance resourceG+Twitterreach out—

Top 5 GCP NEXT breakout sessions on YouTube (so far)

Thursday, April 21, 2016

Posted by Alex Barrett, Editor, Google Cloud Platform BlogGCP NEXTthirty sessionsData Center 360 videoDay One keynotesDay 2 keynotes

What’s the secret behind Google’s legendary uptime? Three words: site reliability engineers. In “How Google Does Planet-Scale Engineering for Planet-Scale Infrastructure” Google engineering director Melissa Bindle lays out the relationship between SREs and developer teams, and how they ensure uptime for Google.com and Google Cloud Platform users. Even if your organization isn’t operating at quite planetary scale, lessons abound for anyone that cares about building and operating resilient environments.

What exactly is machine learning, and what can it do for you? In “Build smart applications with your new superpower: cloud machine learning” Google Developer Advocate Julia Ferraioli walks through the range of Google’s machine learning options, from TensorFlow, to Google Cloud Machine Learning to the Machine Learning APIs. She’s also joined by David Zuckerman, head of developer experience at Wix.com, who describes how it used Google machine learning to mine customer images with the new Vision API.

Cloud Platform users do the darndest things. In Analyzing 25 billion stock market events in an hour with NoOps on GCP, Fidelity Information Systems CTO Neil Palmer and developer Todd Ricker describe how they “ingest, process and analyze the entire U.S. options and equities market in 4 hours using Google Cloud Dataflow, Google Coud Pub/Sub and Google Cloud Bigtable.” That’s 15 TB of data. On a daily basis. That it then makes available for market reanalysis to the regulators.

Cloud Platform isn’t all about crunching large amounts of data for ginormous financial services firms. It’s also about democratizing access to Google’s amazing capabilities to the world at large, wherever they may be. In “Building iOS Apps with Firebase,” Google Cloud Developer Advocate Sara Robinson walks users through how to develop a native iOS app using Firebase, a mobile back-end as a service, and then ties that app into Google Cloud Vision API to provide face and object recognition.

Rounding out the list of most-watched breakout sessions is Painless container management with GKE & Kubernetes. Brendan Burns, Google’s lead engineer on the Kubernetes project, provides context and backstory; Rich Steenberg, principal engineer at WePay, a Kubernetes shop, describes how they’re using Kubernetes today; and Tim Hockin, a Google software engineer, describes what’s new in Kubernetes 1.2, and what’s on tap.

Iron Mountain increases bandwidth to Google Cloud by 10x for faster data ingestion

Wednesday, April 20, 2016

Posted by Ben Chong, Product Manager
Google Cloud Platformoffline data importGoogle Cloud Storage Nearlinehereoffline data import

A simple approach to PCI DSS compliance for Google Cloud Platform

Tuesday, April 19, 2016

Posted by Peter-Mark Verwoerd, Cloud Solutions Architect
Google Cloud Platform— PCI Security Standards CouncilStackdriver MonitoringStackdriver LoggingGoogle BigQuery

A lightweight Google Compute Engine front-end application that accepts credit card information and sends it to an external payment processor. Importantly, that information is never recorded, it's only transmitted.

An external payment processor that charges the credit card if it's accepted or rejects it if it’s not, and notifies your application of the result. Since this is just a notification to your application, no credit card data is transmitted or recorded from the payment processor.

Stackdriver Logging, which logs the actions of every application and server via Squid Proxy which restricts the event traffic and sends them to Stackdriver Monitoring, which monitors the events

BigQuery, which can be used to analyze the logs, run ad-hoc audit queries and create reports.

(click to enlarge)

solution@petermark

Enhanced persistent disks for Google Compute Engine = better Kubernetes and Docker support

Monday, April 18, 2016

Posted by Martin Buhr, Product Manager

Google Cloud Platformnumber of attached persistent disk storage volumestotal quantity of attached persistent disk storageresize persistent disk volumes

(click to enlarge)

networkingColossusperformance and reliabilityKubernetesOpenShiftoffering OpenShift Dedicated on Google Cloud PlatformPods

Google Cloud and Autodesk enable 10x improvement in media rendering efficiency

Monday, April 18, 2016

Posted by Todd Prives, Product Manager, Google Cloud PlatformNABGoogle Cloud Platformin the media and entertainment industryAutodesk Maya

Product teams at Google and Autodesk have developed a cloud-optimized version of Autodesk® Maya®, the popular tool for animation, modeling and rendering. Via a Maya plugin, 3D scenes are transferred to Google Compute Engine in the background, while the artist is working. Because Maya is capable of running on the artist’s workstation and also in the cloud on GCP, this allows artists to run massively parallel rendering far more efficiently than before — taking advantage of the scalability, performance, and price benefits of GCP

Compared with the non-optimized version, Maya® customers see up to 10x improvement in file upload efficiency. This allows many rendering jobs to start instantaneously, cutting wait time and accelerating time to finished shot

RenderMan— Cloud Vision API graduates to General AvailabilityCloud Vision APIGoogle Photos

Logo Detection expanded to cover millions of logos

Updated OCR for understanding more granular text like receipts

Cloud CDN graduates to Beta Cloud CDN Betaglobal load balancing Lytro chooses Cloud Platform and The FoundryLytroThe Foundrywatching their videocloud.google.com/solutions/mediazyncrender.com

Introducing Google Consumer Surveys API: Let users of your app create surveys

Thursday, April 14, 2016

Posted by Dylan Lorimer, Product ManagerEditor's note: Updated April 19, 2016Google Consumer SurveysGoogle Cloud Platform,KantarPredictviaMarketSightgcs-api@google.comAPI documentation site

Introducing the new Google Cloud Load Balancing UI

Wednesday, April 13, 2016

Posted by Prajakta Joshi, Product Manager, Google Cloud PlatformGoogle Cloud PlatformCloud Load Balancing

(click to enlarge)

quickstart video

(click to enlarge)

new UI for a spinGCP NEXTGlobal Load BalancingTCP/UDP network load balancingNSDI

OpenStack users: Backup your Cinder volumes to Google Cloud Storage

Monday, April 11, 2016

Posted by Ben Chong, Product Manager and Mark Lambert, Program Manager, Google Cloud Platform
MitakaRed Hat BiarcaOpenStack Cinder backup driverGoogle Cloud Storage

joinedOpenStack FoundationKubernetes integrationGoogle Cloud Platform

Parameter

Purpose

backup_gcs_credential_file

Denotes the full path of the json file of the Google service account (downloaded from the Google Developer Console in step 3)

backup_gcs_bucket

GCS bucket name to use for backup. Please refer to the official bucket naming guidelines.

backup_gcs_driver

Used for selecting the Google backup driver

backup_gcs_project_id

Denotes the project ID where the backup bucket will be created

backup_gcs_object_size

The size in bytes of GCS backup objects.

default: 52428800 bytes

backup_gcs_block_size

The change tracking size for incremental backup, in bytes.

backup_gcs_object_size has to be a multiple of backup_gcs_block_size

default: 327678 bytes

backup_gcs_user_agent

http user-agent string for the gcs API

backup_gcs_reader_chunk_size

Chunk size for GCS object downloads in bytes.

default: 2097152 bytes

backup_gcs_writer_chunk_size

Chunk size for GCS object uploads in bytes. Pass in a value of -1 to cause the file to be uploaded as a single chunk.

default: 2097152 bytes

backup_gcs_num_retries/td>

Number of times to retry transfers.

default: 3

backup_gcs_bucket_location

Location of GCS bucket.

default: ‘US’

backup_gcs_storage_class

Storage class of GCS bucket.

default: ‘NEARLINE’

backup_gcs_retry_error_codes

List of GCS error codes for which to initiate a retry.

default: [‘429’]

backup_gcs_enable_progress_timer

Enable or Disable the timer to send the periodic progress notifications to Ceilometer when backing up the volume to the GCS backend storage. The default value is True to enable the timer.

default: True

Google Cloud Storage Nearline

Monitor your application errors with Stackdriver Error Reporting

Thursday, April 7, 2016

Posted by Steren Giannini, Product Manager, Google Cloud PlatformStackdriver Error Reporting

Stackdriver Error Reporting: listing errors sorted by occurrence count

Email notification for an error that could not be grouped with previously received errors

Details of an error: understand the root cause

Link an error to an issue URL in your favorite bug tracker

Get startedhttp://console.cloud.google.com/errors

Lessons from a Google App Engine SRE on how to serve over 100 billion requests per day

Wednesday, April 6, 2016

In this blog post we caught up with Chris Jones, a Site Reliability Engineer on Google App Engine for the past three-and-a-half years and SRE at Google for almost 9 years, to find out more about running production systems at Google. Chris is also one of the editors of Site Reliability Engineering: How Google Runs Production Systems, published by O’Reilly and available today. Google App EngineQCJQCJQCJMaglevGoogle Cloud Load BalancerQCJQCJHigh Replication Datastoreslightly wrongQCJwebsitenew bookSREconPosted by Jo Maitland, Managing Editor, Google Cloud Platform

Google and Rackspace co-develop open server architecture based on new IBM POWER9 hardware

Wednesday, April 6, 2016

recently joinedOpen Compute Project48V open rack that we're co-designing with FacebookISAGoogle Cloud PlatformPosted by Maire Mahony, Hardware Engineering Manager at Google & Director, OpenPOWER Foundation

Google Cloud Datastore gets faster cross-platform API

Monday, April 4, 2016

Google Cloud DatastoreGoogle App EngineGoogle Container EngineGoogle Compute Enginenew and simpler pricing for Cloud DatastoreCloud Datastore APIEnable Cloud Datastore APIService Level Agreement New Beta API revision—

Google Cloud Client Librariesclient libraries in our documentation Cloud Datastore Service Level AgreementSLAApp Engine SLACloud Datastoregetting started guidePosted by Dan McGrath, Product Manager, Google Cloud Platform

Google Cloud collaborates with Kinvey for HIPAA-compliant mobile Backend-as-a-Service

Monday, April 4, 2016

Our guest blogs are written by third-party developers, partners and experts with real-world expertise creating and running applications on Google Cloud Platform. They're a great way to dive into the thoughts and opinions of folks at the forefront of software engineering and business development in cloud computing. Today’s guest blog is by Sravish Sridhar, Founder and CEO of mobile Backend-as-a-Service provider, Kinveycollaborating with GoogleGoogle Cloud

An app developer starts to build the UI/UX of their app using the front-end programming language or framework of their choice — Android, Objective-C, Swift, Ionic, Xamarin, PhoneGap, etc.

The developer downloads the Kinvey SDK for the particular language they're using and uses the appropriate Kinvey SDK to take care of client-side functionality like managing and anonymizing auth tokens, marshaling data between the app and Kinvey’s backend APIs, offline caching and sync and data encryption.

The app is wired up to backend functionality by leveraging Kinvey’s backend features, such as an identity service to register/login users, data store to store and retrieve data from the cloud, file store to cache large files like photos and videos, and custom business logic that can be written and provisioned on Kinvey’s Node.js PaaS

In the meantime, owners of backend enterprise systems can connect Kinvey to their enterprise auth and data sources, without writing any code. They use Kinvey’s Mobile Identity Connect (MIC) to connect to auth protocols like Active Directory, OpenID, LDAP, SAML, etc. and Kinvey’s RAPID data connectors and custom data links to connect to enterprise data services like Epic, Cerner, SAP and SharePoint. Services provisioned via MIC and RAPID are then made available to the front-end developers by publishing them in Kinvey’s Service Catalog, with appropriate access policies.

The front-end developer can then "flip a switch" and instruct Kinvey to use a MIC auth service instead of the default Kinvey auth service, and one or more RAPID services instead of sample data stored in collections in the Kinvey data store.

With no front-end app code change, the app then works end-to-end with enterprise auth and data systems.

infrastructureCloud StorageCDN

Plug-in client features for offline caching, network management and RESTful data access to accelerate development

Turn-key backend services for data integration, IAM and orchestration for new mobile use cases

Microservices for interconnectivity between your enterprise systems

Security at every level from mobile client to infrastructure layer

Mobile app analytics and reporting for fine-tuning operations

Kinvey’s HIPAA compliant mobility platformPosted by Sravish Sridhar, Founder/CEO, Kinvey

Introducing Style Detection for Google Cloud Vision API

Friday, April 1, 2016

Google Cloud Platformbeta releaseCloud Vision APICloud Machine Learningunveiled last weekGCP NEXTStyle DetectionherePosted by Miles Ward, Global Head of Solutions