Google Cloud Platform Blog: March 2016

Financial services firm processes 25 billion stock market events per hour with Google Cloud Bigtable

Thursday, March 31, 2016

FISFinTech Top 100 listGoogle Cloud PlatformGoogle Cloud DataflowGoogle Cloud Bigtable

these slidesFISGoogle Cloud BigtablePosted by Misha Brukman, Product Manager for Google Cloud Bigtable

Google Cloud Datastore simplifies pricing, cuts cost dramatically for most use-cases

Thursday, March 31, 2016

Google Cloud Datastore New pricing structureour pricingWritesReadsDeletesFree QuotaNetworkStandard Network costs

New storage usage calculations Building what’s nextGoogle Cloud Datastoregetting started guidePosted by Dan McGrath, Product Manager, Google Cloud Platform

Introducing online resizing of Google Cloud Persistent Disks without downtime

Thursday, March 31, 2016

Persistent Disksonline resizing for Persistent DisksGoogle Compute Engine64 TB volumes¹——Cloud SQL Second GenerationGoogle Cloud SQLPosted by Igor Belianski, Software Engineer, Google Compute Engine
1 Persistent Disk performance depends on the size of the volume and the type of disk you select. Larger volumes can achieve higher I/O levels than smaller volumes.

IAM best practice guides available now

Tuesday, March 29, 2016

Google Cloud Identity & Access ManagemenGoogle Cloud Platform

Using IAM Securely

Designing Resource Hierarchies

Understanding Service Accounts

Using IAM Securely

Least privilege - A set of checks that assist you in restricting your users or applications to not do more than they're supposed to.

Managing Service Accounts and Service Account keys - Provides pointers to help you manage both securely.

Auditing - This covers practices that include reminding you to use Audit logs and cloud logging roles

Policy Management - Some checks to ensure that you're implementing and managing your policies appropriately.

Organization level. The Organization resource represents your company. IAM roles granted at this level are inherited by all resources under the organization.

Project level. Projects represent a trust boundary within your company. Services within the same project have a default level of trust. For example, App Engine instances can access Cloud storage buckets within the same project. IAM roles granted at the project level are inherited by resources within that project.

Resource level. In addition to the existing Google Cloud Storage and Google BigQuery ACL systems, additional resources such as Google Genomics Datasets and Google Cloud Pub/Sub topics support resource-level roles so that you can grant certain users permission to a single resource.

Designing Resource HierarchiesUnderstanding Service Accounts

What resources can the service account access?

What permissions does it need?

Where will the code assuming the identity of the service account be running: on Google Cloud Platform or on-premises?

GCP-iam-feedback@google.comPosted by Grace Mollison, Solutions Architect

Snapchat shares security best practices for running on GCP

Friday, March 25, 2016

GCP NEXTGoogle Cloud PlatformprojectsIAM Roles

AccessControlService can iam.SetPolicy

When bob@ needs access, AccessControlService adds bob@ to policy

AccessControlService removes bob@ after 1 hour

Organizational NodeIAM Service Account APIGCP-iam-feedback@google.comPosted by Jo Maitland, Managing Editor, Google Cloud Platform

Google shares data center security and design best practices

Thursday, March 24, 2016

GCP NEXT 2016 Google Cloud Platform— Security and data protection360 degree data center tour

We build our own hardware and monitoring systems Using machine learning to optimize data center operationsusing our machine learning Powered by renewable energy Google employees operate our data centers, not third parties

Engineering + operation teams are combinedPosted by Joe Kava, VP, Data Center Operations, Google

Google Data Center 360° Tour

Wednesday, March 23, 2016

On desktop using Google Chrome — use your mouse or trackpad to change your view while the video plays

YouTube app on mobile — move your device around to look at all angles while the video plays

And the most immersive way to view — using Google Cardboard (currently supported by the Android YouTube app only, iOS support is coming soon!) Load the video in the YouTube app and tap on the Cardboard icon when the video starts to play. Insert your phone in Cardboard and look around.

A little background . . .Google Cloud Developer Advocacy TeamGoogle Cloud PlatformPosted by Greg Wilson, Head of Developer Advocacy, Google Cloud Platform

How Google is bringing cloud computing innovation to the enterprise

Wednesday, March 23, 2016

GCP NEXT 2016Google Cloud Platformmachine learning and big data analyticsGoogle BigQuery—Google Cloud DataflowGoogle Data Studio 360automating IT operations to remove organizational drag Customer momentumBest BuyDomino’s PizzaSpotifyPocket GemsWixAtomic Fiction Data center expansion US Western region in Oregon and East Asia region in Tokyo, Japan Hybrid cloud management and operational efficiencyBMCPivotalRed HatSAPSplunkTenable Network SecurityVeritasAccentureCI&TCloud Technology PartnersPA ConsultingPwC Enterprise feature enhancements —Audit LoggingGoogle App EngineBigQueryDataflowProjectsService AccountsAPI CredentialsStackdriver LoggingGoogle Cloud StorageGoogle Cloud Pub/SubIAM RolesIAM roles— Customer Supplied Encryption Keyscustomer supplied encryption in beta for Compute Enginesecure storage resourcesNetworkingcross-cloud interconnectSubnetworksCloud RouterCloud VPNCloud Load BalancerhereCloud CDNCommitment to openness and running containers at scaleHadoop MapReduceSpannerSoftware-Defined NetworkingKubernetesDataflowTensorFlowOpen Compute ProjectOpenStack FoundationCloud Native Computing FoundationKubernetes 1.2Google Container Engine
The future of cloud is just beginningGCP NEXT Posted by Brian Stevens, VP, Product Management, Google Cloud Platform

Google takes Cloud Machine Learning service mainstream

Wednesday, March 23, 2016

GCP NEXT 2016Google Cloud PlatformGCP NEXT Machine Learning: Cloud Machine LearningGoogle NowGoogle PhotosTensorFlow

Cloud Machine Learning makes it easy for you to build sophisticated, large scale machine learning models in a short amount of time. It’s portable, fully managed and scalable. Cloud Machine Learning works with data in many formats and is well integrated with other Cloud Platform products such as Google Cloud Dataflow, Google BigQuery, Google Cloud Dataproc, Google Cloud Storage, and Google Cloud Datalab. You can easily build predictive analytics models using your own training data. For example, a financial services app that predicts values using regression models, or a classification service for images. Cloud Machine Learning will take care of everything from data ingestion through to prediction. The result: now any application can take advantage of the same deep learning techniques that power many of Google’s services.

Pre-trained Machine Learning models like Google Translate API and Cloud Vision API are being joined today by Google Cloud Speech API. We're very excited to now offer a full set of APIs that help your applications see, hear and translate. Cloud Speech API brings the same advanced neural network technology that powers voice search in the Google app and voice typing in Google Keyboard. It demonstrates speech-to-text conversion in 80+ languages with unparalleled accuracy, especially in noisy environments, and is blazingly fast. The technology that has already empowered developers to add speech to their Chrome and Android is now available for any application in real-time streaming or batch mode.

Big Data and Analytics:Cloud DataprocApache HadoopApache Spark

BigQuery continues to push the limits of what it means to be a fully managed Analytics Data Warehouse. Today we're announcing lots of exciting new features that make analytics faster, cheaper and easier to use:

Long Term Storage automatically drops the price of your storage by 50% after 90 days.
Automatic Table Partitions (coming soon) simplifies the way you store and query data by partitioning your table by date and querying the date ranges you want.
The new Capacitor storage engine, which accelerates many queries by up to 10x, and Poseidon, a new mechanism that improves data ingest and export speed by 5x.
Direct query and import of Apache AVRO files, simplifying data interoperability
Automatic schema detection of JSON and CSV files.
The new Public Datasets Program to help our community host, share and analyze public datasets.

All these features will come to you automagically, without any upgrades or downtime — the way fully managed is meant to be.

We recently announced Google Data Studio 360, a new reporting and data visualization product that allows you to unify all analytics workflows into one tool. Data Studio allows you to access, transform and share datasets from several different sources such as Google Analytics, BigQuery, and Google Sheets. You can then visualize all of the different sources in the same report and collaborate with your peers to build beautiful and flexible reports. Google Data Studio is available in private Beta (i.e., invite only) for Google Analytics Premium customers.

Open Source:open sourced

TensorFlow, Google’s latest machine learning system, is the #1 Machine Learning project on GitHub today. We’re continuing to develop this ecosystem. For example, you can now use TensorFlow Serving with another of our open-source projects, Kubernetes, to scale and serve ML models. The Cloud Machine Learning product extends these capabilities, enabling you to build powerful Machine Learning models with your data on Cloud Platform.

Earlier this year, we partnered with a number of organizations, including data Artisans, Cloudera, Talend and others to submit the Dataflow model, SDKs and runners for popular OSS distributed systems to the Apache Incubator. This new incubating project, known as Apache Beam, allows you to define portable, powerful and simple data processing pipelines that can execute in either streaming or batch mode.

SpotifyAtomic FictionKhan Academycloud.google.com/Posted by Fausto Ibarra, Director, Product Management

Introducing Google Stackdriver: unified monitoring and logging for GCP and AWS

Wednesday, March 23, 2016

GCP NEXT 2016——Google StackdriverGoogle Cloud Platform¹ A unified view across cloud platforms

Stackdriver Console for hybrid environment

Alerting policy incorporating GCP and AWS capacity metrics

Logs Viewer - search by GCP or AWS service

Eliminate data silos, fix problems faster

Sample Custom Dashboard with AWS and GCP metrics

Alerting policy with ELB Uptime Check and Latency Threshold

Google App Engine

Stackdriver Trace Overview

stream Stackdriver logs to BigQueryGoogle Cloud Datalab Get started in 2 minutes, nothing to maintain or scaleCollectd for metricsFluentd for logs Try Google Stackdriver free during Beta—http://cloud.google.com/stackdriver. Posted by Dan Belcher, Product Manager
1 "Amazon Web Services" and "AWS" are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.^↩

Google Cloud Platform partners with Splunk, BMC and Tenable, making hybrid cloud ops easier

Wednesday, March 23, 2016

GCP NEXT 2016——Google Stackdriverintegrated cloud monitoring and logging serviceSplunkBMCTenable

Easy configurable integration of Cloud Platform with partners

New and complementary capabilities, specifically around Security Information and Event Management (SIEM) and compliance reporting

Google Cloud Platform and SplunkGoogle Cloud Pub/Sub—here

Google Cloud Platform and BMCGCP NEXT

here Google Cloud Platform and Tenable

here Try it out todayGoogle Stackdriver Partner pagePosted by Deepak Tiwari and Joe Corkery, Product Managers, Google Cloud Platform

Getting started with Red Hat OpenShift on Google Cloud Platform

Tuesday, March 22, 2016

announcedOpenShift DedicatedGoogle Cloud PlatformOpenShift clustersKubernetes-based solutionanalyze incoming tweetsGoogle Cloud Pub/SubGoogle BigQueryStep 0: sign-up for Cloud PlatformStep 1:

service account

Step 2Important—Step 3$ base64 -i ~/path/to/downloads/credentials.json‘BASE64_CREDENTIAL_STRING’ImportantStep 4Kubernetes secretapiVersion: v1 kind: Secret metadata: name: google-services-secret type: Opaque data: google-services.json: BASE64_CREDENTIAL_STRING ‘BASE64_CREDENTIAL_STRING’Step 5$ oc create -f google-secret.yamlStep 6https://cloud.google.com/solutions/real-time/kubernetes-pubsub-bigquery

For any pods that need to access Google APIs, modify the pod to create a reference to the secret. The environment variable “GOOGLE_APPLICATION_CREDENTIALS” needs to be exported to the pod — more info on how they work:
https://developers.google.com/identity/protocols/application-default-credentials#howtheywork

In the PubSub-BiqQuery solution, that means you’ll modify two pod definitions:

pubsub/bigquery-controller.yaml
pubsub/twitter-stream.yaml

For example:

apiVersion: v1
kind: ReplicationController
metadata:
  name: bigquery-controller
  labels:
    name: bigquery-controller

    spec:
      containers:
      …
        env:
        … 
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /etc/secretspath/google-services.json
        volumeMounts:
        - name: secrets
          mountPath: /etc/secretspath
          readOnly: true
      volumes:
      - name: secrets
        secret:
          secretName: google-services-secret

Finally, anywhere the solution instructs you to use "kubectl," replace that with the equivalent OpenShift command "oc."

solution—Google Cloud SDKs Join us at GCP Next!GCP NextOpenShift WorkshoponlinePosted by Sami Zuhuruddin, Solutions Architect, Google Cloud Platform

Google Cloud Platform adds two new regions, 10 more to come

Tuesday, March 22, 2016

Google Cloud Platform

US Western region in Oregon

East Asia region in (Tokyo) Japan

openingindustry leading price/performancethis surveythis surveyGeography and RegionsPosted by Varun Sakalkar, Product Manager

Node.js on Google App Engine goes beta

Monday, March 21, 2016

Google App Engine Getting startedgetting started guidessamplesinteractive tutorialsExpressHapiParse-serverMongoDBRedisGoogle Cloud Datastore—Google Container EngineGoogle Compute Enginegcloud npm moduleGoogle BigQueryGoogle Cloud Pub/SubGoogle Cloud Vision APIvar gcloud = require('gcloud')({ projectId: 'my-project', keyFilename: 'keyfile.json' }); var vision = gcloud.vision(); vision.detectText('./image.jpg', function(err, text) { if (text.length > 0) { console.log('We found text on this image...'); } }); Advanced diagnostic toolingGoogle Cloud Debugger