Google Cloud Platform Blog: July 2017

Guest post: How Seenit uses Google Cloud Platform and Couchbase to power our video collaboration platform

Friday, July 28, 2017

By Dave Starling, CTO, SeenitEditor’s Note: In this guest post, Seenit CTO Dave Starling walks us through how they use Google Cloud Platform (GCP) and Couchbase to build their innovative crowdsourced video platform. SeenitCouchbase

We use Google Compute Engine to autoscale between two and 20 servers.

Google Cloud Storage allows for unified object storage and retrieval. Near-infinite scalability means the service is capable of handling everything from small applications to builds of exabyte-scale systems.

Couchbase’s Full Text Search (FTS) enables us to examine all the words in every document and match them with designated criteria.

Cloud Machine Learning APIs sort clips by objects, gender of speakers and sentiment. The APIs all speak the same language so communication is seamless.

TensorFlowCloud Machine Learning Enginejoint talkCouchbaseCloud Launcherfree trial

How to build a conversational app using Cloud Machine Learning APIs, Part 1

Thursday, July 27, 2017

By Chang Luo and Bob Liu, Software Engineerstour guideAPI.AI Google Cloud Machine Learning APIs for Speech, Vision and Translate

Overview

Architecture

API.AI intents

API.AI contexts

Part 2

API.AI webhook with Cloud Functions

Cloud Vision API

Cloud Speech API

Cloud Translation API

Support multiple languages

Part 3

Support the Google Assistant via Actions on Google Integration

Part 2 Architecture

Using API.AIdocumentation and a sample appother platformsCreate AgentCreate Intents

ContextsContexts User: Where am I?Bot: Please upload a nearby picture and I can help find out where you are.[User uploads a picture of Golden Gate Bridge.]Bot: You are near Golden Gate Bridge.User: How much is the ticket?Bot: Golden Gate Bridge is free to visit.User: When does it close today?Bot: Golden Gate Bridge is open 24 hours a day, 7 days a week.User: How do I get there?[Bot shows a map to Golden Gate Bridge.]

Output ContextsInput Contextswherewhere

locationwherewherelocationlocationwherelocationwherewhere

ticketlocationlocationcontexthoursmaplocation
#context.parameter;inquiry.where.location.ticket#inquiry-where-location.location.

hoursmapticket Next timePart 2Cloud Functionspart 3source code

Three steps to Compute Engine startup-time bliss: Google Cloud Performance Atlas

Wednesday, July 26, 2017

By Colt McAnlis, Developer AdvocateGoogle Cloud PlatformCompute EngineInstance GroupsLoad Balancing Where does the time go?profile the official boot stagesofficial documentationprovisioningstagingrunningrequestprovisioningbootingCloud Shell

Request is the time between asking for a VM and getting a response back from the Create Instance API acknowledging that you’ve asked for it. We can directly profile this by timing how long it takes GCP to respond to the insert instance REST command.

Provisioning is the time GCE takes to find space for your VM on its architecture; you can find this by polling the Get Instance API on a regular basis, and wait for the “status” flag to change from “provisioning” to “running.”

Boot time is when your startup scripts, and other custom code, executes; all the way up to the point when the instance is available. Fellow Cloud Developer Advocate Terry Ryan likes to profile this stage by repeatedly polling the endpoint, and timing the change between receiving 500, 400 and 200 status codes

An example graph generated by timing the request, provision and boot stages, measured 183 times

Profiling your startup scriptsboothelpful to profile your boot scriptsStackdriver Custom Metric APIstatsdbrubeckSECONDS commandlinux build

An example graph generated by timing each stage in a linux startup script

Moving to custom imagespublic images—custom images

An example graph generated by timing the startup phases of the instance. Notice that the graph on the right side is in sub-second scale.

Every millisecond countsGoogle Cloud Performance Atlas blog postsvideos

Cloud Shell’s code editor now in beta

Tuesday, July 25, 2017

By Sachin Kotwani, Product Manageran experimental web-based code editor to Cloud ShellCloud Shellthis

Key bindings to navigate, format or edit code. To see the available key bindings go to Options > Keyboard Shortcuts or type Alt+Shift+? (or Ctrl+Shift+? on Mac OS X).

Syntax highlighting for JavaScript, HTML, CSS and Java; basic content assist for JavaScript, CSS and HTML files. Type Ctrl+Space to open content assist at the current cursor position in the editor.

Find and replace

Font and UI customization

The Cloud Shell code editor in actiondeploy and debug an App Engine application using Cloud Shell

Open Cloud Shell by clicking on the Shell icon on the top right section of the toolbar.

Get sample code. Clone the repository that contains the Google Cloud Platform (GCP) Node samples by typing the following command in the prompt, and navigate to the directory for the Hello World code:

git clone 
https://github.com/GoogleCloudPlatform/nodejs-docs-samples

cd nodejs-docs-samples/appengine/hello-world

Install dependencies and start the app.

npm install

npm start

Preview the app. Click on the web preview icon on the top right of the screen, and click to open port 8080:

Modify the app to show the current time.

Open the code editor from the Cloud Shell toolbar.
In the file tree to the left of the editor navigate to the directory ~/nodejs-docs-samples/appengine/hello-world and, click on app.js
Starting at line 23, replace the contents of the app.get function to the snippet below (changes are indicated in bold). As you start to type date.toTimeString(), you’ll see the autocomplete functionality suggest all the functions available under the Date object.
```
app.get('/', (req, res) => {
  var date = new Date();
  var time = date.toTimeString();
  res.status(200).send('Hello, world! It is now ' + time).end();
});
```
On the bottom shell panel, type ctrl+c to stop the previously running app, and then restart the app

npm start

Refresh the tab showing the "Hello World" message to see the new output.

—herehere

CIO’s guide to data analytics and machine learning

Monday, July 24, 2017

By Fausto Ibarra, Director of Product Management, Google Cloud PlatformEditor's Note: Download the new CIO's guide to data analytics and machine learning here.computers besting human world-championsmastering popular video gamesdiagnose skin cancerseeing results“data wrangling,” “data munging” and “data janitor” workCapturing, preparing and analyzing dataCIO’s guide to data analytics and machine learningarticle If Your Company Isn’t Good at Analytics, It’s Not Ready for AICIO’s guide to data analytics and machine learning

What about unstructured data outside of my operational/transactional databases: raw files, documents, images, system logs, chat and support transcripts, social media?

How can I tap the same base of raw data I’ve collected to quickly get answers as new business questions arise?

Rather than processing historical data in batch, what about processes where I need a real-time view of the business? How can I easily handle data streaming in real time?

How can I unify the scattered silos of data across my organization to provide a current, end-to-end view? What about data stored off-premises in the multiple cloud and SaaS providers I work with?

How can I disseminate this capability across my organization — especially to business users, not just developers and data scientists?

opportunity to learn more about your challenge

How to get started with Cloud Spanner in 5 minutes

Friday, July 21, 2017

By Kelsey Hightower, Developer Advocate

The general availability of Cloud Spanner is really good news for developers. For the first time, you have direct access to horizontally-scaling, cloud-native infrastructure that provides global transactions (think apps that involve payments, inventory, ticketing, or financial trading) and that defaults to the “gold standard” of strong/external consistency without compromising latency. Try that with either a traditional RDBMS or non-relational database.

Thanks to the GCP Free Trial that offers $300 in credits for one year, you can get your feet wet with a single-node Cloud Spanner cluster over the course of a couple weeks. Here’s how to do that, using the Spanner API via gcloud. (Click here for the console-based approach.)

In Cloud Console, go to the Projects page and either create a new project or open an existing project by clicking on the project name.
Open a terminal window and set your project as the default for gcloud. Do this by substituting your project ID (not project name) with the command:

gcloud config set project [MY_PROJECT_ID]

Enable billing for your project.
Enable the Cloud Spanner API for your project.
Set up authentication and authorization (Cloud Spanner uses OAuth 2.0 out of the box) with the following command:
```
gcloud auth application-default login
```
API client libraries now automatically pick up the created credentials. You need to run the command only once per local user environment. (Note: This approach is suitable for local development; for production use, you’ll want to use a different method for auth.)

Next, create a single-node instance:

gcloud spanner instances create test-instance
--config=regional-us-central1 \
--description="Test Instance" --nodes=1

Finally, create a database. To create a database called test-db:

gcloud spanner databases create test-db --instance=test-instance

Alternatively, you can download sample data and interact with it using the language of your choice.

That’s it — you now have your very own Cloud Spanner database. Again, your GCP credit should allow you to run it cost-free for a couple weeks. From there, you can download sample data and interact with it using the language of your choice.

TCP BBR congestion control comes to GCP – your Internet just got faster

Thursday, July 20, 2017

By Neal Cardwell, Senior Staff Software Engineer; Yuchung Cheng, Senior Staff Software Engineer; C. Stephen Gunn, Senior Staff Site Reliability Engineer; Soheil Hassas Yeganeh, Senior Software Engineer; Van Jacobson, Research Scientist and Amin Vahdat, Google Fellow
Google Cloud Platform—

"BBR allows the 500,000 WordPress sites on our digital experience platform to load at lightning speed. According to Google’s tests, BBR's throughput can reach as much as 2,700x higher than today's best loss-based congestion control; queueing delays can be 25x lower. Network innovations like BBR are just one of the many reasons we partner with GCP." — Jason Cohen, Founder and CTO, WP Engine

From GCP services to cloud users: First, when GCP customers talk to GCP services like Cloud Bigtable, Cloud Spanner or Cloud Storage, the traffic from the GCP service to the application is sent using BBR. This means speedier access to your data.

From Google Cloud to internet users: When a GCP customer uses Google Cloud Load Balancing or Google Cloud CDN to serve and load balance traffic for their website, the content is sent to users' browsers using BBR. This means faster webpage downloads for users of your site.

make TCP fasterChrome web browserQUIC paper describing the BBR algorithm at a high leveldescribingBBRLinux TCPQUIC What is BBR? BB R—— loss-based congestion control

In shallow buffers, packet loss happens before congestion. With today's high-speed, long-haul links that use commodity switches with shallow buffers, loss-based congestion control can result in abysmal throughput because it overreacts, halving the sending rate upon packet loss, even if the packet loss comes from transient traffic bursts (this kind of packet loss can be quite frequent even when the link is mostly idle).

In deep buffers, congestion happens before packet loss. At the edge of today's internet, loss-based congestion control causes the infamous “bufferbloat” problem, by repeatedly filling the deep buffers in many last-mile links and causing seconds of needless queuing delay.

Benefits for Google Cloud customersCUBIC

Higher throughput: BBR enables big throughput improvements on high-speed, long-haul links. Consider a typical server-class computer with a 10 Gigabit Ethernet link, sending over a path with a 100 ms round-trip time (say, Chicago to Berlin) with a packet loss rate of 1%. In such a case, BBR's throughput is 2700x higher than today's best loss-based congestion control, CUBIC (CUBIC gets about 3.3 Mbps, while BBR gets over 9,100 Mbps). Because of this loss resiliency, a single BBR connection can fully utilize a path with packet loss. This makes it a great match for HTTP/2, which uses a single connection, and means users no longer need to resort to workarounds like opening several TCP connections to reach full utilization. The end result is faster traffic on today's high-speed backbones, and significantly increased bandwidth and reduced download times for webpages, videos or other data.

Lower latency: BBR enables significant reductions in latency in last-mile networks that connect users to the internet. Consider a typical last-mile link with 10 Megabits of bandwidth, a 40 ms round-trip time, and a typical 1000-packet bottleneck buffer. In a scenario like this, BBR keeps queuing delay 25x lower than CUBIC (CUBIC has a median round-trip time of 1090 ms, versus just 43 ms for BBR). BBR reduces queues and thus queuing delays on last-mile links while watching videos or downloading software, for faster web surfing and more responsive video conferencing and gaming. Because of this ability to curb bufferbloat, one might say that BBR could also stand for BufferBloat Resilience, in addition to Bottleneck Bandwidth and Round-trip propagation time.

EspressoJupiterAndromedagRPCMaglevCloud BigtableSpannerTCP BBRGoogle innovations provide industry-leading performancejoin the public bbr-dev e-mail group

Help keep your Google Cloud service account keys safe: taking charge of your security

Wednesday, July 19, 2017

By Grace Mollison, Cloud Solutions Architect Google Cloud Platform—— Introducing the service account service accountGoogle Compute Engine

Create a service account

Grant least-privilege permissions to the service account using IAM. For example, if your application only needs to access Google Cloud Storage, then grant the service account the roles/storage.admin role.

restart it with an alternative service account

Next, install the client library for the language in which your application is written. (You can also use the SDK but the client libraries are the most straightforward and recommended approach.) With this, your application can use the service account credentials to authenticate applications running on the instance. You don’t need to download any keys because you are using a Compute Engine instance, and we automatically create and rotate the keys.

Protecting service account keys outside GCPcreate a new key pairApplication Default Credentialsauthenticate using the service account
Best practices when downloading service account keys manage the key

If you’ve downloaded the key for local development, make sure it's not granted access to production resources.

Rotate keys using the following IAM Service Account API methods:

ServiceAccounts.keys.create()
Replace old key with new key
ServiceAccounts.keys.delete()

Consider implementing a daily key rotation process and provide developers with a cloud storage bucket from which they can download the new key every day.

Audit service accounts and keys using either the serviceAccount.keys.list() method or the Logs Viewer page in the console.

Restrict who is granted the Service Account Actor and Service Account User role for a service account, as they will have full access to all the resources.

Always use the client libraries and the GOOGLE_APPLICATION_CREDENTIALS for local development.

Prevent developers from committing keys to external source code repositories.

And finally, regularly scan external repositories for keys and take remedial action if any are located.

Key rotationKeyrotator

Create a dedicated project setup for shared resources.

Create a bucket in the dedicated project; do NOT make it publicly accessible.

Create a group for the developers who need to download the new daily key.

Grant read access to the bucket using Cloud IAM by granting the storage.objectViewer role to your developer group for the project with the storage bucket.

Google Cloud Key Management Servicemanage secrets
Prevent committing keys to external source code repositories private git repositoriesgit-secrets

‘git commit’{ "type": "service_account", "project_id": "your-project-id", "private_key_id": "randomsetofalphanumericcharacters", "private_key": "-----BEGIN PRIVATE KEY-----\thisiswhereyourprivatekeyis\n-----END PRIVATE KEY-----\n", "client_email": "keyname@your-project-id.iam.gserviceaccount.com", "client_id": "numberhere", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://accounts.google.com/o/oauth2/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/keyname%40your-project-id.iam.gserviceaccount.com" }git secrets --add 'private_key' git secrets --add 'private_key_id'

This screenshot shows a (now deleted) key to illustrate what developers see when they try to commit files that may contain private details.

Scan external repositories for keys trufflehogShannon entropy
ConclusionAuthentication Overview

Introducing Transfer Appliance: Sneakernet for the cloud era

Tuesday, July 18, 2017

By Ben Chong, Product Manager, Transfer ApplianceSneakernetGoogle Cloud Platform

Transfer Appliance offers up to 480TB in 4U or 100TB in 2U of raw data capacity in a single rackmount device

Google Cloud Storage———

Estimated transfer times for given capacity and bandwidth

Compare the transfer times for 1 petabyte of data.

"Google Transfer Appliance moves petabytes of environmental and geographic data for Makani so we can find out where the wind is the most windy." — Ruth Marsh, Technical Program Manager at Makani

"Using a service like Google Transfer Appliance meant I could transfer hundreds of terabytes of data in days not weeks. Now we can leverage all that Google Cloud Platform has to offer as we bring narratives to life for our clients." — Tom Taylor, Head of Engineering at The MillData Transfer servicesdocumentationSign upsign up here