Google Cloud Platform Blog: February 2016

Want to know why your application is slow? Cloud Trace’s new features can help.

Monday, February 29, 2016

Cloud TraceGoogle Cloud Platform 1. Automatic tracing and performance analysis for all App Engine projectsGoogle App Engineinsights

— — 2. Latency shift detection

3. Use Trace API to trace custom workloadsCloud Trace APIJava¹Node.js 4. Intuitive UI with a focus on developer workflows

Logs ViewerGoogle Cloud Monitoring

Google Compute EngineGoogle Container Enginedirect feedbackStack OverflowGCP NEXT 2016Diagnostics - Spend less time diagnosing and more time developing Posted by Sharat Shroff, Product Manager, Google Cloud Platform
1 Java is the registered trademark of Oracle and/or its affiliates.

Announcing the Google Cloud Platform Podcast

Wednesday, February 24, 2016

Google Cloud PlatformGoogle Cloud Platform Podcast—

Google Cloud Platform Developer Advocates (left to right) Mark Mandel and Francesc Campoy

Mark MandelFrancesc CampoyDeveloper Advocacy teamBig DataKubernetesHTTP/2— Ram RamanathanChris Sells—conversations with customers such as Shine TechnologiesRSSiTunesPosted by Mark Mandel, Developer Advocate and Francesc Campoy Flores, Developer Advocate

Spotify chooses Google Cloud Platform to power data infrastructure

Tuesday, February 23, 2016

Google Cloud Platform—

2 billion playlistsCompute EngineGoogle Cloud DatastoreGoogle Cloud BigtableDirect PeeringCloud VPNCloud RouterGoogle Cloud Pub/SubGoogle Cloud DataflowGoogle BigQueryGoogle Cloud DataprocJoin us for the GCP NEXT 2016 opening keynote, where we’ll feature a talk from Nicholas Harteau, VP of Engineering and Infrastructure at Spotify. You can also attend Spotify-led technical sessions where you can learn more about how they’re deploying Google Cloud BigQuery and Dataflow.
- Posted by Guillaume Leygues, Lead Sales Engineer, Google Cloud Platform

Google seeks new disks for data centers

Tuesday, February 23, 2016

2016 USENIX conference on File and Storage Technologies (FAST 2016white paperevery day

collection- Posted by Eric Brewer, VP Infrastructure, Google

Google Cloud Dataproc managed Spark and Hadoop service now GA

Monday, February 22, 2016

Google Cloud DataprocApache HadoopApache Spark———entered beta last yearDevelopers ConsoleGoogle Cloud SDKGoogle BigQueryGoogle Cloud BigtableGoogle Cloud StorageGoogle Cloud Dataflowproperty tuningcluster versioningcustom machine types

Cloud Dataproc tips the scale of running Spark and Hadoop in your favor by lowering cost and complexity while increasing scalability and productivity

—costcomplexity

Low-cost. We believe two things — using Spark and Hadoop should not break the bank and that you should pay for what you actually use. As a result, Cloud Dataproc is priced at only 1 cent per virtual CPU in your cluster per hour, on top of the other Cloud Platform resources you use. Moreover, with per-minute billing and a low 10-minute minimum, you pay for what you actually use, not a rounded (up) approximation.

Speed. With Cloud Dataproc, clusters do not take 10, 15, or more minutes to start or stop. On average, Cloud Dataproc start and stop operations take 90 seconds or less. This can be a 2-10x improvement over other on-premises and IaaS solutions. As a result, you spend less time waiting on clusters and more time hands-on with data.

Management. Cloud Dataproc clusters don't require specialized administrators or software products. Cloud Dataproc clusters are built on proven Cloud Platform services, such as Google Compute Engine, Google Coud Networking, and Google Cloud Logging to increase availability while eliminating the need for complicated hands-on cluster administration. Moreover, Cloud Dataproc supports cluster versioning, giving you access to modern, tested, and stable versions of Spark and Hadoop.

—scaleproductivity

Easy. You can create, monitor, and delete Cloud Dataproc clusters and jobs directly through Google Developers Console and Cloud SDK. For more advanced use cases, you can use the Cloud Dataproc REST API with a programming language, such as Python, to programmatically interact with Cloud Dataproc without hassle.

Modern. Cloud Dataproc is frequently updated with new image versions to support new software releases from the Spark and Hadoop ecosystem. This provides access to the latest stable releases while also ensuring backward compatibility. For general availability we're releasing image version 1.0.0 with support for Hadoop 2.7.2, Spark 1.6.0, Hive 1.2.1, and Pig 0.15.0. Support for other components, such as Apache Zeppelin (incubating) are provided in our GitHub repository for initialization actions.

Integrated. Cloud Dataproc has built-in integrations with other Cloud Platform services, such as BigQuery, Cloud Storage, Cloud Bigtable, and Google Cloud Logging so you have more than just a Spark or Hadoop cluster — you have a complete data platform. You can also use Cloud Dataproc initialization actions to extend the functionality of your clusters.

partner ecosystemArimoAttunityLookerWANdiscoZoomdataMoserPythianTectonic Cloud Dataproc sitegetting started guidepredict keno outcomes with Cloud DataprocStack OverflowPosted by James Malone, Product Manager

Sports Authority handles 2,000 transactions per second with Google Cloud Platform

Friday, February 19, 2016

Sports AuthorityGoogle Cloud Platformsportsauthority.com Choosing the right cloud providerGoogle Apps for WorkGoogle Cloud PlatformGoogle App EngineGoogle Cloud DatastoreGoogle Cloud StorageSOAPUILOADUI

This screenshot shows what customers see when they shop for products on the website — a list of local stores, showing the availability of any given product in each store

In-store kioskGoogle Chromebox

This image shows the in-store kiosk that customers use to locate products available in other stores.

Internal store portalGoogle Custom SearchTempus Nova

This screenshot shows the internal store portal that employees use to monitor daily tasks.

Posted by Jon Byrum, Product Marketing Manager, Google Cloud Platform

Google Cloud Vision API enters Beta, open to all to try!

Thursday, February 18, 2016

Google Cloud Vision API—Google Cloud Storagepricing

Insights from your images: Powered by the same technologies behind Google Photos, Cloud Vision API detects broad sets of objects in your images — from flowers to popular landmarks

Inappropriate content detection: Powered by Google SafeSearch, Cloud Vision API moderates content from your crowd sourced images by detecting different types of inappropriate content.

Image sentiment analysis: Cloud Vision API can analyze emotional attributes of people in your images, like joy, sorrow and anger, along with detecting popular product logos.

Text extraction: Optical Character Recognition (OCR) enables you to detect text within your images, along with automatic language identification across a broad set of languages.

limited previewPhotoFyYik Yakdocumentation¹samplesGoogle Cloud Vision APIhere We couldn't resist showing you our favorite robot again.

- Posted by Ram Ramanathan, Product Manager, Google Cloud Platform

1 Java is the registered trademark of Oracle and/or its affiliates.

Google and Red Hat announce cloud-based scalable file servers

Thursday, February 18, 2016

Google Cloud PlatformGoogle Cloud StorageDataStoreBigQueryPersistent Disksdisks of up to 64TB in sizewith up to 15,000 IOPS

here- Posted by Vic Iglesias, Cloud Solutions Architect

Diagnose problems in your production apps faster with Google Cloud Debugger

Wednesday, February 17, 2016

Google Cloud Debugger
1. Language and runtimes and platforms¹

Java applications running on App Engine, App Engine Managed VMs and Compute Engine

Python applications running on App Engine, App Engine Managed VMs and Compute Engine

Go applications running on Compute Engine

Node.js applications running on App Engine Managed VMs and Compute Engine (Alpha)

2. UI enhanced for debuggingCloud Debugger section

Take snapshots. Cloud Debugger is integrated into common workflows such as deployment.

Setup and select the source code that matches the deployed application by choosing among a variety of source code repositories. Both local and cloud repositories are now supported. You can use it without source code as well.

Traverse a complex source hierarchy using the familiar treeview layout.

Share snapshots and collaborate with other project members, as easily as sharing a URL.

3. Debug using your source code, or none at all

Debug with no access to source at all

We recognize that in many cases, developers may not be able to provide access to their source code. Cloud Debugger now lets you enter just the filename and line number to take a snapshot at that location.

Debug with a source capture

Upload a capture of your source code to help debug your application over multiple sessions without having to connect to a source repository.

Debug with a local source

You can simply point Cloud Debugger to any local source file to take a snapshot. When debugging with local files, the source code is used for that debug session only. No source code is uploaded to Google servers.

Debug with a cloud source repository

Like before, developers can use Cloud Debugger by providing access to the source code for their application using the source code storage and management features provided by Cloud Source Repositories. A source repository provides version control via git and can be managed using the Cloud Console and the new gcloud command-line tool. When a source control system is available, displaying accurate source information is simply a matter of pointing to the correct version of the source code in the source control repository using the developer console.

4. Debug on your terms in your tools

the Cloud Tools for IntelliJ plugindirect feedbackStack Overflow- Posted by Sharat Shroff, Product Manager, Google Cloud Platform
1 Java is the registered trademark of Oracle and/or its affiliates.

Custom Machine Types goes GA, saving you up to 50% on compute costs

Wednesday, February 17, 2016

general availabilityCustom Machine TypesBeta launchour already market-leading prices

Wix has seen an 18% savings in compute to power their media platform that now serves over 75 million users.

Lytics is saving 20% to 50% by accurately matching resource need to each compute type they used to unlock behavior-rich insights with their Customer Data Platform.

iRewind is seeing up to 20% saved in processing cost to power their pipeline that produced more than 500,000 movies just last year.

Per-minute billingSustained Use DiscountsGoogle Container EngineDeployment Manager Compute EngineCreate instance

$ gcloud components update
$ gcloud compute instances create my-custom-vm --custom-cpu 6
--custom-memory 12 --zone us-central1-fwebsite Posted by Sami Iqram, Product Manager, Google Cloud Platform

What it looks like to process 3.5 million books in Google’s cloud

Tuesday, February 16, 2016

Today’s guest blog comes from Kalev Leetaru, founder of The GDELT Project, which monitors the world’s news media in nearly every country in over 100 languages to identify the events and narratives driving our global society.Google BigQuerytwo public BigQuery datasetssample queriesmap two centuries of books

The project began with a single 8-core Google Compute Engine (GCE) instance with a 2TB SSD persistent disk that was used to download the 3.5 million books. I downloaded the books to the instance’s local disk, unzipped them, converted them into a standardized file format, and then uploaded them to Google Cloud Storage (GCS) in large batches, using the composite objects and parallel upload capability of GCS. Unlike traditional UNIX file systems, GCS performance does not degrade with large numbers of small files in a single directory, so I could upload all 3.5 million files into a common set of directories.

Figure 1: Visualization of two centuries of books

Once all books had been downloaded and stored into GCS, I launched ten 16-core High Mem (100GB RAM) GCE instances (160 cores total) to process the books, each with a 50GB persistent SSD root disk to achieve faster IO over traditional persistent disks. To launch all ten instances quickly, I launched the first instance and configured that with all of the necessary software libraries and tools, then created and used a disk snapshot to rapidly clone the other nine with just a few clicks. Each of the ten compute instances would download a batch of 100 books at a time to process from GCS.

Once the books had been processed, I uploaded back into GCS all of the computed metadata. In this way, GCS served as a central storage fabric connecting the compute nodes. Remarkably, even in worst-case scenarios when all 160 processors were either downloading new batches of books from GCS or uploading output files back to GCS in parallel, there was no measurable performance degradation.

With the books processed, I deleted the ten compute instances and launched a single 32-core instance with 200GB of RAM, a 10TB persistent SSD disk, and four 375GB direct-attached Local SSD Disks. I used this to reassemble the 3.5 million per-book output files into single output files, tab-delimited with data available for each year, merging in publication metadata and other information about each book. Disk IO of more than 750MB/s was observed on this machine.

I then uploaded the final per-year output files to a public GCS directory with web downloading enabled, allowing the public to download the files.

access the entire collection todaysample queriesprocessing pipelineI’d like to thank Google, Clemson University, the Internet Archive, HathiTrust, and OCLC in making this project possible, along with all of the contributing libraries and digitization sponsors that have made these digitized books available.Posted by Kalev Leetaru, founder of The GDELT Project

How to build mobile apps on Google Cloud Platform

Wednesday, February 10, 2016

Scale automatically to meet demand

Automatically synchronize data across devices

Handle the offline case gracefully

Send notifications and messages

Build mobile apps using Google Cloud Platform Real-time data synchronization with Firebase

Add managed computation to Firebase apps with Google App Engine

Add flexible computation to Firebase with App Engine Managed VMs

Automatically generate client libraries with App Engine and Google Cloud Endpoints

Have full control with Compute Engine and REST or gRPC

How to build backend services for mobile apps- Posted by Syne Mitchell, Technical Writer, Google Cloud Platform

JGroups-based clustering and node discovery with Google Cloud Storage

Thursday, February 4, 2016

JGroupsGoogle Cloud StorageGoogle Compute Engine Overview of JGroups clustering on Cloud StorageGOOGLE_PING.GOOGLE_PING
Configure Cloud Storage to store JGroups configuration files

In the Cloud Platform Console, go to the Cloud Storage browser.

Click Create bucket.

In the Create bucket dialog, specify the following:

A bucket name, subject to the bucket name requirements
The Standard storage class
A location where bucket data will be stored

GOOGLE_PING

Open the Storage settings page in the Google Cloud Platform Console.

Select the Interoperability tab.

If you have not set up interoperability before, click Enable interoperability access. Note: Interoperability access allows Cloud Storage to interoperate with tools written for other cloud storage systems. Because GOOGLE_PING is based on the Amazon-oriented S3_PING class in JGroups, it requires interoperability access.

Click Create a new key.

Make note of the Access key and Secret values—you'll need them later.

Important Configure your clustered application to use GOOGLE_PINGGOOGLE_PING

Edit your JGroups XML configuration file (jgroups.xml in most cases).

Modify the file to use TCP instead of UDP: <tcp bind_port="7800">

Locate the PING section and replace it with GOOGLE_PING, as shown in the following example. Replace your-jgroups-bucket with the name of your Cloud Storage bucket, and replace your-access-key and your-secret with the values of your access key and secret:

<GOOGLE_PING location="your-jgroups-bucket" access_key="your-access-key" secret_access_key="your-secret" timeout="2000" num_initial_members="3"/>GOOGLE_PING WarningGOOGLE_PING <google_ping ...="" port="443">JGroup subsystemGOOGLE_PING DemonstrationGOOGLE_PINGVert.x$ java -Djava.net.preferIPv4Stack=true
-Djgroups.bind_addr=10.240.0.2 -jar
my-sender-fatjar-3.1.0-fat.jar -cluster -cluster-host 10.240.0.2Note(external_ip)-Djgroups.external_addr=<external_ip>

.listJGROUPS_CLUSTER

$ java -Djava.net.preferIPv4Stack=true
-Djgroups.bind_addr=10.240.0.2 -jar
my-receiver-fatjar-3.1.0-fat.jar -cluster -cluster-host
10.240.0.2.list

.list

Get startedGOOGLE_PINGsigning up for a free trialPosted by Grace Mollison, Solutions Architect

Another Big Data blog, in 2016? Really? Why?

Wednesday, February 3, 2016

Google Cloud Platformunder the hoodSpark and Dataflow programming modelsterabytes in secondsthis blog is for you—Posted by Jo Maitland, Managing Editor, Google Cloud Platform

GCP NEXT 2016: A sneak peak behind the scenes

Tuesday, February 2, 2016

Go programing languageKubernetesGCP NEXT 2016

program

"From idea to market in less than 6 months: Creating a new product with GCP," presented by CI&T — App Developer Track

"Painless container management with Google Container Engine & Kubernetes," presented by Brendan Burns & Tim Hockin, Google — Infrastructure & Operations Track

"Cloud data warehousing with BigQuery featuring Dropbox Nighthawk," presented by Jordan Tigani, Google & Dropbox — Data & Analytics Track

"Security analytics for today's cloud-ready enterprise," presented by Matt O’Connor, Google & PwC — Solutions Showcase

Call for SpeakersRegisterGoogle+TwitterLinkedInPosted by Kelsey Hightower, Developer Advocate, Google Cloud Platform