Google Cloud Platform Blog: January 2016

Cloud9 IDE now supports Google Cloud Platform

Wednesday, January 27, 2016

Google App Engineview Cloud9’s documentation Getting startedauthenticate with Google in Cloud9Cloud Source Repositories

Using gcloud and Google Cloud SDKGoogle Cloud SDK

Edit, build and deploy directly from Cloud9

What’s next for Cloud9 and Cloud PlatformCloud9’s community sitePosted by Chris Sells, Product Manager

Google and Red Hat integrate OpenShift Dedicated and Google Cloud Platform to make adopting containers easier

Thursday, January 21, 2016

OpenShift Dedicatedthe Kubernetes projectCloud Native Computing FoundationOpenShift Dedicated Helping Customers Adopt and Operationalize Containers

Improved security: Confidence that containerized applications are developed, deployed and maintained on validated platforms with appropriate provenance and governance.

Services and ecosystem: Delivering lifecycle services and open interfaces for partners to give developers and operators the ability to build and execute a broad array of microservice based applications.

Dynamic scheduling: Provide frictionless resources and management to enable flexible deployment of containers as workloads change.

Storage: Resilient access to application data regardless of container deployment locality.

Cross cloud portability and hybrid deployments: Consistent container deployment frameworks, resources and platforms wherever development and deployment occurs.

operational expertise around containers The Best of Open Source and Cloudhere Posted by Martin Buhr, Product Manager, Google Cloud Platform

Dataflow and open source - proposal to join the Apache Incubator

Wednesday, January 20, 2016

Editor's update February 9, 2016: The Dataflow submission to the Apache Incubator was accepted on February 1, 2016, and the resulting project is now called Apache Beam.Apache Hadoop MapReduceApache SparkApache FlinkDataflow Java SDKcreated one for Apache Flinkdid it for Apache SparkGoogle’s hosted Cloud Dataflow serviceClouderadata ArtisansTalendCasksent a proposal for DataflowApache Software Foundation (ASF)Incubator projectGoogle Cloud Dataflow

Pipeline first, runtime second – With the Dataflow model and SDKs, you focus first on defining your data pipelines, not how they'll run or the characteristics of the particular runner executing them.

Portability – Data pipelines are portable across a number of runtime engines. You can choose a runtime based on any number of considerations, such as performance, cost or scalability.

Unified model – Batch and streaming are integrated into a unified model with powerful semantics, such as windowing, ordering and triggering.

Development tooling – The Dataflow SDK contains the tools you need to create portable data pipelines quickly and easily using open-source languages, libraries and tools.

The World Beyond Batch: Streaming 102

Watch the Dataflow overview presentation from the 2015 @Scale Conference

Take a look at the Dataflow Java SDK GitHub repository, which would be moved to the Apache Software Foundation as a part of our proposal

Read the Dataflow model VLDB paper, which provides a detailed overview of the Dataflow model

Posted by Frances Perry (Software Engineer) and James Malone (Product Manager)

Build a mobile gaming analytics platform

Thursday, January 14, 2016

reference architecture

Batch processing: This pattern helps you process game logs and other large files in a fast, parallelized manner. For example, leading mobile gaming company DeNA moved to BigQuery from Hadoop to get faster query responses for their log file analytics pipeline. In this GDC Lightning Talk video they explain the speed benefits of Google’s analytics tools and how the team was able to process large gaming datasets without the need to manage any infrastructure.

Real-time processing: Use this pattern when you want to understand what's happening in the game right now. Cloud Pub/Sub and Cloud Dataflow provide a fully managed way to perform a number of data-processing tasks like data cleansing and fraud detection in real-time. For example, you can highlight a player with maximum hit-points outside the valid range. Real-time processing is also a great way to continuously update dashboards of key game metrics, like how many active users are currently logged in or which in-game items are most popular.

But why choose between one or the other pattern? A key benefit of this architecture is that you can write your data pipeline processing once and execute it in either batch or streaming mode without modifying your codebase. So if you start processing your logs in batch mode, you can easily move to real-time processing in the future. This is an advantage of the high-level Cloud Dataflow model that was released as open source by Google.

Cloud Dataflow loads the processed data into one or more BigQuery tables. BigQuery is built for very large scale, and allows you to run aggregation queries against petabyte-scale datasets with fast response times. This is great for interactive analysis and data exploration, like the example screenshot above, where a simple BigQuery SQL query dynamically creates a Daily Active Users (DAU) graph using Google Cloud Datalab.

And what about player engagement and in-game dynamics? The BigQuery example above shows a bar chart of the ten toughest game bosses. It looks like boss10 killed players more than 75% of the time, much more than the next toughest. Perhaps it would make sense to lower the strength of this boss? Or maybe give the player some more powerful weapons? The choice is yours, but with this reference architecture you'll see the results of your changes straight away. Review the new reference architecture to jumpstart your data-driven quest to engage your players and make your games more successful, contact us, or sign up for a free trial of Google Cloud Platform to get started.

Further Reading and Additional Resources

- Posted by Oyvind Roti, Solutions Architect

Last chance to present at NEXT: Call for speakers closing

Wednesday, January 13, 2016

announcedGCP NEXT 2016Submitcall for speakers

Data and Analytics: Learn how Google Cloud Platform can help you build more intelligent applications and make better, more timely decisions.

Infrastructure and Operations: See how Google’s infrastructure — including our networks, storage, security, data center operations and DevOps tools — gives you scale, security and reliability.

App and Services Development: Understand how different components of Google Cloud Platform can work together to help you develop and deploy powerful apps.

Solutions Showcase: Learn how our customers and other developers are using Google Cloud Platform in production.

RegisterGoogle+TwitterLinkedIn- Posted by Julia Ferraioli, Developer Advocate, Google Cloud Platform

Top 5 Power Features of the Google Cloud CLI

Tuesday, January 12, 2016

Google Cloud SDK #1. Using cloud service emulatorsgcloud emulators$ gcloud beta emulators datastore start ... [datastore] To connect, set host to http://localhost:8967/datastore [datastore] Admin console is running at http://localhost:8851/_ah/adminlocalhostgcloud-nodegcloud-rubygcloud-pythongcloud-javaDATASTORE_LOCAL_HOSTgcloud emulators$ $(gcloud beta emulators datastore env-init) $ echo $DATASTORE_HOST http://localhost:8967gcloud emulatorsPro tip from Vilas, Engineer on the Cloud SDK #2. Type like the wind with autocompletiongcloud

(Pro tip from Mark, Engineer on the Cloud SDK) #3. Using --format to filter, sort and transform output to CSV, JSON, and more--format--formatGoogle Compute Engine$ gcloud compute instances list \ --format='csv(zone:sort=1,name:sort=2,networkInterfaces[0].networkIP)' > list.csv

--format--format--format(Pro tip from Glenn, Engineer on the Cloud SDK) #4. Using the gcloud tool with PowerShellPowerShell--formatPS> (gcloud compute instances list --format=json | Out-String | ConvertFrom-Json) | ?{$_.zone -match 'asia'} | select name name ---- asia-ops-1 asia-ops-2PS> (gcloud compute instances list --format=json | Out-String | ConvertFrom-Json) | ?{$_.zone -match ’asia’} | %{gcloud compute instances reset $_.name --zone $_.zone}(Pro tip from Valentin, Engineer on the Cloud SDK) #5. Easily ssh to your Compute Engine instances$ gcloud compute ssh my-instance-name$ gcloud compute config-ssh~/.ssh/configsshscpsftp$ ssh myvm.asia-east1-c.myproject$ sftp myvm.asia-east1-c.myprojectgcloud compute sshgcloud compute config-ssh(Pro tip from Stephen, Engineer on the Cloud SDK) Posted by the Google Cloud SDK team

Happy New Year from Google Cloud Platform - still the price/performance leader in public cloud!

Friday, January 8, 2016

Custom Machine Types

You might have noticed that the price is a little closer for high-memory instance types: AWS provides a particularly high ratio of ram to CPU on these, so for our comparison instance we’re using one with 4 cores rather than 2… and it’s still over 15% less expensive.understanding cloud pricingTCO ToolPosted by Miles Ward, Global Head of Solutions, Google Cloud Platform

With Amadeus, Cloud is in the Air

Tuesday, January 5, 2016

Today we hear from Olivier Favorel, Senior Manager, Airline IT at Amadeus. Operating in 195 countries, Amadeus is a leading technology company dedicated exclusively to the global travel industry. When an increase in CPU consumption of just 100 microseconds can mean thousands of dollars of extra hosting, Amadeus turned to Google Cloud Platform to offer new alternatives to its airline customers.1000:1—Given the challenge to maintain accurate and consistent airline offers across all distribution channels, how can we ensure high performance in dynamic content distribution for massive volumes?Amadeus Airline Cloud Availability

Figure 1: Amadeus Airline Cloud Availability architecture

Demonstrate the scalability and performance of flight availability requests using Google Compute Engine. Amadeus is currently handling requests for 4M+ flights per second in its private data center in Munich, for more than 140 airline carriers. This traffic increases by 50% every year.

Contain infrastructure cost of flight availability traffic.

100 microseconds per transaction translates into several thousands of dollars in extra hosting costs over a one-year period The 1,500 cores challenge

Reducing the CPU path-length per transaction thanks to several C++ low level optimizations, and usage of Google’s tcmalloc memory allocator.

Increasing the IO throughput towards Couchbase data store to keep our application cores busy. We were quite impressed by the stability and very low latency of the internal Compute Engine network (stable sub-millisecond round-trip to Couchbase cluster nodes).

Enabling NOOP scheduler on VMs hosting our Couchbase cluster (optimal IO scheduling pattern to increase throughput to SSD drives).

Adjusting the VMs size (CPU/Memory ratio) to ensure that our servers were running constantly between 85-90% CPU usage (n1-highcpu-16 for application servers and n1-highmem-4 for Couchbase cluster nodes).

Figure 2: GCP Console and Performance Reports

The results Going further
Preemptible VMs

Dynamic capacity adjustment is being implemented thanks to Kubernetes (Google’s container orchestration and cluster management solution) that’s being rolled out in the pilot framework to dynamically spawn or shut down application VMs in line with flight availability traffic fluctuation. Kubernetes is shipped by our PaaS partner, Red Hat, as part of their OpenShift offer (we’re building our internal application platform, Amadeus Cloud Services, on top of these strategic products, to ensure our independence to the underlying IaaS provider). Per-minute billing of instances further optimizes the hosting costs.

Preemptible VMs, released in May 2015, run at a much lower price than standard VMs (70% off) but might be terminated, or preempted, by Compute Engine if it requires access to those resources for other tasks. Our plan is to oversize the number of computation VMs by 10% and use exclusively preemptible instance types, assuming that a fraction of those VMs will be terminated on a daily basis but still keeping our overall processing power at the required level to handle the flight availability traffic. Significant cost savings are anticipated with this new feature as well.

Custom machine types, released in November 2015, are being setup to replace our standard instance types (n1-highcpu-16 and n1-highmem-4). Custom VMs will be sized with only the required amount of cores and minimal memory requirement (per GB). The objective is to avoid any waste of CPU/memory.

Return on experience

Performance: Network latency, throughput and stability have astonishing performance. Also, the on-going migration of VMs to next-generation Intel architecture (Haswell) in many regions will bring even more CPU gains to flight availability request processing.

Stability: We faced very few VM outages over the 6-month pilot duration. The maintenance notification process is working great and the live VM migration is really transparent.

Monitoring: The Stackdriver framework is awesome to report both system metrics (CPU, Memory, IOs) and user-defined KPIs (like the rate of airline flights processed per second). Coupled with an efficient alerting system and the “Cloud Console” mobile app, we rapidly ended up with a production-grade monitoring solution.

Pace of innovation: During the six month duration of the pilot, three major announcements were made that helped our project: introduction of preemptible VMs, rollout of custom machine types and most importantly a 15% price drop in May 2015.

Summary