Google Cloud Platform Blog: January 2015

In case you missed it in January: Cloud pricing, complex business challenges in the cloud and our new series unpacks containers

Friday, January 30, 2015

Digging into containersKubernetescAdvisornamed Open Source Rookies of the YearAn introduction to containers, Kubernetes, and the trajectory of modern cloud computingDockerKubernetesGoogle Container EngineWhat makes a container clusterEverything you wanted to know about Kubernetes but were too afraid to askGoogle Container EngineGoogle Container Registry Demystifying cloud pricingGoogle Cloud Platform pricing compared to Amazon Web Services Tech tips on tips: Dataflow Big Data pipelines, verify MongoDB backups, diagnose bottlenecks...

Run Dataflow Big Data pipelines everywhere, thanks to Cloudera

Automate MongoDB backup validation jobs on Google Compute Engine

Use Google Cloud Monitoring to help gain insight into application performance

Diagnose service performance bottlenecks with Google Cloud Trace, which was introduced at Google I/O 2014

Run Hadoop tasks more seamlessly than ever on the cloud with Hortonworks Data Platform

Deploy Elasticsearch, a tool for searching and analyzing large volumes of data

From genomics to website design: how to’s with our customers

Alacris Theranostics, a Berlin-based spin-off of the Max Planck Institute for Molecular Genetics is using Google Cloud Platform to better match cancer patients with the most promising drug therapies.

Aucor, based in Finland, transitioned customer websites onto Google Cloud Platform, providing them the capacity to scale with their expanding customer base and focus on what they do best: design awesome websites.

Shine Technologies, a digital consultancy based in Australia, uses Google BigQuery to help businesses make sense of the billions of ad clicks, ad impressions and other data that guide business decisions.

Aerospike, an open-source NoSQL database based here in Mountain View, pushes the limits of Local SSD technology to offer blazing performances: fully 95% of local SSD reads complete in under 1 ms. In fact, benchmarks show that Aerospike delivers a 15x price advantage in storage costs with Local SSD compared with RAM.

New year, new series: Introducing the Learn with Google Cloud Platform Webinar Serieswebinar

Everything you wanted to know about Kubernetes but were afraid to ask

Friday, January 30, 2015

a postcontainer cluster management Everything you wanted to know about Kubernetes but were afraid to askannounced

Given that Google already has its own mature, robust cluster management systems (which handle around two billion new containers a week), why did you create Kubernetes?

How does Kubernetes relate to Docker? How does it differ from Docker Swarm?

What insures that Google is committed to the Kubernetes open source project over the long run?

How does Kubernetes fit in with and augment your overarching strategy for Google Cloud Platform?

What incentive does Google have to make Kubernetes great outside of Google Cloud Platform for deployment on premise or on other public clouds?

What is the relationship between Kubernetes and Google Container Engine, now and in the future?

Kubernetes G+ page Why Kubernetes?technologiesconceptsChromiumcgroupscadvisorUrs Hölzlenotedgravity well How does Kubernetes relate to Docker? How does it differ from Docker Swarm? Kubernetes pods

we’ve highlightedpost Is Google committed to Kubernetes?an integral part of OpenShift version 3contributors How does Kubernetes fit into Google’s cloud strategy? Kubernetes on other clouds? On-premise?¹

Kubernetes and Google Container Engine?Google Container Engine We're Stoked!blog post While we have a lot of experience in this space, Google doesn't have all the answers. There are requirements and considerations that we don't see internally. With that in mind, please check out what we are building and get involved!Try it outfile bug reportsask for helpsend a pull request (PR)
1 The theories of supply chain diversification and vendor risk management both recommend against relying on a single supplier for any critical component of one’s business or infrastructure. This has been borne out by the experience of numerous customers over the years with large vendors of proprietary IT systems and software. Part of the appeal of Docker and Kubernetes is the degree to which they significantly lower the friction involved in moving applications between various resource pools (laptop to server, server to server, data center to data center, cloud to cloud, etc.).

VMware to make Google Cloud Platform available to its customers

Thursday, January 29, 2015

(Cross-posted on the Google for Work Blog)today’s announcementinfrastructureGoogle BigQueryGoogle Cloud DatastoreGoogle Cloud StorageGoogle Cloud DNS

vCloud Air customers will have access to Google Cloud Platform under their existing service contract and existing network interconnect with vCloud Air, and will simply pay for the Google Cloud Platform services they consume.

Google Cloud Platform services will be available under the VMware vCloud Air terms of service, and will be fully supported by VMware’s Global Support and Services (GSS) team.

Certain Google Cloud Platform services are also fully covered by VMware’s Business Associate Agreement (BAA) for US customers who require HIPAA-compliant cloud service.

Today’s announcementChromebooks and VMware ViewKubernetes

Shine Technologies Reels in Big Data Using Google’s BigQuery

Wednesday, January 28, 2015

Today’s guest blog comes from Graham Polley, Senior Consultant for Shine Technologies, a digital consultancy in Melbourne, Australia. Shine builds custom enterprise software for companies in many industries, including online retailers, telecom providers, and energy businesses. Google’s DoubleClickGoogle Cloud StorageGoogle BigQuery

bloggedContributed by Graham Polley, Senior Consultant, Shine Technologies

Understanding Cloud Pricing

Wednesday, January 28, 2015

Part 1 - Virtual Compute

An API frontend that mobile devices will contact for requests and actions. This portion will consume the majority of the compute cycles.

A static marketing and blog front end.

An application layer that will process and store images as they come in or are accessed.

And on the back end, a Cassandra cluster to store operational metadata.

The API frontend instances can respond to roughly 80 requests per second. We expect about 350 requests per second given this number of users. Therefore we should only need four regular instances for this layer.

The marketing front end shouldn’t need more than two instances for redundancy.

The application layer will need four instances for image processing and storage control.

The Cassandra cluster will need five instances with a higher memory footprint. Let’s assume for now that the workload is entirely static, and autoscaling isn’t being used (oh don’t worry, we’ll add that and more back in later).

Please note that we completed these calculations on January 12, 2015, and have included the output prices in this post. Any discrepancies are likely due to pricing or calculator changes following the publishing of this post. Here is the output of the pricing calculatorsGoogle Cloud Platform estimateAmazon Web Services estimate Reserved InstancesAmazon Web Services, no-upfront, 1 year estimate:Amazon Web Services, partial upfront, 1 year estimate:Amazon Web Services, all upfront, 1 year estimate:Partial upfront, 3 year estimate:All upfront, 3 year estimate:

Infrastructure prices will drop, either for Google (which has happened 3 times in the last 12 months, as we've reintroduced Moore’s law to the cloud), or for Amazon (which has happened 2 times in the last 12 months). For 2014, this worked out to an average of a 4.85% price reduction per month on Google Cloud Platform. Due to on-demand pricing, any reduction in prices is something you automatically receive on GCP.

Also, don’t forget, capital is expensive! Most businesses pay a ~7% per year cost of capital, which reduces the value of these up-front purchases significantly. For this example, that adds an effective $11,823.63 to the 3-year all up-front Reserved Instance price from Amazon.

By combining conservative evaluations of the basic facts of public cloud pricing dynamics (3% per month price reductions, 7% cost of capital) even 3-year all-upfront RI’s from AWS are not cost efficient compared to on-demand Sustained Use Discounts from Google Cloud Platform.
Flexibility

New instance types might make your old choices inefficient (c3 instances from AWS are substantially more cost efficient for some workloads than older m3 instances, for example).

Your software might change. For example, what if you improve the efficiency of your software to reduce your infrastructure requirements by 50%? Or what if you re-platform from Windows to Linux? (Reserved Instances require a commit on OS type) Or what if your memory needs to grow, and instances need to switch from standard to high-memory variants?

Your needs might change. For example, what if a new competitor arrives who takes ½ of your customers, which reduces the load on your infrastructure by 50%?

What if you picked everything right but the geography, and your app is suddenly popular in Asia or Europe?

Google Compute Engine estimateAmazon Web Services Partial upfront, 1 year, estimateGoogle$13,527.54Amazon$29,584.58RI marketplace

Are the RI’s you’re selling, for instance, types that are now clearly inefficient for many workloads and therefore not desirable to other customers?

Will your RI’s sell for full price, or some discount to encourage a sale?

How many buyers are there in the marketplace, and how quick will your RI’s sell, if at all?

What if you didn’t start out in the US? The RI Marketplace is only available for customers with a US bank account.

Automatic Scalingtoothbrush test100xGoogle Compute Engine estimateAmazon Web Services estimate SummarySustained Usage DiscountsStarMakerReach out

Kubernetes gathering on Wednesday February 25th

Tuesday, January 27, 2015

TimeLocationRegister here

Google Cloud Projects Named Amongst Open Source Rookies of the Year

Tuesday, January 27, 2015

Black Duck SoftwareKubernetescAdvisor

KubernetesDockerGoogle Cloud PlatformAmazon Web ServiceMicrosoft Azurestart

Google Container Engine

cAdvisorInfluxDBBigQuery

Continuously updating view of a container's resource usage

Aucor transitions its customers’ websites onto Google Cloud Platform

Monday, January 26, 2015

AucorGoogle Cloud PlatformGoogle App Enginehere

Live webinar Jan. 28: zulily turns big data into business advantage with Google Cloud Platform

Sunday, January 25, 2015

In 2015, we're introducing a monthly webinar series to take an in-depth look at diverse elements that help us solve complex business challenges in the cloud and nurture business growth. We’ll cover unique IT management and implementation strategies and the people, tools, and applications that increase impact. We're opening it up to a live online and global forum with the aim to foster collaborative learning through use cases we can all relate to and real-time Q/A sessions. Our first webinar features, zulily, a high-growth online retailer that leverages big data to provide a uniquely tailored product and customer experience to a mass market around the clock.

Best practices and implementation strategies to drive value from big data using products such as Google BigQuery and Hadoop

How zulily uses Google Cloud Platform to improve customer experience, increase sales, and increase relevance via marketing initiatives

Key leadership and technical benefits and risks to be aware of as you plan, execute and optimize your big data implementation strategy across one or multiple business units

Live Webinar: zulily turns big data into a big advantage with Google Cloud Platform

Wednesday, January 28, 2015

10:30 - 11:00 a.m. PT

Speakers: William Vambenepe, Lead Product Manager for Google Cloud Big Data Services and Sudhir Hasbe, Director Software Engineering for Data Services, BI and Big Data Analytics for zulily

View the recording here.

Secure hosting of private Docker repositories in Google Cloud Platform

Friday, January 23, 2015

we kicked offcontainerDockerrich supportGoogle Container EngineKubernetescontainer optimized VMsGoogle Compute EngineManaged VMsGoogle App EngineGoogle Container Registry

Access control: The registry service hosts your private images in Google Cloud Storage under your Google Cloud Platform project. This ensures by default that your private images can only be accessed by members of your project, enabling them to securely push and pull images through the Google Cloud SDK command line. Container host VMs can then access secured images without additional effort.

Server-side encryption: Your private images are automatically encrypted before they are written to disk.

Fast and reliable deployment: Your private images are stored in Google Cloud Storage and cached in our datacenters, ready to be deployed to Google Container Engine clusters or Google Compute Engine container optimized VMs over Google Cloud Platform’s Andromeda based network fabric.

zulilycharges for storage and networkfree trialDockerGoogle Cloud SDKdocumentationdirect feedback

Hortonworks HDP 2.2 on Google Cloud Platform

Thursday, January 22, 2015

computestoragenetworkbdutilGoogle Cloud StorageBigQueryHDP 2.2Google and Hortonworks./bdutil deploy -e platforms/hdp/ambari_env.sh

bdutil Hortonworksziptar.gzhere

What makes a container cluster?

Thursday, January 22, 2015

a post What makes a container cluster?DockerKubernetespostIt is a dynamic system that reacts to conditions and inputs in real time. Why run a container cluster?

Microservices in order to keep moving parts manageable. Having a cluster manager enables us to break down an application into smaller parts that are separately manageable and scalable. This lets us scale up our organization by having clear interfaces between smaller teams of engineers.

Self healing systems in the face of failures. The cluster manager automatically restarts work from failed machines on healthy machines.

Low friction horizontal scaling. A container cluster provides tools for horizontal scaling, such that adding more capacity can be as easy as changing a setting (replication count).

High utilization and efficiency rates. Google was able to dramatically increase resource utilization and efficiency after moving to containers.

Specialized roles for cluster and application operations teams. Developers are able to focus much more on the service they are building rather than on the underlying infrastructure that supports it. For example, the GMail operations and development teams rarely have to talk directly to the cluster operations team. Having a separation of concerns here allows (but doesn't force) operation teams to be more widely leveraged.

you Ingredient 1: Dynamic container placementThe cluster management system should decide where to actually run your workload¹concept known as a pod

Ingredient 2: Thinking in setslabels and replication controllerslabels

Figure 1 - Filtering pods using labels

Scalingreplication controllerobject replication_controller { property num_replicas property template property label_selector runReplicationController(num_desired_pods, template, label_selector) { loop forever { num_pods = length(query(label_selector)) if num_pods > num_desired_pods { kill_pods(num_pods - num_desired_pods) } else if num_pods < num_desired_pods { create_pods(template, num_desired_pods - num_pods) } } } } desired state

Figure 2 - The Replication Controller enforces desired state

Ingredient 3: Connecting within a clusterservicesgraphnaming resolution systemshard assignment¹service proxyKubernetes roadmap Voila!hereroadmap to v1.0Try it outfile bug reportsask for helpsend a pull request (PR) 1 This is the classic knapsack problem which is NP-hard in the general case. 2 The "Watch API pattern" is a way to deliver async events from a service. It is common on lock server systems (zookeeper, etc.) that are derived from the original Google Chubby paper. The client essentially reaches out and "hangs" a request until there are changes. This is usually coupled with version numbers so that the client stay current on any changes.

Aerospike demonstrates RAM-like performance with Local SSDs

Wednesday, January 21, 2015

Today’s post is by Sunil Sayyaparaju, Director of Product and Technology at Aerospike, the open source, flash-optimized, in-memory NoSQL database.Click to Deploy of AerospikeSSDsClick to Deploy Aerospike documentationhard disksGoogle Compute EngineAerospike Hits 1 Million Writes Per Second With Just 50 NodesAerospike certifies Local SSDs on Google Compute EngineACT

Reads small objects (default 1500 bytes) using multiple threads (default 16).

Writes large blocks (default 128KB) to simulate a buffered write mechanism in DBMS.

Reads large blocks (default 128KB) to simulate typical background processing.

recommended SSDsAerospike Certification Tools (ACT) for SSDs Setup

Machine: n1-standard-4 with 1 Local SSD provisioned (4 vCPU, 15 GB memory)

SSD size: 375GB

Read/Write size: 1500 bytes (all reads hit disk, but writes are buffered)

Large block read size: 128KB

Load: 6000 reads/s, 3000 writes/s, 71 large block reads per sec

ACT results show that 95% of Local SSD reads complete in under 1 ms

only 5% take > 1 ms

only 3% take > 2 ms

only 1% take > 4 ms

a negligible number take > 8 ms

Comparing Aerospike performance on Local SSDs vs. RAMearlier post

15x price advantage in storage costs with Local SSDs vs RAM

Achieved roughly the same write throughput using Local SSDs compared to RAM

Achieved half the read throughput using Local SSDs compared to RAM

Aerospike delivers 15x storage cost advantage with Local SSDs vs. RAM

Aerospike demonstrates RAM-like Latencies for Local SSDs vs. RAM

Aerospike demonstrates RAM-like Throughput for Writes on Local SSDs vs. RAM

1.0x write throughput (while doing 100% writes) using Local SSDs compared to RAM. Aerospike is able to achieve the same write throughput because of buffered writes, where writes are first written in memory and subsequently flushed to disk.

0.5x read throughput (while doing 100% reads) using Local SSDs compared to RAM. Aerospike is able to achieve such high performance using Local SSDs because it stores indexes in RAM and they point to data on disk. The disk is accessed exactly once per read operation, resulting in highly predictable performance.

SummaryDeploy an Aerospike clusterfree trialAerospike is the registered trademark of Aerospike, Inc.. All other trademarks cited here are the property of their respective owners.