Google Cloud Platform Blog
Product updates, customer stories, and tips and tricks on Google Cloud Platform
DataStax Enterprise feels right at home in Google Compute Engine
Monday, December 9, 2013
Today’s guest post comes from Martin Van Ryswyk, Vice President of Engineering at DataStax.
The cloud promises many things for database users: transparent elasticity and scalability, high availability, lower cost and much more. As customers evaluate their cloud options -- from porting a legacy RDBMS to the cloud to solutions born in the cloud -- we would like to share our experience from running more than 300+ customers’ live systems in a cloud-native way.
At
DataStax
, we drive
Apache CassandraTM
. Designed for the cloud, Cassandra is a massively scalable, open-source NoSQL database designed from the ground up to excel at serving modern online applications. Cassandra easily manages the distribution of data across multiple data centers and cloud availability zones, can add capacity to live systems without impacting your application’s availability and provides extremely fast read/write operations.
One of the advantages of Google Compute Engine is its use of Persistent Disks. When an instance is terminated, the data is still persisted and can be re-connected to a new instance. This gives great flexibility to Cassandra users. For example, you can upgrade a node to a higher CPU/Memory limit without re-replicating the data or recover from the loss of a node without having to stream all of the data from other nodes in the cluster.
DataStax and Google engineers recently collaborated on running DataStax Enterprise (DSE) 3.2 on Google Compute Engine. The goal was to understand the performance customers can expect on Google’s Persistent Disk, which recently
announced
new performance and pricing tiers.
DataStax Enterprise
supports a purely cloud-native solution and can span on-premise and cloud instances for customers wanting a hybrid solution.
Tests and results of DataStax Enterprise on Google Compute Engine
We were very interested to see how consistent the latency would be on Persistent Disks, as it represents a highly consistent storage with predictable and highly competitive pricing. Our tests started at the operational level and then moved into testing the robustness of our cluster (Cassandra ring) during failure and I/O under heavy load. All tests were run by DataStax, with Google providing configuration guidance. The resulting configuration file and methodology can be found
here
.
The key to consistent latency in Google Compute Engine is sizing one’s cluster so that each node stays within the throughput limits. Taking that guidance with our recommended configuration, we believe the results are readily replicable and applicable to your application. We tested three scenarios, all with positive outcomes:
Operational stability
of 100 nodes spread across two physical
zones
.
Objective: longevity test at 6,000 record per second (60 record/sec/node) for 72 hours.
Results: we saw trouble-free operation, where data tests completed without issue. Replication completed, where data streamed effortlessly across dual zones.
Robustness during a reboot/failure
through reconnecting Persistent Disks to an instance.
Objective: measure impact of terminating a node and re-connecting its disk to a new node.
Results: new nodes joined the Cassandra ring without having to be
repaired
and with no data loss (no streaming required). We did need to manage IP address changes for the new node.
Push the limits of disk performance
for a three node cluster.
Objective: measure response under load when approaching the disk throughput limit.
Results: Our tests showed a good distribution of latency during the tests, and 90% of the I/O write times were less than 8ms (see figures below depicting the medium latency and latency distribution). These results were while our load did not exceed the published throughput (I/O) thresholds (see
caps
for thresholds).
What’s next
We find Google Compute Engine and the implementation of Persistent Disks to be very promising as a platform for Cassandra. The next step in our partnership will be more extensive performance benchmarking of DataStax Enterprise. We look forward to publishing the results in a future blog post.
Figures for reference
The graph below shows median latency, a figure of merit indicating how much time it takes to satisfy a write request (in milliseconds):
The figure below depicts the distribution of latencies (ms) for write latencies. As noted above, 90% of write latencies were below 8ms, indicating the consistency of performance. The tight distribution within 1-4ms speaks to the predictability of performance.
-Contributed by Martin Van Ryswyk, Vice President of Engineering, DataStax
Why We Deployed Zencoder on Google Cloud Platform
Monday, December 9, 2013
Today's guest post comes from Jon Dahl, VP of Encoding Services at Brightcove.
Brightcove’s Zencoder transcodes millions of video and audio files each month, all in the cloud. We've worked hard to establish the Zencoder service as the cloud encoding performance leader and are constantly investigating ways to optimize the application and to architect the service around the best infrastructure available. So, we are very excited to announce the Beta availability of the
Zencoder cloud encoding service
running on Google Compute Engine.
The Zencoder service offers developers APIs for the fastest and most reliable live and file video encoding in the cloud. Thousands of customers, such as AOL, PBS, Khan Academy and the Wall Street Journal, have built their media workflows around our service.
Starting today, developers building applications and video workflows on the Google Cloud can use the Zencoder API to transcode video for a wide variety of Internet-connected devices. Users will be able to programatically select a
Cloud Platform region
for their transcoding in addition to previously supported regions. Initially, transcoding will be limited to video on-demand jobs, but we will expand to include live transcoding in the future.
Based on our usage and testing to date, there are a few specific things about Cloud Platform that we're most excited about, and as we scale up usage, we hope to release more metrics:
Multi-cloud
At a high level, it's fundamentally important to have a multi-cloud approach to our infrastructure. Having a diversity of cloud resources makes the service more reliable and resilient.
Fast launch times
We built the Zencoder service from the ground up to scale dynamically based on demand. Our goal is to obviate the notion of the queue, or to effectively have infinite lanes in which jobs can be slotted. When a server isn't available for a job, we have to spin one up. Compute Engine instances boot extremely fast, and the faster we spin up instances, the better the experience for our customers.
Consistent performance
It's one thing to be really fast, but Compute Engine boot times are consistently fast. For example, from our preliminary testing, we found that if you run 100 instances, each instance has the same characteristics.
Fast I/O
With a service running at the scale of Brightcove’s Zencoder, even the smallest performance advantages in underlying infrastructure help. Video encoding is fundamentally a CPU-bound process, but in aggregate, improvements in disk I/O make a difference. Video encoding jobs typically consist of a single input file going to multiple output renditions. Improvements in disk read/write time will reduce latency and decrease transcoding time.
Intelligent caching
Content providers should use Google Cloud Storage in conjunction with the Zencoder service on Compute Engine. Storing and processing content in the same cloud infrastructure ensures fast, reliable transfer. Additionally, Cloud Storage improves transfer performance by optimizing data placement and caching across its global infrastructure.
Super network
We're extremely impressed with the network consistency and speed, which are particularly important for content ingest and egress, as well as latency-sensitive service video functions such as live streaming.
The GA release of Google Compute Engine is big news
for those of us with our heads and services in the cloud. Google has released storage and compute services that raise the bar (and maybe set the standard) for performance and reliability in some key areas. We're excited to see what types of video apps developers build that take advantage of the Zencoder service and the unique characteristics of Google Cloud Platform.
-Contributed by Jon Dahl, VP of Encoding Services, Brightcove
Scalr and Google Compute Engine
Sunday, December 8, 2013
Today's guest post is from Sebastian Stadil, CEO of Scalr. The company providies a web-based control panel for cloud infrastructure that serves as an interface between end users and the multiple cloud platforms that they use. In this post, Sebastian discusses benchmarks they conducted to analyze Compute Engine performance.
At Scalr, we build a web-based control panel for cloud infrastructure, which serves as an interface between end users and the multiple cloud platforms that they use. Engineers use Scalr to achieve significant productivity gains, and IT departments use it to drive and control cloud adoption.
One of our customers —
grandcentrix
— used
Scalr
with Google Compute Engine a few months back. They were building the backend of the companion mobile application for the
Eurovision song contest
using both Compute Engine and Scalr. Our experience was documented on the Google Cloud Platform blog.
No random hiccups with Google. Only high and stable performance.
In a few words: this was the first time Eurovison had a companion app, so they had no idea how much traffic they’d have. Fortunately, our load tests had shown that Compute Engine was a predictable high performer with fast provisioning times, so all we had to do was ensure that the application architecture would scale horizontally.
Eurovision was a success, and we’re looking forward to taking on such a challenge again. Why? Because we feel very comfortable using Google Compute Engine. It just doesn’t surprise you, and delivers extremely consistent performance.
We’ve recently conducted performance benchmarks on persistent volumes across multiple cloud providers. For volumes, performance is only part of the story. Stability matters a lot, too. What good is a high performing volume if it fails to perform 1/10th of the time? Not much!
Using Google Compute Engine, every single volume performs the same, every hour of every day. If their throughput is sufficient when you run your tests, you can know for sure that Google volumes won’t let you down when you need them.
By the numbers: the benchmarks
Below, you’ll find graphs that compare the performance
dispersion
for IOPS, bandwidth, and latency for Google Compute Engine and EC2 volumes. This is basically a measurement of how consistent disk performance has been over hundreds of 10-minute disk-performance measurements.
What you’ll see is that Google volumes offer significantly more consistent performance than their AWS counterparts, including PIOPs volumes!
Here are the graphs. Lower dispersion means more consistent performance.
Bandwidth Dispersion
Latency Dispersion
IOPS Dispersion
Note: we’re still in the process of adding benchmarks on sequential workloads for PIOPs volumes, but they have been benchmarked for random workloads.
Are you trying out Google Compute Engine?
Google Compute Engine, EC2, and Rackspace aren’t API-compatible, so if you intend to start using Google’s Cloud for your business today, you’ll probably need to rewrite quite a few integrations.
If that’s the case, you might want to look at the Scalr Cloud Management Platform. Using Scalr lets you design infrastructure and policies once, and use them with any cloud platform.
We actually used Scalr to run the benchmarks we presented here. If you’re interested, you could
watch this talk from the OpenStack Summit
, where we presented how we did it.
You can of course
learn more about Scalr on our website, or request a POC
.
-Contributed by Sebastian Stadil, CEO and Founder at Scalr
Cloud9 IDE on Google Compute Engine
Saturday, December 7, 2013
Today's guest post comes from Ivar Pruijin, Product Manager at Cloud9 - a popular cloud-based IDE. In this post, Ivar discusses their work with Google Compute Engine.
Cloud9 IDE
moves your entire development flow onto the cloud by offering an online development environment for web and mobile applications. With Cloud9 IDE developers write code, debug, and deploy their applications, and easily collaborate with others - all right in the cloud. We’ve worked hard to ensure that our online development environment is easy, powerful, and a thrill to use.
Google Compute Engine
recently got us very excited about the possibilities for Cloud9 IDE. So much so, that we built support for Compute Engine into the backend of the soon-to-be-released major update of Cloud9 IDE! We’ve seen major improvements in speed, provisioning and the ability to automate deployments and management of our infrastructure. In this article we’ll talk about those experiences and what benefits Compute Engine offers to a complex web application like ours.
Speed
When building a hosted IDE, latency is a big concern. You want files to open instantly, step debug through your code without delays, and interact with the terminal as if it all was running locally. Every millisecond of delay we reduce is an immediate usability improvement. Due to the global reach of Google’s fiber network and its huge ecosystem of peering partners, we’ve been able to achieve major speed improvements!
Let’s look at this in some more detail. We’ve optimized our architecture to require just one hop between the hosted workspace and the browser running Cloud9. This intermediate layer is our virtual file system server (VFS). VFS connects to the hosted workspaces and provides a REST & WebSocket interface to the client running in the browser. Initially we expected the lowest latency when placing the VFS server close to the hosted workspaces and possibly in the same data center. Surprisingly it turned out that placing the VFS server as close as possible to the user resulted in way smaller latencies. This is where Google’s advantage in networking really comes into play. A round trip from our Amsterdam office to a hosted workspace at the US east coast is about 50% faster when connecting through a VFS server in a Google European data center!
In order to select the closest VFS server we use latency based DNS routing. This alone perceivably improves the responsiveness of Cloud9 in general. Users who use SSH workspaces, live outside of the US or are travelling a lot will especially feel the difference.
Fast & easy Provisioning & Infrastructure Management
There are other things we like about Google Compute Engine. The management interface is great:
It is very responsive and offers metrics on the running VM, like live CPU metrics. Everything you can do in the web console can be performed using Google’s REST interface and the very powerful command line client, too. Throughout the UI you’ll find hints of how to perform the same action using REST or the command line. This allowed us to completely script the deployment of new servers and the management of our infrastructure.
The integrated login system is another strong point of Google Compute Engine (and for Google Cloud Platform as a whole). It allows us to assign different set of permissions to every user being part of the c9.io domain. We can sign in to Google Compute Engine using our regular email/password credentials (we already use Google Apps), and a two factor authentication policy for strong security is available as well.
What’s next?
Our upcoming release is scheduled to go GA in Q1 of 2014. The backend of this new version supports Google Compute Engine next to the infrastructure that we already have and will allow our users to get a workspace in the closest region near them. So far we couldn't be happier. The performance of the VMs and virtual disks are great, the pricing is competitive and Google is really helpful and responsive. As we roll out the new version of Cloud9 IDE we’ll continue working on our Compute Engine support.
-Contributed by Ivar Pruijn, Product Manager, Cloud9 IDE
Fishlabs architects upcoming game with Compute Engine and RightScale
Saturday, December 7, 2013
Today’s guest blog comes from Alex Pop, a cloud integration engineer at RightScale. RightScale is a Google Cloud Platform partner that enables leading enterprises to accelerate delivery of cloud-based applications while optimizing cloud usage to reduce risk and costs. With RightScale, IT organizations can deliver instant access to a portfolio of public, private, and hybrid cloud services across business units and development teams while maintaining enterprise control. Since 2007, leading enterprises including Pearson International, Intercontinental Hotels Group, and PBS have launched millions of servers through RightScale.
Fishlabs is a Hamburg-based developer and publisher of premium games for smartphones and tablets. They are building their next game, Galaxy on Fire - Alliances with a new approach to their cloud strategy. Fishlabs infrastructure architects are working with Google and
RightScale
to ensure maximum performance and uptime, key factors include:
Single Reference deployment that can be redeployed across iOS and Android platforms
Scalable Database layer, scaling the DB layer vertically and horizontally with slaves.
Compute Engine performance and stability, particularly in a high memory gaming scenario
Competitive pricing across compute and networking
Deep experience and knowledge of the gaming space
For this project, Fishlabs augmented their cloud strategy with the addition of Google to meet compute and DB needs and RightScale to provide consistent deployment across platforms. “With Google and RightScale we were able to provide Fishlabs a new level of performance and cloud deployment consistency for this new title” said Bruno Ciscato, Cloud Solutions Engineer at RightScale
“We’ve chosen to work with the Google Compute Engine and RightScale teams to meet our DB scaling and repeatable deployment requirements across platforms. Using GCE and RightScale, the team had a fully functional deployment up and running in a week, we’re enthusiastic about measuring results and performance as we head from beta to launch,” saod Uli Sesselmann, IT Supervisor at Fishlabs.
-Contributed by Alex Pop, Integration Engineer, RightScale
An ode to Sharkon
Friday, December 6, 2013
In addition to
introducing Compute Engine in GA this week
, we launched a
new website
for Google Cloud Platform and a new set of Cloud Platform logos:
Now, none of this changes anything for you. New logos aren’t going to help you serve more requests-per-second (good news,
you can already top 1 million
). And they aren’t going to allow you to scale your caching capacity indefinitely or reduce your datastore costs (good news,
dedicated memcache in App Engine already does that
). And they aren’t going to provide you with an analytics tool that lets you query terabytes of data in seconds (
that’s what Big Query is for
).
But, they do allow us to reflect on what we have been able to do over the past 5 years with Google Cloud Platform.
In April 2008, when we launched Google App Engine, we introduced the first modern Platform as a Service. In fact, the term
barely existed at the time
. And, given that we were launching a new product, we needed a new icon. The marketing team decided that what App Engine needed was an engine. And because it was Google, the engine should be in the shape of a ‘G’. So they did what marketers do, and made a bunch of versions of this idea:
There was only one problem: none of the people who had built App Engine actually liked the logo. It didn’t represent the next-generation technology that they were building. In fact, it looked like a combination of a printing press (which
Gutenberg first started tinkering with in 1436
) and an internal combustion engine (conceived of by
Huygens in 1680
and first built by
Rivaz in 1807
). It didn’t reflect a product that allowed you to deploy an application with one-click, scale it effortlessly to serve millions of users, and fundamentally change how web applications are developed.
So, three of App Engine’s early engineers went back to the whiteboard. Literally. Rafe Kaplan, Alon Levi and Brett Slatkin thought about what the product should look like. And it wasn’t a printing press. And it wasn’t an internal combustion engine. And it wasn’t shaped like a G. These are some of the concepts that Brett sketched on the whiteboard in Building 44 at the Googleplex in Mountain View:
The team was much happier with these; they felt like they were inspiring. And these whiteboard sketches were sent to a graphic designer who came back with three concepts for a final logo:
Well, no one really loved these either. In the words of one member of the App Engine team, “A looks like a fan, B looks like a washing machine, and C looks like a washing machine with fins.” So, to cut a long story short, they did some more revisions, and eventually Google designer Micheal Lopez landed on a logo that everyone (well, most people) loved:
It evoked both the power as well as simplicity of App Engine. To Rafe Kaplan, the new icon looked like a shark, so he called it ‘Sharkon’ - a name that quickly spread among the team.
And, for the past 5 years, Sharkon has been the face of App Engine. It’s found it’s way into many forms - whether made into a plush doll, knit into yarn, painted using acrylics by artist
Nan Washare
, done in a single brushstroke of calligraphy, poured into latte art, or made into a punch-card reader when we announced on April Fools Day in 2009, that
App Engine would be supporting Fortran
:
Over these 5 years, a lot else has happened. App Engine has grown to support new runtimes, including
Python
,
Java
,
Go
and
PHP
. And, we’ve introduced a host of other cloud computing products, including
Compute Engine
,
Cloud Storage
,
BigQuery
and others. Together, this family makes it easy for you to take advantage of the scale, speed, and consistency of Google’s infrastructure. And, these services work great together - so that you can truly take advantage of an integrated Cloud Platform.
The new App Engine logo is designed to fit in alongside the rest of the Cloud Platform family, while still paying homage to Sharkon. These new logos represent the toolkit in your garage. They’re the nuts and bolts with which you can create just about anything. So, pick up any one of them (or all of them) and start building.
Oh, and watch this space. We’ve got more exciting announcements coming up - the kind that go deeper than a logo refresh.
-Posted by Benjamin Bechtolsheim, Product Marketing Manager
SaltStack for Google Compute Engine
Thursday, December 5, 2013
SaltStack
was built for the performance, reliability, security and scale delivered by
Google Compute Engine
. Needless to say we are very excited to support the launch and general availability of Google Compute Engine.
SaltStack is a systems automation and configuration management framework written in Python and built from the start to be fast, scalable and open while being simple to use yet flexible enough to meet the challenge of any data center automation task. In just two years SaltStack became the eighth biggest and most active
open source project on GitHub
at the end of 2012, and thousands of the largest infrastructures in the world are already heavily Salted.
Together SaltStack and Google are working to help developers get code into production faster and to help engineers command and control compute resources quickly and easily. If you are new to SaltStack or Salt Cloud, let’s get on the same page. There is a good chance you are using SaltStack and Google Compute Engine together with plans to build really big, large-scale things that will most likely require:
Rapid access to substantial compute resources;
Cost-aware and efficient compute consumption;
High-quality, hardened and secure VMs;
Flexibility to do things in unique and fluid ways;
Software that can keep up with the speed and scale of cloud;
Easy, open and portable infrastructure control through a data-centric approach to automation;
Autonomic, event-driven compute resources that autoscale to demand or lack thereof.
SaltStack can be first used to deploy Google Compute Engine resources and then to provide ongoing management and automation of Compute Engine VMs and configuration management of the application stacks running in that environment.
So let’s dive a little deeper to see what else is possible now within a Salted Compute Engine environment. To start managing Compute Engine with SaltStack, here is a handy
“Getting Started with Google Compute Engine” tutorial
to help. Dependencies simply require:
Source install of
Libcloud
(or greater than 0.14.0-beta3 when available)
A Google Cloud Platform account with Google Compute Engine enabled;
A registered Google service account for authorization;
SaltStack
.
Configure Salt Cloud next:
Once you have a Salted GCE environment it is time to utilize SaltStack to deploy and control virtual machines in milliseconds while managing application configurations and doing continuous code integration and deployment. SaltStack users often command and control Google Compute Engine resources alongside other cloud environments and infrastructure. Join us at
SaltConf
in January to see all of this in action.
Google is the epitome of scale, while SaltStack was designed from the start to deliver systems automation for Web scale, cloud and big enterprise IT. So let’s do big things together.
-Contributed by Thomas Hatch, CTO and co-founder, SaltStack
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Firebase Blog
Apigee Blog
Popular Posts
dotCloud provides faster, more reliable PaaS with Google Cloud Platform
Introducing Google Cloud Storage Nearline: (near)online data at an offline price
World's largest event dataset now publicly available in BigQuery
A look inside Google’s Data Center Networks
Google Compute Engine is now Generally Available with expanded OS support, transparent maintenance, and lower prices
Labels
Announcements
193
Big Data & Machine Learning
134
Compute
271
Containers & Kubernetes
92
CRE
27
Customers
107
Developer Tools & Insights
151
Events
38
Infrastructure
44
Management Tools
87
Networking
43
Open
1
Open Source
135
Partners
102
Pricing
28
Security & Identity
85
Solutions
24
Stackdriver
24
Storage & Databases
164
Weekly Roundups
20
Feed
Subscribe by email
Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.
Learn More
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow
.png)
.png)
Follow