Google Cloud Platform Blog
Product updates, customer stories, and tips and tricks on Google Cloud Platform
One click to deploy a RabbitMQ cluster handling over 1 million msg/sec
Monday, June 23, 2014
If you develop scalable applications, you often want to use a messaging system, but you may be concerned that setting up such a system is time consuming and its throughput limits are not sufficient for your growing needs. But what if the whole setup required just a few minutes and benchmarks showed that the system was capable of processing over one million messages per second? This is exactly what we are announcing today with Pivotal.
When we talked about high-throughput, low-latency messaging scenarios with our customers, they were particularly interested in
RabbitMQ
, a popular open source messaging system that serves a variety of use cases.
We recently made it super easy for our users to deploy a
dedicated RabbitMQ cluster
on Google Compute Engine. All it takes is one click after providing some basic information, such as the size of the cluster and RabbitMQ username / password.
In a matter of a few minutes, you can get a RabbitMQ cluster deployed and configured. You will also get two load balancers set up for your cluster. As a next step you can securely access RabbitMQ web management console and start developing your app that uses RabbitMQ. Even if you need a very large 64-node cluster, the deployment usually takes less than 10 minutes.
In fact, the RabbitMQ cluster is so easy to deploy and configure that we’ve received feedback that for some developers the experience may be similar to having their own "Rabbit service" consisting of multiple nodes.
We also wanted to demonstrate that a RabbitMQ cluster running on Google Compute Engine is a good fit for high-throughput and low-latency scenarios. To do so, we ran a series of benchmarks in various configurations. These test clusters were able to sustain a throughput of over 1 million messages published and consumed per second (a sustained combined ingress/egress of over two million messages per second).
To put this throughput in context, one million messages per second translates to 86 billion messages per day. U.S. text messages reached
6 billion per day
in 2012. Apple processes about
40 billion iMessages per day
, and WhatsApp recently hit a new daily record in December when it sent 20 billion daily messages.
Google and Pivotal Benchmarks
We then invited engineers from Pivotal, the company behind RabbitMQ, to validate and endorse our results.
For our joint tests on Google Compute Engine we provisioned a cluster of 32 virtual machines with 8 vCPUs and 30GB of RAM each, and deployed RabbitMQ on Debian.
To generate load we used RabbitMQ
PerfTest
client tool running on a set of GCE virtual machines separate from the ones housing the RabbitMQ cluster nodes. As per
RabbitMQ clustering recommendations
the clients were configured to target the IP address of a GCE load balancer that was automatically created through click to deploy. This is a more flexible approach than having the clients know the IP addresses or host names of every node in the Rabbit cluster.
After the traffic generating clients warmed up, we reached a steady state as shown in RabbitMQ’s web management console: over 1.3 million messages published and consumed per second.
Under multiple hours of such load, the RabbitMQ cluster and its underlying GCE VMs remained stable, with average throughput staying at approximately the same levels as shown in the screenshot.
Conclusion
We are excited to offer our customers an easy way to deploy RabbitMQ. We are proud that both Google and
Pivotal benchmarks
validate that Compute Engine is an excellent choice for running RabbitMQ in a public cloud
To deploy your own dedicated RabbitMQ cluster, go to
Google Developer Console
and navigate to an existing project, or create a new one, and you will have an option to deploy RabbitMQ from the project dashboard:
You can also find a link at the bottom of the project landing page:
There is no extra charge for using click to deploy - you will be only billed for the underlying Compute Engine resources, so why not deploy your cluster today and tell us what you think.
-Posted by Grzegorz Gogolowicz, Cloud Solutions Architect
P.S. If you don’t want to have your own dedicated RabbitMQ cluster deployed as described in this post, and rather would like to use a managed RabbitMQ, you may want to check out
CloudAMQP offering
that runs on Google Compute Engine.
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Firebase Blog
Apigee Blog
Popular Posts
Understanding Cloud Pricing
World's largest event dataset now publicly available in BigQuery
A look inside Google’s Data Center Networks
New in Google Cloud Storage: auto-delete, regional buckets and faster uploads
Enter the Andromeda zone - Google Cloud Platform’s latest networking stack
Labels
Announcements
193
Big Data & Machine Learning
134
Compute
271
Containers & Kubernetes
92
CRE
27
Customers
107
Developer Tools & Insights
151
Events
38
Infrastructure
44
Management Tools
87
Networking
43
Open
1
Open Source
135
Partners
102
Pricing
28
Security & Identity
85
Solutions
24
Stackdriver
24
Storage & Databases
164
Weekly Roundups
20
Feed
Subscribe by email
Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.
Learn More
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow
Follow