Google Cloud Platform Blog

Scaling SongPop to 60 million users with App Engine and Google Cloud Storage

Monday, February 4, 2013

Continuing our Developer Insights series, today’s guest bloggers are Olivier Michon, CTO, and Alexis Hanicotte, software engineer, from Fresh Planet, maker of the popular mobile application SongPop. SongPop is a social application where players compete to be the fastest to guess the name of a song or artist.
SongPop is a social mobile app where players compete on who will be the fastest to guess a songs’ artist or title. It is a huge success for our very small team of just 6 engineers. We now have more than 60 million users, were the number 5 most downloaded iOS game of 2012, and went from 0 to more than 10k queries/second on our servers in less than 6 months. This has been made possible in large part because we run on Google App Engine.
App Engine allows us to quickly prototype, iterate and release our games. We’ve been accumulating experience with the platform since 2009, but really saw the power of autoscaling once SongPop became a hit. Our experience scaling with App Engine

SongPop was released in May 2012. During our growth to our first 100,000 daily active users (DAU), our App Engine backend scaled smoothly. This allowed us to spend our time making actual improvements to the game experience, while our user growth continued at a rapid pace.

We opened a Premier Account with App Engine around the time when we reached 100,000 DAU which gave us access to live customer support. It came in handy when we encountered two downtimes, with one lasting just 10 minutes and the other for 1 hour. But we were able to promptly reach Premier Support and had Google engineers investigate these issues with us.

Once we reached 500,000 DAU, we applied a variety of optimization ideas to reduce latency. For example, we used to have user data spread over many models, but we combined them into a single entry in the Datastore to reduce read operations. We also often needed the list of a user’s opponents, so instead of querying every time, we cached this result using Memcache. It only took one engineer and just 4 days of work to reduce the latency with these optimizations.

As we reached the milestone of 1 million DAU, some Datastore queries (used to find random opponents in the game) showed high latencies and a high rate of timeouts. We had to enforce deadlines, implement better fallbacks and identified, with the help of Premier Support, that degraded performance came from the fact that our queries were relying on many different indexed properties. So the solution was simple - either add a composite index with all the properties we needed or combine into a single one.

Using App Engine + Google Cloud Storage

For each game session, our users need to download song samples in order to play. It is critical that this data gets delivered fast and reliably wherever the user is located. We chose to use Google Cloud Storage for this use case. It has proved to provide high performance content delivery, as we have been able to serve 17 terabytes/day of songs and images worldwide.

In addition to its reliability, Cloud Storage is great because of its integration with App Engine. We can easily read and write files from our application to Cloud Storage using the same syntax as we would use to write local files (using Python). We found it intuitive and convenient because you do not have to manage opaque keys to retrieve your files (just use the path you specified), and you can browse your files through a directory-like structure. Cloud Storage also allows you to manage access rights, can be used with Google BigQuery, and it is priced affordably compared to other solutions we considered.

Architecture diagram of SongPop

Long live App Engine!
When we speak to other game developers, we always recommend that they use App Engine. We’ve used other services such as EC2 from Amazon Web Services for other games before, but we’ve found App Engine to be a better service for our needs. We don’t want to spend time setting up servers and load balancing, when we could instead use that time to build great games and let our service provider handle the infrastructure for us.

When we compare the development of SongPop to stories of other apps, we’re thankful that App Engine allowed us to have only one engineer working full-time on the backend portion of our app. Even better, he was able to do additional work on adding new features to the game instead of solely focusing on infrastructure issues. With App Engine, scaling our game was easy. Others things we want to share

Do not worry if documented resources limits and rates look too small, they are just there to make sure one app does not abuse any resource, but they can scale. Most of our limits have been increased by 18-fold! We had days where we made 230,000,000 UrlFetch API calls, for instance.
The Location headers is a really great feature, because it is easily accessible for a wide variety of use cases, such as selecting users’ opponents or building their game profile.

Contributed by Olivier Michon and Alexis Hanicotte, FreshPlanet

Posted by Zafir Khan, Product Marketing Manager, Google App Engine

Find sample code and more for Google Cloud Platform, now on GitHub

Tuesday, January 22, 2013

Today's post introduces the Google Cloud Platform organization on GitHub, which includes samples and tools for App Engine developers.
You can now find Google Cloud Platform on GitHub! The GitHub organization for Google Cloud Platform is your destination for samples and tools relating to App Engine, BigQuery, Compute Engine, Cloud SQL, and Cloud Storage. Most existing open source tools will be migrated to the organization over time. You can quickly get your app running by forking any of our repositories and diving into the code.

Currently, the GitHub organization for Google Cloud Platform has 36 public repositories, some of which are currently undergoing their initial code reviews, which you can follow on the repo. The Google Cloud Platform Developer Relations Team will be using GitHub to maintain our starter projects, which show how to get started with our APIs using different stacks. We will continue to add repositories that illustrate solutions, such as the classic guest book app on Google App Engine. For good measure, you will also see some tools that will make your life easier, such as an OAuth 2.0 helper.

From getting started with Python on Google Cloud Storage to monitoring your Google Compute Engine instances with App Engine, our GitHub organization is home to it all.

Trick of the trade: to find samples relating to a specific platform, try filtering on the name in the “Find a Repository” text field.

We set up this organization not only to give you an easy way to find and follow our samples, but also to give you a way to get involved and start hacking alongside. We’ll be monitoring our repositories for any reported issues as well as for pull requests. If you’re interested in seeing what a code review looks like for Google’s open source code, you can follow along with the discussion happening right on the commits.

Let us know your suggestions for samples. We look forward to seeing what you create!
- Posted by Julia Ferraioli, Developer Advocate

Calling all coders: Hardcode, the secure coding contest for App Engine

Wednesday, January 16, 2013

Originally posted on the Google Online Security Blog

Protecting user security and privacy is a huge responsibility, and software security is a big part of it. Learning about new ways to break applications is important, but learning preventative skills to use when building software, like secure design and coding practices, is just as critical. To help promote secure development habits, Google is once again partnering with the organizers of SyScan to host Hardcode, a secure coding contest on the Google App Engine platform.

Participation will be open to teams of up to 5 full-time students (undergraduate or high school). Contestants will be asked to develop open source applications that meet a set of functional and security requirements. The contest will consist of two rounds: a qualifying round over the Internet, with broad participation from any team of students, and a final round, to be held during SyScan on April 23-25 in Singapore.

During the qualifying round, teams will be tasked with building an application and describing its security design. A panel of judges will assess all submitted applications and select the top five to compete in the final round.

At SyScan, the five finalist teams will be asked to develop a set of additional features and fix any security flaws identified in their qualifying submission. After two more days of hacking, a panel of judges will rank the projects and select a grand prize winning team that will receive $20,000 Singapore dollars. The 2nd-5th place finalist teams will receive $15,000, $10,000, $5,000, and $5,000 Singapore dollars, respectively.

Hardcode begins on Friday, January 18th. Full contest details will be be announced via our mailing list, so subscribe there for more information!

Posted by Parisa Tabriz, Security Team

Mobile voucher sales terminal in Africa powered by App Engine

Thursday, December 20, 2012

Today’s guest blogger is Dale Humby, CTO of Nomanini. Nomanini is a startup based in South Africa which provides a platform to sell prepaid products such as airtime, electricity and insurance electronically. In this post, Dale explains how the backend systems of their flagship device, Lula, run on App Engine.
Introduction
In rural markets, it is often difficult to distribute physical vouchers, which can be used to provide access to services such as electricity, insurance, and airtime for mobile phones. Nomanini enables entrepreneurs in Africa to earn income selling prepaid vouchers in their local communities. We do this by distributing a portable, user-friendly voucher sales terminal, known as the Lula, which can be used on-the-go by people ranging from taxi drivers to street vendors.

The Lula, Nomanini's portable voucher sales terminal

Running Lula on App Engine

Nomanini uses Google App Engine to support the backend system for our network of point of sale terminals.

Terminals in the field connect to our App Engine application through the GSM mobile network. The terminals synchronize when a connection becomes available, allowing sales to be processed even when devices are offline.

The devices make an https post to a URL endpoint. Any data uploaded by the device is queued for processing as multiple tasks, and the App Engine application sends information back to the device within the returned body. Often the only responsibility a URL endpoint has is to create a task. If work can be broken into discrete areas we fan out to other tasks which run in parallel on separate App Engine instances. By controlling how many concurrent tasks are run in each queue, we are able to prioritize specific parts of our application ensuring the best quality of service for our customers.

By utilizing cross entity group transactions, transactional tasks and appropriate key naems for entities in the Datastore, we have been able to build an extremely resilient application capable of processing data.

Services we use

High Replication Datastore, NDB and Memcache

Extensive use of push Task Queues

Blobstore for storing file records

Logs API, Mail API and XMPP for reporting and real-time notification of issues to our development team

URL Fetch API to link to third-party API’s such as sending SMS’s and integration to mobile networks for airtime voucher purchase

Google Cloud Storage and Google BigQuery for mining our data for interesting trends

ProdEagle (another application built on top of Google App Engine) for real time metrics visualization

Benefits of building on App Engine
Saves time
App Engine’s High Replication Datastore gives us the peace of mind that our data will be available and accurate to a degree that we couldn’t easily replicate ourselves. As a start up, capital and time are in short supply. With App Engine, we can focus on building our unique application rather than worrying about infrastructure. App Engine Task Queues allow parallel data execution and retrying on failure, with little code overhead for developers.

Scalability
Nomanini has very cyclical traffic patterns: our peak traffic occurs in the early morning and late afternoon, with a monthly peak around payday in South Africa. Google App Engine automatically scales our application so that we don’t have to pay for excess server capacity during off-peak times, but have capacity available when we need it.

Ease of operation
Deployment is a breeze on App Engine. With just a few scripts tied in to our continuous integration server we can:

write the version number and app name into app.yaml

deploy automatically

run data migration scripts

change the default version once all indexes are serving

Monitoring is simplified by using built in dashboards. We can also export logs to Google Cloud Storage and run ad-hoc queries on analytics using Google BigQuery.

Conclusion
To reduce our time to market we used as many off-the-shelf components as possible, including Google App Engine.

We chose App Engine because it offers a platform with a consistent, well documented, ready-to-use set of services and allows our developers to test within an environment that is identical to our production environment - a practice that used to be prohibitively expensive.

Using the Google App Engine platform has saved our small development team time that we would have had to use to design, build and test a highly reliable backend to support our network of vending terminals in farflung places. Instead, we can focus our time on building a device that we hope will impact the way local commerce is done in rural economies.

To read more about the story of Nomanini, check out the post about us on the Official Google Blog.

-Contributed by Dale Humby, CTO, Nomanini

Posted by Zafir Khan, Product Marketing Manager, Google App Engine

App Engine 1.7.4 Released

Thursday, December 13, 2012

The Google App Engine team has been busy putting together our final release of 2012. This release includes a number of features graduating from Experimental status as well as the usual batch of bug fixes and improvements. We’ll be taking a short break from our monthly release cycle over the holidays, but we’ll be back to our normal schedule starting in February.
Expanded EU Support
We’re happy to announce that we are expanding European Union datacenter support, based on positive feedback from early users. You can sign up here. Please note, deployment is currently limited to billing-enabled applications. We understand that data locality and latency are important to developers and are committed to further expanding this support in the coming months.

Java
We've made a new Maven plugin available, added the source to the SDK for easier IDE debugging, and made significant performance improvements to the JSP compilation process. Please see our Java release notes for more information.

Python
We’ve made a big push to bring a number of new features to GA, upgraded the interpreter to version 2.7.3, and added several new experimental features. For more details, please see our Python release notes.

New and Upgraded Features
We also have a handful of notable features in this release:

Task Queue statistics (General Availability): You can see statistics such as the current number of tasks in a queue, the number of tasks executed in the last hour, and more.

Traffic splitting (General Availability): You can split requests amongst different versions of your app.

LogsReader and Logs API (General Availability): You can now fetch requests based on a list of request IDs.

Expanded Datastore query support (Experimental): We’ve added ‘DISTINCT’ support to Datastore queries.

Full release notes and Google Cloud Platform newsletter
The complete list of features and a list of bug fixes for 1.7.4 can be found in our release notes. For App Engine coding questions and answers check us out on Stack Overflow, and for general discussion and feedback, find us on our Google Group.

Finally, to stay up to date with Google Cloud Platform, sign up for our newsletter where you’ll get the latest news, announcements and event information.
- Posted by the Google App Engine Team

Get started at no cost with a faster, larger Google Cloud SQL database

Thursday, November 8, 2012

You want your application to be fast. Databases play a big role in your overall application speed and end-users’ experience. Reads and writes should execute instantly, even as your user base grows to millions of users, and this requires faster, larger databasesGoogle App Engine is designed to scale. And now Google Cloud SQL, a MySQL database that lives in Google’s cloud, is scaling up alongside App Engine to meet that demand for faster access to more data. With the features we are launching today, you can now work with bigger, faster MySQL databases in the cloud:

More Storage: We’re increasing the available storage on Cloud SQL to 100 GB - ten times more than before.

Faster Reads: We’re increasing the maximum size of instances to 16 GB RAM. This amounts to a 4 times increase in the amount of data you can cache.

Faster Writes: We’re adding optional asynchronous replication, which gives the write performance of a non-replicated database, but the availability of a replicated one.

EU datacenter availability: If you sign up to be a Premier customer, you can choose to store your data and run your Cloud SQL database instance in either our US or EU data centers.

Integration with Google Apps Script: We’re making it quick and easy for businesses using Google Apps to use Cloud SQL. Publish and share your data with Google Sheets, add data to your Google Sites pages or create simple Google Forms without worrying about hosting or configuring servers.

Introducing a new trial offer
We’ve received many requests from users for a trial offer to test out Cloud SQL. Today, we’re introducing a 6- month trial offer at no charge, effective until June 1, 2013, and will include one Cloud SQL instance with 0.5 GB of storage. Sign up now to get started.
Posted by Joe Faith, Product Manager, Google Cloud SQL

Upgrading to Python 2.7: simple and practical

Tuesday, November 6, 2012

We launched a major improvement to the Python runtime when we introduced support for Python 2.7 last February. It has since proven to be astonishingly popular – just a few months later more than half of all active Python applications on App Engine are using Python 2.7.

What’s so great about Python 2.7? It isn’t just a language update, it also includes a number of features that can help your apps run much more efficiently.

Increase efficiency with concurrent requests. Now a single frontend instance can handle multiple requests at the same time in separate threads. This means App Engine requires fewer frontend instances to handle the traffic, which can lead to significant cost savings.

Rapidly growing mobile gaming company Pocket Gems reduced the cost of running their front-end instances by 78% when they switched to to the Python 2.7 runtime. “We immediately saw a dramatic difference” said engineering director David Underhill “We only need one third as many instances under Python 2.7 as we did under 2.5 to handle the same traffic”.

Move faster with more native C modules. In Python 2.7 we’ve added more native C modules to App Engine. Core libraries like cPickle run much faster than the equivalent pure Python library available in the Python 2.5 runtime.

Take advantage of popular third-party libraries like PIL and numpy. We’ve greatly expanded the number of third party libraries available for use in App Engine in Python 2.7. So far we’ve added the Python Imaging Library (PIL), lxml, numpy, and several others.

Best of all, upgrading won't require lots of code changes. “It was pretty straightforward,” says Underhill. “We had to be careful, but we ended up needing to change less than 0.4% of the code”.

If you're ready to make the switch, check out our migration guide, find out more about the other great features of the Python 2.7 runtime and get in-depth tips from the App Engine Python team by watching their talk on Migrating to Python 2.7 at Google I/O 2012.

- Posted by Andrew Jessup, Product Manager, Google App Engine