This post is part of Who's at Google I/O, a series of guest blog posts written by developers who are appearing in the Developer Sandbox at Google I/O. It's also cross-posted to the Google Code blog which has similar posts for all sorts of Google developer products.
Evite is one of web's oldest social planning services. Since the launch in 1998, Evite has delivered over a billion party invitations. Although it has served us well, after ten years of operation it became necessary to replace the aging platform, and position Evite for another decade of successful party planning.
The reengineering effort took well over a year. Our development team replaced Java with Python and saw a significant increase in developer productivity. However, dealing with a ten year-old Oracle database remained a challenge. To overcome scalability limitations and inflexibility of a large relational DBMS, we designed and implemented a proprietary data store. We conducted a series of A/B tests and gradually started migrating production traffic to the new system.
Our initial motivation to use Google App Engine was fast provisioning. We introduced App Engine as a temporary solution while we increased scale and optimized our proprietary data store. However, once we imported user profile data and put App Engine backend services in production, we never looked back.
Importing a large data set of user profiles from Oracle RAC into App Engine was challenging at first because we had to perform a bulk import and then keep the data synchronized between the two datastores. As we developed our data synchronization tools we gained better understanding of the API and performance characteristics of the App Engine datastore.
Once we synchronized profile data and enabled production traffic, App Engine really started to impress. We would watch our daily traffic grow and observe App Engine’s automatic scaling in action. Additional server instances would come online to meet increased demand and disappear as traffic lowered, without any sysadmin intervention. Despite our proficiency in the use of cloud computing resources and system automation, it was never this easy to provision new servers. As Grig, Evite’s devops guru, likes to say "It's not in production until it's monitored and graphed." App Engine’s dashboard automatically manages instances and graphs system usage data.
Following our positive experience with Evite profiles, we have continued to use App Engine for other services. Occasional frustrations with elevated error rates on the Master/Slave Datastore disappeared as we switched to the High Replication Datastore. At this point we see no technical obstacles to using App Engine more extensively. Architectural decisions and best practices implemented by the App Engine team are very well aligned with the choices we made in designing Evite’s new platform. This makes it easy to deploy additional services and data sets to App Engine.
The real obstacle to using App Engine exclusively for most Evite services is risk management. As reliable and cost-effective as App Engine has been for us, it is difficult to depend completely on a single vendor/service provider. To ensure availability of our application we use multiple cloud providers. Unfortunately, this strategy prevents us from using certain App Engine features and API's because we do not have adequate replacements for them in other deployment environments.
Evite’s new platform built on Python, NoSQL and App Engine has been a success. Our new web application has been well received by users, and our first iPhone app is receiving positive reviews. (Android version is in the works). We look forward to continued use of App Engine for data warehousing, rapid launches of new services and other projects.
Come see Evite in the Developer Sandbox at Google I/O on May 10-11.
Dan Mesh is a Vice President of Engineering at Evite. His extended responsibilities include espresso deliveries and release day food orders.
Posted by Scott Knaster, Editor
This is part of our on going series of blog posts from guest authors highlighting success stories from applications and services built on or targeting App Engine developers. Today, we have a post from Bruno Morency of Context.IO. Bruno has been involved in startups since graduating from McGill Engineering in 2001. And since being introduced to Pine on UNIX terminals, he’s had a love-hate relationship with email.
Email mailboxes contain years of important conversations and business information yet there are no easy ways for App Engine developers to find and use that information. This is what Context.IO does. It’s the missing email API that turns mailboxes into a data source developers can leverage. This is particularly interesting for Google App Engine developers since it makes content of Gmail accounts (and any other IMAP accessible email account) available through a set of HTTP calls.
In this post, we’re going to walk you through using Context.IO to build an app that lets users search for contacts from their inbox then easily get the history of recent emails and attachments exchanged with these contacts. A working demo is available at http://contextio-demo.appspot.com. The code for this application is available on Context.IO’s GitHub account.
Our demo is quite simple in its functionality: there’s a search box used to find contacts, and once a contact has been found, we list recent emails and attachments associated with the contact. To do this, the application offers 3 urls that are called by the JavaScript running in the browser to obtain the data: search.json, messages.json and files.json.
The actual UI formatting is all implemented in the JavaScript running in the browser, but our focus here is how we get that data out of the mailbox and return it to the browser.
Let’s see how we respond to the request to get message history for a given contact. This is done by calling /messages.json which accepts an email address as a GET parameter. Note, this functionality requires an authentication step not shown here. The code behind that call is as follows:
class MessagesHandler(webapp.RequestHandler): def get(self): current_user = users.get_current_user() current_email = current_user.email() emailAddr = self.request.get('email') contextIO = ContextIO(api_key=settings.CONTEXTIO_OAUTH_KEY, api_secret=settings.CONTEXTIO_OAUTH_SECRET, api_url=settings.CONTEXTIO_API_URL) response = contextIO.contactmessages(emailAddr,account=current_email) self.response.out.write(simplejson.dumps(response.get_data()))
The code simply uses the contactmessages.json API call of and returns all the messages including the subject, other recipients, thread ID, and even attachments in JSON format. Check the documentation for a complete breakdown of the response.
You can see how we handle the other cases similarly in handlers.py. To get a complete overview of other calls offered by Context.IO, please refer to the documentation available here http://context.io/docs/latest.
The complete code for this demo application has been made available by the Context.IO team on our GitHub account. Feel free to use it as the base for your own application. To get started, you’ll need to setup your application with your own Google API authorization and Context.IO API keys.
If you have questions about using Context.IO to embed your user’s email in your app, feel free to contact us through http://support.context.io.
Posted by Bruno Morency, Co-founder of Context.IO
Elastic Path develops a very flexible enterprise ecommerce platform. Many global brands rely on the Elastic Path platform to power their ecommerce solutions.
Many ecommerce sites are actually complex web applications. Catalog management, shopping cart functionality, promotion engine, order fulfillment, and backend integrations are just some of the challenges involved in running a full-fledged online store.
Since 2008, our Java-based platform has been the ecommerce backbone of a couple of online stores that are being migrated to run on App Engine. Like many complex web applications, these stores used to run in a multi-server environment (Apache Tomcat with a MySQL database) hosted in a colocation center.
As the diagram above shows, our goal is to have Elastic Path running entirely on the App Engine cloud. The storefronts have already been migrated, and the database and remaining parts of the Elastic Path platform will be fully on the cloud soon.
Why are we doing this? There are many benefits to being on App Engine:
Our migration’s high-level approach was to move everything except the persistence layer onto App Engine, and then resolve issues with the technical limitations such as the class whitelist and request length. We also had to modify some third-party libraries to work around App Engine’s restrictions on operations such as class loading, threads, and sockets.
We didn’t migrate the persistence layer because Elastic Path uses a relational database; converting our entire object graph to the Datastore is not feasible now. We are working closely with Google on alternatives. In the interim, we are still using a MySQL database and have kept our persistence layer running within a Tomcat application in the colo. We implemented a creative solution: the non-persistence layers of Elastic Path run on App Engine and communicate with the Tomcat-hosted persistence services via Spring Remoting. The back-and-forth remoting was expensive and impacted the performance of our application so we implemented some data caching. For this, we turned to App Engine’s Memcache, which improved performance by an order of magnitude (less than 2 seconds average response times vs. 2 minutes or more without Memcache).
Other App Engine technologies we use heavily include AppStats for performance tuning, URL Fetch for the Spring Remoting described above, and the fantastic Maven GAE plugin that we use for packaging and automated deployments. As we continue to push our platform up to the cloud, we hope to utilize more of App Engine’s cool features. If you’d like to learn more about Elastic Path, how we are migrating our Java platform to run on the cloud, and how you might be able to migrate your application to App Engine, drop by our booth in the App Engine section of the Developer Sandbox. See you there!
Come see Elastic Path Software in the Developer Sandbox at Google I/O on May 10-11.
Eddie Chan is an ecommerce developer at Elastic Path Software in beautiful Vancouver, Canada. He and his brilliant team work closely with Google and are currently focused on migrating existing online stores to App Engine.
Mojo Helpdesk from Metadot is an RDBMS-based Rails application for ticket tracking and management that can handle millions of tickets. We are migrating this application to run on Google App Engine (GAE), Java, and Google Web Toolkit (GWT). We were motivated to make this move because of the application’s need for scalability in data management and request handling, the benefits from access to GAE’s services and administrative tools, and GWT’s support for easy development of a rich application front-end.
In this post, we focus on GAE and share some techniques that have been useful in the migration process.
Task failure management
Our application makes heavy use of the Task Queue service, and must detect and manage tasks that are being retried multiple times but aren’t succeeding. To do this, we extended Deferred, which allows easy task definition and deployment. We defined a new Task abstraction, which implements an extended Deferrable and requires that every Task implement an onFailure method. Our extension of Deferred then terminates a Task permanently if it exceeds a threshold on retries, and calls its onFailure method.
Deferred
Task
Deferrable
onFailure
This allows permanent task failure to be reliably exposed as an application-level event, and handled appropriately. (Similar techniques could be used to extend the new official Deferred API).
Appengine-mapreduce
Mojo Helpdesk needs to run many types of batch jobs, and appengine-mapreduce is of great utility. However, we often want to map over a filtered subset of Datastore entities, and our map implementations are JDO-based (to enforce consistent application semantics), so we don’t need low-level Entities prefetched. So, we made two extensions to the mapper libraries. First, we support the specification of filters on the mapper’s Datastore sharding and fetch queries, so that a job need not iterate over all the entities of a Kind. Second, our mapper fetch does a keys-only Datastore query; only the keys are provided to the map method, then the full data objects are obtained via JDO. These changes let us run large JDO-based mapreduce jobs with much greater efficiency.
appengine-mapreduce
Supporting transaction semantics
The Datastore supports transactions only on entities in the same entity group. Often, operations on multiple entities must be performed atomically, but grouping is infeasible due to the contention that would result. We make heavy use of transactional tasks to circumvent this restriction. (If a task is launched within a transaction, it will be run if and only if the transaction commits). A group of activities performed in this manner – the initiating method and its transactional tasks – can be viewed as a “transactional unit” with shared semantics.
We have made this concept explicit by creating a framework to support definition, invocation, and automatic logging of transactional units. (The Task abstraction above is used to identify cases where a transactional task does not succeed). All Datastore-related application actions – both in RPC methods and "offline" activities like mapreduce – use this framework. This approach has helped to make our application robust, by enforcing application-wide consistency in transaction semantics, and in the process, standardizing the events and logging which feed the app’s workflow systems.
Entity Design
To support join-like functionality, we can exploit multi-valued Entity properties (list properties) and the query support they provide. For example, a Ticket includes a list of associated Tag IDs, and Tag objects include a list of Ticket IDs they’re used with. This lets us very efficiently fetch, for example, all Tickets tagged with a conjunction of keywords, or any Tags that a set of tickets has in common. (We have found the use of "index entities" to be effective in this context). We also store derived counts and categorizations in order to sidestep Datastore restrictions on query formulation.
Ticket
Tag
Tickets
These patterns have helped us build an app whose components run efficiently and robustly, interacting in a loosely coupled manner.
Come see Mojo Helpdesk in the Developer Sandbox at Google I/O on May 10-11.
Amy (@amygdala) has recently co-authored (with Daniel Guermeur) a book on Google App Engine and GWT application development. She has worked at several startups, in academia, and in industrial R&D labs; consults and does technical training and course development in web technologies; and is a contributor to the @thinkupapp open source project.
This is a guest post by Mike Johnston and Fred Cheng, co-founders, Simperium. This post is part of Who's at Google I/O, a series of guest blog posts written by developers who are appearing in the Developer Sandbox at Google I/O. It's also cross-posted to the Google Code blog which will have similar posts for all sorts of Google developer products.
We originally created Simplenote both as a learning exercise and to address what we thought were shortcomings in the original Notes app for the iPhone (Marker Felt font, no ability to search, etc.) The very first version of Simplenote didn't even have syncing!
We've certainly come a long way since then. The Simplenote backend now synchronizes data across devices, the web, and third-party apps while also handling in-app purchases, sharing, and basic metrics. About a year ago, we were accepted to the Y Combinator startup accelerator with something like 20,000 users. Today, with hundreds of thousands of users, we're currently serving 15 million requests daily and providing access to over 500 gigabytes (!) worth of text notes.
Google App Engine is at the heart of it. We made a decision early on to use App Engine so we wouldn't have to worry about scaling, or deploying more servers, or systems administration of any kind. Being able to instantly deploy new versions of code has allowed us to iterate quickly based on feedback we get from our users, and easily test new features in our web app, like the newly added Markdown support.
We consider our syncing capabilities to be core features of Simplenote. They are, in and of themselves, largely responsible for attracting and retaining many of our users. Our goal is to give other developers access to great syncing, too. The next version of our backend is named after our company, Simperium. As a general-purpose, realtime syncing platform intended for third-party use, Simperium's architecture is much more expansive than the Simplenote backend. Yet App Engine still plays a key role. It powers the Simplenote API that is used by dozens of great third-party apps like Notational Velocity. And it continues to power auxiliary systems, like processing payments with Stripe, while bridging effectively with externally hosted systems, like our solution for storing notes as files in the wonderful Dropbox.
We suspected we might outgrow App Engine, but we haven't. Instead, our use of it has evolved along with our needs. Code we wrote for App Engine a year ago continues to hum along today, providing important functionality even as new systems spring up around it.
In fact, we still come up with entirely new ways to use App Engine as well. Just last week we launched an internal system that uses APIs from Twitter, Amazon Web Services, Assistly, and HipChat to pump important business data into our private chat rooms. This was a breeze to write and deploy using App Engine. Such is the mark of a versatile and trustworthy tool: it's the first thing you reach for in your tool belt.
Come see Simperium in the Developer Sandbox at Google I/O on May 10-11.
Mike Johnston was a senior designer and programmer at Irrational Games where he worked on numerous games and prototypes for PC and Xbox 360. Before that he built security software at Entrust.
Fred Cheng hails from Cantaloupe Systems, a venture-backed startup, where he built their infrastructure for wirelessly tracking tens of thousands of vending machines.
Here on the App Engine team, we’re always looking for new ways to make it easier for developers to build applications and services. Today, I’m happy to introduce ProtoRPC, a new tool for creating simple Python services, which requires minimal set up and configuration to create new services.
What can you use ProtoRPC web-services for? Most web applications have the need to send and receive data between different components and/or applications. Typically, developers come up with ad-hoc ways of doing this as quickly as possible. As the application grows larger and the need to share information across components grows, it becomes more difficult to manage. URL end-points are defined and appear inconsistent from one another and a lot of boiler plate code is added checking parameters. This quickly becomes a maintenance nightmare and is a problem that ProtoRPC is built to solve.
ProtoRPC makes it easy to write consistent, reliable web interfaces that can be used, for example, to do the following:
Using ProtoRPC, you can define structured web-services right in the application’s Python code without having to first learn and write a new interface definition language such as Thrift and Protocol Buffers, however still retain the same powerful features such as interface introspection and automatic client generation.
The way to go about defining a web service should be familiar to you if you already have experience working with App Engine db.Models and the webapp framework. The data sent between client and web service are defined in a similar way as Datastore models. The services classes that handle requests are defined similarly to webapps RequestHandler classes. Let’s take a look at a simple example from the ProtoRPC getting started guide. This simple web service says hello to its client:
class HelloRequest(messages.Message): my_name = messages.StringField(1, required=True) class HelloResponse(messages.Message): hello = messages.StringField(1, required=True) class HelloService(remote.Service): @remote.method(HelloRequest, HelloResponse) def hello(self, request): return HelloResponse(hello='Hello there, %s!' % request.my_name)
If this web services was used as the URL end-point for an AJAX based form, Javascript to communicate with the service might look like this:
$.ajax({url: ‘/helloservice.hello’, type: 'POST', contentType: 'application/json', data: ‘{ my_name: Bob }’, dataType: 'json', success: function(response) { // The response is { hello: “Hello there, Bob!” } alert(response.hello); } });
As you can see from the example, ProtoRPC can speak JSON right out of the box. In addition, it is compatible with the protocol buffer binary format and can therefore communicate with clients and servers written using traditionally compiled .proto files.
Right now, ProtoRPC is available as an separate project that can be downloaded here. It’s still considered experimental and may change in substantial ways before being integrated in to the SDK. But even though it’s in preview, it already has a number of useful features, such as a remote service discovery mechanism and a forms interface for easy testing. At the moment, ProtoRPC is only available in Python. As always, we plan to offer it for Java developers in the near future.
Check out the getting started guide for a more complete overview to try out writing a few services of your own!
Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.
Follow