Google Cloud Platform Blog
Product updates, customer stories, and tips and tricks on Google Cloud Platform
High Replication Datastore: A Solid Choice for Enterprise-class Applications
Tuesday, March 29, 2011
WebFilings
delivers a revolutionary cloud-based application that enables companies to collaboratively build and file financial reports with the SEC.
Built on the Google App Engine platform
, companies of all sizes use WebFilings software to take control of their financial reporting processes. WebFilings greatly
reduces the time and stress
required to file financial reports with the SEC.
Enterprise Requirements
As an enterprise class application supporting
many Fortune 100 customers
with strict financial reporting requirements, we simply cannot afford to have maintenance outages or service disruptions. Google App Engine has proven to be a robust platform for us to build on and has allowed us to scale seamlessly to support a rapid increase in customer demand. We’re pleased to report that a recent innovation by Google has allowed us to achieve an even higher level of availability and performance.
This recent innovation is the
High Replication Datastore
or HR Datastore. HR Datastore allows us keep our application live even during system upgrades and other maintenance operations that Google may be performing. In addition, it delivers a new level of reliability that has given our application significantly more consistent response times and 100% availability since making the move.
Decisions... Decisions...
We were originally a bit hesitant to move to HR Datastore because it was a relatively new offering from the App Engine team. Even though the underlying technology has been around for a long time, we weren’t sure initially how the eventual consistency model might affect our application. Well, we did our homework and read the Google posts on HR Datastore, followed the App Engine news groups, pinged Googlers via the App Engine Office Hours Chat on IRC, and did some prototyping with entity groups and eventually consistent queries. From this we got a better handle on how entity groups worked to enforce strong consistency with ancestor queries and also learned of the potential performance issues of doing large parallel db.get() operations across multiple entity groups. We also reviewed our code, and made a few changes to our application where we needed to use entity groups to have strong consistency. In the end, we found the move to be much less work than we had anticipated.
The Migration
The actual migration to HR Datastore was a fairly simple process. Essentially, we just set up a a new appspot instance, and selected the High Replication Datastore option during creation. Then we migrated our data to the new instance.
During our migration process we:
Created our new instance and deployed a new version of code to work with the new entity groups
Scheduled our maintenance window with our customers
Put our current master/slave instance into read-only mode
Copied the data from our master/slave instance to our HR Datastore instance. (Due to the size of our data, this was accomplished with some help from the App Engine team)
Ran a migration mapreduce to iterate over a few entities that now needed to be in entity groups. These were saved with a new parent making them part of the entity group. This enables queries to be consistent.
Implemented a site alias that routes customers to the new instance, webfilings-hrd.appspot.com, from webfilngs.appspot.com. Note: This feature is not generally available so you should contact the App Engine team directly if you need similar functionality.
In short, we went through a complete release and testing cycle and got our application migrated to HR Datastore in under a month. The actual migration happened overnight when we moved a litte over 200GB of data to our new HR Datastore instance, executed our migration mapreduce, and did our verification testing in about five hours. Since moving to HR Datastore, we are wondering why we ever even thought twice. Our customers and our Customer Success Team that works with customers every day have given us feedback that the application is performing better than ever and we have had no downtime since the move.
A Solid Choice
The High Replication Datastore was designed with the needs of large enterprises in mind. HR Datastore provides maximum uptime, multiple datacenter redundancy, consistent response times, along with scalability and performance that has met the needs of our fast growing company. It does take some planning to make the move, however. You should be prepared to review your existing code and test thoroughly beforehand. We found that this effort alone had a significant payback in terms of overall application performance and reliability. Even though the average latency on HR Datastore calls can sometimes be a little bit longer, we were able to optimize and end up with an overall experience that was better for the end user. If you have an application that has these same demands then my only question for you is
“When are you moving to HR Datastore?”
. If you are like us, you will be very glad you took advantage of this latest innovation from the Google team.
Posted by Dave Tucker
Director of Product Development
WebFilings (
www.webfilings.com
)
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Firebase Blog
Apigee Blog
Popular Posts
Understanding Cloud Pricing
World's largest event dataset now publicly available in BigQuery
A look inside Google’s Data Center Networks
New in Google Cloud Storage: auto-delete, regional buckets and faster uploads
Enter the Andromeda zone - Google Cloud Platform’s latest networking stack
Labels
Announcements
193
Big Data & Machine Learning
134
Compute
271
Containers & Kubernetes
92
CRE
27
Customers
107
Developer Tools & Insights
151
Events
38
Infrastructure
44
Management Tools
87
Networking
43
Open
1
Open Source
135
Partners
102
Pricing
28
Security & Identity
85
Solutions
24
Stackdriver
24
Storage & Databases
164
Weekly Roundups
20
Feed
Subscribe by email
Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.
Learn More
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow
Follow