Google Cloud Platform Blog
Product updates, customer stories, and tips and tricks on Google Cloud Platform
Project WOW
Monday, September 26, 2011
Today's post
is contributed by
Edward Hartwell Goose of PA Consulting, who is working on an App for the UK’s
Met Office
to report everyone's favorite bit of small talk, the
weather.
W
e hope you find the discussion of his team's experience using App Engine illuminating
.
The UK’s
Met Office
is one of the world’s leading organisations in weather forecasting, providing reports throughout the day for the UK and the rest of the world. The weather information they provide is consumed by a variety of industries from shipping to aircraft, and powers some of the UK’s leading media organisations, such as the BBC.
Although the Met Office is the biggest provider of weather data, they aren’t the only ones collecting information. Thousands of enthusiasts worldwide collect their own weather data, from a wide variety of weather stations - either simple temperature sensors or highly sophisticated stations that rival the Met Office’s own equipment. The question of course is: how do you harness the power of this crowd?
Enter the Weather Observations Website
The Met Office and our team from PA Consulting worked together to answer this question late last year. The end result was
The Weather Observations Website
, or “
WOW
” for short. In the 3 months since launch on the 1st June 2011, WOW has recorded
5.5 million weather reports
from countries throughout the world. Furthermore, we can retrieve current reports in sub second times, providing a real time map of worldwide weather. We haven’t got the whole globe covered just yet and the UK carries the most sites, but most countries in Western Europe are reporting. We also have reports from a medley of countries throughout the world, from Mauritius, Brazil and as far away as New Zealand. We even have one site reporting at regular intervals in
Oman
(it’s hot!).
Better yet, as a development team of 2, since launch, we’ve spent almost no time at all doing anything but casual monitoring of WOW. No one carries a pager, and the one time we did have problems (we underestimated demand, and our quota ran out), I was able to upgrade the quota in a minute or so. And I did it from my sofa. On my phone.
How good is that?
WOW - Showing Live Temperature Data Across Europe
Lessons Learn
t Building for App Engine
We learnt a lot building WOW. A huge amount in fact. And we’d love to share our insights with you. We’d also love to tell you what we love about App Engine - and why you should use it too. And so you know we’re honest - we’ll tell you what we don’t like too!
Firstly - the good stuff. We think App Engine is a fantastic tool for prototyping and any team working in an agile environment. We’re big fans of
SCRUM
at PA Consulting, and App Engine was a dream to work with. Compared to some of our colleagues working with difficult build procedures and environments, our full release procedure never took more than 5 minutes. Better yet, App Engine’s deployment tools all hook up with
ANT
and
CruiseControl
(our continuous build system), allowing us to run unit tests and deploy new code on every check-in to our code repository. This allowed us as developers to get on with what we do best: develop!
The APIs are great too. The documentation is fantastic and is regularly updated. We use all but the Channel API at the moment (sadly, it uses Google Talk Infrastructure behind the scenes and this gets blocked by some corporate environments). If I could offer any advice to a budding App Engine developer it would be to thoroughly read the documentation, and place the following three questions and their solutions at the forefront of everything you do:
1. Can I do it in the background? - Task Queue API
2. Can I use the low-level API? - Datastore API
3. Can I cache the result? - Memcache API
These three principles have given WOW the performance it has, and will allow it to scale effectively over the coming years. We’ve got some big changes coming over the next couple of months too that should provide even higher performance as well as exciting new features.
Future Developments
So, how about improvements App Engine could make? There are definitely a few, although how important they are will depend on the problems you’re trying to solve.
To begin with, improvements to some of the non technical elements are needed before App Engine becomes truly mainstream. Needing a credit card to pay is a showstopper for some organisations. Backup and restore is also missing, unless you implement it yourself. This is perfectly possible, but can add significant man days to your development effort if your data structure is complex and fast changing.
We also struggled with how to estimate quota to begin with too. One of the brilliant features of App Engine is how easy it is to spool up (and down) new instances to deal with demand. Unfortunately, this also means it can be quite easy to
accidentally
spool up too many instances and burn quota quickly. Although this has never affected the actual data, it can cause an unpleasant spike in the amount of dollars spent. We also had a similar problem with a MapReduce job getting stuck overnight that caused a scare the next morning. Hopefully the new monitoring API should provide a bit more visibility of these issues as well as automatic email notifications to help catch these issues.
Aside from that, other features will probably depend on the application you’re trying to build. Built in support for geo-queries would be invaluable for WOW. Currently we use an external library, but this adds some extra overhead on our development. Another common feature request is full text search which is essential for projects dealing with large text corpa. Both of these features would allow us to provide better search facilities for our users - for example search by site name or geographic location. These queries can be implemented in App Engine as it is now, but achieving optimal performance and optimal cost are difficult problems that we struggle to complete ourselves.
Final Thoughts
Overall, we’re really impressed by App Engine. The App Engine team regularly releases new versions, and although it does have limitations it has allowed us to concentrate on what really matters to us - the weather. We know WOW will scale without any problems, and we don’t have to worry about any of the hardware configuration or system administration that can easily consume time. Our small team of developers spends all of their time understanding the business problems and improving WOW.
We’re really looking forward to taking WOW forward in the future, we hope you can join us:
http://wow.metoffice.gov.uk
.
Edward Hartwell Goose (
@edhgoose
)
Developer
PA Consulting
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Firebase Blog
Apigee Blog
Popular Posts
World's largest event dataset now publicly available in BigQuery
A look inside Google’s Data Center Networks
Enter the Andromeda zone - Google Cloud Platform’s latest networking stack
Using labels to organize Google Cloud Platform resources
New in Google Cloud Storage: auto-delete, regional buckets and faster uploads
Labels
Announcements
193
Big Data & Machine Learning
134
Compute
271
Containers & Kubernetes
92
CRE
27
Customers
107
Developer Tools & Insights
151
Events
38
Infrastructure
44
Management Tools
87
Networking
43
Open
1
Open Source
135
Partners
102
Pricing
28
Security & Identity
85
Solutions
24
Stackdriver
24
Storage & Databases
164
Weekly Roundups
20
Feed
Subscribe by email
Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.
Learn More
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow