Google Cloud Platform Blog

Wanted: App Engineers

Wednesday, July 13, 2011

You may have heard the news that 2011 is Google’s biggest hiring year yet. And the App Engine team is looking for a few great Software Engineers to join us in San Francisco to code, collaborate on innovative ideas for platform computing, and get burritos with us every Friday.

The App Engine team in San Francisco.

We think we have some of the best and most enthusiastic developers out there-- so we thought we’d ask you to come work with us. The App Engine team needs talented developers to help build the platform by developing the features on our road map and issue tracker, as well as that amazing feature idea you have that will revolutionize App Engine for our users.If you are a backend engineer who is experienced with C++ or Java, lives and breathes distributed systems, and is obsessed with scalability, please visit our jobs website to submit your resume along with a cover letter describing your experience with App Engine. We’ll be in touch if we think your experience matches our open positions.Thanks for continuing to help make App Engine a great platform!Posted by The App Engine Team

Highlights from our Google I/O Presentations

Wednesday, June 22, 2011

Now that things have settled down from last month’s Google I/O, we thought it would be a good time to highlight some of the technical talks given by the App Engine team. At this year’s I/O, we emphasized some of the big themes we’ve been focusing on this past year: availability and productionization of our platform, removing limits while maintaining performance at scale, and developing new APIs.Availability and Productionization of our PlatformSince App Engine automatically handles all of the sharding and distribution of your application across datacenters, we thought it would be important to talk about the strategies we use to run and maintain our platform. These talks also give you an opportunity to meet some of the engineers who carry pagers so you don’t have to.

More 9s Please: Under the Covers of the High Replication Datastore - We were excited to announce the HR Datastore in January, since it provides a high level of availability, even during unplanned outages, and even if an entire datacenter goes offline. HRD uses the paxos algorithm to distribute data across multiple datacenters. Alfred Fuller and Matt Wilder talk about the high level distribution strategy used and how this helps keep your app serving.

Life in App Engine Production - Let’s face it - everyone has some great war stories about running live applications. And the stories of Google’s SRE team probably rank with the best of them. Michael Handler and Alan Green talk about what goes right and what can go wrong running in Production.

Removing Limits while maintaining performance at scaleHow can you design your application so you can transition from serving 1 user to 1 million users with the minimum amount of work? How do you make sure your user experience stays consistent for those users?

App Engine Backends - At I/O this year we launched App Engine Backends, which allow you to write long running processes on App Engine. Crunch numbers, process data, and build large in-memory models! Justin Haugh and Greg Darke give an overview of this new feature and cover best practices.

Scaling App Engine Applications - We always get lots of question on what it means and how it works to scale an app using App Engine (hint: low latency is the key!). Guido van Rossum and Justin Haugh explain how instances work and cover best practices for scaling your application.

Developing New APIs

Full Text Search - It took 8 days after App Engine’s initial launch for this feature request to be filed - and now work on the Full Text Search API is nearing completion. Bo Majewski and Ged Ellis give a preview of the upcoming App Engine Full Text Search API.

App Engine MapReduce- So you just want to analyze 5TB of data? You’ll probably need MapReduce to do so! Mike Aizatsky talks about the progress being made on the App Engine MapReduce framework and how you can use it to generate reports for your data.

Large-scale Data Analysis using the App Engine Pipeline API - Once you’ve written one MapReduce, you’ll probably want to write more, and then you’ll need a way to manage your application’s analysis pipeline. In this talk, Brett Slatkin discusses a lightweight API for managing MapReduce workflows.

Our team always enjoys participating in Google I/O because our engineers get to share their vision and discuss their work with a large audience. If you are interested in what some of our customers had to say at I/O, check out the Google Enterprise Blog. Of course, with App Engine featured in 23 talks for Google I/O, we couldn’t cover them all here. See the complete session list for I/O and discover all the talks on App Engine and more!Posted by The App Engine Team

App Engine 1.5.1 Release

Monday, June 20, 2011

It’s been about a month since Google I/O 2011, so we thought it was time for another release. This month we are releasing ProtoRPC as an official Python API, making the High Replication Datastore (HRD) available in the SDK to help developers better understand the consistency model, adding presence to the Channel API, plus a few more goodies. Check them out below:Serving Changes

Geolocation Header: App Engine will now include a header with every client request that represents a best-effort attempt to identify the country from which the request originated. This header, “X-AppEngine-country,” can allow you to customize content based on the origin of the user. We hope this means no more “Select your location” drop downs for users!

Updated Java and Python APIs

Channel API (Presence Support): You can now configure an inbound service for your application to detect user presence for the Channel API. This lets you take an action when a user connects or disconnects from the Channel.

Images API (WebP Support): The Images API now supports WebP images. If you haven’t checked it out, WebP is a new image format that Google open sourced earlier this year which provides lossy compression for images and can be up to 39% smaller than JPEG images of a similar quality!

New Python APIs

ProtoRPC: ProtoRPC is an open source framework for creating a well defined and easy to use web-based RPC service. While ProtoRPC is similar to using Google Protocol Buffers to define services, the goal of ProtoRPC is to make it easy for developers to start defining web based services, while still allowing those services to scale and evolve over time.

Datastore

High Replication in the SDK: Since releasing the High Replication Datastore, we’ve wanted to provide tools that help developers understand and test the new consistency model while developing their applications. The 1.5.1 SDKs for Java and Python can now emulate the HRD consistency model. This means that now, by setting the appropriate SDK config options, queries across entity groups will occasionally return results that don’t reflect the most recent data written. This should allow you to develop your application to be more resilient to this consistency model.

As with all releases, 1.5.1 contains some additional small features and many bugfixes, the full list of which is available in our release notes for Python and Java. Hopefully you’ll find this release useful. Either way feel free let us know what you think by posting in the App Engine group.Thanks!The App Engine Team

Getting Started with the Tasks API on Google App Engine

Friday, June 17, 2011

This is cross-posted from the Google Apps Developer BlogGoogle TasksGoogle Tasks APIGetting Started with the Tasks API on Google App Engine

Create a simple App Engine app using the Google APIs Client Library.

Provide Google Tasks users a mechanism to authorize your app to access their tasks.

Make authorized requests to the Tasks API.

Check out the tutorialGoogle Tasks API forumPosted by Johan Euphrosine, Developers Program Engineer

App Engine at I/O 2011, Day 2

Wednesday, May 11, 2011

Good morning I/O-ers and welcome to day two! We hope you got a chance to see all of the great sessions on day one and, if you missed our big announcements from yesterday, take a look at all the great features in 1.5.0 including Backends and our plans for App Engine to leave preview later this year.We’ve got a great set of sessions lined up for day two including updates on our progress with Full-text Search and MapReduce. We’ve also got two great sessions on a subject close to developers’ hearts: reliability.Under the Covers with the High Replication Datastore - At 10:45 am, we jump into the internals of the High Replication Datastore (HRD), explaining how it works, how it differs from the Master/Slave configuration, and why developers love it. Now that HRD is the default configuration and cheaper to use, come find out what you’re missing.Life in App Engine Production - At 3:00 pm, come meet the team wearing the pagers so you don’t have to, App Engine’s Site Reliability Engineers (SREs). In this session, they’ll give you a view of what life behind the scenes is like and why you should concentrate on your application and let the SREs take care of keeping the lights on. Learn how building on App Engine means your application gets its very own Devops team.For those of you that couldn't make it to I/O this year, don’t stress. While we wish you were here, the I/O video team will soon have videos of all our sessions available so you can catch up from the comfort of your own home. We’ve even captured a few of our Developer Sandbox companies so you’ll get the full experience!Posted by the App Engine team

The Year Ahead for Google App Engine!

Tuesday, May 10, 2011

Google App Enginelaunched in Preview status in 2008Google App Engine for BusinessGoogle App Engine for Business (GAE4B) was announced at Google I/O last year

Our customers are really excited about many of the features of GAE4B: SLA, SSL for custom domains, SQL, etc

The per-user, per-app pricing is not appropriate for many customers that are excited about the GAE4B features because they are not always building applications focused on users within their organization.

Because App Engine is currently a Preview product, some businesses are reluctant to heavily invest in the platform.

App Engine graduating from Preview later this year!

We will be leaving Preview to become an official Google product! This is the top priority for our team and we are on schedule to graduate in the second half of this year. This does not mean we’ve stopped development of other much needed features, for example today we are releasing Backends.

Most of the GAE4B features are going to be rolled into App Engine, enabling all paying customers to take advantage of them. This includes: SLA, SSL for custom domains, SQL, Operational and Developer support, and the new business-oriented Terms of Service. The domain console is currently in trusted tester and will remain so for the time being. GAE4B will no longer be offered as a standalone product.

In order to become an official Google product we must restructure our pricing model to obtain sustainable revenue. Based on customer feedback this means focusing on usage-based pricing and placing per-user, per-app pricing on hold until further notice.

99.95% uptime service level agreementnew Terms of Service agreementNew App Engine Pricing Modeldetails of the new pricing model

Free Apps: Similar to the Free Apps of today but with more restrictive quotas

Paid Apps: Similar to Apps which have Billing Enabled today, except they will have a 99.95% SLA and cost $9/app/month in addition to usage fees. Customers will need to opt-in to this.

Premier Accounts: A new offering which will allow a company to not pay for individual Apps but rather use as many as they need. Premier Accounts will be eligible to receive Operational Support for $500/month in addition to usage fees.

Instances: Due to customer feedback and to better service memory intensive applications, we will be eliminating CPU hours. Instead, our serving infrastructure will charge for the number of Instances running as a new unit called “Instance-hours (IH)” (1 instance running for 1 hour). These instances will be similar to the instances you can see in the Admin Console today with the exception that we will be improving our scheduler to ensure each instance has an appropriate level of utilization. IHs can either be pay-as-you-go ($0.08/IH) or you can commit to a minimum number of IHs over the course of a week and pay less for those reserved IHs ($0.05/IH).

APIs: All APIs which are currently charged as CPU hours will instead be charged per operation. This is intended to make it easier to predict and understand the cost of an application. Our website contains the details of the API pricing.

Datastore Storage: High Replication Datastore (HRD) storage is being reduced in price from $0.45/G/month to $0.24/G/month as of today. HRD has delivered essentially 99.999% uptime since we launched it in January. As of today HRD is the default option for new applications, and we strongly encourage users of the M/S Datastore to migrate to HRD. Master/Slave Datastore storage will increase to $0.24/G/month when App Engine leaves Preview. We are actively working on better tools to make this migration easier.

Thank you!The App Engine Team

App Engine 1.5.0 Release

Tuesday, May 10, 2011

The App Engine team has been working furiously in preparation for Google I/O time and today, we are excited to announce the release of App Engine 1.5.0, complete with a bunch of new features. This release brings a whole new dimension to App Engine Applications with the introduction of Backends, some big improvements to Task Queues, a completely new, experimental runtime for the Go language, High Replication Datastore as the new default configuration (and a lower price!), and even more tweaks and bug fixes.Serving Changes

Backends: Until now all App Engine applications have been running on short-lived dynamic instances that we spin up and down in response to application requests. This is great for building scalable web applications, but has a number of limitations if you are looking to build larger, long-lived, and/or memory intensive infrastructure. With 1.5.0, we are introducing Backends which will allow developers to do precisely this! Backends are developer-controlled, long-running, addressable sets of instances which are as large as the developer specifies. There are no request deadlines, they can be started and stopped (or dynamically start when called), they can use between 128M and 1G of memory and proportional CPU. If you’d like to find out more, have a read through our Backend docs for Java and Python.

Pull Queues: Most of our users are heavily using Task Queues in their applications today, but there is lots of room for more flexibility. With 1.5.0 we are introducing Pull Queues to allow developers to “pull” tasks from a queue as applications are ready to process them, rather than relying on Task Queues to push tasks at a pre-configured rate. This means you can write a Backend to do some background processing and pull 1, 10, or 100s of tasks off the Pull Queue when the Backend is ready for more. In addition, we’ve introduced a REST API which will allow external services to do the same thing. For example, if you have an external server running to do image conversion or OCR, you can now use the REST API to pull tasks off, run them, and return the results. In conjunction with these 2 improvements, we’ve also increased the payload limits and processing rate. We are excited both about expanding the use of Task Queues as well as improving the ease of integration between App Engine and the rest of the cloud.

Datastore

High Replication Datastore as default: After months of usage and feedback on the High Replication datastore (as well as a record of 99.999% uptime so far) we are now confident that it is the right path forward for the majority of our users. So, today we are doing two things: setting HRD as the default for all new apps created, lowering the price of HRD storage from $0.45 down to $0.24, and encouraging everybody to begin plans to migrate. We really appreciate all the time that early users of HRD put into trying it out and finding issues and have fixed a number of those issues with this release.

Changed APIs

In response to popular demand, the HTTP request and response sizes have been increased to 32 MB.

Mail API: We have added a few restrictions to the Mail API to improve the reliability and reputation of the service for all applications. First, emails must be sent from email accounts managed by Google (either Gmail, or a domain signed up for Google Apps). Second, we’ve reduced the number of free recipients per day from 2000 to 100 for newly created applications. Both of these will help ensure mail from your application arrives at the destination reliably.

Administration

Code downloads: As of 1.5.0, we have expanded the ability to download an Application’s source code to include both the user who uploaded the code to download it as well as the Owner(s) of the project as listed in the Admin Console. Owners were introduced in 1.4.2 as an admin role.

New runtime: With 1.5.0 we are launching an experimental runtime for the Go Programming Language. Go is an open source, statically typed, compiled language with a dynamic and lightweight feel. It’s also an interesting new option for App Engine because Go apps will be compiled to native code, making Go a good choice for more CPU-intensive tasks. As of today, the App Engine SDK for Go is available for download, and we will soon enable deployment of Go apps into the App Engine infrastructure. If you’re interested in starting early, sign up to be first through the door when we open it up to early testers. See the announcement on the Go Blog for more details.

JavaPythonother announcement at I/O 2011The App Engine Team