What Exactly Is Google App Engine?

Last week was Google App Engine announcement and brouhaha.  This week is deeper analysis and understanding of Google App Engine. I spent some time last week trying to understand this better, read a bunch of stuff, and spent some time having a top secret special meeting that I can’t talk about with some of my friends at Microsoft where this was discussed.

Following are three interesting things for you this morning (all courtesy of my friend Scott Moody) if you are interested in learning more:

1. The Google App Engine Q&A – an in-depth blogger-created FAQ that provides great links to other blog posts on the topic and summarizes various opinions and known facts.

2. Google App Engine for developers – Nial Kennedy’s overview from his meeting with the App Engine team leads.

3. A high level comparison (via email) from Scott Moody where he compares App Engine and Amazon EC2.  Since Scott doesn’t keep a blog, following is the pertinent text from his email.

Google hides infrastructure from AppEngine users. AE programmers never (and, in fact, aren’t allowed to) think about database scaling and configuration, load balancing , fail-over, etc. In theory, the complexity of writing a highly scalable app completely disappears.

With EC2, you still have to set-up load balancers, configure multiple replicated database servers, implement scalability hacks if things grow too fast (such as distributed caching of data via memcached), keep distros and apps up-to-date, etc. Bottom Line: EC2-based companies still require sys admins, AppEngine companies don’t. That will certainly change as more companies begin offering EC2 server management services.

Google provides a non-relational datastore and that’s the only datastore available (no traditional file system, no relational databases). With EC2, people generally use MySQL or Postgresql. Amazon offers a non-relational datastore called SimpleDB, but it’s a bit *too* simple. For example, it does not support sorting of results sets. Huh? That makes it non-workable in my opinion. There’s also an issue with using EC2 virtual machines for your database servers — Amazon says that when a virtual machine crashes, all the data managed by it disappears, so virtual machine crash = hard drive crash.

With EC2, programmers can use any (non-Microsoft) language to develop their apps. AppEngine users must code in Python. Also, Google does not support sockets at this time. All cross-app communication must be done via HTTP.

At *this* moment in time, it would be difficult to move apps off of AppEngine. Doing that in EC2 is trivial. This, to me, is the biggest issue, as I believe it could make startups less-interesting from an acquisition perspective by anyone other than Google. This will most likely change as people develop compatibility layers. However, Google has yet to provide any information about how to migrate data from their datastore the best I can tell. If you have a substantial amount of data, you can’t just write code to dump it because they will only let any request run for a short period before they terminate it.

Some people are complaining about Google having access to their source code. I don’t see this as an issue. I’d rather have it be stored at Google than at some small hosting company.

One final nice little thing in AppEngine’s favor: Websites that store less than 500MB of data and get roughly 5MM pageviews per month or less can use AppEngine for free. The downside is that Google has yet to say what they’ll charge if apps go over that quota, but I have to believe that it will be reasonable.  Right now, you’re prevented from going above the free-level quotas.

If you are into this and have other good links, please leave a comment with them.

  • http://www.brianberliner.com/ Brian Berliner

    [Ugh. Sorry. Last time. Really. 3 Timeouts...]

    I've written numerous articles on Amazon EC2, and took a close look at Google App Engine. You are right that there is a switching cost (and some risk surrounding that at the present), but for the vast majority of the new class of Web-based startups, Google App Engine clearly wins. Hands down. And, I'm a big EC2 fan.

    Most recent articles about Google App Engine:
    http://www.brianberliner.com/2008/04/09/an-exampl
    http://www.brianberliner.com/2008/04/09/google-ap

    Enjoy!

    -Brian

  • http://www.brianberliner.com/ Brian Berliner

    [Ugh. Sorry. Last time. Really. 3 Timeouts...]

    I've written numerous articles on Amazon EC2, and took a close look at Google App Engine. You are right that there is a switching cost (and some risk surrounding that at the present), but for the vast majority of the new class of Web-based startups, Google App Engine clearly wins. Hands down. And, I'm a big EC2 fan.

    Most recent articles about Google App Engine:
    http://www.brianberliner.com/2008/04/09/an-exampl
    http://www.brianberliner.com/2008/04/09/google-ap

    Enjoy!

    -Brian

  • http://www.brianberliner.com/ Brian Berliner

    I've written numerous articles on Amazon EC2, and took a close look at Google App Engine. You are right that there is a switching cost (and some risk surrounding that at the present), but for the vast majority of the new class of Web-based startups, Google App Engine clearly wins. Hands down. And, I'm a big EC2 fan.

    Most recent articles about Google App Engine:
    http://www.brianberliner.com/2008/04/09/an-exampl
    http://www.brianberliner.com/2008/04/09/google-ap

    Enjoy!

    -Brian

  • http://www.brianberliner.com/ Brian Berliner

    I've written numerous articles on Amazon EC2, and took a close look at Google App Engine. You are right that there is a switching cost (and some risk surrounding that at the present), but for the vast majority of the new class of Web-based startups, Google App Engine clearly wins. Hands down. And, I'm a big EC2 fan.

    Most recent articles about Google App Engine:
    http://www.brianberliner.com/2008/04/09/an-exampl
    http://www.brianberliner.com/2008/04/09/google-ap

    Enjoy!

    -Brian

  • http://www.mindtouch.com/blog Steve Bjorg

    EC2 and App Engine are different facets of Cloud Software. They serve a different purpose and work best when put together. More about this on a recent in a recent post:
    http://www.mindtouch.com/blog/2008/04/11/google-a

  • http://www.brianberliner.com/ Brian Berliner

    I've written numerous articles on Amazon EC2, and took a close look at Google App Engine. You are right that there is a switching cost (and some risk surrounding that at the present), but for the vast majority of the new class of Web-based startups, Google App Engine clearly wins. Hands down. And, I'm a big EC2 fan.

    Most recent articles about Google App Engine:
    An Example of Why Google App Engine Will Win
    Google App Engine Body-Slams Amazon Web Services

    Enjoy!

    -Brian

  • http://www.brianberliner.com/ Brian Berliner

    I've written numerous articles on Amazon EC2, and took a close look at Google App Engine. You are right that there is a switching cost (and some risk surrounding that at the present), but for the vast majority of the new class of Web-based startups, Google App Engine clearly wins. Hands down. And, I'm a big EC2 fan.

    Most recent articles about Google App Engine:
    An Example of Why Google App Engine Will Win
    Google App Engine Body-Slams Amazon Web Services

    Enjoy!

    -Brian

  • http://blog.jeffreymcmanus.com/ Jeffrey McManus

    Who says that memcached is a “hack”? Tons of giant web properties use it every day.

    • Scott Moody

      I didn't mean that memcached is a hack — it is an amazing piece of infrastructure software. What I meant was this: the way in which memcached is sometimes applied to the problem of relational database scaling is a hack.

  • Zeroth

    Basically, GAPE(Google App Engine), is a tool for rapid deployment of an interactive website. Because it supports Python(and by extension, Django) Basically… it can give a startup first mover advantage. It lets someone build a Digg-like site, or a wiki site in almost literally hours. Again, this is great, lowers costs of startups, lets VCs like yourself fund more startups, and logically make more money(You get a return on about 1/10 of all startups… if you fund more… more will succeed. )

    EC2 is scalable, cheap(ridiculously cheap), and yes, you still do need sysadmins. But I consider EC2 a second stage solution. Basically, GAPE lets people get set up, get their ideas operational quickly, and thus, hopefully, making money. Once they reach that point, and want to scale up, then they can move to EC2. Of course, the third stage is where they run localized data centers all over the world like Google. Most startups will never reach that point.

    One of the old Paul Graham articles(Older is better with him… his new stuff isn't so good) mentioned one key difference between his company and the competition. They used Lisp, which made them faster programmers. They used a better technology, to move faster. This is, I think, part of what makes a startup succeed or fail.

  • http://www.naffziger.net/blog Dave Naffziger

    Good writeup. It is worth noting that Amazon *just* announced persistent storage for EC2 instances:____http://www.allthingsdistributed.com/2008/04/persi

  • http://blog.gardeviance.org Simon Wardley

    The real innovation in GoogleAppEngine is the provision of the Open SDK which is basically an open sourced standard. This open SDK can be re-implemented on other environments, even AWS, and hence is a first real step towards portability in the SaaS, PaaS, FaaS or whatever other aaS world.

    This or that language, this or that platform, misses the point by miles of what GAE can really become.

    http://blog.gardeviance.org/2008/04/run-rabbit-ru

  • http://blog.gardeviance.org Simon Wardley

    The real innovation in GoogleAppEngine is the provision of the Open SDK which is basically an open sourced standard. This open SDK can be re-implemented on other environments, even AWS, and hence is a first real step towards portability in the SaaS, PaaS, FaaS or whatever other aaS world.

    This or that language, this or that platform, misses the point by miles of what GAE can really become.

    http://blog.gardeviance.org/2008/04/run-rabbit-ru

  • Paul Freet

    For a 'real' web services company, Amazon is vastly superior. Here are just a few MAJOR issues with GAE: no cron jobs, no SSL!, must use Google login, no image processing, hard fail when 'over-quota'. I am sure there are more, but this is enough to say no way Google.

  • http://theconvergingnetwork.com Mitchell Ashley

    GAE is all about getting developers to use Google's toolkit of APIs and build software on Google's platform. Your friend Scott Moody is right that it locks developers into Google's platform but in the long run that may or may not be such a bad thing.

    GAE is more akin to what Salesforce did with force.com, letting you create applications with their own technology. It's not portable, but if the ecosystem is alive and well portability might not be as important. As a general development platform for startups, GAE could be too limiting unless it is some type of play around Google.

    I see GAE as more of the beginnings by Google to create third party applications around Google's own offerings, building a much richer ecosystem. It may also be a good way to help Google Android phone app developers build the cloud-side of their offerings on Google's cloud. GAE could enable many parts of Google's strategy. Rumors are that it took Ray Ozzie quite by surprise but I don't know if that's true or not.

    I also blogged about GAE when it was announced. http://www.networkworld.com/community/node/26719

  • http://intensedebate.com/people/mitchell_as5509 mitchell_as5509

    GAE is all about getting developers to use Google's toolkit of APIs and build software on Google's platform. Your friend Scott Moody is right that it locks developers into Google's platform but in the long run that may or may not be such a bad thing.

    GAE is more akin to what Salesforce did with force.com, letting you create applications with their own technology. It's not portable, but if the ecosystem is alive and well portability might not be as important. As a general development platform for startups, GAE could be too limiting unless it is some type of play around Google.

    I see GAE as more of the beginnings by Google to create third party applications around Google's own offerings, building a much richer ecosystem. It may also be a good way to help Google Android phone app developers build the cloud-side of their offerings on Google's cloud. GAE could enable many parts of Google's strategy. Rumors are that it took Ray Ozzie quite by surprise but I don't know if that's true or not.

    I also blogged about GAE when it was announced. http://www.networkworld.com/community/node/26719

  • http://theconvergingnetwork.com Mitchell Ashley

    GAE is all about getting developers to use Google's toolkit of APIs and build software on Google's platform. Your friend Scott Moody is right that it locks developers into Google's platform but in the long run that may or may not be such a bad thing.

    GAE is more akin to what Salesforce did with force.com, letting you create applications with their own technology. It's not portable, but if the ecosystem is alive and well portability might not be as important. As a general development platform for startups, GAE could be too limiting unless it is some type of play around Google.

    I see GAE as more of the beginnings by Google to create third party applications around Google's own offerings, building a much richer ecosystem. It may also be a good way to help Google Android phone app developers build the cloud-side of their offerings on Google's cloud. GAE could enable many parts of Google's strategy. Rumors are that it took Ray Ozzie quite by surprise but I don't know if that's true or not.

    I also blogged about GAE when it was announced. http://www.networkworld.com/community/node/26719

  • Charlie O'Keefe

    I'd be surprised, and very disappointed, if Google didn't offer a way to get your application's data out of Data Store. After all, they have differentiated themselves by letting you easily get your email out of Gmail. Other webmail providers, notably Microsoft, have not been so accommodating. That's been a point of pride for Google. Not allowing this with App Engine would quite literally go against everything they stand for.

    As for the programming interface they provide, it makes perfect sense to me. They've designed a scalable architecture that works a certain way, and if you want to use their cloud, why shouldn't you expect to be writing for that architecture? I expect the short request constraint will be a non issue once they add support for submitting your own MapReduce jobs against your data store objects.

    There are open source projects already modeled after MapReduce and BigTable (Hadoop, HBase and Hypertable for instance), and I fully expect that the App Engine API will be duplicated on other cloud computing hosting companies, either by startups acting as intermediaries or by the hosting companies themselves.

    Another thing I expect to see are python tools (plugins to Django perhaps, or even new python web frameworks) made to support App Enigne. There is no longer a need for object-relational mapping libraries, to name the biggest difference in the platform. Another tool that I hope will be widely adopted would be an OpenID plugin. There is no reason why an App Engine application needs to tie itself to Google Accounts – that has simply been provided by Google as the default, built-in option for user accounts. It should be fairly straightforward to write an application, for example, that lets users sign up with their choice of an OpenID account or a Google account. Now that's a nice thought!

    • Craig

      take at look at cohesiveft. They have already implemented the APP Engine SDK on AWS

  • http://www.friarminor.com friarminor

    Cloud wars just starting and other big players are ready to jump in at this point. Expect open source a another big winner in all of these offerings but why do i get the feeling it won't be too open in the future once the big hands start dipping in?

    Best.
    alain

  • http://elliotross.wordpress.com Elliot Ross

    I agree with Mitchell – It is perfectly aligned with their strategy. You are in Google Docs or what have, you, and a simple link takes you to a Project Management tool, another one into a work force tool etc – Microsoft will be looking over their shoulder :-)

  • http://www.batterylaptoppower.com battery

    [...]The offline scripting tool might add some relief here, but it’s still sort of a pain. I’ve had to separate “build” from “configuration” steps.[...]

  • http://intensedebate.com/people/zeroth8105 zeroth8105

    Basically, GAPE(Google App Engine), is a tool for rapid deployment of an interactive website. Because it supports Python(and by extension, Django) Basically… it can give a startup first mover advantage. It lets someone build a Digg-like site, or a wiki site in almost literally hours. Again, this is great, lowers costs of startups, lets VCs like yourself fund more startups, and logically make more money(You get a return on about 1/10 of all startups… if you fund more… more will succeed. )

    EC2 is scalable, cheap(ridiculously cheap), and yes, you still do need sysadmins. But I consider EC2 a second stage solution. Basically, GAPE lets people get set up, get their ideas operational quickly, and thus, hopefully, making money. Once they reach that point, and want to scale up, then they can move to EC2. Of course, the third stage is where they run localized data centers all over the world like Google. Most startups will never reach that point.

    One of the old Paul Graham articles(Older is better with him… his new stuff isn't so good) mentioned one key difference between his company and the competition. They used Lisp, which made them faster programmers. They used a better technology, to move faster. This is, I think, part of what makes a startup succeed or fail.

  • Jeffrey McManus

    Who says that memcached is a "hack"? Tons of giant web properties use it every day.

  • Paul Freet

    For a 'real' web services company, Amazon is vastly superior. Here are just a few MAJOR issues with GAE: no cron jobs, no SSL!, must use Google login, no image processing, hard fail when 'over-quota'. I am sure there are more, but this is enough to say no way Google.

  • Scott Moody

    I didn't mean that memcached is a hack — it is an amazing piece of infrastructure software. What I meant was this: the way in which memcached is sometimes applied to the problem of relational database scaling is a hack.

  • http://intensedebate.com/people/charlie_o038903 charlie_o038903

    I'd be surprised, and very disappointed, if Google didn't offer a way to get your application's data out of Data Store. After all, they have differentiated themselves by letting you easily get your email out of Gmail. Other webmail providers, notably Microsoft, have not been so accommodating. That's been a point of pride for Google. Not allowing this with App Engine would quite literally go against everything they stand for.

    As for the programming interface they provide, it makes perfect sense to me. They've designed a scalable architecture that works a certain way, and if you want to use their cloud, why shouldn't you expect to be writing for that architecture? I expect the short request constraint will be a non issue once they add support for submitting your own MapReduce jobs against your data store objects.

    There are open source projects already modeled after MapReduce and BigTable (Hadoop, HBase and Hypertable for instance), and I fully expect that the App Engine API will be duplicated on other cloud computing hosting companies, either by startups acting as intermediaries or by the hosting companies themselves.

    Another thing I expect to see are python tools (plugins to Django perhaps, or even new python web frameworks) made to support App Enigne. There is no longer a need for object-relational mapping libraries, to name the biggest difference in the platform. Another tool that I hope will be widely adopted would be an OpenID plugin. There is no reason why an App Engine application needs to tie itself to Google Accounts – that has simply been provided by Google as the default, built-in option for user accounts. It should be fairly straightforward to write an application, for example, that lets users sign up with their choice of an OpenID account or a Google account. Now that's a nice thought!

  • http://intensedebate.com/people/dave_naffzi2081 dave_naffzi2081

    Good writeup. It is worth noting that Amazon *just* announced persistent storage for EC2 instances:____http://www.allthingsdistributed.com/2008/04/persi

  • http://intensedebate.com/people/steve_bjorg3568 steve_bjorg3568

    EC2 and App Engine are different facets of Cloud Software. They serve a different purpose and work best when put together. More about this on a recent in a recent post:
    http://www.mindtouch.com/blog/2008/04/11/google-a

  • http://intensedebate.com/people/friarminor8877 friarminor8877

    Cloud wars just starting and other big players are ready to jump in at this point. Expect open source a another big winner in all of these offerings but why do i get the feeling it won't be too open in the future once the big hands start dipping in?

    Best.
    alain

  • Elliot Ross

    I agree with Mitchell – It is perfectly aligned with their strategy. You are in Google Docs or what have, you, and a simple link takes you to a Project Management tool, another one into a work force tool etc – Microsoft will be looking over their shoulder :-)

  • Craig

    take at look at cohesiveft. They have already implemented the APP Engine SDK on AWS

  • battery

    [...]The offline scripting tool might add some relief here, but it’s still sort of a pain. I’ve had to separate “build” from “configuration” steps.[...]