Waxy.org is the sandbox of Andy Baio, an
independent journalist and programmer living in
Portland, Oregon. I created One of the biggest criticisms of Google's App Engine have been cries of lock-in, that the applications developed for the platform won't be portable to any other service. This morning, Chris Anderson, the Portland-based cofounder of the Grabb.it MP3 blog service, just released AppDrop — an elegant hack proving that's not true.
AppDrop is a container for applications developed with the Google App Engine SDK, running entirely on Amazon's EC2 infrastructure. Just like Google's Appspot, anyone can use a modified SDK to deploy their App Engine apps directly to Amazon EC2 instead of Google, and they work without modification.
This proof-of-concept was built in only four days and can be deployed in virtually any Linux/Unix hosting environment, showing that moving applications off Google's servers isn't as hard as everyone thought.
How does it work? Behind the scenes, AppDrop is simply a remote installation of the App Engine SDK, with the user authentication and identification modified to use a local silo instead of Google Accounts. As a result, any application that works with the App Engine SDK should work flawlessly on AppDrop. For example, here's Anderson's Fug This application running on Google App Engine and the identical code running on EC2 at AppDrop.
Of course, this simple portability comes at the cost of scalability. The App Engine SDK doesn't use BigTable for its datastore, instead relying on a simple flat file on a single server. This means issues with performance and no scalabity to speak of, but for apps with limited resource needs, something as simple as AppDrop would work fine.
I spoke to Chris this morning about his project and where he wants it to go. "AppDrop is open-source just like the Google SDK, so I'm hoping someone will come along and take it to the next level," he said. "It wouldn't be hard for a competent hacker to add real database support. It wouldn't be that hard to write a Python adapter to MySQL that would preserve the BigTable API. And while that wouldn't be quite as scalable as BigTable, we've all seen that MySQL can take you pretty far. On top of that, you could add multiple application machines connecting to the central database, and load-balancing, and all that rigamarole."
While this is only a hack, it demonstrates that App Engine developers don't need to live in fear of Google's reprisal. "The upshot is that if you put a lot of time into an App Engine app, and then run afoul of Google, you have alternatives, even if they are more work."
Update: Chris announced the project on his own blog, with some design notes.
April 20, 2008: Addressing concerns with data lock-in, Google announced that large-scale data import and export for will be coming soon to App Engine.
1:18 PM
Interesting hack, cleverly done by Mr. Anderson. Wonder how long it'll take for someone to make this even easier, and 'fix' the scalability problem. Not long, I'm betting.
1:20 PM
I love when stuff like this comes along: all the hand wringing just goes away because someone got off their ass and wrote some code.
1:21 PM
And presumably if you were very clever you would swap out the flat file for Hbase, or Hypertable so you don't have to pay the penalty of metaphor mismatch.
1:43 PM
Shouldn't be too hard to port BigTable class interface to Amazon SimpleDB!
2:19 PM
What about data? If you have a bunch of users of your app on Google, will you be able to get your data out and use it on EC2 instead?
2:28 PM
Thanks for the write-up, Andy. As far as data export goes, I think the standard we'll see is people writing web-service layers with XML, JSON, or CSV export support, into their App Engine apps. Then they can download from big table and upload to App Drop, or vice versa, without worrying about the backend.
3:19 PM
I think (I believe) this misses the point of the lock-in concerns.
Porting the python/wsgi code is *trivial*
Getting the data that is already in Big-Table out, and migrating away is *not* especially when there is a 10 second time out on any script that runs on GAE...
So this doesn't really prove anything at all -- in fact it answers the wrong question.
3:22 PM
In what way is this an exclusive? Chris already posted to the appengine list & blogged about it himself.
3:27 PM
Edward - After Andy encouraged me to go through with this hack on Friday, I told him I'd give him the scoop on it, so I waited to announce it until his article went up. I guess that makes it exclusive. Can't keep a secret forever...
4:16 PM
Edward: What Chris said. I was the first to write about AppDrop and the first to interview him about it. And, for a short time after I posted, my site was the first result for "AppDrop," since the official site hadn't been indexed yet. Exclusive!
4:44 PM
Except of the DB storage part there is nothing really new in the appengine. It is a packaging of python libraries and python is a very portable language. If you are not planning to use Google for storage you may as well use a more advanced open source python framework like web2py (http://mdp.cti.depaul.edu) which can seamlessly switch between MySQL, PostgreSQL and Oracle. Here is a video: http://www.vimeo.com/875433
5:32 PM
I wonder how hard it would be to graft hypertable onto it?
5:37 PM
This is missing the point. The whole idea of App Engine is that it complements EC2. See blog post about Cloud Software here: http://is.gd/5wY
9:42 PM
This is cool. I like Greg's comment: all hand wringing goes away because someone got off his ass. Isn't this always the way!
12:56 AM
Put me in the camp that thinks the hand wringing is kind of weird.
First of all, I don't believe that anyone creating applications that live in the cloud expects perfect portability from platform to platform. Whether you're living on Ning, Amazon, AppEngine, or something else, you're working in a sandboxed API that might just happen to look like an existing language runtime. If you're not willing to bind to a provider's cloud abstraction API, then that begs the question of what is acceptable and has semantics that magically port to the universe at large?
Secondly, I fail to see how the complaint in the linked arstechnica article is addressed at all by being able to dump a Google AppEngine app onto EC2. Don't you need to pay for EC2 from the get-go? Additionally, who is to say that the costs that Google will charge will be priced monopolistically instead of competitively with other cloud platforms? It will probably be enough to keep people from abusing its storage and processing capacity, but it will also probably be inexpensive enough to consider seriously, if a developer is truly serious about a project. At least if they're not, hopefully Google will be kind enough to show a "Out of Resources" screen instead of running up a tremendous overage on your account.
So, to me, this is all kind of a silly argument. Promoting the idea that all cloud platforms should speak some form of scripting and data Esperanto is probably just about as effective as filing a bug against the MySQL developers asking them to "work like Oracle."
6:58 AM
I have to agree with the naysayers. Porting the code will be the least of your problems if you start a popular web service on AppEngine and then want to leave.
You'll be forever tied to using google accounts, the google database etc... which is all they cared about anyway.
So while I like the idea of the AppEngine for small projects and as a learning sandbox, there are still many serious considerations you should make before deciding to use it to host a big time project.
8:33 AM
Does using GAE require one to use Google accounts for authentication (or is it optional)? I haven't seen this in the documentation yet.
1:20 PM
The scalability issue is a perfect candidate for the Google Summer of Code ;))