Getting around GAE’s 1000-result limit

UPDATE: Version 1.3.1 of the SDK has removed the 1000-result (and added cursors), so there’s no longer any reason to use this technique.

When you consider  Google App Engine‘s posted hard cap on returning a maximum of 1000 results per query and their alluded-to workaround, it seems like the GAE engineers are trying to send a message. Combining the two facts there, I interpret it as saying something like this:

We know more about how GAE and BigTable work than you do, and we have a much better idea of what each operation costs. Based on the platform we’re giving you, we don’t think you want to be returning more that many results. But if you’ve thought about it hard and you’re convinced you do want to be doing this, there’s a roundabout way to do it.

I thought about it for a while in the context of a project I’m working on, switched my design around a bit (as I get my head around the differences between BigTable and SQL), and still want to get more than 1000 results. So I took the example linked to above, and implemented it in a generator for ease of use.

Read the rest of this entry »

Profiling Google App Engine

Google Appe Engine is an interesting platform for hobbyist development – it’s got good tool and documentation, and you can deploy (for free) to Google’s cloud with some pretty generous quotas. The downside, especially at the start, is that it uses a database model that you’re probably not familiar with: BigTable, as opposed to something SQL-like. Sure, BigTable has GQL, but it’s really just syntactic sugar to make you feel comfortable. There’s nothing here like GROUP BY or UPDATE, and there are hard limits on the number of results you can return from a query.

I’ve been doing some work on a submission for the Apps for America 2 contest, and have been running into the dreaded runtime._apphosting_runtime___python__apiproxy.Wait
line at the top of my profiling results – this means that GAE is stalling waiting for some API process to complete, and as a result it’s timing out. What to do about this?

Read the rest of this entry »