One of my pet peeves with Google App Engine is its horrible support for geospatial indexes. Although you can store a GeoPt in the Datastore, you can’t really query it. You can use various hacks such as Geomodel, but they end up being slow and potentially expensive.
Last year Google released the beta API for Google App Engine search. This lets you search documents for text, HTML, numbers, dates, and location. However, it searches documents instead of datastore entities.
If documents are separate from your datastore, how do you use the new search API to do geospatial queries on your database? Simply store the location for each entity inside a document instead. To do this, make a document with a GeoField with the location and a string that contains the id of the associated datastore entity’s key (this code is based on Google’s own location example):
key = str(entity.key())
geopoint = search.GeoPoint(lat, lng)
index = search.Index(_INDEX_NAME)
document = search.Document( fields=[search.TextField(name='key', value=str(key)),
search.GeoField(name='loc', value=geopoint)])
search.Index(name=_INDEX_NAME).put(document)
Note that you have to store the key’s id as a string since you can’t store a long in a document.
Now, when you perform a geolocation search like this:
index = search.Index(_INDEX_NAME)
query = "distance(loc, geopoint(" + str(lat) + "," + str(lng) + ")) < 1000"
try:
results = index.search(query)
for doc in results:
logging.info('Document! ' + str(doc.field("key")))
except search.Error:
logging.exception('Error!')
You can grab the id field from the document and query the datastore for it to get the rest of your data.
There are several problems with this method. First, it doesn’t work on the local dev_app_server. Currently, GeoField searches only work on the appspot production servers. Also, because the API is in beta you are restricted to the free quotas which don’t allow for very many operations. Finally, when Google reveals pricing changes, it can have disastrous results. It’s very risky to build an app using this method when you have no idea how much it costs.
At least it works! It’s still a mystery to me why they can’t add this feature to the datastore itself.
Hi Ralph, I’m working on the geospatial side at Nokia and would be interested in hearing what you’re doing in the space.
Best,
Peter
I’ve been knee-deep in geospatial tech for awhile now. This was one of my first presentations on the subject:
http://www.slideshare.net/ralphbarbagallo/ralph-login-2011
I’ll send you an email.
Pingback: Social Recruiting: Google App Engine Experiments 3 | Ralph Barbagallo's Self Indulgent Blog
We tested the api yesterday but we ran out off quota in 5 minutes with 3 users 😦 the reason:
“Search API Complex Searches 100% 100 of 100 Limited”
Yeah, the free quota makes the API useless for production. But, Google announced the paid API terms so you’ll be able to use it for production soon (for $).
I’ve been tinkering with GeoSpatial bits and this article keeps coming up when I’m searching. Thank you for writing it. One correction, geo searches do work via the sdk now. I’m not sure what release it happened, but they do indeed work – both via the api and via the _ah/ tools.
Recently, Geo Search is no longer counted against “Search API Complex Searches” on the quota page. Either logging is broken or Google have changed this aspect of their policy.
is this still the same story – these queries don’t work on the datastore directly?
Now they have Google App Engine Search, which can search by location. I wrote a blog about it here. There’s also this API page about it: https://cloud.google.com/appengine/docs/python/search/
I really would like to know:
why would I use this google search API knowing that I have to create document and indexes which for me is a kind of second database on which I have to perform queries.
Why not just use an SQL database in this case?
They do have Google Cloud SQL, which supports geospatial indexes: http://googlecloudplatform.blogspot.com/2014/05/mysql-56-now-available-in-cloud-sql.html So you can still use SQL and keep the easy scalability of Google App Engine. The Search API is for document store type data, so it’s only relevant if you are dead set on a NoSQL database.
How would one go about querying by multiple properties on top of GeoPoint without having to put all the datastore properties in a search document?
I haven’t looked at this in years, so I’m not sure. It might be a better idea to use Google Cloud SQL instead–especially now that it uses geospatial extensions.