Big Data Bootstrapping Beware

I suppose this a dumb observation, but the one thing I learned in building ZenThousand is that bootstrapping a big data startup can be expensive. Obviously, it’s due to all of that data you have to deal with before having a single user.

First, there’s the problem of collecting the data. In the case of ZenThousand, I am looking for social network profiles of programmers. Although sites like Github, LinkedIn, and others have collected a treasure trove of personal details on engineers, it’s not like they are just going to let you walk in and take it!

In LinkedIn’s case, over half of their revenue is from their recruiting features. Essentially, the information they have on you is worth nearly a billion dollars. Which is why use of their people search API has strict licensing restrictions. Github and other social sites don’t let you do simple searches for users either–you have to use what tools they give you in their API to sniff info out.

Collecting your own data can be very expensive. Data intensive services such as mapping require massive effort. This is why companies sitting on large datasets are so valuable. People scoff at Foursquare’s valuation, but while the app might not have great user numbers the location database they’ve built is of immense value.

Secondly, there’s the cost to store and process all of this data. With most startups, the amount of data you store is directly proportional to the amount of users you have. Scaling issues become a so-called “good problem to have” as it usually means your app has a lot of traction. If these are paying users, even better–Your data costs are totally covered.

With a big data startup, you have massive amounts of data to store and process with no users. This gets costly really fast. In my case, Google App Engine service fees quickly became prohibitively expensive. My future strategy involves moving off of GAE and on to either Google Compute Engine or a physical box. I know of at least one big data startup that migrated out of the cloud to a colocation facility for both cost and performance reasons.

This doesn’t mean big data isn’t possible without a large investment. It’s just that two of the first big problems you need to solve are how to cost-effectively collect and analyze lots of data before you have any revenue.

Donut Vision: Google App Engine Experiments 2

Some (well, very few) of you may remember my previous post on Google App Engine. Developing a GAE app using JSP was a trip down memory lane, using a technology that has seemingly been left unchanged since 2001.

I recently began a project that involves using Vine and Twitter to sort through video clips. I decided to build on Google App Engine again. This time I’m using Python. My initial hacking has resulted in Donut Vision–a search portal for donut videos on Vine. Hey, don’t laugh. These guys are trying to build an actual business off of the same type of sites–Presumably with cokehead money.

Using Python (GAE’s original language) has been an absolute pleasure. On GAE, it really does seem much faster than using Java. GAE’s built in webapp2 framework and Django templates make building sites and APIs a breeze. I swear not having to type brackets has given me some kind of minor productivity boost–Or not. But placebo is a real thing.

My general “get off my lawn” nitpicks with Python are mostly due to it being a weird hybrid of a dynamic language, yet strongly typed. This gives PyDev in Eclipse a problem performing autocomplete since it really doesn’t know what type you’re referring to in most cases. PyDev and Eclipse is a decent combination due to the convenience of deploying to GAE within the IDE. I’d switch to something else with better autocomplete support, though.

As for the details of how this works, it’s really pretty simple. There’s no Vine API yet, so I simply use the Twitter API to search for Vines with relevant hashtags and pull the URLs out of them. Originally I was using Vine’s new embed code to display videos, but I eventually resorted to grabbing the URL of the MP4 file in the S3 bucket it’s stored in to have more control over the video when playing it with video-js. I expect Vine to shut down this method since I’m just running up their AWS bill with no benefit to them–not even a link back to the Vine app. Hey, if Vine provides a proper API, I’d use it.

Oh also, in my earlier post I stated that Google App Engine is not available in China. This is only partially true. The default appspot domain is indeed blocked in China. Yet, when putting my custom domain, donuts.pw, through GreatFirewallOfChina.org I get nothing but green status. Yes, I’m boldly sparking a democratic revolution one French Cruller at a time. So, if you want to serve Chinese customers via GAE, just map a custom domain to it.

I’m seriously considering using Google App Engine as a backend for a new game. The only problem is cost estimation. I have constant paranoia of real-world usage patterns running up my bill. Especially with improperly indexed datastores, you can rack up charges pretty fast. Still, simply writing an app and uploading it to Google’s cloud is significantly easier than fiddling with Amazon Web Services and Beanstalk. If you haven’t checked it out since the early days, GAE is worth another look.

Oh, also the latest version of GAE has sockets support. It’s still experimental, but this may lead to GAE being suitable for real-time applications such as multiplayer game servers.

Google App Engine Experiments

For a while I’ve been complaining about the fact that sites such as AppAnnie and AppFigures don’t send daily summary emails of not just your apps, but the top apps in the App Store. I want to know what’s trending and topping the charts every day.

I could have made something in PHP to do this in a matter of hours, but I like to use side projects to learn something new. As an excuse to learn Google App Engine I built UpTopR: a site that emails a daily summary of the top 10 apps for iOS and iPad. It’s slow and ugly, but does what I need it to.

I used the Java API since I couldn’t find a way to deploy Python projects to GAE as easily as the Google plug-in for Eclipse does. I only had to learn how to use Google’s NoSQL App Engine Datastore and caching APIs. Otherwise, getting up and running on GAE is as easy, if not easier, than deploying a servlet on Tomcat. The whole process of learning GAE and finishing the app took about 4 days.

I’m big on PaaS now. Writing an application that magically scales inside Google’s environment is much easier than managing a cluster of EC2 instances as virtual infrastructure. Of course, writing a giant scaling servlet isn’t appropriate for a lot of tasks–but for the back-end of an asynchronous mobile game it makes a lot of sense.

Although last year’s pricing changes caused a revolt with long time GAE users, low traffic applications fall under the free usage quotas. Noodling around on GAE costs you nothing. This is great for prototyping.

Unfortunately, Google App Engine doesn’t work in China. The vast majority of IAPs in China are fraudulent, but China is kind of a big deal. Also, as useful as Google’s Datastore is, it still can’t search using geolocation without some suspect hacks. Amazon Web Services is available in China, and I can attach any kind of database I want to Amazon’s GAE equivalent, Beanstalk. This includes the geohash-supporting MongoDB. For these reasons I’m most likely going to use Amazon’s Beanstalk as a GAE alternative on future projects.

PROTIP: I had this problem for a while when trying to use uptopr.com as the domain for the app. Here’s what you have to know about using custom domains for GAE apps:

  • Only domain aliases for your main domain your App Engine Account is hosted on can be used with Google App Engine apps.

  • For mysterious reasons, naked domains can’t be used. You have to use a subdomain such as www.uptopr.com and use a URL redirect to point the naked domain at the subdomain.

  • Once your domain alias is registered with Google Apps, you have to type in the main active domain the alias is for on the Domain settings page for the GAE app. Then it will direct you to your Google Apps administration panel where you will be able select the alias from a dropdown.

I wish I knew this earlier! It took a few days of banging my head against a wall to figure out how to host my App Engine app on a custom domain.