The Next Problems to Solve in Augmented Reality

I’m totally amped up about Project Tango. After having worked with augmented reality for a few years, most of the problems I’ve seen with current platforms could be solved with a miniaturized depth-sensing Kinect-style sensor. The Myriad 1 is a revolutionary chip that will dramatically change the quality of experience you get from augmented reality applications–both on mobile devices and wearables.

There’s a few other issues in AR I’d like to see addressed. Perhaps they are in research papers, but I haven’t seen anything real yet. Maybe they require some custom hardware as well.

Real-world lighting simulation.

One of the reasons virtual objects in augmented reality look fake is because AR APIs can’t simulate the real-world lighting environment in a 3D engine. For most applications, you place a directional light pointing down to and turn up the ambient for a vague approximation of overhead lighting. This is assuming the orientation of the object you’re tracking is upright, of course.

Camera Birds AR mode using an overhead directional light.

Camera Birds AR mode using an overhead directional light.

What I’d really like to use is Image Based Lighting. Image based Lighting is a computationally efficient way to simulate environmental lighting without filling a scene up with dynamic lights. It uses a combination of cube maps built from HDR photos with custom shaders to produce great results. A good example of this is the Marmoset Skyshop plug-in for Unity3D.

Perhaps with a combination of sensors and 360 cameras you can build HDR cubemaps out of the viewer’s local environment in real-time to match environmental lighting. Using these with Image Based Lighting will be a far more accurate lighting model than what’s currently available. Maybe building rudimentary cubemaps out of the video feed is a decent half-measure.

Which object is moving?

In a 3D engine, virtual objects drawn on top of image targets are rendered with two types of cameras. Ether the camera is moving around the object, or the object is moving around the camera. In real life, the ‘camera’ is your eye–so the it should move if you move your head. If you move an image target, that is effectively moving the virtual object.

Current AR APIs have no way of knowing whether the camera or the object is moving. With Qualcomm’s Vuforia, you can either tell it to always move the camera around the object, or to move the objects around the camera. This can cause problems with lighting and physics.

For instance, on one project I was asked to make liquid pour out of a virtual glass when you tilt the image target it rest upon. To do this I had to force Vuforia to assume the image target was moving–so then the image target tilted, so would the 3D object in the game engine and liquid would pour. Only problem is, this would also happen if I had moved the phone as well. Vuforia can’t tell what’s actually moving.

There needs to be a way to accurately track the ‘camera’ movement of either the wearable or mobile device so that in the 3D scene the camera and objects can be positioned accurately. This will allow for lighting to be realistically applied and for moving trackable objects to behave properly in a 3D engine. Especially with motion tracking advances such as the M7 chip, I suspect there are some good algorithmic solutions to factoring out the movement of the object and the observer to solve this problem.

Anyway, these are the kind of problems you begin to think about when staring at augmented reality simulations for years. Once you get over the initial appeal of AR’s gimmick, the practical implications of the technology poses many questions. I’ve applied for my Project Tango devkit and really hope I get my hands on one soon!

Ludum Dare: Ten Seconds of Thrust

This past weekend I participated in Ludum Dare 48, a contest where you make a game by yourself in 48 hours. The theme is revealed at the start of the contest–Iron Chef style. All code, graphics, and sounds have to be made from scratch. Voting began on Sunday night and will extend for a few weeks. I’m not even sure what you win, but that’s not the point. It’s an awesome experience in GETTING IT DONE.

Ten Seconds of Thrust!

My entry is the Lunar Lander-esque Ten Seconds of Thrust. (Please rate it!) Attempt to land at the bottom of increasingly difficult randomly generated space caverns with only ten seconds of thruster time. It’s crude, ugly, and buggy–especially on Windows where it doesn’t seem to detect landing. I didn’t have time to fix this bug as I only discovered it in the last half hour, but it does seem like a strange Unity Web Player bug since it works fine on OSX browsers. (PROTIP: Make sure you have a few friends around during the weekend to test your game!)

One of the best things about the contest is watching games evolve quickly through Twitter, Vine, Facebook, and Instagram posts. I put up a few videos in progress over the weekend.

I used a lot of the tools mentioned in my rapid prototyping posts, including a new tool I found called Sprite Gen which creates randomly generated animated character and tile sprites in tiny 12×12 blocks. Naturally, the game was developed in Unity along with 2DToolkit and HOTween for plug-ins.

I’d like to fix the landing bug as it makes the game useless on Windows, but the rules are somewhat unclear on bug-fixes that don’t add any content. This game was actually based on an idea for a Lunar Lander roguelike I was developing earlier this year. The LD48 version is highly simplified and way more fun. I abandoned my prototype in disgust back in February. This quick and dirty version is much better–I might run with it and make a full game.

Big Data Bootstrapping Beware

I suppose this a dumb observation, but the one thing I learned in building ZenThousand is that bootstrapping a big data startup can be expensive. Obviously, it’s due to all of that data you have to deal with before having a single user.

First, there’s the problem of collecting the data. In the case of ZenThousand, I am looking for social network profiles of programmers. Although sites like Github, LinkedIn, and others have collected a treasure trove of personal details on engineers, it’s not like they are just going to let you walk in and take it!

In LinkedIn’s case, over half of their revenue is from their recruiting features. Essentially, the information they have on you is worth nearly a billion dollars. Which is why use of their people search API has strict licensing restrictions. Github and other social sites don’t let you do simple searches for users either–you have to use what tools they give you in their API to sniff info out.

Collecting your own data can be very expensive. Data intensive services such as mapping require massive effort. This is why companies sitting on large datasets are so valuable. People scoff at Foursquare’s valuation, but while the app might not have great user numbers the location database they’ve built is of immense value.

Secondly, there’s the cost to store and process all of this data. With most startups, the amount of data you store is directly proportional to the amount of users you have. Scaling issues become a so-called “good problem to have” as it usually means your app has a lot of traction. If these are paying users, even better–Your data costs are totally covered.

With a big data startup, you have massive amounts of data to store and process with no users. This gets costly really fast. In my case, Google App Engine service fees quickly became prohibitively expensive. My future strategy involves moving off of GAE and on to either Google Compute Engine or a physical box. I know of at least one big data startup that migrated out of the cloud to a colocation facility for both cost and performance reasons.

This doesn’t mean big data isn’t possible without a large investment. It’s just that two of the first big problems you need to solve are how to cost-effectively collect and analyze lots of data before you have any revenue.

Extremely Rapid Prototyping Tools Part 2

It’s been awhile since my last post on extremely rapid prototyping tools. So I figured I’d list a few gems I’ve discovered along the way. A lot of times having the right tools to create placeholder content, find clip art, or other resources can get you over the hump when trying to create a quick prototype. Here’s some tools that can help.

placehold.it

A Placeholder!

When laying out HTML/CSS, or otherwise mocking up interfaces, sometimes you need placeholder icons, pictures, and other bitmaps. I discovered this incredibly simple to use and absolutely ingenious site awhile back, placehold.it, which does just that. You simply pass the dimensions of the bitmap to the URL like placehold.it/350×150 and it will create a bitmap of that size. Why didn’t I think of this?

autotracker.py

Need temp music for your game but don’t want to find a composer or pay for library tracks? Easy, use autotracker. This free python script will create surprisingly good randomly generated chiptune music in IT tracker format. This is a very compact format that can be used directly in engines such as Unity or converted to MP3 or WAV files. It even randomly generates a name for each composition.

sfxr / Cfxr / Bfxr

Need sounds for your prototype? These open source tools will create randomly generated 8-bit style sound effects using tunable parameters and sound types (power up, damage, jump, etc.) I find the quality of the flash version, Bfxr, to be the best. Especially if you are doing a retro 8-bit style game, as so many neckbeardy hipsters are, these sounds are good enough for production use too. I’ve been toying around with the idea of porting it to C# as a Unity3D editor extension.

App Engine Geospatial Datastore Search: A New Way!

One of my pet peeves with Google App Engine is its horrible support for geospatial indexes. Although you can store a GeoPt in the Datastore, you can’t really query it. You can use various hacks such as Geomodel, but they end up being slow and potentially expensive.

Last year Google released the beta API for Google App Engine search. This lets you search documents for text, HTML, numbers, dates, and location. However, it searches documents instead of datastore entities.

If documents are separate from your datastore, how do you use the new search API to do geospatial queries on your database? Simply store the location for each entity inside a document instead. To do this, make a document with a GeoField with the location and a string that contains the id of the associated datastore entity’s key (this code is based on Google’s own location example):

key = str(entity.key())
geopoint = search.GeoPoint(lat, lng)
index = search.Index(_INDEX_NAME)
document = search.Document( fields=[search.TextField(name='key', value=str(key)),
                                                    search.GeoField(name='loc', value=geopoint)])
search.Index(name=_INDEX_NAME).put(document)

Note that you have to store the key’s id as a string since you can’t store a long in a document.

Now, when you perform a geolocation search like this:

index = search.Index(_INDEX_NAME)
query = "distance(loc, geopoint(" + str(lat) + "," + str(lng) + ")) < 1000"
     
try:
     results = index.search(query)
     for doc in results:
         logging.info('Document! ' + str(doc.field("key")))
except search.Error:
     logging.exception('Error!')

You can grab the id field from the document and query the datastore for it to get the rest of your data.

There are several problems with this method. First, it doesn’t work on the local dev_app_server. Currently, GeoField searches only work on the appspot production servers. Also, because the API is in beta you are restricted to the free quotas which don’t allow for very many operations. Finally, when Google reveals pricing changes, it can have disastrous results. It’s very risky to build an app using this method when you have no idea how much it costs.

At least it works! It’s still a mystery to me why they can’t add this feature to the datastore itself.

Donut Vision: Google App Engine Experiments 2

Some (well, very few) of you may remember my previous post on Google App Engine. Developing a GAE app using JSP was a trip down memory lane, using a technology that has seemingly been left unchanged since 2001.

I recently began a project that involves using Vine and Twitter to sort through video clips. I decided to build on Google App Engine again. This time I’m using Python. My initial hacking has resulted in Donut Vision–a search portal for donut videos on Vine. Hey, don’t laugh. These guys are trying to build an actual business off of the same type of sites–Presumably with cokehead money.

Using Python (GAE’s original language) has been an absolute pleasure. On GAE, it really does seem much faster than using Java. GAE’s built in webapp2 framework and Django templates make building sites and APIs a breeze. I swear not having to type brackets has given me some kind of minor productivity boost–Or not. But placebo is a real thing.

My general “get off my lawn” nitpicks with Python are mostly due to it being a weird hybrid of a dynamic language, yet strongly typed. This gives PyDev in Eclipse a problem performing autocomplete since it really doesn’t know what type you’re referring to in most cases. PyDev and Eclipse is a decent combination due to the convenience of deploying to GAE within the IDE. I’d switch to something else with better autocomplete support, though.

As for the details of how this works, it’s really pretty simple. There’s no Vine API yet, so I simply use the Twitter API to search for Vines with relevant hashtags and pull the URLs out of them. Originally I was using Vine’s new embed code to display videos, but I eventually resorted to grabbing the URL of the MP4 file in the S3 bucket it’s stored in to have more control over the video when playing it with video-js. I expect Vine to shut down this method since I’m just running up their AWS bill with no benefit to them–not even a link back to the Vine app. Hey, if Vine provides a proper API, I’d use it.

Oh also, in my earlier post I stated that Google App Engine is not available in China. This is only partially true. The default appspot domain is indeed blocked in China. Yet, when putting my custom domain, donuts.pw, through GreatFirewallOfChina.org I get nothing but green status. Yes, I’m boldly sparking a democratic revolution one French Cruller at a time. So, if you want to serve Chinese customers via GAE, just map a custom domain to it.

I’m seriously considering using Google App Engine as a backend for a new game. The only problem is cost estimation. I have constant paranoia of real-world usage patterns running up my bill. Especially with improperly indexed datastores, you can rack up charges pretty fast. Still, simply writing an app and uploading it to Google’s cloud is significantly easier than fiddling with Amazon Web Services and Beanstalk. If you haven’t checked it out since the early days, GAE is worth another look.

Oh, also the latest version of GAE has sockets support. It’s still experimental, but this may lead to GAE being suitable for real-time applications such as multiplayer game servers.

Unity3D 4 Pet Peeves

I’ve been updating my older apps to use the newly released Unity3D 4 engine, as well as starting an entirely new project. I haven’t used many of Unity3D 4’s new features yet, but I figured this is as good a time as any to list a few of my pet peeves with Unity3D 4 as I did with Unity3D 3 a few years back.

It’s time Unity3D had a package manager.

Unity3D plug-ins and assets purchased from the Asset Store are invaluable. It’s becoming the most important feature that makes Unity3D the superior choice. However, managing projects with multiple plug-ins is can be a nightmare. A lot of this is how Unity3D handles file deletions.

If you click the “update” button to overwrite an existing plug-in with the latest version from the Asset Store, it may wreak havoc upon your entire project. Unity3D’s file hashing system will sometimes fail to overwrite files with the same name, even if you are importing a newer one. You’ll end up with a mess of old and new plug-in files causing chaos and mayhem. The only way to prevent this is to manually find delete all the old plug-in files before updating with the latest version.

Not to mention the fact that native plug-ins either require you to manually setup your own XCode project with external libraries or have their own proprietary scripts that edit your XCode project. Unity3D should provide an API and package manager that lets plug-ins forcibly delete and update their own files as well as modify settings in the XCode project Unity3D generates.

Let me import files with arbitrary extensions.

A minor annoyance is how Unity3D will only accept files with specific extensions in your project. If you want a custom binary data file you HAVE to give it the txt extension. It’s the only way you can drag the file in to the project. Unity3D should allow you to import files with any extension you want, but provide a method in the AssetPostprocessor API to be called when an unknown file extension is detected.

Where’s the GUI?

Come on now. It’s 2013. The new GUI has been “coming soon” for years. Unity hired the NGUI guy, which leads me to believe the mythical Unity3D 4 GUI is merely the stuff of legends and fantasies. I like NGUI but I’m really looking forward to an official solution from Unity. Although I’m not looking forward to re-writing all my GUIs once it arrives. Let’s just get it over with. Bring it on.

Monodevelop sucks.

My god. Monodevelop sucks. Lots of people use other text editors for code, but you still can’t avoid touching Monodevelop when it comes to debugging on OSX. I’m sure it can be whipped into shape with a minor overhaul, but it’s been awful for so long perhaps this is unlikely. Aside from the crashes and interface weirdness, how much human productivity has been destroyed waiting for Monodevelop to reload the solution every time so much as a single file has been moved to a different folder?

Is it time to update Mono?

While we’re at it, Mono recently updated to C# 5.0. I’m not sure if this is a big performance drag or not, but I’d love to see Unity3D’s Mono implementation updated to the latest. There are some C# 5.0 features I’ve been dying to use in Unity3D.

Tough Love

Don’t take it personally, Unity3D is still my engine of choice. This list of annoyances is pretty minor compared to previous ones. Every year, Unity gives me fewer and fewer things to whine about. It seems competing solutions are having trouble keeping up.

Google App Engine Experiments

For a while I’ve been complaining about the fact that sites such as AppAnnie and AppFigures don’t send daily summary emails of not just your apps, but the top apps in the App Store. I want to know what’s trending and topping the charts every day.

I could have made something in PHP to do this in a matter of hours, but I like to use side projects to learn something new. As an excuse to learn Google App Engine I built UpTopR: a site that emails a daily summary of the top 10 apps for iOS and iPad. It’s slow and ugly, but does what I need it to.

I used the Java API since I couldn’t find a way to deploy Python projects to GAE as easily as the Google plug-in for Eclipse does. I only had to learn how to use Google’s NoSQL App Engine Datastore and caching APIs. Otherwise, getting up and running on GAE is as easy, if not easier, than deploying a servlet on Tomcat. The whole process of learning GAE and finishing the app took about 4 days.

I’m big on PaaS now. Writing an application that magically scales inside Google’s environment is much easier than managing a cluster of EC2 instances as virtual infrastructure. Of course, writing a giant scaling servlet isn’t appropriate for a lot of tasks–but for the back-end of an asynchronous mobile game it makes a lot of sense.

Although last year’s pricing changes caused a revolt with long time GAE users, low traffic applications fall under the free usage quotas. Noodling around on GAE costs you nothing. This is great for prototyping.

Unfortunately, Google App Engine doesn’t work in China. The vast majority of IAPs in China are fraudulent, but China is kind of a big deal. Also, as useful as Google’s Datastore is, it still can’t search using geolocation without some suspect hacks. Amazon Web Services is available in China, and I can attach any kind of database I want to Amazon’s GAE equivalent, Beanstalk. This includes the geohash-supporting MongoDB. For these reasons I’m most likely going to use Amazon’s Beanstalk as a GAE alternative on future projects.

PROTIP: I had this problem for a while when trying to use uptopr.com as the domain for the app. Here’s what you have to know about using custom domains for GAE apps:

  • Only domain aliases for your main domain your App Engine Account is hosted on can be used with Google App Engine apps.

  • For mysterious reasons, naked domains can’t be used. You have to use a subdomain such as www.uptopr.com and use a URL redirect to point the naked domain at the subdomain.

  • Once your domain alias is registered with Google Apps, you have to type in the main active domain the alias is for on the Domain settings page for the GAE app. Then it will direct you to your Google Apps administration panel where you will be able select the alias from a dropdown.

I wish I knew this earlier! It took a few days of banging my head against a wall to figure out how to host my App Engine app on a custom domain.

How To Prevent Performance Spikes in Unity3D When a Model is First Visible

In my latest Unity3D app I dynamically load assets from the Resources folder and place them in the world after the initial scene load. These assets use new materials and textures that must be uploaded to the GPU. I thought I was being slick by caching prefabs to prevent a loading hiccup when I needed to instantiate. However, that’s only part of the problem. After placing the object in the scene, my game would freeze up for a frame or two when the newly created object first became visible. The profiler showed this spike attributed to a function called AwakeFromLoad.

It turns out Unity3D does not load the GPU with your new object’s assets until it’s first visible. Apparently, this is what AwakeFromLoad does. This is an optimization technique presumably to prevent thrashing on the GPU by loaded assets that won’t be visible immediately. The downside is you’ll see a pause as Unity3D uploads data to the GPU. From what I can tell, this can even mean compiling the shader if it hasn’t been used in the scene yet.

Unity doesn’t provide a function to force the GPU to load assets. From looking at Unity forum threads, the most common solution is to put up a loading screen and show newly instantiated assets to the main camera for a frame to force a GPU load. Once all the assets have been made visible, the loading screen is dropped.

Putting up a loading screen just seemed like a huge pain in the ass, not to mention an ugly hack. So, I came up with a solution using Unity Pro’s RenderTexture and a second camera. Now, my game scene has two cameras: the Main Camera and a disabled secondary camera with a tiny 32×32 RenderTexture as its target. Whenever I instantiate a new asset in the world, I position the second camera in front of it and render a frame to this texture. This forced rendering does the trick of uploading all necessary data to the GPU. Yes, there still is a loading spike, but you decide when it’s going to happen and you don’t have to reposition your object in view of the main camera for a frame.

I put this in a behavior called AssetGPULoader, you can grab it here. It only works with Unity3D Pro as it needs RenderTexture. As far as I can tell, this does the trick. It has removed my unpredictable performance spikes. For an alternative solution, I also found this technique in the Unity forums.

Extremely Rapid Prototyping Tools

Perhaps you’re competing in a hackathon or you’re trying to deliver a minimum viable product in a compressed timeframe. This list of prototyping and development resources might help you crank out wireframes, demos, and even full products quicker and cheaper.

Interface Mockups: Balsamiq

Balsamiq is the first tool I use when trying to design a UI flow for an application. It’s is a simple layout tool that lets you drag sketches of widgets, controls, and other UI elements on a canvas to quickly build mockups of interface screens.

The best thing about Balsamiq is that the end result looks like a hand drawn sketch. This way it’s clear to the UI artist that the design is just for layout and flow, not for the actual look and style. I’ve seen horrible GUI designs delivered as a result of prototyping tools that use actual widget images for mockup layouts. This can confuse artists and clients, making them think the mockup is how the interface is actually supposed to look.

The original Balsamiq mockups for Brick Buddies

Brick Buddies as mocked up in Balsamiq

Icons: iconfinder.com

If you need interface graphics or even a simple app icon for a prototype, just use this search engine. It contains a large amount of royalty-free icons you can use right away or massage in Photoshop to your liking. Iconfinder is invaluable for delivering professional looking interfaces with no graphic designers in a hackathon crunch. I wish I knew about it before doing NTheMiddle!

Sounds: freesound.org

Freesound.org is a great resource for free sounds and much easier to use than professional sound libraries you pay thousands of dollars for. The sounds here vary in quality greatly, but most have pretty loose terms that allow you to use them commercially or otherwise. If I can’t find what I’m looking for here, then I drop a few bucks on Soundsnap for more professional sound effects.

Server Back End: Parse

It seems you end up writing the same back-end code over and over for most mobile apps. In recognition of this pain point, a bunch of cloud APIs for common mobile back-end functions have popped up this year. I’m still evaluating them, but from a cursory glance I think Parse is the best.

Parse has the clearest billing terms, the easiest to use API regardless of platform (including a Unity3D plug-in!), and a solid feature set. If you need to make a user account system, send out push notifications, or store and query geolocation data, Parse is pretty easy to use. For anything real-time, you’ll need another solution.