The State of Augmented Reality

My new app is out: Camera Birds–Simultaneous release on iOS and Android. Although, I have been working on Augmented Reality projects for quite some time, this is the first game I’ve released using AR with Qualcomm’s Vuforia technology. I have to say, Vuforia is the best written Unity3D plug-in I’ve ever worked with. It’s extremely easy to use, fast, and works without a hitch on both Android and iOS–no changes needed.

Anyway, I figured I’d jot down some notes about AR and its possible future while my Camera Birds experience is still fresh.

I’m specifically talking about Augmented Reality on phones. AR glasses such as Google’s Project Glass and whatever Valve is cooking up have a completely different set of challenges–most notably rendering graphics not over video, but directly on top of your vision. This requires absolutely no latency and the additional difficulty of drawing graphics over a transparent display (glasses). It sounds like a fun project–ever since playing Heavy Rain I’ve wanted to build Norman Jayden’s ARI goggles in real life.

So, those notes…

We Are Still 2 Hardware Generations Away From AR Being Good

Around the release of the iPad 2 we started getting mobile devices that could run AR robustly enough to be truly usable. Slower devices such as earlier iPhones or even the 3DS suffer from finicky tracking and recognition performance. The iPhone 4S is even smoother, but in order to have enough processing power left over to do something interesting while recognizing and tracking objects in video we’re going to need better cameras and more cores to crunch through the data coming from them.

We Need More Sensors

Recognizing and tracking objects in 2D video has been around for a long time. In fact, a lot of the algorithms used for AR grew out of technologies developed for computer based match moving in movies such as Forrest Gump. Instead of carefully recorded footage, we’re dealing with a much more unpredictable real-time source of video: handheld cameras on phones and tablets.

This introduces all sorts of problems such as motion blur, lighting changes, and specular highlights that can trip up recognition and tracking algorithms. Although there are some decent solutions to this problem, newer AR technologies often go beyond visual tracking. Compass and gyroscope hardware in modern mobile devices can now be used to compensate for visual tracking loss. We still need more.

If Microsoft could shrink the Kinect down enough to include in a mobile phone, AR would take a big leap. Having a depth camera eliminates a lot of AR’s issues with visual tracking. Also, using stereo cameras can help determine the real-world size of objects in video. Most solutions currently require you to manually specify the size of glyphs in real-world units instead of determining this on the fly.

Markers Suck

Most AR solutions right now require markers or pre-set images for natural feature tracking. This means the system is trained on a 2D image (such as a QR Code or trading card) that can be recognized and tracked in footage. For this reason, most AR games on phones require you to print out markers in order to use the app.

There are markerless technologies in AR, most notably the so-called SLAM algorithm which has popped up in a few examples such as’s licensable library, Ogmento’s tank demo, and Sony’s Vita air hockey game from this past GDC. In these examples, arbitrary details in video are tracked geometry is built out of them. (I say so-called because there’s more at work here, but let’s not get nerdy!)

In most cases this means finding a simple plane out of points of contrast on a flat surface such as a tablecloth or a poster. This geometry is then used as a collision volume in a 3D game played over the video feed. The system also remembers where features were relative to the user’s position–so when details are lost due to movement, they can be easily re-tracked when moved back into view. An example you can download and play right now is 13thLab’s Ball Invasion–it came out last Summer and is still the only good example in the App Store.

Markers still have their place. Cards can be a collectible gimmick used for monetization (see Nintendo’s use of them in Kid Icarus for the 3DS) as well as a useful tool to instruct users on how to properly use an AR app. Still, they usually are a burden and limit the audience.

AR Is Still A Gimmick

Advertisers love AR because it involves users having to physically interact with a product: pick up a cereal box and you are 3/4 of the way to buying it. For most applications however, AR is still a gimmick. Having to wave your phone around at an object to play a game is a huge hassle. A compelling business case hasn’t been made yet–this is evidenced by the fact that most AR conferences feel more like amateur science fairs than bustling hives of commerce. Although, at the pace Vuforia has been advancing, I’m sure Qualcomm are working hard to change this.

One thought on “The State of Augmented Reality

  1. Good post… Perhaps a bit harsh towards VR/AR researchers and conferences. When I go to conferences like that, I like to get that “science fair” feel: research is at the edge, and usually, if it is too polished, it is probably not research anymore.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s