The Next Problems to Solve in Augmented Reality

I’m totally amped up about Project Tango. After having worked with augmented reality for a few years, most of the problems I’ve seen with current platforms could be solved with a miniaturized depth-sensing Kinect-style sensor. The Myriad 1 is a revolutionary chip that will dramatically change the quality of experience you get from augmented reality applications–both on mobile devices and wearables.

There’s a few other issues in AR I’d like to see addressed. Perhaps they are in research papers, but I haven’t seen anything real yet. Maybe they require some custom hardware as well.

Real-world lighting simulation.

One of the reasons virtual objects in augmented reality look fake is because AR APIs can’t simulate the real-world lighting environment in a 3D engine. For most applications, you place a directional light pointing down to and turn up the ambient for a vague approximation of overhead lighting. This is assuming the orientation of the object you’re tracking is upright, of course.

Camera Birds AR mode using an overhead directional light.

Camera Birds AR mode using an overhead directional light.

What I’d really like to use is Image Based Lighting. Image based Lighting is a computationally efficient way to simulate environmental lighting without filling a scene up with dynamic lights. It uses a combination of cube maps built from HDR photos with custom shaders to produce great results. A good example of this is the Marmoset Skyshop plug-in for Unity3D.

Perhaps with a combination of sensors and 360 cameras you can build HDR cubemaps out of the viewer’s local environment in real-time to match environmental lighting. Using these with Image Based Lighting will be a far more accurate lighting model than what’s currently available. Maybe building rudimentary cubemaps out of the video feed is a decent half-measure.

Which object is moving?

In a 3D engine, virtual objects drawn on top of image targets are rendered with two types of cameras. Ether the camera is moving around the object, or the object is moving around the camera. In real life, the ‘camera’ is your eye–so the it should move if you move your head. If you move an image target, that is effectively moving the virtual object.

Current AR APIs have no way of knowing whether the camera or the object is moving. With Qualcomm’s Vuforia, you can either tell it to always move the camera around the object, or to move the objects around the camera. This can cause problems with lighting and physics.

For instance, on one project I was asked to make liquid pour out of a virtual glass when you tilt the image target it rest upon. To do this I had to force Vuforia to assume the image target was moving–so then the image target tilted, so would the 3D object in the game engine and liquid would pour. Only problem is, this would also happen if I had moved the phone as well. Vuforia can’t tell what’s actually moving.

There needs to be a way to accurately track the ‘camera’ movement of either the wearable or mobile device so that in the 3D scene the camera and objects can be positioned accurately. This will allow for lighting to be realistically applied and for moving trackable objects to behave properly in a 3D engine. Especially with motion tracking advances such as the M7 chip, I suspect there are some good algorithmic solutions to factoring out the movement of the object and the observer to solve this problem.

Anyway, these are the kind of problems you begin to think about when staring at augmented reality simulations for years. Once you get over the initial appeal of AR’s gimmick, the practical implications of the technology poses many questions. I’ve applied for my Project Tango devkit and really hope I get my hands on one soon!

From Bits to Atoms: Creating A Game In The Physical World

Some of you may recall last year’s post about 3D printing and my general disappointment with consumer-grade additive manufacturing technology. This was the start of my year-long quest to turn bits into atoms. Since that time there has been much progress in the technology and I’ve learned a lot about manufacturing. But first, a little about why I’m doing this, and my new project titled: Ether Drift.

Ether Drift AR App

A little over a year ago, I met a small team of developers who had a jaw-dropping trailer for a property they tried to get funded as a AAA console game. After failing to get the game off the ground it was mothballed until I accidentally saw their video one fateful afternoon.

With the incredible success of wargaming miniatures and miniature-based board game campaigns on Kickstarter, I thought one way to launch this awesome concept would be to turn the existing game assets into figurines. These toys would work with an augmented reality app that introduces the world and the characters as well as light gameplay elements. This would be a way to gauge interest in the property before going ahead with a full game production.

A lot of this was based on my erroneous assumption that I could just 3D print game models and ship them as toys. I really knew nothing about manufacturing. Vague memories of Ed Fries’ 3D printing service that made figurines out of World of Warcraft avatars guided my first steps.

3D printers are great prototyping tools. Still, printing the existing game model took over 20 hours and cost hundreds of dollars in materials and machine time. Plus, 3D prints are fragile and require a lot of hand-finishing to smooth out. When manufacturing in quantity, you need to go back to old-school molding.

You can 3D print just about any shape, but molding and casting has strict limitations. You have to minimize undercut by breaking the model up into smaller pieces that can be molded and assembled. The game model I printed out was way too complicated to be broken down into a manageable set of parts.

Most of these little bits on the back and underside would have to be individual molded parts to be re-assembled later--An expensive process!

Most of these little bits on the back and underside would have to be individual molded parts to be re-assembled later–An expensive process!

So I scrapped the idea of using an existing game property. Instead, I developed an entirely new production process. I now create new characters from scratch that are designed to be molded. This starts as a high detail 3D model that is printed out in parts that molds are made from. Then, I have that 3D model turned into something that can be textured and rigged for Unity3D. There are some sacrifices made in character design since the more pieces there are, the more expensive it is to manufacture. Same goes for the painting process–the more detailed the game texture is, the more costly it becomes to duplicate in paint on a plastic toy.

We're working on getting a simple paint job that matches the in-game texture.

We’re working on getting a simple paint job that matches the in-game texture.

So, what is Ether Drift? In short: it’s Skylanders for nerds. I love the concept of Skylanders–but, grown adult geeks like toys too. The first version of this project features a limited set of figures and an augmented reality companion app.

The app uses augmented reality trading cards packed with each figure to display your toy in real-time 3D as well as allowing you to use your characters with a simple card battle game. I’m using Qualcomm’s Vuforia for this feature–the gold standard in AR.

The app lets you add characters to your collection via a unique code on the card. These characters will be available in the eventual Ether Drift game, as well as others. I’ve secured a deal to have these characters available in at least one other game.

If you are building a new IP today, it’s extremely important to think about your physical goods strategy. Smart indies have already figured this out. The workflow I created for physical to digital can be applied to any IP, but planning it in advance can make the process much simpler.

In essence, I’m financing the development of a new IP by selling individual assets as toys while it is being built. For me, it’s also a throwback to the days before everything was licensed from movies or comic books and toy store shelves were stocked with all kinds of crazy stuff. Will it work? We’ll see next month! I am planning a Kickstarter for the first series in mid-March. Stay Tuned to the Ether Drift site, Facebook page, or Twitter account. Selling atoms instead of bits is totally new ground for me. I’m open to all feedback on the project, as well as people who want to collaborate.

The State of Augmented Reality

My new app is out: Camera Birds–Simultaneous release on iOS and Android. Although, I have been working on Augmented Reality projects for quite some time, this is the first game I’ve released using AR with Qualcomm’s Vuforia technology. I have to say, Vuforia is the best written Unity3D plug-in I’ve ever worked with. It’s extremely easy to use, fast, and works without a hitch on both Android and iOS–no changes needed.

Anyway, I figured I’d jot down some notes about AR and its possible future while my Camera Birds experience is still fresh.

I’m specifically talking about Augmented Reality on phones. AR glasses such as Google’s Project Glass and whatever Valve is cooking up have a completely different set of challenges–most notably rendering graphics not over video, but directly on top of your vision. This requires absolutely no latency and the additional difficulty of drawing graphics over a transparent display (glasses). It sounds like a fun project–ever since playing Heavy Rain I’ve wanted to build Norman Jayden’s ARI goggles in real life.

So, those notes…

We Are Still 2 Hardware Generations Away From AR Being Good

Around the release of the iPad 2 we started getting mobile devices that could run AR robustly enough to be truly usable. Slower devices such as earlier iPhones or even the 3DS suffer from finicky tracking and recognition performance. The iPhone 4S is even smoother, but in order to have enough processing power left over to do something interesting while recognizing and tracking objects in video we’re going to need better cameras and more cores to crunch through the data coming from them.

We Need More Sensors

Recognizing and tracking objects in 2D video has been around for a long time. In fact, a lot of the algorithms used for AR grew out of technologies developed for computer based match moving in movies such as Forrest Gump. Instead of carefully recorded footage, we’re dealing with a much more unpredictable real-time source of video: handheld cameras on phones and tablets.

This introduces all sorts of problems such as motion blur, lighting changes, and specular highlights that can trip up recognition and tracking algorithms. Although there are some decent solutions to this problem, newer AR technologies often go beyond visual tracking. Compass and gyroscope hardware in modern mobile devices can now be used to compensate for visual tracking loss. We still need more.

If Microsoft could shrink the Kinect down enough to include in a mobile phone, AR would take a big leap. Having a depth camera eliminates a lot of AR’s issues with visual tracking. Also, using stereo cameras can help determine the real-world size of objects in video. Most solutions currently require you to manually specify the size of glyphs in real-world units instead of determining this on the fly.

Markers Suck

Most AR solutions right now require markers or pre-set images for natural feature tracking. This means the system is trained on a 2D image (such as a QR Code or trading card) that can be recognized and tracked in footage. For this reason, most AR games on phones require you to print out markers in order to use the app.

There are markerless technologies in AR, most notably the so-called SLAM algorithm which has popped up in a few examples such as’s licensable library, Ogmento’s tank demo, and Sony’s Vita air hockey game from this past GDC. In these examples, arbitrary details in video are tracked geometry is built out of them. (I say so-called because there’s more at work here, but let’s not get nerdy!)

In most cases this means finding a simple plane out of points of contrast on a flat surface such as a tablecloth or a poster. This geometry is then used as a collision volume in a 3D game played over the video feed. The system also remembers where features were relative to the user’s position–so when details are lost due to movement, they can be easily re-tracked when moved back into view. An example you can download and play right now is 13thLab’s Ball Invasion–it came out last Summer and is still the only good example in the App Store.

Markers still have their place. Cards can be a collectible gimmick used for monetization (see Nintendo’s use of them in Kid Icarus for the 3DS) as well as a useful tool to instruct users on how to properly use an AR app. Still, they usually are a burden and limit the audience.

AR Is Still A Gimmick

Advertisers love AR because it involves users having to physically interact with a product: pick up a cereal box and you are 3/4 of the way to buying it. For most applications however, AR is still a gimmick. Having to wave your phone around at an object to play a game is a huge hassle. A compelling business case hasn’t been made yet–this is evidenced by the fact that most AR conferences feel more like amateur science fairs than bustling hives of commerce. Although, at the pace Vuforia has been advancing, I’m sure Qualcomm are working hard to change this.