The Next Problems to Solve in Augmented Reality

I’m totally amped up about Project Tango. After having worked with augmented reality for a few years, most of the problems I’ve seen with current platforms could be solved with a miniaturized depth-sensing Kinect-style sensor. The Myriad 1 is a revolutionary chip that will dramatically change the quality of experience you get from augmented reality applications–both on mobile devices and wearables.

There’s a few other issues in AR I’d like to see addressed. Perhaps they are in research papers, but I haven’t seen anything real yet. Maybe they require some custom hardware as well.

Real-world lighting simulation.

One of the reasons virtual objects in augmented reality look fake is because AR APIs can’t simulate the real-world lighting environment in a 3D engine. For most applications, you place a directional light pointing down to and turn up the ambient for a vague approximation of overhead lighting. This is assuming the orientation of the object you’re tracking is upright, of course.

Camera Birds AR mode using an overhead directional light.

Camera Birds AR mode using an overhead directional light.

What I’d really like to use is Image Based Lighting. Image based Lighting is a computationally efficient way to simulate environmental lighting without filling a scene up with dynamic lights. It uses a combination of cube maps built from HDR photos with custom shaders to produce great results. A good example of this is the Marmoset Skyshop plug-in for Unity3D.

Perhaps with a combination of sensors and 360 cameras you can build HDR cubemaps out of the viewer’s local environment in real-time to match environmental lighting. Using these with Image Based Lighting will be a far more accurate lighting model than what’s currently available. Maybe building rudimentary cubemaps out of the video feed is a decent half-measure.

Which object is moving?

In a 3D engine, virtual objects drawn on top of image targets are rendered with two types of cameras. Ether the camera is moving around the object, or the object is moving around the camera. In real life, the ‘camera’ is your eye–so the it should move if you move your head. If you move an image target, that is effectively moving the virtual object.

Current AR APIs have no way of knowing whether the camera or the object is moving. With Qualcomm’s Vuforia, you can either tell it to always move the camera around the object, or to move the objects around the camera. This can cause problems with lighting and physics.

For instance, on one project I was asked to make liquid pour out of a virtual glass when you tilt the image target it rest upon. To do this I had to force Vuforia to assume the image target was moving–so then the image target tilted, so would the 3D object in the game engine and liquid would pour. Only problem is, this would also happen if I had moved the phone as well. Vuforia can’t tell what’s actually moving.

There needs to be a way to accurately track the ‘camera’ movement of either the wearable or mobile device so that in the 3D scene the camera and objects can be positioned accurately. This will allow for lighting to be realistically applied and for moving trackable objects to behave properly in a 3D engine. Especially with motion tracking advances such as the M7 chip, I suspect there are some good algorithmic solutions to factoring out the movement of the object and the observer to solve this problem.

Anyway, these are the kind of problems you begin to think about when staring at augmented reality simulations for years. Once you get over the initial appeal of AR’s gimmick, the practical implications of the technology poses many questions. I’ve applied for my Project Tango devkit and really hope I get my hands on one soon!

Ludum Dare: Ten Seconds of Thrust

This past weekend I participated in Ludum Dare 48, a contest where you make a game by yourself in 48 hours. The theme is revealed at the start of the contest–Iron Chef style. All code, graphics, and sounds have to be made from scratch. Voting began on Sunday night and will extend for a few weeks. I’m not even sure what you win, but that’s not the point. It’s an awesome experience in GETTING IT DONE.

Ten Seconds of Thrust!

My entry is the Lunar Lander-esque Ten Seconds of Thrust. (Please rate it!) Attempt to land at the bottom of increasingly difficult randomly generated space caverns with only ten seconds of thruster time. It’s crude, ugly, and buggy–especially on Windows where it doesn’t seem to detect landing. I didn’t have time to fix this bug as I only discovered it in the last half hour, but it does seem like a strange Unity Web Player bug since it works fine on OSX browsers. (PROTIP: Make sure you have a few friends around during the weekend to test your game!)

One of the best things about the contest is watching games evolve quickly through Twitter, Vine, Facebook, and Instagram posts. I put up a few videos in progress over the weekend.

I used a lot of the tools mentioned in my rapid prototyping posts, including a new tool I found called Sprite Gen which creates randomly generated animated character and tile sprites in tiny 12×12 blocks. Naturally, the game was developed in Unity along with 2DToolkit and HOTween for plug-ins.

I’d like to fix the landing bug as it makes the game useless on Windows, but the rules are somewhat unclear on bug-fixes that don’t add any content. This game was actually based on an idea for a Lunar Lander roguelike I was developing earlier this year. The LD48 version is highly simplified and way more fun. I abandoned my prototype in disgust back in February. This quick and dirty version is much better–I might run with it and make a full game.

A Few Quick Notes: GDC2013 Edition

Before we get started, vote for evolve.la

Blatant plug!–please vote for evolve.la in the My LA2050 grant contest. I’m in the running to build a social gaming experiment that will attempt to analyze social media activities of Los Angelenos to determine how they want the future of Los Angeles to look. I need your votes to get evolve.la off the ground! We now continue with your irregularly scheduled blog post.

GDC 2013 Rundown

GDC has become increasingly irrelevant over the past 5 years or so as influence has moved away from the realm of cloistered AAA console game teams and to so-called “indie” developers and the disruptive platforms of mobile and social. Because of this, you can get much better information having conversations with other developers. I spent most of GDC talking to people–you can always watch the good presentations on the GDC Vault.

The trend for 2013 is an industry wide panic over free2play. Presentations and panels worried over whether f2p games are ethical and how the game industry is supposed to survive through this disruption. Considering this is a conversation game developers have been having since 2009, it just goes to show how long it takes for GDC to catch on to major trends.

“Indie” developers were the big celebrities this year. So much so that formerly closed platforms from Nintendo and Sony bent over backwards to encourage garage developers to create content. Nintendo greatly loosened requirements for their development program and even revealed HTML5 support for the Wii U. Sony eliminated concept approval. This shows there are some radical changes ahead for the next generation–Changes I suggested years ago on this blog.

The biggest star of the show was Oculus VR. The wait time to try the Oculus Rift headset grew to over 2 and a half hours by the final day of GDC. I got in to see it and came away hopeful, but unimpressed. The current prototype headset is uncomfortable, but I didn’t spent much time adjusting it. The display resolution is low, causing a screen door effect. When I turned my head, the screen smeared to the point where I couldn’t see anything.

These problems are being addressed. They showed me the physical part for the new screen–the retail version of Oculus will fix the resolution and latency issues. The current kit is strictly for developers and mega-nerdy early adopters. It’s pretty neat for a $300 prototype, but far from a finished product.

I was more impressed with Infinite Z’s zSpace virtual holography system that was on display at Unity3D’s booth. It costs over 10X what Oculus does for no apparent reason. Still, being able to draw 3D splines in thin air and look around them was really cool.

Overall, GDC had a lot of opportunity on display as far as new devices, markets, and tools–but a lot of uncertainty on how to actually make money producing games.

Favorite Quotes of GDC

  • “Cokeheads are better than publishers.”

  • “They said they’d publish my game if I turn it into a Skinner-box.”

  • “The reason why you won’t close the deal is because you’re too competent.”

Displaying Maps in Unity3D

There have been a few recent examples of real-world maps displayed in Unity3D apps. The first one I noticed was the playfield in the infamous Halo 4 iPhone app that came out late last year. For unknown reasons, I was really into this game for a few months. I hung around my local 7-11 scanning bags of Doritos so much that I thought I was going to get arrested for shoplifting. Eventually this obsession led to me wanting to duplicate the map display used in the game. Here’s how I did it.

Google Maps Plug-In

Naturally the first place I looked was the Asset Store. It turns out there is a free Google Maps plug-in available. The only catch is that it requires UniWeb to work. UniWeb lets you call REST APIs and generally have more control over HTTP requests than Unity’s own WWW class allows. It can be a necessity if you’re using REST API calls but it restricts your code stripping options. This will bump up your binary size.

This asset’s sample scene works flawlessly. It downloads a map from the Google Static Map API and textures it on a cube. The code is clean and well documented, featuring the ability to request paths and markers to be added to the static map. Most attributes can be tweaked through the inspector–such as map resolution, location, etc.

I made a lot of changes to this package. I really wish it was open source. Free code assets really should be in most cases. I will try to isolate my changes into another C# file and post a Gist.

The first change I made was to add support for themed Static Maps. If you look at this wizard, you can see that there are a lot of styling options. This appears to be the same technique used in the Halo 4 app because with the right set of options you can get something that looks really close. Supporting styling in Unity3D is just a simple act of appending the style parameters to the end of the URL used by the Google Maps plug-in.

Displaying Markers in 3D

The next thing I wanted to do is display the markers as 3D objects on top of the map instead of having them inside the texture itself. This requires 3 steps:

  1. Determine where the markers are in pixel coordinates in the static map texture.
  2. Calculate the UV coordinate of the pixel coordinate.
  3. Calculate the world coordinate of the texel the UV coordinate resides at.

Step 1 can be tricky. You have to project the latitude and longitude of the marker with the Mercator projection Google Maps uses to get the pixel coordinate. Luckily, this guy already did it in PHP to create image maps from static maps. I adapted this code to C# and it works perfectly. You can grab the Google Maps utility functions here. (All this great free code on the net is making me lazy–but I digress)

Step 2 is easy. This code snippet does the trick. The only catch is that you have to flip the V so that it matches with how Unity uses UV coordinates.

Step 3 is also tricky. However, someone with much better math skills than I wrote a JavaScript method to compute the world coordinate from a UV coordinate. It searches through each triangle in the mesh and sees if the UV coordinate is contained inside it. If so, it then calculates the resultant world coordinate. The key to using this is to put the static map on a plane (the default scene in the plug-in uses a cube) and use the C# version of this function I wrote here.

3D objects floating over marker locations on a Google Static Map.

3D objects floating over marker locations on a Google Static Map.

Here’s the end result–in this case it’s a display for the Donut Dazzler prototype. 3D donuts are floating over real-world donut shops and cupcakes over cupcake bakeries. I got the locations from the Foursquare API. This is quite easy to do using UniWeb.

Slippy Maps

The aforementioned technique works great if you just want a static map to display stuff around the user’s current location. What if you want to be able to scroll around and see more map tiles, just like Google Maps when you move around with your mouse? This is called a Slippy Map. Slippy Maps are much more elaborate–they require dynamically downloading map tiles and stitching them together as the user moves around the world.

Thankfully Jonathan Derrough wrote an amazing free Slippy Map implementation for Unity3D. It really is fantastic. It displays markers in 3D and pulls map tiles from multiple sources–including OpenStreetMap and Bing/VirtualEarth. It doesn’t use Google Maps because of possible TOS violations.

I couldn’t find a way to style map tiles like Google Static Maps can. So the end result was impressive but kind of ugly. It is possible with OpenStreetMap to run your own tile server and run a custom renderer to draw styled tiles. I suspect that’s how Rescue Rush styles their OpenStreetMap tiles–unless they are doing some image processing on the client.

Either Or

For my prototype I ended up using Google Static Maps because Slippy Maps were overkill. Also, pulling tiles down from the servers seemed much slower than grabbing a single static map. I suppose I could add some tile caching, but in the end static maps worked fine for my purposes.

Keep in mind that Google Maps has some pretty fierce API usage costs. If your app goes viral, you will likely be on the hook for a huge bill. Which is why it might be worth figuring out how to style free OpenStreetMap tiles.

Unity3D 4 Pet Peeves

I’ve been updating my older apps to use the newly released Unity3D 4 engine, as well as starting an entirely new project. I haven’t used many of Unity3D 4’s new features yet, but I figured this is as good a time as any to list a few of my pet peeves with Unity3D 4 as I did with Unity3D 3 a few years back.

It’s time Unity3D had a package manager.

Unity3D plug-ins and assets purchased from the Asset Store are invaluable. It’s becoming the most important feature that makes Unity3D the superior choice. However, managing projects with multiple plug-ins is can be a nightmare. A lot of this is how Unity3D handles file deletions.

If you click the “update” button to overwrite an existing plug-in with the latest version from the Asset Store, it may wreak havoc upon your entire project. Unity3D’s file hashing system will sometimes fail to overwrite files with the same name, even if you are importing a newer one. You’ll end up with a mess of old and new plug-in files causing chaos and mayhem. The only way to prevent this is to manually find delete all the old plug-in files before updating with the latest version.

Not to mention the fact that native plug-ins either require you to manually setup your own XCode project with external libraries or have their own proprietary scripts that edit your XCode project. Unity3D should provide an API and package manager that lets plug-ins forcibly delete and update their own files as well as modify settings in the XCode project Unity3D generates.

Let me import files with arbitrary extensions.

A minor annoyance is how Unity3D will only accept files with specific extensions in your project. If you want a custom binary data file you HAVE to give it the txt extension. It’s the only way you can drag the file in to the project. Unity3D should allow you to import files with any extension you want, but provide a method in the AssetPostprocessor API to be called when an unknown file extension is detected.

Where’s the GUI?

Come on now. It’s 2013. The new GUI has been “coming soon” for years. Unity hired the NGUI guy, which leads me to believe the mythical Unity3D 4 GUI is merely the stuff of legends and fantasies. I like NGUI but I’m really looking forward to an official solution from Unity. Although I’m not looking forward to re-writing all my GUIs once it arrives. Let’s just get it over with. Bring it on.

Monodevelop sucks.

My god. Monodevelop sucks. Lots of people use other text editors for code, but you still can’t avoid touching Monodevelop when it comes to debugging on OSX. I’m sure it can be whipped into shape with a minor overhaul, but it’s been awful for so long perhaps this is unlikely. Aside from the crashes and interface weirdness, how much human productivity has been destroyed waiting for Monodevelop to reload the solution every time so much as a single file has been moved to a different folder?

Is it time to update Mono?

While we’re at it, Mono recently updated to C# 5.0. I’m not sure if this is a big performance drag or not, but I’d love to see Unity3D’s Mono implementation updated to the latest. There are some C# 5.0 features I’ve been dying to use in Unity3D.

Tough Love

Don’t take it personally, Unity3D is still my engine of choice. This list of annoyances is pretty minor compared to previous ones. Every year, Unity gives me fewer and fewer things to whine about. It seems competing solutions are having trouble keeping up.

How To Prevent Performance Spikes in Unity3D When a Model is First Visible

In my latest Unity3D app I dynamically load assets from the Resources folder and place them in the world after the initial scene load. These assets use new materials and textures that must be uploaded to the GPU. I thought I was being slick by caching prefabs to prevent a loading hiccup when I needed to instantiate. However, that’s only part of the problem. After placing the object in the scene, my game would freeze up for a frame or two when the newly created object first became visible. The profiler showed this spike attributed to a function called AwakeFromLoad.

It turns out Unity3D does not load the GPU with your new object’s assets until it’s first visible. Apparently, this is what AwakeFromLoad does. This is an optimization technique presumably to prevent thrashing on the GPU by loaded assets that won’t be visible immediately. The downside is you’ll see a pause as Unity3D uploads data to the GPU. From what I can tell, this can even mean compiling the shader if it hasn’t been used in the scene yet.

Unity doesn’t provide a function to force the GPU to load assets. From looking at Unity forum threads, the most common solution is to put up a loading screen and show newly instantiated assets to the main camera for a frame to force a GPU load. Once all the assets have been made visible, the loading screen is dropped.

Putting up a loading screen just seemed like a huge pain in the ass, not to mention an ugly hack. So, I came up with a solution using Unity Pro’s RenderTexture and a second camera. Now, my game scene has two cameras: the Main Camera and a disabled secondary camera with a tiny 32×32 RenderTexture as its target. Whenever I instantiate a new asset in the world, I position the second camera in front of it and render a frame to this texture. This forced rendering does the trick of uploading all necessary data to the GPU. Yes, there still is a loading spike, but you decide when it’s going to happen and you don’t have to reposition your object in view of the main camera for a frame.

I put this in a behavior called AssetGPULoader, you can grab it here. It only works with Unity3D Pro as it needs RenderTexture. As far as I can tell, this does the trick. It has removed my unpredictable performance spikes. For an alternative solution, I also found this technique in the Unity forums.