HoloLens is Ready for Prime Time

Microsoft recently invited me to try out HoloLens at their Venice space on Abbot-Kinney. Having just won the AR/VR award in the Tango App Contest for InnAR Wars, I jumped at the chance.  After developing for Google Tango for over a year, I had a long wishlist of features I’m looking for in an AR platform. In the unlikely event that HoloLens was for real, I could make InnAR Wars exactly how I envisioned it at the start of the project.

Entering the Hololens Demo Zone


My skepticism was well warranted. Having worked in AR for the past 5 years, I’ve seen my share of fake demo videos and smoke-and-mirrors pitches. Every AR company is obligated to put out faked demo videos that they inevitably fail to live up to. Just look at this supercut of utterly ridiculous promotional videos. I was curious if the staged HoloLens demos weren’t much better.

I had heard many polarizing opinions about HoloLens from people who have tried it. Some felt it was an incredible experience while others told me it was the single most overhyped disappointment in consumer electronics history.

After trying it for myself, I can say HoloLens is the real deal.

The demo I tried was the “X-Ray” game shown at BUILD no too long ago. This version was a little simpler than this staged demonstration–your arm isn’t magically turned into a plasma cannon. Instead, your hold your hand out in front of the device and “air tap” to fire at enemies that appear to be coming out of cracks in the walls. Occasionally you can give a voice command to freeze time, Matrix-style, and take out enemies while they are vulnerable.

A simple game, for sure, but a great demonstration of the HoloLens’ capabilities.  

The device is clearly a prototype. It’s bulky, looks like a vision of the future from a ’90s sci-fi flick, and it even BSODed on me which was kind of hilarious.  Still, I was thoroughly impressed with HoloLens and here’s why:


When the game starts, you have to look around the room and watch it build geometry out of the walls and other objects in the area. HoloLens builds a mesh out of the real world with a depth camera that is then used in gameplay. It’s kind of like building a video game level out of all the walls and floors in your room. This mesh is then used to place virtual wall cracks and spawn points for enemies during gameplay. Once you’ve build a complete mesh out of the room, the game begins.

This same meshing process is possible with Google Tango, but it’s slow and temperamental. Still, very impressive given it’s in a tablet you can buy right now. In fact, I used Tango’s meshing capabilities to place floor traps in InnAR Wars.

I was impressed with HoloLens’ rapid meshing as I moved around my environment. It even handled dynamic objects, such as the woman guiding me through the demo. When I looked at her during the meshing phase, she quickly transformed into a blocky form of red polygons.


Initially I was disappointed in the display. Much like Epson’s Moverio or ODG’s R7 glasses, it projects the augmentation on a part of the “glass” in front of your eyes. This means you see a distracting bright rectangle in the middle of your vision where the augmentation will appear. Compared to ODG’s R7s, HoloLens seemed to have higher contrast between the part of the display that’s augmented and the part that’s not. There also is an annoying reflection that looks like a blue and green “shadow” of the augmentation above and below the square.

While playing the game this didn’t matter at all. Although everything is somewhat translucent, if the colors are bright enough the virtual objects appeared solid. Combined with rock solid tracking on the environment, I soon forgot all about contrast issues and internal reflection problems on the display. A these issues can be dealt with through art and the lighting of the room you are playing in. Plus, a Microsoft representative assured me the display is even better in the current version still in their labs.

Field of View

The top issue people have with HoloLens is the field of view. People complain that it only shows graphics in a postcard sized space in front of your vision. I had rock bottom expectations for this having developed applications on previous generation wearable AR displays. Although HoloLens’ augmentation is limited to a small square in front of your vision, this space is easily 2X the size of other platforms. In the heat of the action while playing X-Ray, I mostly forgot about this restriction.

Field of view is not an easy thing to solve–it’s a fundamental problem with the physics of waveguide optics. I’m not sure how much progress Microsoft can make here. But the FOV is already wide enough for a lot of applications.

I’m All In

Part of the pitch is the “Windows Holographic” platform. That is, in the future you won’t have screens. With HoloLens you’ll be able to place Windows applications in mid-air anywhere in your office. Applications will virtually float around you like monitors hovering in space. (Magic Leap’s fake demo video shows an example of how this would work) This can actually be done right now with HoloLens and existing Windows applications. Supposedly, some Microsoft engineers wear HoloLens and have integrated the “holographic” workspace into their daily use.

Beyond gaming applications, I am on board with this screen-free future. Your tablet, computer, even your phone or smartwatch will merely be a trackable object to display virtual screens on. Resolution and screen size will be unlimited, and you can choose to share these virtual screens with other AR users for collaboration.  No need to re-arrange your office for that huge 34 inch monitor. You can simply place it in 3D space and overlay it on top of reality. Think of all the extra stuff your phone could do if it didn’t have to power a giant LCD display! It’s definitely on its way. I’m just not sure exactly when.

VR with a Gamepad Sucks

I was kind of bummed my first day of Oculus Connect 2.


Last year’s Oculus Connect was revelatory to me. Despite having worked on two different Gear VR titles at the time, the Crescent Bay demo was incredible in comparison. From Oculus’ own vignette demos to Epic’s Showdown sequence–the leap in quality from DK2 to Crescent Bay was astounding. Everyone walked out of that demo with a huge simile on their faces.

The first demos I tried at OC2 were the Gamepad demos. Oculus spent an absurd amount of time at their E3 keynote talking about how amazing it was that they were launching with the XBox 360 controller as the input device. At Oculus Connect, I put this claim to the test.

Every game from EVE Valkyrie to Edge of Nowhere seemed like playing a regular video game strapped to my face. I felt like I was playing an XBOX One through binoculars. In fact, a few of the games made me a little queasy–which I’m usually not susceptible to.

Maybe I’m just jaded having been developing gamepad VR experiences on Gear VR for a while, I thought.

Later on I tried Toybox which is a cool tech demo but doesn’t really illustrate how you’d play an actual game for any length of time with the Touch controllers. In fact, I found the controllers a little hard to use compared to the Vive. They have tons of confusing buttons and getting the finger gestures right seemed to take a little bit of work.

I was leaving the demo area and getting ready to head home when a friend of mine who works for Oculus stopped to ask what I thought. I told him honestly that I felt last year’s demos were better–they were more immersive and interesting. Although a little taken aback at my impressions, he strongly suggested I come by tomorrow for the next set of demos. He couldn’t tell me what they were, but promised they’d be awesome.

The Oculus Connect app sent a notification alerting me that new demo registrations would be available at 8 AM. I set my alarm and woke up the next morning to register for the Touch demos via my iPhone. I promptly slept through the keynote and arrived on the scene at noon for my demo.

We were only allowed to try two games, and it was heavily suggested I try Epic’s “Bullet Train” experience. Having not seen the keynote, I had no idea what I was getting into.

Bullet Train is mind blowing.

Bullet Train is essentially Time Crisis in VR. When I saw the Showdown demo last year I thought a game like this in VR would be a killer app. One of my favorite coin-ops of all time is Police 911–which motion tracks your body with a pair of cameras to duck behind obstacles. I thought doing this in VR would be amazing. However, last year there were no hand tracking controls–it was just a vague idea.

Here, Epic took the Touch controllers and made an incredible arcade shooter experience that should be a killer app should Epic choose to develop this further. Oculus really needs to do everything in their power to get Epic to produce this as a launch title for the Touch controllers.

The touch controls make all the difference. From handling weapons and grenades to plucking bullets out of the air in slow motion, Bullet Train really drives home how flexible the Touch controls are. Unlike Vive which is like holding a set of tools, these let you reach out and grab stuff–Even pump a shotgun.

The combination of standing in a motion tracked volume and visceral interaction with the world using your hands–even with Touch’s primitive finger gesture technology–really immerses you in an experience way beyond what’s possible sitting in a chair with an XBox controller.

It’s disappointing that Touch won’t launch with Oculus’ headset. Hand tracking is absolutely required for a truly immersive experience. Developing games that support both Gamepad and Touch control is going to be difficult without diluting features for one or the other. I’ve experienced a similar issues developing games that work with Gear VR’s touchpad and Bluetooth gamepad.

I left Oculus Connect 2 I reinvigorated with the feeling that VR with hand tracking is the One True VR Experience. Gamepad is fine for mobile VR at the moment, but all of my PC and Console VR projects are now being designed around Touch and hand tracked input. It’s the only way!

The Challenge of Building Augmented Reality Games In The Real World

InnAR Wars Splash Image - B

Last week I submitted the prototype build of my latest augmented reality project, InnAR Wars, to Google’s Build a Tango App Contest. It’s an augmented reality multiplayer space RTS built for Google’s Tango tablet that utilizes the environment around you as a game map. The game uses the Tango’s camera and Area Learning capabilities to superimpose an asteroid-strewn space battlefield over your real-world environment. Two players holding Tangos walk around the room hunting for each other’s bases while sending attack fleets at the other player’s structures.

Making InnAR Wars fun is tricky because I essentially have no control over the map. The battlefield has to fit inside the confines of the real-world environment the tablets are in. Using the Tango’s Area Learning capabilities with the positions of players, I know the rough size of the play area. With this information I adjust the density of planetoids and asteroids based on the size of the room. It’s one small way I can make sure the game at least has an interesting number of objects in the playfield regardless of the size of the area. As you can see from the videos in this post, it’s already being played in a variety of environments.

This brings up the biggest challenge of augmented reality games–How do you make a game fun when you have absolutely no control over the environment in which it’s played? One way is to require the user to set up the play space as if she were playing a board game. By using Tango’s depth camera, you could detect the shapes and sizes of objects on a table and use those as the playfield. It’s up to the user to set it up in a way that’s fun–much like playing a tabletop war game.

For the final release, I’m planning on using Tango’s depth camera to figure out where the room’s walls, ceilings, and floors are. Then I can have ships launch from portals that appear to open on the surfaces of the room. Dealing with the limited precision and performance of the Tango depth camera along with the linear algebra involved in plane estimation is a significant challenge. Luckily, there are a few third-party solutions for this I’m evaluating.

Especially when looking at augmented reality startups’ obligatory fake demo videos, the future of AR gaming seems exciting. But the practical reality of designing a game to be played in reality–which is itself rather poorly designed–can prevent even the most amazing technology from enabling great games. It’s probably going to take a few more hardware generations to not only make the technology usable, but also develop the design language to make great games that work in AR.

If you want to try out the game, I’ll have a few Tangos on hand at FLARB’s VRLA Summer Expo table. Stop by and check it out!

How To Support Gear VR and Google Cardboard In One Unity3D Project

Google Cardboard is a huge success. Cardboard’s userbase currently dwarfs that of Gear VR. Users, investors, and collaborators who don’t have access to Gear VR often ask for Cardboard versions of my games. As part of planning what to do next with Caldera Defense, I decided to create a workflow to port between Gear VR and Cardboard.

Always keep a Cardboard on me at ALL TIMES!

I used my VR Jam entry, Duck Pond VR, as a test bed for my Unity3D SDK switching scripts. It’s much easier to do this on a new project. Here’s how I did it:

Unity 4 vs. Unity 5

Google Cardboard supports Unity 4 and Unity 5. Although Oculus’ mobile SDK will technically work on Unity 5, you can’t ship with it because bugs in the current version of Unity 5 cause memory leaks and other issues on the Gear VR hardware. Unity is working on a fix but I haven’t heard any ETA on Gear VR support in Unity 5.

This is a bummer since the Cardboard SDK for Unity 5 supports skyboxes and other features in addition to the improvements Unity 5 has over 4. Unfortunately you’re stuck with Unity 4 when making a cross-platform Gear VR and Cardboard app.

Dealing With Cardboard’s Lack of Input

Although Gear VR’s simplistic touch controls are a challenge to develop for, the vast majority of Cardboards have no controls at all! Yes, Google Cardboard includes a clever magnetic trigger for a single input event. Yet, the sad fact is most Android devices don’t have the necessary dock connector to use this.

You have a few other control options that are universal to all Android devices: the microphone and Bluetooth controllers. By keeping the microphone open, you can use loud sounds (such as a shout) to trigger an action. You can probably use something like the Pitch Detector plug-in for this. Or, if your cardboard has a head strap for hands-free operation, you can use a Bluetooth gamepad for controls.

Because of this general lack of input, many Cardboard apps use what I call “stare buttons” for GUIs. These are buttons that trigger if you look at them long enough. I’ve implemented my own version. The prefab is here, the code is here. It even hooks into the new Unity UI event system so you can use it with my Oculus world space cursor code.

Gear VR apps must be redesigned to fit within Cardboard’s constraints. Whether it’s for limited controls or the performance constraints of low end devices. Most of my Cardboard ports are slimmed down Gear VR experiences. In the case of Caldera Defense, I’m designing a simplified auto-firing survival mode for the Cardboard port. I’ll merge this mode back into the Gear VR version as an extra game mode in the next update.

Swapping SDKs

This is surprisingly easy. You can install the Cardboard and Gear VR SDKs in a single Unity project with almost no problems. The only conflict is they both overwrite the Android manifest in the plugin folder. I wrote an SDK swapper that lets you switch between the Google Cardboard and Oculus manifests before you do a build. You can get it here. This editor script has you pick where each manifest file is for Cardboard and Gear VR and will simply copy over the appropriate file to the plugin folder. Of course for iOS Cardboard apps this isn’t an issue.

Supporting Both Prefabs

Both Oculus and Cardboard have their own prefabs that represent the player’s head and eye cameras. In Caldera Defense, I originally attached a bunch of game objects to the player’s head to use for traces, GUI positioning, HUDs, and other things that need to use the player’s head position and orientation. In order for these to work on both Cardboard and Oculus’ prefabs, I placed all objects attached to the head on another prefab which is attached to the Cardboard or Oculus’ head model at runtime.

Wrapping Both APIs

Not only do both SDK’s have similar prefabs for the head model, they also have similar APIs. In both Cardboard and Oculus versions, I need to refer to the eye and head positions for various operations. To do this, I created a simple class that detects which prefab is present in the scene, and grabs the respective class to wrap the eye position reference around. The script is in the prefab’s package.


For the final step, I made separate Cardboard versions of all my relevant Gear VR scenes which include the Cardboard prefabs and modified gameplay and interfaces. If no actual Oculus SDK code is in any of the classes used in the Cardboard version, the Oculus SDK should be stripped out of that build and you’ll have no problem running on Cardboard. This probably means I really need to make an Oculus and Cardboard specific versions of that CameraBody script.

The upcoming Unity 5.1 includes native Oculus support which may make this process a bit more complicated. Until then, these steps are the best way I can find to support both Cardboard and Gear VR in one project. I’m a big fan of mobile VR, and I think it’s necessary for any developer at this early stage of the market to get content out to as many users as possible.

Why I’m All In On Mobile VR

Last month I released Caldera Defense, a Virtual Reality tower defense game on Gear VR. This is the second Gear VR title I’ve worked on, and the first I’ve built and published from the ground up. (Not including my Oculus Mobile VR Jam submission) Caldera Defense is a free early access demo–basically a proof of concept of the full game–and the reaction has been great. Thousands of people have downloaded, rated, and given us valuable feedback. We’re busy incorporating it into the first update.

Caldera Defense featured on the Gear VR store

Originally I planned to use this as a demo to fund an expanded PC and Morpheus launch version of the game with greatly improved graphics, hours of gameplay, and additional features such as multiplayer and second-screen options.

However, pitching even a modestly budgeted console and PC VR game experience to publishers, or even the platforms themselves, is a tough sell. I’m sure at E3 next month we will see all sorts of AAA VR announcements. Yet, many traditional funding avenues for games remain skeptical of the opportunity VR presents.

Since the Caldera project began last year, mobile VR has morphed into a unique opportunity. With over a million Google Cardboards in the wild and new versions of the Gear VR headset in retail stores worldwide, there will be millions of mobile VR users before there’s comparable numbers on Oculus desktop, Vive, and Morpheus.

Is it possible that mobile VR will be a viable business before it is on PC and consoles? Most of my colleagues are skeptical. I’m not.

The economics work out. Due to the mobile nature of the experience, games and apps for these platforms tend towards the bite-sized. This greatly reduces the risk of mobile VR since assets optimized for mobile are simpler and casual VR experiences require less content to be built overall.

I can make a dozen mobile VR minimum viable products for the same budget of one modestly scoped Morpheus experience. From these MVPs I can determine what types of content gains the most traction with VR users and move in that direction. I can even use this data to guide development of larger AAA VR experiences later.

By this time next year it will be possible to monetize these users significantly, whether through premium content or advertising. It may be more valuable to collect a lot of eyeballs in mobile VR than breaking even on a multi-million dollar AAA launch tile. As we’ve seen in the past, acquiring a huge audience of mobile players can lead to tremendous revenue streams.

Being on the Oculus desktop, Vive, or Sony’s Morpheus deck at launch is an enormous opportunity. In fact, I’m still searching for ways to produce the console and desktop version of Caldera Defense. However, if you lack the capital to produce at that scale, smaller mobile projects are much easier to bootstrap and the upside is huge.

The Coming Public Point Cloud

One of the most important elements of Augmented Reality is the ability to seamlessly mesh 3D graphics with the real world.  Current AR technology simply overlays graphics on top of video–even when tracking and recognizing objects like cards and markers. The AR SDK gives the position and orientation of the tracked object to a 3D engine which then renders geometry on top of the video frame coming from the device’s camera.

A 3D scan of myself overlaid on an AR card with Vuforia.

A 3D scan of myself overlaid on an AR card with Vuforia.

New technologies like Google’s Tango Tablet use Kinect-style depth cameras to store not only the color of each pixel, but the depth and position, too. (Well, sort of–the depth camera’s resolution is much lower than that of the color camera). This means that you can build a 3D model out of what the tablet’s camera sees as you move around an environment.

Tango displaying point cloud data of what it currently sees.

Tango displaying point cloud data of what it currently sees.

This feature has huge ramifications. Tango uses this data to do what is called “localization.” This means once an area is scanned, the tablet can compare the internal 3D model of the current environment it has stored with what the camera is currently seeing. When fused with compass and gyro data, the Tango tablet can compute its precise location inside the scanned area. This doesn’t take long either. Tango starts building the model immediately. Walk back to where you started using the tablet, and Tango knows where it is.

Usually this 3D data is stored as a point cloud. This is basically a 3D point for every position the 3D camera records.  Hence, a sufficiently complicated area will look like a cloud of dots–a point cloud. You can see an example of the Tango building a point cloud with the Room Scanner Tango app.

These point clouds are important for not only localization, but AR graphical effects such as occluding rendered 3D objects with the real world.  This is because a 3D mesh can be built out of these points which can be used for occlusion, collision and other features. Having objects in between you and the augmentation occlude the 3D render is essential to nailing the feeling that an AR object is really there.

Point clouds are awesome, but building them can be frustrating. Current point cloud scanners are bulky and slow, not to mention their accuracy issues can lead to jitter and other artifacts. Also, some depth cameras run at a frame rate low enough to make it hard to create point clouds without moving very slowly through an environment. Who wants to play a game where you have to walk around and meticulously scan a room before you can start?

In order for AR games and apps to succeed, devices need to effortlessly be able to sense and detect the 3D geometry of their surroundings. Yet, quick and instant generation of point clouds is far beyond the capabilities of current mobile sensor technology.

That’s where the public point cloud comes in.

A truly great Augmented Reality platform needs to upload point clouds generated by devices to the cloud.  Then, when a user uses some hot new wearable AR glasses, it can pull down a pre-made point cloud for the current location off of a server and use that until the glasses can update it from its own sensors. The device will then upload a fresh point cloud which can be used to refine the version stored online.

You can kind of see this already–Google and Apple Maps’ 3D satellite mode use similar point cloud reconstruction techniques presumably from aerial photos and other sources. Whereas these 3D models often look like something you’d see on the original PlayStation, the public point cloud will have to be much more detailed.  As sensors on mobile devices become more advanced, the crowdsourced point cloud data will become incredibly detailed.

Apple Maps' 3D reconstruction kind of looks like an original PlayStation game. The public point cloud will have to be higher resolution.

This 3D reconstruction kind of looks like an original PlayStation game. The public point cloud will have to be higher resolution. Oh, this is also where you can bet the best pastrami sandwich on the planet.

A massive, publicly accessible point cloud is not just necessary for the next generation of AR wearable devices. But also for self driving cars, drone navigation, and robotics (which is indeed where many of these algorithms came from in the first place). Privacy implications do exist, but perhaps not more so than Google Maps’ street view, or other current technologies that give you very precise information about your location.

In the near future, almost every public place on the planet will be stored in the cloud as 3D reconstructed geometry–passively built up and constantly refined by sensors embedded on countless mobile and wearable devices, perhaps without the user even knowing.

Adult Contemporary Video Games

One of my favorite Combat Jack podcasts of 2014 is when they interviewed legendary hip hop producer, Marley Marl over the Summer.  Marly Marl invented the modern hip-hop sound most take for granted and created the Juice Crew, one of the most important groups of MCs ever.

The Juice Crew

Before producing hit records, Marley had a career as an on-air DJ, starting on Mr. Magic‘s show on KISS-FM in New York.  In the ’90s he went on to host “Future Flavas” with Pete Rock on Hot 97.  Marley Marl was also still producing hit albums for the likes of LL Cool J and Lords of the Underground.

Times change, and Marley Marl isn’t producing music for 20 year olds anymore.  While many DJs desperately hang on to their fading youth, Marley tried another tactic.  He moved over to WBLS which plays old school hip hop for a mature audience.

it just so happens, rap fans in their fourties and beyond have far more disposable income than those in their teens and twenties.  His WBLS show has gone on to be a great success.  It turns out that despite being a youth-powered movement, there’s plenty of advertising dollars in hip-hop appealing to older rap fans.

This got me thinking about video games.

A lot of veteran developers are debating about the decline of AAA games in the face of the disruptive waves of free2play and mobile.  Many gamers in their demographic agree.  If that’s the case, why not appeal to this older audience?

The challenge to monetizing these gamers is that although they have the same taste in games they may have had over a decade ago, their play styles are vastly different due to lifestyle changes.  If you’ve got kids or a demanding job, perhaps you no longer have 120+ hours to spend playing an RPG. However, you might digest the same style of game in shorter episodic bursts on a tablet or smartphone.

Some developers have caught on to this and produce what I call Adult Contemporary Video Games.  A good example is the 1980s pencil and paper RPG, Shadowrun.  Microsoft’s attempt at AAA shooter based on Shadowrun was an abject failure (although I quite liked it).  Five years later, Harebrained Schemes went from a surge of support on Kickstarter for “Shadowrun Returns” to a series of popular mobile and PC downloadable games based on the franchise.

Shadowrun for iPad

This is a smart strategy–delivering content aimed at an older audience on newer devices.  Those of us who grew up not on just the original RPG, but the SNES and Genesis games were ripe for a new entry in the series.  This model has also seen success with Wasteland 2, and surely the upcoming Bard’s Tale sequel will continue the trend.

It remains to be seen if you can develop a new IP targeted at this audience.  A lot of what you hear on Adult Contemporary radio is old artists making new music.  In games it may be the same. So far, the genre seems to bank on nostalgia by resurrecting classic franchises for an older audience on new devices with updated play styles. Especially if you include teh current wave of retro remakes. While some veteran developers excel at creating games for the new mobile f2p masses, others may be more suited for this viable slice of the market.