VRLA Mixed Reality Easter Egg Hunt: Behind the Scenes

Late last year, John Root of Virtual Reality Los Angeles approached me with a crazy idea: what if we built a giant fake forest, placed it in the middle of the Los Angeles Convention Center, and used HoloLens to allow people to hunt for virtual Easter eggs inside a mixed reality experience? I wasn’t sure exactly how I’d go about creating it, but based on my time building and demoing Ether Wars at VRLA’s previous event, I knew it was possible and people would love it. I was all in.

VRLA_Easter_Egg

The project got a late start–a mere matter of weeks before the show. Regardless, everything came together at the right time. We were able to have Microsoft on as a partner who donated all the HoloLens devices for the event as well as brought in their own developers to help with technical issues, project management, and the logistics of running a large public HoloLens experience.

 

LESSONS LEARNED

There haven’t been many projects of this kind. Certainly very few people have experience designing real-world mixed reality installations. I figured I’d give a brain dump of what I learned which might be useful for developers attempting similar feats.

DESIGNING FOR REALITY

Before anything could be built, we had to work out the math behind the user flow of the whole experience. People such as Disney Imagineers who design amusement park rides are very familiar with this process. We had to figure out how many people we could fit in the exhibit’s space and how long it would take to get users in and out of the experience. From here we determined how many people could move through the Easter Egg hunt per day, how much staff would be needed to run it, and how many HoloLens devices we’d need in total. We also had to design the set to accommodate potentially large lines that wouldn’t descend into chaos and mayhem by the middle of the day.

PHYSICAL SET DESIGN IS HARD

The VRLA Mixed Reality Easter Egg Hunt involved typical software development issues and the added challenge of building a unique physical set that works properly with mixed reality technology.

The first step was designing the set itself. Mike Murdock of TriHelix designed the set, using a Vive to make sure it fit inside the dimensions set by the booth. This allowed him to not only judge the size, but preview what it would be like to walk through the set before construction began.

IMG_0779

Choosing paint colors to match Mike’s virtual set design.

It’s important to get the art director of the set and the app on the same page–Primarily to make sure the colors of the real world work best against the HoloLens’ additive display while still being highly trackable.  Unifying the look of the virtual and physical objects is also critical to creating a seamless experience. This means you need to organize the art process far in advance. Nathan Fulton, my art director on the app, made sure the color swatches for the set matched his vision of the virtual objects in the experience.

Lighting is also very important. We spent a lot of time designing the placement and type of lights so the space would be as trackable as possible without having the display overpowered by the environment.

 

To build the physical set, we turned to Fonco Studios, an experienced Hollywood set and prop design company who constructed an amazing N64-stylized forest out of literally a ton of styrofoam. This set was then sliced up into chunks and transported to the LA Convention Center where it was reassembled.

ANCHORING TO THE REAL WORLD

One of the first challenges when designing the software was to determine how the virtual objects were going to be placed on the set. In the beginning I thought we’d simply drop spatial anchors on the physical location and then share these anchors with each HoloLens. This proved to be impractical for a number of reasons.

Firstly, I’ve had lots of issues sharing spatial anchors across devices. They often appear in the wrong places. Not only that, but anyone who has been to a conference will tell you wifi access is spotty, if available at all. Having to use the Internet to transfer spatial anchors around might not be possible.

Plus, without having the physical set to scan we would have no way to test the final layout of eggs and other objects until we got to the location a day or so before the event.

For most of the app’s development, we used Mike’s 3D model of the set design in Unity3D to place the objects on. Then we used AfterNow’s technique of placing three spatial anchors in the corners of the set, saving them, and then spawning the game level at the center of these points. This real world alignment process is a little error prone, but does work.

These anchors are saved locally to the HoloLens so the setup process only has to be done once. Whenever players put on the headset after the app is restarted, they can jump right into the experience–no alignment necessary.

IMG_0854 2

The final set, assembled at the Los Angeles Convention Center

We also attempted to match lighting as accurately as possible by taking spherical photos using a RICOH THETA camera at different points inside the set and using them as cubemaps in Unity3D. These cubemaps provided convincing reflections, and can also be used with IBL shaders to help make virtual objects match the real world environment.

TO OCCLUDE OR NOT TO OCCLUDE

The added bonus of having a fixed set is we know the exact shape of the world. This meant we had the option of using a LIDAR scan of the environment as an occlusion mesh instead of the one built internally by HoloLens.

There are lots of advantages to this. Most importantly, the 3D scan can be optimized for performance by reducing the number of polygons and adding detail where the scan may have missed a few things. This makes the occlusion much more accurate while giving a bonus in performance.

Mimic3D made an amazing scan of the set once it was finally set up in the convention center. We saw quite a reduction in poly count from our own retopologized mesh vs. the dynamically generated HoloLens one. This highly optimized mesh also let us do a neat “The Matrix” style grid effect that appears to pour out over the world with a simple shader.

IMG_0828

Mimic3D’s amazing LIDAR scan

Perhaps most importantly, with over a dozen HoloLens devices to set up before the show, not having to generate a good occlusion mesh on each headset saved a lot of preparation time.

WHERE DO I LOOK?

The limited FOV of the HoloLens made it very important to guide the user’s gaze to where the action is. In some cases our assumptions of where users would look and how they moved through the environment was not exactly lined up with reality. One prime example is the rabbits.

In the back right corner of the forest there is a group of three rabbits that scurry away when you get close. We spent a lot of time tweaking the trigger size, position, and animation of the rabbits to make sure people had plenty of time to notice them. We even had Somatone create a positional audio effect to call attention to this. However, it was far too easy for someone to back into the trigger volume and have the bunnies escape without seeing them. A lot of participants reported not seeing rabbits at all–perhaps, the most important creature in the entire experience!

IN CONCLUSION

The lines were huge but everyone seemed to have a smile on their face when leaving the booth. The press loved it, too. I consider this a huge success for all parties involved and a leading example of how Mixed Reality can be used for things far more interesting than enterprise apps.

Building on what we’ve learned at VRLA, I’m totally ready to tackle a much more elaborate Mixed Reality entertainment experiences. Since the event, I’ve recieved a lot of interest in doing this type of project on a larger scale. Who knows what we might build next!

Why I Don’t Care About Your New Mixed Reality Headset

I’m often approached by entrepreneurs in the AR/MR space offering me demos of new hardware.  Competition in this space is fierce. You need three major elements for me to take a new platform seriously.

vrlatryon_2

You Need These Three Things To Have A Successful Mixed Reality Device

The three requirements for any successful AR (or more specifically MR) device are: Display, Computer Vision, Operating System

Display

This is the first element of an AR/MR wearable, and usually this what all hardware companies have. There are a number of different displays out there, but they all seem to share the same limitations: additive translucent graphics, small FOV, and relatively low resolution. Often times devices with claims of wider FOVs end up with even lower resolution visuals as a compromise. Both low and high resolution displays I’ve seen are all additive, thus images appear as translucent. Some companies claim to have solved these problems. As far as I’ve seen, we’re a long ways off from a commercial reality.

getimage

Operating System

When I got my HoloLens devkits, the first thing that impressed me is that Microsoft ported the entirety of Windows 10 to Mixed Reality. Up until now, most AR headsets had simple gaze-optimized skins for Android. Windows Holographic makes even traditional 2D applications able to be run in mixed reality as application windows floating in space or attached to your walls. It’s all tied to a bulletproof content delivery ecosystem (Windows App Store) so distribution is solved as well.

2786932-hololens

Your device needs to be more than just something worn only to run a specific app. Mixed reality wearables will one day replace your computer, phone, and just about anything with a screen. You need a complete Mixed Reality operating system that can run everything from the latest games to a browser and your email client in this inevitable use case.

Computer Vision

I can’t tell you how many device manufacturers have shown me their new display but “just don’t have the computer vision stuff in.” Sorry, but this is the most important element of mixed reality. Amazing localization, spatialization, tracking, and surface reconstruction features are what puts HoloLens light years ahead of its nearest competition.

This stuff is hard to do. Computer Vision was formerly an obscure avenue of computer science not many people studied. Now augmented reality has created a war for talent in this sector, with a small (but growing) number of Computer Vision PhDs commanding huge salaries from well funded startups. There are very few companies that have the Computer Vision expertise to make mixed reality work, and this talent is jealously guarded.

[BONUS] Cloud Super-intelligence

The AR headset of the future is a light, comfortable, and truly mobile device you wear everywhere. This requires a constant, fast connection to the Internet. HoloLens is Wifi only for now, but LTE support must be on the horizon. Not only is this critical for everyday-everywhere use, but many advanced computer vision functions such as object recognition need cloud-based AI systems to analyze images and video. With the explosion of deep learning and machine learning technology, a fast 5G connection to these services will make Mixed Reality glasses something you never want to leave the house without.

Don’t Waste My Time

A lot of people seem impressed with highly staged demos of half baked hardware. It’s only when you begin to develop mixed reality apps that you understand what’s really needed to make these platforms successful. Demos without the critical elements listed in this post will be harder to impress with once more people are familiar with the technology.

How To Demo HoloLens Apps In Public

Last week’s VRLA Summer Expo was the first time the public got a look at my current HoloLens project, Ether Wars. Tons of people lined up to try it. I must have done well over 100 demos over the two day event. Since then, I’ve showed it to a variety of developers, executives, and investors ranging from zero experience to those who have used HoloLens quite a bit. Combined with all the demoing done at HoloHacks a few months ago, I’ve learned a lot of common sense tips when demoing mixed reality apps. I figured I’d sum up some of my presentation tricks here.

Know Your Space

HoloLens can be a very temperamental device. Although it features the most robust tracking I’ve ever seen with an AR headset, areas with a lot of moving objects (pets, crowds of people), featureless walls, windows, and mirrors can really mess things up. Also, rooms that are too dark or too bright can make the display look not so great.

If you are travelling somewhere to show your app, try to find out ahead of time what the room is like that you are demoing in. It might be possible to ask for a few alternative rooms if the space they’ve got you in is inappropriate.

And, how do you know if the space is inappropriate? Scan the room before the demo starts. In the case of Ether Wars, you have to scan the room before you play the game. This scan is saved, so subsequent games don’t have to go through that process. When I demo the game, I scan the room myself to make sure the room works before I let others use it. This not only lets me know if the room works but allows the rest of the users to skip this sometimes lengthy step.

Consider building demo-specific safety features. For instance, Ether Drift needs ceilings to spawn space stations from. In the case of a room with a vaulted ceiling the HoloLens can’t scan, a safety feature would be one that automatically spawns the bases at ceiling height for demo purposes.

Teach The Air Tap

Microsoft’s mantra for HoloLens the interfaces is “Gaze, Gesture, and Voice“-essentially a conroller-free interface for all HoloLens apps. Very cool in concept, but I find at least half the people who try the device can’t reliably perform the air tap. It’s a tricky and unnatural gesture. Most people want to reach out and poke the holograms with their finger. It takes quite a bit of explanation to teach users that they must aim with their head and perform that weird air tap motion to click on whatever is highlighted by the cursor.

airtap

Teach the user how to perform the air tap before the demo–perhaps by having them actually launch and pin the app on a wall. It might help to put a training exercise in the app itself. For instance, to start Ether Wars you have to gaze and air tap on a button to start the experience. I use this moment to teach the player how to navigate menus and use the air tap.

Worst case scenario, you can stick your arm over the player’s shoulder in view of the HoloLens and perform the air tap yourself if the user just can’t figure it out.

Check The Color Stack

Unlike VR, it’s difficult to see what the user is viewing when demoing a HoloLens app. You can get a live video preview from the Windows Device Portal. However, this can affect the speed and resolution of the app. Thus, degrading the performance of your demo. One trick I’ve used to figure out where the user currently is in the demo is to learn what the colors of the stacked display look like on different screens.

IMG_2687

Each layer of the display shows different colors

If you look at the side of the HoloLens display you’ll see a stack of colored lights. These colors change depending on what is being shown on the screen. By observing this while people are playing Ether Wars, I’ve learned to figure out what screen people are on based on how the lights look on the side of the device. Now I don’t have to annoyingly ask “what are you seeing right now” during the demo.

None of this is rocket science–just some tips and tricks I’ve learned while demoing Hololens projects over the past month or so. Let me know if you’ve got any others to add to the list.