VRLA Mixed Reality Easter Egg Hunt: Behind the Scenes

Late last year, John Root of Virtual Reality Los Angeles approached me with a crazy idea: what if we built a giant fake forest, placed it in the middle of the Los Angeles Convention Center, and used HoloLens to allow people to hunt for virtual Easter eggs inside a mixed reality experience? I wasn’t sure exactly how I’d go about creating it, but based on my time building and demoing Ether Wars at VRLA’s previous event, I knew it was possible and people would love it. I was all in.

VRLA_Easter_Egg

The project got a late start–a mere matter of weeks before the show. Regardless, everything came together at the right time. We were able to have Microsoft on as a partner who donated all the HoloLens devices for the event as well as brought in their own developers to help with technical issues, project management, and the logistics of running a large public HoloLens experience.

 

LESSONS LEARNED

There haven’t been many projects of this kind. Certainly very few people have experience designing real-world mixed reality installations. I figured I’d give a brain dump of what I learned which might be useful for developers attempting similar feats.

DESIGNING FOR REALITY

Before anything could be built, we had to work out the math behind the user flow of the whole experience. People such as Disney Imagineers who design amusement park rides are very familiar with this process. We had to figure out how many people we could fit in the exhibit’s space and how long it would take to get users in and out of the experience. From here we determined how many people could move through the Easter Egg hunt per day, how much staff would be needed to run it, and how many HoloLens devices we’d need in total. We also had to design the set to accommodate potentially large lines that wouldn’t descend into chaos and mayhem by the middle of the day.

PHYSICAL SET DESIGN IS HARD

The VRLA Mixed Reality Easter Egg Hunt involved typical software development issues and the added challenge of building a unique physical set that works properly with mixed reality technology.

The first step was designing the set itself. Mike Murdock of TriHelix designed the set, using a Vive to make sure it fit inside the dimensions set by the booth. This allowed him to not only judge the size, but preview what it would be like to walk through the set before construction began.

IMG_0779

Choosing paint colors to match Mike’s virtual set design.

It’s important to get the art director of the set and the app on the same page–Primarily to make sure the colors of the real world work best against the HoloLens’ additive display while still being highly trackable.  Unifying the look of the virtual and physical objects is also critical to creating a seamless experience. This means you need to organize the art process far in advance. Nathan Fulton, my art director on the app, made sure the color swatches for the set matched his vision of the virtual objects in the experience.

Lighting is also very important. We spent a lot of time designing the placement and type of lights so the space would be as trackable as possible without having the display overpowered by the environment.

 

To build the physical set, we turned to Fonco Studios, an experienced Hollywood set and prop design company who constructed an amazing N64-stylized forest out of literally a ton of styrofoam. This set was then sliced up into chunks and transported to the LA Convention Center where it was reassembled.

ANCHORING TO THE REAL WORLD

One of the first challenges when designing the software was to determine how the virtual objects were going to be placed on the set. In the beginning I thought we’d simply drop spatial anchors on the physical location and then share these anchors with each HoloLens. This proved to be impractical for a number of reasons.

Firstly, I’ve had lots of issues sharing spatial anchors across devices. They often appear in the wrong places. Not only that, but anyone who has been to a conference will tell you wifi access is spotty, if available at all. Having to use the Internet to transfer spatial anchors around might not be possible.

Plus, without having the physical set to scan we would have no way to test the final layout of eggs and other objects until we got to the location a day or so before the event.

For most of the app’s development, we used Mike’s 3D model of the set design in Unity3D to place the objects on. Then we used AfterNow’s technique of placing three spatial anchors in the corners of the set, saving them, and then spawning the game level at the center of these points. This real world alignment process is a little error prone, but does work.

These anchors are saved locally to the HoloLens so the setup process only has to be done once. Whenever players put on the headset after the app is restarted, they can jump right into the experience–no alignment necessary.

IMG_0854 2

The final set, assembled at the Los Angeles Convention Center

We also attempted to match lighting as accurately as possible by taking spherical photos using a RICOH THETA camera at different points inside the set and using them as cubemaps in Unity3D. These cubemaps provided convincing reflections, and can also be used with IBL shaders to help make virtual objects match the real world environment.

TO OCCLUDE OR NOT TO OCCLUDE

The added bonus of having a fixed set is we know the exact shape of the world. This meant we had the option of using a LIDAR scan of the environment as an occlusion mesh instead of the one built internally by HoloLens.

There are lots of advantages to this. Most importantly, the 3D scan can be optimized for performance by reducing the number of polygons and adding detail where the scan may have missed a few things. This makes the occlusion much more accurate while giving a bonus in performance.

Mimic3D made an amazing scan of the set once it was finally set up in the convention center. We saw quite a reduction in poly count from our own retopologized mesh vs. the dynamically generated HoloLens one. This highly optimized mesh also let us do a neat “The Matrix” style grid effect that appears to pour out over the world with a simple shader.

IMG_0828

Mimic3D’s amazing LIDAR scan

Perhaps most importantly, with over a dozen HoloLens devices to set up before the show, not having to generate a good occlusion mesh on each headset saved a lot of preparation time.

WHERE DO I LOOK?

The limited FOV of the HoloLens made it very important to guide the user’s gaze to where the action is. In some cases our assumptions of where users would look and how they moved through the environment was not exactly lined up with reality. One prime example is the rabbits.

In the back right corner of the forest there is a group of three rabbits that scurry away when you get close. We spent a lot of time tweaking the trigger size, position, and animation of the rabbits to make sure people had plenty of time to notice them. We even had Somatone create a positional audio effect to call attention to this. However, it was far too easy for someone to back into the trigger volume and have the bunnies escape without seeing them. A lot of participants reported not seeing rabbits at all–perhaps, the most important creature in the entire experience!

IN CONCLUSION

The lines were huge but everyone seemed to have a smile on their face when leaving the booth. The press loved it, too. I consider this a huge success for all parties involved and a leading example of how Mixed Reality can be used for things far more interesting than enterprise apps.

Building on what we’ve learned at VRLA, I’m totally ready to tackle a much more elaborate Mixed Reality entertainment experiences. Since the event, I’ve recieved a lot of interest in doing this type of project on a larger scale. Who knows what we might build next!

My Week with PSVR

Full disclosure, I’ve had a PSVR devkit for some time now, so this isn’t my first experience with the device. However, this certainly is my first taste of most PSVR launch content. I figured I’d post my impressions after a week with my PSVR launch bundle.

Best Optics In the Business

PSVR does not use fresnel lenses, thus you don’t see any god rays and glare on high contrast screens. Vive and Rift both suffer from these problems, which makes PSVR look a lot better than the competition. Many cite the lower resolution of the PSVR display as a problem, but I don’t think numbers tell the whole story. The screen door effect is not very noticeable, and I suspect there’s some way PSVR is packing those pixels together that make the slightly lower resolution a non-issue. PSVR looks great.

Fully Integrated With Sony’s Ecosystem

The great thing about the platform is they are combining a mature online store and gaming social network with VR. In many cases PSVR is ahead of the competition in community features. When you first don the PSVR headset, you’ll see the standard PlayStation 4 interface hovering in front of you as a giant virtual screen. Thus, all current PSN features are available to you in VR already. You can even click the Share button and stream VR gameplay live. There’s also a pop up menu to manage your friends list, invites, etc. inside any VR experience. The only weird thing is when you get an achievement you hear the sound, but don’t see any overlay telling you what you did.

Tracking Issues

PSVR uses colored LED lights for optical tracking–essentially the same solution Sony created for their PS3 Move controllers in 2010. In fact, the launch bundle comes with what seem to be new, deadstock Move controllers as its hand tracking solution.

Tracking is iffy. It seems that lamps, bright lights, and sunlight streaking through windows can throw PSVR’s tracking off. I find that it works much better at night with the room lights visible to the PS4 Eye camera turned off. I also replaced my original PlayStation 4 Eye camera with the V2 version in the launch bundle to no avail.

Even more annoying is calibration. Holding the PSVR up in precise positions so that the lights are visible to the camera can be quite a pain. Not only that, but many games require their own calibration involving standing in a place where your head fits inside a camera overlay representing the best position to play in.

The hand controllers are jittery even under the best circumstances. Some games seem to have smoother tracking than others–probably via filtering Move input data. Still, given the price of the bundle, Move is an acceptable solution. Just not ideal.

One advantage to this approach is PSVR can also track the DualShock 4 via that previously annoying light bar on the back. Having a positionally tracked controller adds an element of immersion to non-hand tracked games previously unseen.

The Content

Despite PSVR using a PS4 which pales in power compared to, say, a juiced up Oculus-ready PC, the PSVR launch experiences are second to none. Sony is an old pro at getting together strong titles to launch a new platform. They have made some great choices here.

Worlds

The amount of free content you get with the Launch Bundle is staggering. In addition to the new VR version of Playroom and a disc filled with free demos, you also get Worlds–Sony London’s brilliant showcase of VR mini games and experiences. The Deep is a perfect beginner’s VR introduction–a lush, underwater experience that rivals anything I’ve seen on Rift or Vive. London Heist is my favorite, combining storytelling and hand-tracked action in what is often compared to a VR Guy Ritchie film.

Arkham VR

This is the single coolest VR experience I’ve ever had. It’s really more like a narrative experience with some light gameplay elements. Some are complaining that this barely qualifies as a game and is way too short for $20, but I disagree. This is the gold standard in VR storytelling–a truly unique experience that a lot of developers can learn from. It combines puzzle solving, story, interactive props, and immersive environments into a VR experience that makes you really feel like the Caped Crusader. This is the game I use to showcase PSVR and nobody has left disappointed.

cvb9h4uusae0dk4

Battlezone

Battlezone is my other favorite launch title right now, if I can find other people online (a definite problem given the small, but growing PSVR user base). This is a VR update to Atari’s coin-op classic in the form of a co-op multiplayer vehicle shooter. Guide a team of futuristic tank pilots over a randomly generated hexagonal map as you journey on a quest to destroy the enemy base. This game requires great teamwork and voice communication, which makes it all the more immersive. The positionally tracked DualShock 4 adds to the immersion in the cockpit as well.

Rigs

Guerilla does everything wrong (including uninterruptible tutorials) in VR here, defying all conventions. I have no problems with it, but this makes almost everyone I know violently ill. Apparently I am immune to VR sickness. Rigs is probably unplayable by the vast majority of players even with all the comfort modes turned on. If you want to test your so-called “VR Legs”, then try this game. If you can manage to play this without puking, you’re in for a great competitive online experience–that is, if you can find other players easily.

Wayward Sky

This game started out last year as a Gear VR launch title called Ikarus, which was pulled from the store shortly after its release. Uber’s small mobile VR demo has now reappeared on PSVR as the expanded and enhanced Wayward Sky–an innovative take on point-and-click adventure games in VR. The first stage is essentially a remixed and remastered version of the short Gear VR demo that came out last year. Once you complete this stage, the game opens up with a lot more levels and an all new story line. This is another gentle introduction to VR as it doesn’t involve a lot of movement or complicated mechanics. It’s largely point and click puzzle solving affair, with a few areas that require you to use your hands to manipulate objects.

In Conclusion

img_3678

My dream VR platform would be PSVR’s optics, Vive’s tracking, and Oculus’ controllers. Until that singularity happens, we’re stuck with all of these different systems. PSVR is incredibly compelling, and the platform I recommend to most people. It’s cheap and surprisingly good. Most of my current favorite VR games are on PSVR right now. I personally don’t find its limitations a problem–but it will be interesting to see how the average gaming public responds. Initial sales are promising, and there is way more high profile VR content on the horizon. Dare I say Sony has won this first round?

My Favorite VR Experiences So Far

Now that I’ve had plenty of time to go through the launch content of both Oculus and Vive, I figured I’d highlight my favorite experiences you can try for both devices instead of a typical product review. Many of these games are available for both platforms, while some are exclusive.

IMG_2589

My Retail Vive Finally Arrived!

Adr1ft (Oculus)

This is the flagship release for Oculus and deservedly so. Although not a pure VR experience (it also works as a standard game), it’s an absolutely wild trip in VR. Billed as a First Person Experience (FPX), it ranks somewhere between a walking simulator like Firewatch and an adventure such as Bioshock on the “Is It a Game?” scale.

This is consistently one of the top-selling Oculus titles, yet ranks near the bottom on comfort. I had no nausea issues at all, but I generally don’t feel uncomfortable in most VR games. I can see how free-floating in zero gravity, desperately grasping at oxygen canisters as you slowly suffocate to death inside a claustrophobic space suit can cause issues with those prone to simulation sickness. Regardless, this shows that it pays to be hardcore when making VR experiences–especially at this early adopter stage of the market.

A stunning debut for Adam Orth’s threeonezero studio.

Firma (Oculus)

This perhaps one of my absolute favorite pure VR games so far. Think Lunar Lander, Space Taxi or Thrust in VR. If this was a standard video game, it would be mundane, but as a VR experience I really do feel like I have a job piloting a tiny lander craft on a desolate moon base. It actually sort of achieves presence for me–but not the feeling of being in another reality…more like being in an ‘80s sci-fi movie.

Originally available via Oculus Share for years–it’s obvious that a lot of work has been put into this game to get it here for the commercial Oculus release. There are tons of missions, great voice acting, and a lot of fun mechanics and scenarios. This game is giving me plenty of ideas on how to adapt my old Ludum Dare game to VR.

Strangely, this game is in Oculus’ Early Access section, even though I consider it a complete game.

The Apollo 11 Virtual Reality Experience (Oculus, Vive)

An astounding educational journey through America’s moon landing told via VR. This is better than any field trip I took as a kid to the Boston Museum of Science, that’s for sure. This is just the tip of the spear when it comes to education and VR.

Hover Junkers (Vive)

Hover Junkers requires the most physical activity out of any VR game I’ve played–So much so that after 20 minutes of shooting, cowering behind my hovercraft’s hull for cover, and frantically speeding around post-apocalyptic landscapes, my Vive was soaked in sweat. One thing is for sure, public VR arcades are going to need some kind of solution to keep these headsets sanitary. Hover Junkers certainly is the most exciting multiplayer experience I’ve had in VR so far.

Budget Cuts (Vive)

The absolute best example of room scale VR. I didn’t really get it when watching the videos, but when I was finally able to try the demo on my own Vive….wow. This is the game I let everyone try when they first experience Vive. It really nails the difference between seated, controller-based VR and a room scale hand-tracked experience. This is the first “real game” I’ve played that uses all of these elements. So many firsts here, and done so well.

The past month has been a very encouraging start for VR. At this early stage there are already several games that give me that “just one more try” lure. This is surprising given that many current VR titles are small in scope, and in some cases partially-finished early access experiences. With the launch of PSVR later this year, we’re sure to see more full-sized VR games…whatever that means.

VR with a Gamepad Sucks

I was kind of bummed my first day of Oculus Connect 2.

FullSizeRender

Last year’s Oculus Connect was revelatory to me. Despite having worked on two different Gear VR titles at the time, the Crescent Bay demo was incredible in comparison. From Oculus’ own vignette demos to Epic’s Showdown sequence–the leap in quality from DK2 to Crescent Bay was astounding. Everyone walked out of that demo with a huge simile on their faces.

The first demos I tried at OC2 were the Gamepad demos. Oculus spent an absurd amount of time at their E3 keynote talking about how amazing it was that they were launching with the XBox 360 controller as the input device. At Oculus Connect, I put this claim to the test.

Every game from EVE Valkyrie to Edge of Nowhere seemed like playing a regular video game strapped to my face. I felt like I was playing an XBOX One through binoculars. In fact, a few of the games made me a little queasy–which I’m usually not susceptible to.

Maybe I’m just jaded having been developing gamepad VR experiences on Gear VR for a while, I thought.

Later on I tried Toybox which is a cool tech demo but doesn’t really illustrate how you’d play an actual game for any length of time with the Touch controllers. In fact, I found the controllers a little hard to use compared to the Vive. They have tons of confusing buttons and getting the finger gestures right seemed to take a little bit of work.

I was leaving the demo area and getting ready to head home when a friend of mine who works for Oculus stopped to ask what I thought. I told him honestly that I felt last year’s demos were better–they were more immersive and interesting. Although a little taken aback at my impressions, he strongly suggested I come by tomorrow for the next set of demos. He couldn’t tell me what they were, but promised they’d be awesome.

The Oculus Connect app sent a notification alerting me that new demo registrations would be available at 8 AM. I set my alarm and woke up the next morning to register for the Touch demos via my iPhone. I promptly slept through the keynote and arrived on the scene at noon for my demo.

We were only allowed to try two games, and it was heavily suggested I try Epic’s “Bullet Train” experience. Having not seen the keynote, I had no idea what I was getting into.

Bullet Train is mind blowing.

Bullet Train is essentially Time Crisis in VR. When I saw the Showdown demo last year I thought a game like this in VR would be a killer app. One of my favorite coin-ops of all time is Police 911–which motion tracks your body with a pair of cameras to duck behind obstacles. I thought doing this in VR would be amazing. However, last year there were no hand tracking controls–it was just a vague idea.

Here, Epic took the Touch controllers and made an incredible arcade shooter experience that should be a killer app should Epic choose to develop this further. Oculus really needs to do everything in their power to get Epic to produce this as a launch title for the Touch controllers.

The touch controls make all the difference. From handling weapons and grenades to plucking bullets out of the air in slow motion, Bullet Train really drives home how flexible the Touch controls are. Unlike Vive which is like holding a set of tools, these let you reach out and grab stuff–Even pump a shotgun.

The combination of standing in a motion tracked volume and visceral interaction with the world using your hands–even with Touch’s primitive finger gesture technology–really immerses you in an experience way beyond what’s possible sitting in a chair with an XBox controller.

It’s disappointing that Touch won’t launch with Oculus’ headset. Hand tracking is absolutely required for a truly immersive experience. Developing games that support both Gamepad and Touch control is going to be difficult without diluting features for one or the other. I’ve experienced a similar issues developing games that work with Gear VR’s touchpad and Bluetooth gamepad.

I left Oculus Connect 2 I reinvigorated with the feeling that VR with hand tracking is the One True VR Experience. Gamepad is fine for mobile VR at the moment, but all of my PC and Console VR projects are now being designed around Touch and hand tracked input. It’s the only way!

Why I’m All In On Mobile VR

Last month I released Caldera Defense, a Virtual Reality tower defense game on Gear VR. This is the second Gear VR title I’ve worked on, and the first I’ve built and published from the ground up. (Not including my Oculus Mobile VR Jam submission) Caldera Defense is a free early access demo–basically a proof of concept of the full game–and the reaction has been great. Thousands of people have downloaded, rated, and given us valuable feedback. We’re busy incorporating it into the first update.

Caldera Defense featured on the Gear VR store

Originally I planned to use this as a demo to fund an expanded PC and Morpheus launch version of the game with greatly improved graphics, hours of gameplay, and additional features such as multiplayer and second-screen options.

However, pitching even a modestly budgeted console and PC VR game experience to publishers, or even the platforms themselves, is a tough sell. I’m sure at E3 next month we will see all sorts of AAA VR announcements. Yet, many traditional funding avenues for games remain skeptical of the opportunity VR presents.

Since the Caldera project began last year, mobile VR has morphed into a unique opportunity. With over a million Google Cardboards in the wild and new versions of the Gear VR headset in retail stores worldwide, there will be millions of mobile VR users before there’s comparable numbers on Oculus desktop, Vive, and Morpheus.

Is it possible that mobile VR will be a viable business before it is on PC and consoles? Most of my colleagues are skeptical. I’m not.

The economics work out. Due to the mobile nature of the experience, games and apps for these platforms tend towards the bite-sized. This greatly reduces the risk of mobile VR since assets optimized for mobile are simpler and casual VR experiences require less content to be built overall.

I can make a dozen mobile VR minimum viable products for the same budget of one modestly scoped Morpheus experience. From these MVPs I can determine what types of content gains the most traction with VR users and move in that direction. I can even use this data to guide development of larger AAA VR experiences later.

By this time next year it will be possible to monetize these users significantly, whether through premium content or advertising. It may be more valuable to collect a lot of eyeballs in mobile VR than breaking even on a multi-million dollar AAA launch tile. As we’ve seen in the past, acquiring a huge audience of mobile players can lead to tremendous revenue streams.

Being on the Oculus desktop, Vive, or Sony’s Morpheus deck at launch is an enormous opportunity. In fact, I’m still searching for ways to produce the console and desktop version of Caldera Defense. However, if you lack the capital to produce at that scale, smaller mobile projects are much easier to bootstrap and the upside is huge.