The Basics of Hand Tracked VR Input Design

Ever since my revelation at Oculus Connect I’ve been working on a project using hand tracking and VR. For now, it’s using my recently acquired Vive devkit. However, I’ve been researching design techniques for PSVR and Oculus Touch to keep the experience portable across many different hand tracking input schemes. Hand tracking has presented a few new problems to solve, similar to my initial adventures in head tracking interfaces.

The Vive's hand controller

Look Ma, No Hands!

The first problem I came across when designing an application that works on both Vive and Oculus Touch is the representation of your hands in VR. With Oculus Touch, most applications feature a pair of “ghost hands” that mimic the current pose of your hands and fingers. Since Oculus’ controllers can track your thumb and first two fingers, and presumably the rest are gripped around the handle, these ghost hands tend to accurately represent what your hands are doing in real life.

Oculus Touch controller

This metaphor breaks down with Vive as it doesn’t track your hands, but the position of the rod-like controllers you are holding. Vive games I’ve tried that show your hands end up feeling like waving around hands on a stick–there’s a definite disconnect between the visual of your hands in VR and where your brain thinks they are in real life. PSVR has this problem as well, as the Move controllers used with the current devkit are similar to Vive’s controllers.

You can alleviate this somewhat. Because there is a natural way most users tend to grip Move and Vive controllers, you can model and position the “hand on a stick” in the most likely way the controllers are gripped. This can make static hands in VR more convincing.

In any case, you have a few problems when you grab an object.

For Oculus, the act of grabbing is somewhat natural–you can clench your first two fingers and thumb into a “grab” type motion to pick something up. In the case of Bullet Train, this is how you pick up guns. The translucent representation of your hands means you can still see your hand pose and the gripped object at the same time. There’s not much to think about other than where you attach the held object to the hand model.

It also helps that in Bullet Train the objects you can grab have obvious handles and holding points. You can pose the hand to match the most likely hand position on a grabbed object without breaking immersion.

With Vive and PSVR you have a problem if you are using the “hand on a stick” technique. When you “grab” a virtual object by pressing the trigger, how do you show the hand holding something? It seems like the best answer is, you don’t! Check this video of Uber Entertainment’s awesome Wayward Sky PSVR demo:

Notice anything? When you grab something, the hand disappears. All you can see is the held object floating around in front of you.

This is a great solution for holding arbitrary shaped items because you don’t have to create a potentially infinite amount of hand grip animations. Because the user isn’t really grabbing anything and is instead clicking a trigger on a controller, there is no “real” grip position for your hand anyway. You also don’t have the problem of parts of the hands intersecting with the held object.

This isn’t a new technique. In fact, one of the earliest Vive demos, Job Simulator, does the exact same thing. Your brain fills in the gaps and it feels so natural that I just never noticed it!

Virtual Objects, Real Boundaries

The next problem I encountered is what do you do when your hand passes through virtual objects, but the objects can’t? For instance, you can be holding an object, and physically move your real, tracked hand through a virtual wall. The held object, bound by the engine’s physics simulation, will hit the wall while your hand continues to drag it through. Chaos erupts!

You can turn off collisions while an object is held, but what fun is that? You want to be able to knock things over and otherwise interact with the world while holding stuff. Plus, what happens when you let go of an object while inside a collision volume?

What I ended up doing is making the object detach, or fall out of your virtual hand, as soon as it hits something else. You can tweak this by making collisions with smaller, non-static objects less likely to detach the held object since they will be pushed around by your hand.

For most VR developers these are the first two things you encounter when designing and experience for hand-tracking VR systems. It seems Oculus Touch makes a lot of these problems go away, but we’ve just scratched the surface of the issues needed to be solved when your real hands interact with a virtual world.

Why I’m All In On Mobile VR

Last month I released Caldera Defense, a Virtual Reality tower defense game on Gear VR. This is the second Gear VR title I’ve worked on, and the first I’ve built and published from the ground up. (Not including my Oculus Mobile VR Jam submission) Caldera Defense is a free early access demo–basically a proof of concept of the full game–and the reaction has been great. Thousands of people have downloaded, rated, and given us valuable feedback. We’re busy incorporating it into the first update.

Caldera Defense featured on the Gear VR store

Originally I planned to use this as a demo to fund an expanded PC and Morpheus launch version of the game with greatly improved graphics, hours of gameplay, and additional features such as multiplayer and second-screen options.

However, pitching even a modestly budgeted console and PC VR game experience to publishers, or even the platforms themselves, is a tough sell. I’m sure at E3 next month we will see all sorts of AAA VR announcements. Yet, many traditional funding avenues for games remain skeptical of the opportunity VR presents.

Since the Caldera project began last year, mobile VR has morphed into a unique opportunity. With over a million Google Cardboards in the wild and new versions of the Gear VR headset in retail stores worldwide, there will be millions of mobile VR users before there’s comparable numbers on Oculus desktop, Vive, and Morpheus.

Is it possible that mobile VR will be a viable business before it is on PC and consoles? Most of my colleagues are skeptical. I’m not.

The economics work out. Due to the mobile nature of the experience, games and apps for these platforms tend towards the bite-sized. This greatly reduces the risk of mobile VR since assets optimized for mobile are simpler and casual VR experiences require less content to be built overall.

I can make a dozen mobile VR minimum viable products for the same budget of one modestly scoped Morpheus experience. From these MVPs I can determine what types of content gains the most traction with VR users and move in that direction. I can even use this data to guide development of larger AAA VR experiences later.

By this time next year it will be possible to monetize these users significantly, whether through premium content or advertising. It may be more valuable to collect a lot of eyeballs in mobile VR than breaking even on a multi-million dollar AAA launch tile. As we’ve seen in the past, acquiring a huge audience of mobile players can lead to tremendous revenue streams.

Being on the Oculus desktop, Vive, or Sony’s Morpheus deck at launch is an enormous opportunity. In fact, I’m still searching for ways to produce the console and desktop version of Caldera Defense. However, if you lack the capital to produce at that scale, smaller mobile projects are much easier to bootstrap and the upside is huge.

A Weekend at Oculus Connect

I spent this past weekend at Oculus Connect and have just now had the time to process what I saw. For Oculus to go from a humble Kickstarter project a few years ago to a capacity filled conference rife with amazing demos and prototypes by countless developers is mind-boggling. I know I said VR in 2014 is like Mobile in 2002, but the pace of progress is staggering. The maturation path for VR is going to be MUCH quicker. Is it 2005 already?

...and all I got was this lousy t-shirt.

…and all I got was this lousy t-shirt.

As I stated before, Gear VR is the most important wearable platform in the universe. I’ve been developing Gear VR games for a while and am thoroughly convinced this wireless, lightweight platform will have far more reach than VR tethered to your desktop.

The GearVR demo area.

The GearVR demo area.

The apps on display were great, but I even saw a few Gear VR demos from random developers in the hotel hallways that blew away what were officially shown in Samsung’s display area. Developer interest for Gear VR is very high. Once it’s commercially available, a flood of content is soon upon us.

Despite the intense interest in the platform, I spoke to a few desktop and console developers who dismissed Gear VR as a distraction and are ignoring it–which I think is really short-sighted.

It’s true that there may be a division in audiences. Gear VR may be the larger, casual audience while apps built around Oculus’ astounding Crescent Bay platform could be for a highly monetizable market of core enthusiasts. Either route is smart business. Depending on how long you can hold out for customer traction, that is.

Oh, and Crescent Bay…was a revolution. There’s probably not much more to be said about it that hasn’t already–but the ridiculous momentum behind Oculus’ path from the DK1 to Crescent Bay makes me question the competition. Oculus has hired all of the smartest people I know and have billions of dollars to spend on VR R&D–which is their main business, not a side project. Will competitors like Sony really commit enough resources to compete with the relentless pace of Oculus’ progress?