My Week With HoloLens

holobox

Microsoft ships the HoloLens and Clicker accessory in the box

My HoloLens development kits finally arrived a week ago. I’ve spent a great deal of time using the device over the past week. I figured I’d post my impressions here.

This Really Works

When I first put my HoloLens on, I placed an application window floating above my kitchen table. Suddenly, I realized I hadn’t taken out the garbage. Still wearing the device, I ran downstairs to put something in the trash. I narrowly missed my neighbor–successfully avoiding an awkward conversation about what this giant contraption on my face does.

When I returned to my kitchen, the application was still there–hovering in space.

As I’ve stated before, Microsoft blew me away. HoloLens is an absolutely incredible leap from previous generation AR glasses (and MUCH cheaper, believe it or not). It also does everything Tango does but at a much higher level of performance and precision. Which means most applications built on Tango can be directly moved over to HoloLens.

HoloLens is fast and intuitive enough to attempt getting actual work done with it. Yet, a lot of my time spent is just trying to make silly videos like this.

It’s A Full Blown Windows Machine

HoloLens isn’t just a prototype headset–it’s a full featured desktop Windows PC on your face. Not only can you run “Windows Holographic” apps, but any Universal Windows App from the Windows Store. Instead of these applications running in a window on a monitor, they float around in space–positioned wherever you choose.

Although HoloLens really does need a taskbar of some kind. It’s way too easy to forget where Skype physically is because you launched it in the bathroom.

It also helps to connect a Bluetooth keyboard and mouse when running standard applications. Gestures can’t give you the input fidelity of a traditional mouse, and typing in the air is a chore.

HoloLens’ narrow FOV makes using a regular Windows app problematic–as the screen will get cut off and require you to move your head around to see most of it. Also, if you push a window far enough into the background so you can see the whole thing, you’ll notice HoloLens’ resolution is a little low to read small text. We’re going to need a next generation display for HoloLens to really be useful for everyday computing.

Microsoft Has Created A New Input Paradigm

HoloLens can seemingly only recognize two gestures: bloom and “air tap”. Blooming is cool–I feel like a person in a sci-fi movie making the Windows start menu appear in the air by tossing it up with a simple gesture.

The air tap can be unintuitive. Most people I let try the HoloLens poke at the icons by stabbing them with a finger. That’s not what the air tap is for. You still have to gaze at a target by moving your head and then perform the lever-like air tap gesture within the HoloLens camera’s view to select what the reticule is on.

HoloLens can track the motion of your finger and use it as input to move stuff around (such as application windows), but not detect collisions between it and virtual objects. It’s as if it can detect the amount your finger moves but not its precise location in 3D space.

Using apps while holding your hand out in front of the headset is tiring. This is why Microsoft includes the clicker. This is a simple Bluetooth button that when pressed triggers the air tap gesture. Disappointingly, the clicker isn’t trackable–so you can’t use it as a true finger replacement.

Microsoft has adapted Windows to the holographic model successfully. This is the first full blown window manager and gesture interface for augmented reality I’ve ever seen and it’s brilliant. After a few sessions with the device, most people I’ve let use it are launching apps and moving windows around the room like a pro.

This Thing Is Heavy

Although the industrial design is cool in a retro ‘90s way, this thing is really uncomfortable to use for extended periods of time. Maybe I don’t have it strapped on correctly, but after a 20 minute Skype session I had to take the device off. I felt pain above the bridge of my nose. When I looked in the mirror, I saw what can only be described as ‘HoloHead’

holohead

The unfortunate symptom of “HoloHead”

The First Generation Apps Are Amazing

There already are great free apps in the Windows Store that show off the power of the HoloLens platform. Many made by Asobo Studio–a leader in Augmented Reality game development.

Young Conker

Young Conker is a great example of HoloLens as a games platform. The game is simple: after scanning your surroundings, play a familiar platform game over the floors, walls, tables and chairs as Conker runs, jumps and collects coins scattered about your room.

Conker will jump on top of your coffee table, run into walls, or be occluded by a chair as if he were walking behind it–well, depending on how accurate your scan is. The fact that this works as well as it does is amazing to me.

Fragments

One of the first true game experiences I’ve ever played in augmented reality. You play the part of a futuristic detective, revisiting memories of crimes as their events are re-created holographically in your location. Characters sit on your furniture. You’ll hunt for pieces of evidence scattered about your room–even under tables. It really is an incredible experience, As with Conker, it requires some pre-scanning of your environment. However, applications apparently can share scans between each other as Fragments was able to re-use a scan of my office I previously made with another app.

Skype

When Skyping with a person not using HoloLens, you simply place their video on a wall in your surroundings. It’s almost like talking to someone on the bridge of the Enterprise, depending on how big you make the video window.

When Skyping with another HoloLens user, you can swap video feeds so either participant can see through the other’s first person view. While looking at someone else’s video feed as a floating window, you can sketch over it with drawing tools or even place pictures from your photos folder in the other person’s environment. 2D lines drawn over the video feed will form around the other user’s real-world in 3D–bending around corners, or sticking to the ceiling. 

Conclusion

As a consumer electronics device, HoloLens is clearly beta–maybe even alpha, but surprisingly slick. It needs more apps. With Wave 2 underway, developers are working on just that. In my case, I’m moving all of my Tango projects to HoloLens–so you’ll definitely be seeing cool stuff soon!

HoloLens is Ready for Prime Time

Microsoft recently invited me to try out HoloLens at their Venice space on Abbot-Kinney. Having just won the AR/VR award in the Tango App Contest for InnAR Wars, I jumped at the chance.  After developing for Google Tango for over a year, I had a long wishlist of features I’m looking for in an AR platform. In the unlikely event that HoloLens was for real, I could make InnAR Wars exactly how I envisioned it at the start of the project.

Entering the Hololens Demo Zone

 

My skepticism was well warranted. Having worked in AR for the past 5 years, I’ve seen my share of fake demo videos and smoke-and-mirrors pitches. Every AR company is obligated to put out faked demo videos that they inevitably fail to live up to. Just look at this supercut of utterly ridiculous promotional videos. I was curious if the staged HoloLens demos weren’t much better.

I had heard many polarizing opinions about HoloLens from people who have tried it. Some felt it was an incredible experience while others told me it was the single most overhyped disappointment in consumer electronics history.

After trying it for myself, I can say HoloLens is the real deal.

The demo I tried was the “X-Ray” game shown at BUILD no too long ago. This version was a little simpler than this staged demonstration–your arm isn’t magically turned into a plasma cannon. Instead, your hold your hand out in front of the device and “air tap” to fire at enemies that appear to be coming out of cracks in the walls. Occasionally you can give a voice command to freeze time, Matrix-style, and take out enemies while they are vulnerable.

A simple game, for sure, but a great demonstration of the HoloLens’ capabilities.  

The device is clearly a prototype. It’s bulky, looks like a vision of the future from a ’90s sci-fi flick, and it even BSODed on me which was kind of hilarious.  Still, I was thoroughly impressed with HoloLens and here’s why:

Meshing

When the game starts, you have to look around the room and watch it build geometry out of the walls and other objects in the area. HoloLens builds a mesh out of the real world with a depth camera that is then used in gameplay. It’s kind of like building a video game level out of all the walls and floors in your room. This mesh is then used to place virtual wall cracks and spawn points for enemies during gameplay. Once you’ve build a complete mesh out of the room, the game begins.

This same meshing process is possible with Google Tango, but it’s slow and temperamental. Still, very impressive given it’s in a tablet you can buy right now. In fact, I used Tango’s meshing capabilities to place floor traps in InnAR Wars.

I was impressed with HoloLens’ rapid meshing as I moved around my environment. It even handled dynamic objects, such as the woman guiding me through the demo. When I looked at her during the meshing phase, she quickly transformed into a blocky form of red polygons.

Display

Initially I was disappointed in the display. Much like Epson’s Moverio or ODG’s R7 glasses, it projects the augmentation on a part of the “glass” in front of your eyes. This means you see a distracting bright rectangle in the middle of your vision where the augmentation will appear. Compared to ODG’s R7s, HoloLens seemed to have higher contrast between the part of the display that’s augmented and the part that’s not. There also is an annoying reflection that looks like a blue and green “shadow” of the augmentation above and below the square.

While playing the game this didn’t matter at all. Although everything is somewhat translucent, if the colors are bright enough the virtual objects appeared solid. Combined with rock solid tracking on the environment, I soon forgot all about contrast issues and internal reflection problems on the display. A these issues can be dealt with through art and the lighting of the room you are playing in. Plus, a Microsoft representative assured me the display is even better in the current version still in their labs.

Field of View

The top issue people have with HoloLens is the field of view. People complain that it only shows graphics in a postcard sized space in front of your vision. I had rock bottom expectations for this having developed applications on previous generation wearable AR displays. Although HoloLens’ augmentation is limited to a small square in front of your vision, this space is easily 2X the size of other platforms. In the heat of the action while playing X-Ray, I mostly forgot about this restriction.

Field of view is not an easy thing to solve–it’s a fundamental problem with the physics of waveguide optics. I’m not sure how much progress Microsoft can make here. But the FOV is already wide enough for a lot of applications.

I’m All In

Part of the pitch is the “Windows Holographic” platform. That is, in the future you won’t have screens. With HoloLens you’ll be able to place Windows applications in mid-air anywhere in your office. Applications will virtually float around you like monitors hovering in space. (Magic Leap’s fake demo video shows an example of how this would work) This can actually be done right now with HoloLens and existing Windows applications. Supposedly, some Microsoft engineers wear HoloLens and have integrated the “holographic” workspace into their daily use.

Beyond gaming applications, I am on board with this screen-free future. Your tablet, computer, even your phone or smartwatch will merely be a trackable object to display virtual screens on. Resolution and screen size will be unlimited, and you can choose to share these virtual screens with other AR users for collaboration.  No need to re-arrange your office for that huge 34 inch monitor. You can simply place it in 3D space and overlay it on top of reality. Think of all the extra stuff your phone could do if it didn’t have to power a giant LCD display! It’s definitely on its way. I’m just not sure exactly when.