My Week With Project Tango

A few weeks back I got into Google’s exclusive Project Tango developers program. I’ve had a Tango tablet for about a week and have been experimenting with the available apps and Unity3D SDK.

Project Tango uses Movidius’ Myriad 1 Vision Processor chip (or “VPU”), paired with a depth camera not too unlike the original Kinect for the XBOX 360. Except instead of being a giant hideous block, it’s small enough to stick in a phone or tablet.

I’m excited about Tango because it’s an important step in solving many of the problems I have with current Augmented Reality technology. What issues can Tango solve?

POSITIONAL TRACKING

First, the Tango tablet has the ability to determine the tablet’s pose. Sure, pretty much every mobile device out there can detect its precise orientation by fusing together compass and gyro information. But by using the Tango’s array of sensors, the Myriad 1 processor can detect position and translation. You can walk around with the tablet and it knows how far and where you’ve moved. This makes SLAM algorithms much easier to develop and more precise than strictly optical solutions.

Also, another problem with AR as it exists now is that there’s no way to know whether you or the image target moved. Rendering-wise, there’s no difference. But, this poses a problem with game physics. If you smash your head (while wearing AR glasses) into a virtual box, the box should go flying. If the box is thrown at you, it should bounce off your head–big distinction!

Pose and position tracking has the potential to factor out the user’s movement and determine the motion of both the observer and the objects that are being tracked. This can then be fed into a game engine’s physics system to get accurate physics interactions between the observer and virtual objects.

OCCLUDING VIRTUAL CHARACTERS WITH THE REAL WORLD

Anyway, that’s kind of an esoteric problem. The biggest issue with AR is most solutions can only overlay graphics on top of a scene. As you can see in my Ether Drift project, the characters appear on top of specially designed trading cards. However, wave your hand in front of the characters, and they will still draw on top of everything.

Ether Drift uses Vuforia to superimpose virtual characters on top of trading cards.

Ether Drift uses Vuforia to superimpose virtual characters on top of trading cards.

With Tango, it is possible to reconstruct the 3D geometry of your surroundings using point cloud data received from the depth camera. Matterport already has an impressive demo of this running on the Tango. It allows the user to scan an area with the tablet (very slowly) and it will build a textured mesh out of what it sees. When meshing is turned off the tablet can detect precisely where it is in the saved environment mesh.

This geometry can possibly be used in Unity3D as a mesh collider which is also rendered to the depth buffer of the scene’s camera while displaying the tablet camera’s video feed. This means superimposed augmented reality characters can accurately collide with the static environment, as well as be occluded by real world objects. Characters can now not only appear on top of your table, but behind it–obscured by a chair leg.

ENVIRONMENTAL LIGHTING

Finally, this solves the challenge of how to properly light AR objects. Most AR apps assume there’s a light source on the ceiling and place a directional light pointing down. With a mesh built from local point cloud data, you can generate a panoramic render of where the observer is standing in the real world. This image can be used as a cube map for Image-based lighting systems like Marmoset Skyshop. This produces accurate lighting on 3D objects which when combined with environmental occlusion makes this truly a next generation AR experience.

A QUICK TEST

The first thing I did with the Unity SDK is drop the Tango camera in a Camera Birds scene. One of the most common requests for Camera Birds was to be able to walk through the forest instead of just rotating in place. It took no programming at all for me to make this happen with Tango.

This technology still has a long way to go–it has to become faster and more precise. Luckily, Movidius has already produced the Myriad 2, which is reportedly 3-5X faster and 20X more power efficient than the chip currently in the Tango prototypes. Vision Processing technology is a supremely nerdy topic–after all it’s literally rocket science. But it has far reaching implications for wearable platforms.

Samsung Gear VR Development Challenges with Unity3D

As you may know, I’m a huge fan of Oculus and Samsung’s Gear VR headset. The reason isn’t about the opportunity Gear VR presents today. It’s about the future of wearables–specifically of self-contained wearable devices. In this category, Gear VR is really the first of its kind. The lessons you learn developing for Gear VR will carry over into the bright future of compact, self-contained, wearable displays and platforms. Many of which we’ve already started to see.

The Gear VR in the flesh (plastic).

The Gear VR in the flesh (plastic).


Gear VR development can be a challenge. Rendering two cameras and a distortion mesh on a mobile device at a rock solid 60fps requires a lot of optimization and development discipline. Now that Oculus’ mobile SDK is public and having worked on a few launch titles (including my own original title recently covered in Vice), I figured I’d share some Unity3D development challenges I’ve dealt with.

THERMAL ISSUES

The biggest challenge with making VR performant on a mobile devices is throttling due to heat produced by the chipset. Use too much power and the entire device will slow itself down to cool off and avoid damaging the hardware. Although the Note 4 approaches the XBOX 360 in performance characteristics, you only have a fraction of its power available. This is because the phone must take power and heat considerations in mind when keeping the CPU and GPU running at full speed.

With the Gear VR SDK you can independently tell the device how fast the GPU and CPU should run. This prevents you from eating up battery when you don’t need the extra cycles, as well as tune your game for performance at lower clock speeds. Still, you have to be aware of what types of things eat up GPU cycles or consume GPU resources. Ultimately, you must choose which to allocate more power for.

GRAPHICAL DETAIL

The obvious optimization is lowering graphical detail. Keep your polycount under 50k triangles. Avoid as much per pixel and per vertex processing as possible. Since you have tons of RAM but relatively little GPU power available–opt for more texture detail over geometry. This includes using lightmaps instead of dynamic lighting. Of course, restrict your usage of alpha channel to a minimum–preferably for quick particle effects, not for things that stay on the screen for a long period of time.

Effects you take for granted on modern mobile platforms, like skyboxes and fog, should be avoided on Gear VR. Find alternatives or design an art style that doesn’t need them. A lot of these restrictions can be made up for with texture detail.

A lot of standard optimizations apply here–for instance, use texture atlasing and batching to reduce draw calls. The target is under 100 draw calls, which is achievable if you plan your assets correctly. Naturally, there are plenty of resources in the Asset Store to get you there. Check out Pro Draw Call Optimizer for a good texture atlasing tool.

CPU OPTIMIZATIONS

There are less obvious optimizations you might not be familiar with until you’ve gone to extreme lengths to optimize a Gear VR application. This includes removing as many Update methods as possible. Most update code spent waiting for stuff to happen (like an AI that waits 5 seconds to pick a new target) can be changed to a coroutine that is scheduled to happen in the future. Converting Update loops to coroutines will take the burden of waiting off the CPU. Even empty Update functions can drain the CPU–death by a thousand cuts. Go through your code base and remove all unnecessary Update methods.

As in any mobile game, you should be pooling prefabs. I use Path-o-Logical’s PoolManager, however it’s not too hard to write your own. Either way, by recycling pre-created instances of prefabs, you save memory and reduce hiccups due to instantiation.

IN CONCLUSION

There’s nothing really new here to most mobile developers, but Gear VR is definitely one of the bigger optimization challenges I’ve had in recent years. The fun part about it is we’re kind of at the level of Dreamcast-era poly counts and effects but using modern tools to create content. It’s better than the good old days!

It’s wise to build for the ground up for Gear VR than to port existing applications. This is because making a VR experience that is immersive and performant with these parameters requires all disciplines (programming, art, and design) to build around these restrictions from the start of the project.

My Week with the Ouya

I’ve been skeptical of the Ouya since the beginning. Who would want to play mobile games scaled up on a TV with a box that’s about as powerful as a mid-range Android phone? Last year’s smashing Kickstarter success and resultant investor stampede was a mystery to me. Criticism of Ouya’s launch has been harsh. For a mere $99, I had to see if this microconsole ‘revolution’ is going to take off.

My Ouya

My Ouya

The console itself is a nice piece of industrial design–except for the fact that the vent is on the bottom. There also are shielding issues forcing some users to lay the box on its side or position it away from cabling to prevent interference with Bluetooth and WiFi. Contrasting Ouya’s sleek form factor is a cheap looking controller with no obvious way to put batteries in. At $99, you have to cut corners somewhere.

Setting up the Ouya is easy. It requires the creation of an Ouya account and a credit card link before playing anything. Some are complaining about this, but there’s no point in getting users on the platform if they aren’t going to pay. Making sure every user is billable and ready to buy stuff is a top priority for Ouya.

Despite having to create user accounts, Ouya appears to have no social network based around them. There aren’t any friends lists, achievements, or leaderboards. This is a glaring omission. Ouya’s somewhat unwieldy catalog is organized by curated arrays of games from celebrity editors and lists sorted by genre. At the very least, I’d like to see what some of my friends are playing.

What about the games? Frankly, they are atrocious. With over 100 launch titles, this is to be expected. Ouya’s games include half-baked ports from mobile (some don’t even bother to change the screen to landscape!), suspect exclusives, and tragically hip indie games about abused children.

As I dove deep into the Ouya catalog, I actually began to enjoy the experience. The diversity of games is charming. Plus, I’m a big connoisseur of b-movies and the Ouya feels like the MST3k of consoles. Blowing up games originally built for a 4-inch screen on my 60-inch TV really magnifies polish issues and sub-par graphics. Seeing just how bad some of these games are can be hilarious. However, I’m not quite sure that’s what Ouya had in mind.

Where Ouya shines is playing mobile game ports that originally had virtual buttons and d-pads. Games like Gunslugs and the unusually polished Ice Rage play MUCH better with physical controls. My current obsession is Ravensword: Shadowlands. Despite having bought it on the iPad last year, I purchased the Ouya version and have been ironically enjoying it as a PS2-looking Roger Corman-esque Z-grade Skyrim. It’s hilariously unpolished, absurdly ambitious, and infinitely more playable with a joypad despite the game being somewhat incompetently designed.

Ouya’s biggest flaw is the business model. All games are a hybrid of premium and f2p with a timed demo period. I learned way back in the feature phone era that demos are death for casual games. With an constant stream of content and bite-sized gameplay, users get their fill of your title for free and move on to the next demo. Forcing this model on developers is a major mistake. It’s the hallmark of a team clueless as to how to build a thriving ecosystem. If your mantra is “free the games,” then free the business model too. Straight f2p and video previews for premium titles is a much better option.

A lot of these early problems are to be expected from a company launching its first consumer electronics product so quickly. Also, people seem to have unrealistic expectations for a $99 box. It remains to be seen if the emerging category of microconsoles will be a market of any significance. Perhaps Ouya can break off some of their war chest for quality exclusive titles, or else they are doomed to drown in a sea of similar devices.

Native Code is Dead

Although Android has a larger market share when you count it by pure number of devices and users, iOS still dominates monetization. Research I did earlier this year about apathetic Android users still rings true. However, vast improvements in Android as an OS and Google Play as a way to monetize apps are changing this. Not to mention changes to the new iOS 6 App Store are making app discovery even more difficult on Apple devices. A lot of developers are grumbling about their fortunes on iOS and are looking elsewhere.

Platforms are volatile. Five years ago, Facebook was the ultimate destination for game developers. Now it’s a ghost town. iOS is the hot ticket now, but Android is becoming increasingly competitive. As a developer you need to be prepared to move platforms in an instant. For this reason, native code is dead.

Using a solution such as Unity3D, Flash, or HTML5 allows you to easily move apps from one device to another. Or, from mobile to the Web. Or, from Web to desktop. You get the idea. It’s true that each one of these solutions has tradeoffs in features or performance to accomplish frictionless cross-platform porting. However, most studios can’t afford to double their engineering staff to multiply the amount of platforms they deploy on.

If you’re starting a new project from scratch, you have to consider your cross-platform options:

Unity3D

As one of the (very few) detractors said of my recent GDCO 2012 presentation on Unity3D: “This guy was a bigger Unity fan-boy than the company would have been.” It’s true! I am a self-declared Unity3D zealot. My experience moving between platforms has been incredibly easy. You can check this older post on the process I went through to bring Brick Buddies to Android. Unity3D has issues, but it’s the best solution I’ve found yet.

Corona

I’ve not used Corona myself, but did research it a bit when deciding which platform to hang my hat on. I know other developers who have created very successful apps with it. The major drawbacks are it uses Lua as its scripting language and it still doesn’t allow native code extensions. Yeah, I know I said native was dead–but not totally dead. I occasionally have to write native code plug-ins for Unity3D to access parts of a platform’s API that aren’t abstracted in Unity itself. This is a critical feature. Also, Corona can’t be used on the web or desktop platforms.

Flash

Flash has a tragic branding problem. The declaration of mobile Flash’s death doesn’t mean Flash is dead on mobile. This means the browser plug-in on Android is defunct. Good riddance. Flash made the mobile browsing experience on Android unusable.

Adobe has stepped their game up with Flash’s iOS and Android exporters. The packager allows Flash projects to be exported as apps on the target device. Flash’s CS5 exporter was atrocious, but I’ve seen some impressive work with the latest version. Flash even supports native code extensions. Adobe’s extortionate demands for revenue share mean Flash is out of the running for me if I intend to use their more advanced features. Otherwise, it is a superior option over Corona.

HTML5

For game development, HTML5 is insane. If you really want to give it a shot, there are some relatively performant libraries such as impact.js that might help you out. I don’t recommend it. HTML5 isn’t much of a standard, needing a lot of workarounds for various browsers. Not to mention its horrible performance on mobile browsers. You just can’t win.

For non-game GUI-based apps (like Yelp or Evernote) HTML5 makes a lot of sense. PhoneGap/Cordova makes this possible by providing a framework for running HTML5/CSS/JS based applications inside a mobile web view and packaged as a native app. Coming from a native code background, constructing interfaces in HTML5/CSS seems absolutely insane. Friends don’t let friends write HTML/CSS. It should remain purely the output of tools such as Handheld Designer. HTML/CSS is becoming the Assembly Language of the web–it’s good to know, but hopefully you’ll never have to touch it.

Etc.

There are plenty of other options I haven’t mentioned: Moai, Marmalade, Titanium Studio, UDK, and the list goes on. The important thing is to research your platform independent option and find what’s best for you. For games, I’m biased towards Unity–but other options are just as valid…I guess. Obviously there are applications for which native code will always be the solution. Yet, for the incredible glut of dying console game studios “pivoting to mobile,” this is an increasingly remote option.

Kakao is the new Asian mobile gaming powerhouse

Kakao has leveraged their insanely popular mobile group messaging app, Kakao Talk, to launch a social gaming network called Game Center in Korea. This is similar to Tencent’s strategy of using their Chinese instant messaging network, QQ, to drive traffic to f2p PC games. Yet in the West, Apple’s iMessage seems to have killed off most group messaging startups, and the only one to pursue a gaming strategy released a tower defense game based on Shannon Tweed. No, really.

Kakao has successfully commissioned established game developers to produce high quality games for Game Center. The slickly produced Match-3 game, Anipang, recently rose to the top of the Korean App Store due to its use of Kakao Talk as a viral messaging channel. (It’s also available for Android) Encouraging users to message friends in order to get extra plays, Anipang uses social news update notifications similar to the early days of Facebook games to drive reach. Korean friends of mine have deleted the app from their iPhones to avoid the onslaught of game messages–reminiscent of the bad old days of “Lost Sheep” Farmville spam. Regardless, Kakao has mined a passionate and highly monetizable social network of gamers from Kakao Talk.

With over 55 million users, Kakao may eclipse the size of DeNA’s Mobage platform. It remains to be seen what their plans for the US are. Although Game Center will probably have to be renamed before it launches in the West. I hear Apple can be quite litigious when it comes to their intellectual property.

Come to my session next week at Austin GDC

By the way…On Wednesday, October 10th at 5:35 PM at Austin GDC come see my talk on cross-platform Unity3D mobile games development. Hey, it’s only 25 minutes long. How bad can it be?

Detecting Android Tablets and Phones in Unity3D

There’s been a few cases when I’ve needed to know whether one of my apps is running on an Android phone or a tablet. With Camera Birds’ gyro virtual camera, I encountered the fact that orientations are flipped differently on Android tablets and phones. By default, an Android tablet’s “natural” orientation is landscape, while a phone is portrait. This means that a 90 degree rotation is landscape on a phone, while on a tablet this becomes portrait. Get it? Neither do I. It’s another supremely awful decision that is simply par for the course with Android.

I ended up adapting a method from a Stack Overflow post used to determine natural orientation. I wrote it with Unity3D’s ability to access Android’s Java classes via the AndroidJavaClass object. This is a great feature of Unity3D that allows you to access the Android API through JNI without having to write a native plug-in.

The code is here. With this, you can tell if you are running on a tablet or a phone by checking for the natural orientation: landscape on a tablet, portrait on a phone. Even if you don’t need to flip gyro rotations, you might want to do this to separate tablet ad units from phone ads, for instance.

Android Users Are Apathetic

I launched the virtual pets spoof, Brick Buddies, on Android and iOS last month with zero PR. It was a crazy idea I wanted to make for no particular reason. Since both versions were launched with the same minimal PR effort (a mere tweet and a Facebook post), I figured I’d use this as an opportunity to analyze both platforms from a new perspective.

The top line: Brick Buddies on iOS gets 10X the downloads of the Android version. I got more iOS downloads in the first day than I did in 3 weeks on Google Play. Although both versions have earned a pittance, iOS users spend more money too.

Brick Buddies’ iOS release got picked up as a news story on at least 3 websites, including The Guardian, with no PR. I saw far more Facebook likes, shares, and retweets about Brick Buddies on iOS through my social graph than the Android release.

But wait, isn’t Android beating iOS in market share?

Pondering this, I thought about everyone I know that has an Android phone. They are my friends least into mobile tech. When I get a peek at their home screen, they hardly have any apps installed. They are seemingly content to have a slick-looking phone with a giant screen that makes calls and sends messages. iPhone users (myself included) appear to be platform zealots and voracious consumers of apps.

I had a hunch that most Android users just aren’t into their phones–which makes sense. If you aren’t into mobile tech, you’d probably settle for an Android device. Let’s face it–Aside from the Google ‘pure’ handsets, most really aren’t so great.

I put a survey up on Mechanical Turk to unscientifically poll the public about the habits of iPhone and Android users. I wanted to see how they like their phones and how excited they are about apps. I only got about 200 responses, so this really isn’t statistically significant.

Hey–it’s good fodder for an inflammatory linkbait post about Android users! Here are some results:

WHO ARE ANDROID USERS?

Both Android and iOS had the same ratio of men to women users.

My results supported what I’ve heard from other studies–Android users trend younger than on the iPhone. 60.6% of Android users polled were under 29 as opposed to 47.3% of iPhone users.

Android users trend younger than iPhone

Android users also reported lower incomes and education levels than iPhone users. Not that this is relevant information–Unless you are an Instagram snob.

ANDROID USERS SEEM LESS SATISFIED WITH THEIR PHONE

When asked if they would buy the same kind of phone again, 89.3% of iPhone users said yes, while 78.9% of Android users did. Android owners also seem a little less satisfied with their phone when asked–89.2% of iPhone users were satisfied or extremely satisfied with their devices versus 81.7% for Android.

ANDROID USERS DON’T CARE ABOUT APPS

A chart showing the app recommendation habits of iPhone and Android users.

Android users seem less likely to blab about apps to their friends.

Android users seem slightly less likely to recommend apps to friends, with 37.5% of iPhone users recommending apps to friends often or extremely often and only 19.7% for Android. Hey, Android users just aren’t into apps–why talk about them?

Nearly 43% of iPhone users reported using apps extremely often, compared to 31% of Android owners.

ANDROID USERS ARE CHEAPSKATES

Android vs. iPhone users: Have you paid for an app?

iPhone users pay for apps, Android users don't.

Although the vast majority of Android and iPhone users have downloaded free apps, only 47.5% of Android users have ever paid to download an app vs. 80.4% of iPhone users. Also, 77.5% of Android users reported never having made an in app purchase in a freemium game versus 58.9% of iPhone owners. Hey, it’s been said before.

CONCLUSION

This data is based on too small a sample to really make a conclusion. I still think it’s decent data to expand on my hunch–Android users just aren’t into apps. This presents a marketing challenge. Android users are out there, but how do you get them excited about your content?

According to this final chart, users of both platforms look for information about apps in similar places.

Info sources for new apps ranked by popularity for Android and iPhone users

Android and iPhone users have similar tastes as far as info sources for new apps.

The audience exists. Perhaps you have to address them differently in the same channels.

Some developers seemed to have cracked this code. For most, monetizing apathetic Android users remains a challenge.