​ARKit, ARCore, Facebook and Snapchat or THE BATTLE FOR SMARTPHONE AR WORLD SUPREMACY

I haven’t written a blog post in awhile. Over the past 6 months, I’d try to pontificate on the topic of Augmented Reality but some major new development would always occur. I have a bunch of scrapped posts sitting in Google Drive that are now totally irrelevant. Cruising through December, I figured the coast was clear. I was considering writing a dull year in review post when the final paradigm shift occurred with Snap’s release of Lens Studio. So, let’s try and get this out before it’s obsolete!

The Return of Smartphone AR

Smartphone AR is definitely back.  After Apple’s announcement, everyone wanted to talk about ARKit. Despite developing the award-winning Holographic Easter Egg Hunt for HoloLens with Microsoft this past Spring, discussions with clients and investors became laser-focused on smartphone AR instead of mixed reality.

It looks like 2018 will be a big year for these platforms while mixed reality headset makers gear up for 2019 and beyond. Because of this renewed interest in smartphone AR, this is a good time to investigate your options if you’re looking to get into this platform.

ARKit and ARCore

Despite being announced after Facebook’s AR Camera Effects platform, it really was Apple’s ARKit’s announcement that set off this new hype cycle for smartphone AR. Google’s announcement of ARCore for Android was seemingly a me-too move, but also quite significant.

This isn’t about ARKit versus ARCore since there is no competition. They both do similar things on different platforms. ARCore and ARKit have a common set of features but implement them in ways that are subtly different from the user’s perspective. Because of this, it’s not super difficult to port applications between the two platforms if you are using Unity.

The biggest limitation of both ARKit and ARCore is that when you quit the application, it forgets where everything is. Although you can place anchors in the scene to position virtual objects in the real world, there is no persistence between sessions. I suspect ARCore might advance quicker in this department as Google’s ill-fated Tango technology had this in their SDK for years. I’m assuming we’ll see more and more Tango features merged into ARCore in 2018. Rumors suggest ARKit 2.0 will also see similar improvements.

ARKit does one up ARCore with the addition of face tracking for the iPhone X. This is the most advanced facial tracking system currently available on mobile phones. However, it’s only on one device–albeit a wildly popular one. ARKit’s facial tracking seems to produce results far beyond current mask filter SDKs as it builds a mesh out of your face using the TrueDepth camera. However, there doesn’t seem to be a reason why many of the basic facial tracking features can’t be brought over to phones with standard cameras. Maybe we’ll see a subset of these features trickle down into other iOS devices in the near future.

ARKit has far more penetration than ARCore. ARCore runs on a tiny fraction of Android devices, and this isn’t likely to improve. ARKit requires an iPhone 6S and above, but that’s still a large chunk of iOS devices. There probably is zero business case for focusing on ARCore first. If you truly need to develop a standalone AR app, your best bet is to target iOS primarily and Android second (if at all). If ARCore starts to get some of Tango’s features added to it ahead of ARKit, then there will be compelling use cases for ARCore exclusive apps.

Facebook Camera Effects Platform vs. Snapchat World Lens

When ARKit was first announced, I had a few meetings at large companies. They all thought it was cool, but didn’t want to develop standalone apps. Getting users to download yet another app is expensive and somewhat futile as most go unused after a few tries. There’s a lot more interest in distributing AR experiences inside apps people already have installed. Before Facebook Camera Effects was announced, the only option was Blippar. Which really isn’t an option since hardly anyone uses it.

I got access to Facebook Camera Effects early on and was really impressed with the tools. Leading up to the public release, Facebook has added a lot of features. I’ve seen everything from simple masks to full-blown multiplayer games built with Facebook’s AR Studio.

Screen Shot 2017-12-18 at 6.09.19 PM

Facebook’s AR Studio

Facebook developed an entire 3D engine inside the Facebook Camera. It has an impressive array of features such as a full-featured JavaScript API, facial tracking, SLAM/plane detection, bones (sadly only animated in code), 2D sprite animation, particles, shaders, UI, and advanced lighting and material options. You also can access part of the Facebook graph as well as any external URL you want. If you can fit it inside the filter’s size, poly count, and community guideline restrictions–you can make a fairly elaborate AR app far beyond simple masks.

The great thing about Camera Effects Platform is you are able to distribute an AR experience through an app that already has hundreds of millions of users. Because of this reach, a filter must be tested on a wide variety of phones to account for per-platform limitations and bugs. This is because Facebook AR filters run on a huge number of devices–whether they have native AR SDKs or not.

What’s tricky is after getting approval for distribution of your filter, you still have to somehow tell users to use it. Facebook provides a few options, such as attaching a filter to a promoted Facebook page, but discovery is still a challenge.

As Camera Effects Platform opened to all, Snap released Lens Studio for both Windows and Mac. This platform allows developers to create World Lens effects for Snapchat. I was really excited about this because a lot of clients were just not very enthusiastic about Facebook’s offering. I kept hearing that the valuable eyeballs are all on Snapchat and not Facebook, despite Snapchat’s flatlining growth. Brands and and marketers were chomping at the bit to produce content for Snapchat without navigating Snap’s opaque advertising platform.

Screen Shot 2017-12-18 at 6.07.56 PM

Snap’s Lens Studio

Lens Studio shares many similarities to Facebook’s AR Studio, including the use of JavaScript as a language. The big difference here is that Lens Studio does not expose Snapchat’s facial tracking features. You can only make World Lenses–basically placing animated 3D objects on a plane recognized by the rear camera.

World Lenses also have much tighter size and polycount restrictions than Facebook Camera Effects. However, Lens Studio supports the importing of FBX bone animations and morph targets, along with a JavaScript API to play and blend simultaneous animations. Lens Studio also supports Substance Designer for texturing and a lot of great material and rendering options that make it easier to build a nice looking World Lens despite having lower detail than Facebook.

As for distribution, you still have to go through an approval process which includes making sure your lens is performant on low-end devices as well as current phones. Once available you can link your lens to a Snapcode which you can distribute any way you want.

Which should you develop for? Unlike ARCore and ARKit, Facebook and Snapchat have wildly different feature sets. You could start with a Facebook Camera Effect and then produce a World Lens with a subset of features using detail reduced assets.

The easier path may be to port up. Start with a simple World Lens and then build a more elaborate Facebook AR filter with the same assets. Given how few people use Facebook’s stories feature, I feel that it may be smarter to target Snapchat first. Once Facebook’s Camera Effects Platform works on Instagram I’d probably target Facebook first. It really depends on what demographic you are trying to hit.

App vs. Filters

Should you develop a standalone AR app or a filter inside a social network platform? It really depends on what you’re trying to accomplish. If you want to monetize users, the only option is a standalone ARKit or ARCore app. You are free to add in-app purchases and ads in your experience as you would any other app. Facebook and Snap’s guidelines don’t allow this on their respective platforms. Are you using AR to create branded content? In the case of AR filters, they are usually ads in themselves. If you are trying to get as much reach as possible, a properly marketed and distributed AR filter is a no-brainer. A thorough mobile AR strategy may involve a combination of both native apps and filters–and in the case of Facebook’s Camera Effects Platform, they can even link to each other via REST calls.

spectrum

How each platform ranks sorted by feature complexity

2018 is going to be an exciting year for smartphone AR. With the explosive growth of AR apps on the AppStore and the floodgates opening for filters on social media platforms, you should be including smartphone AR into your mixed reality strategy. Give your users a taste of the real thing before the mixed reality revolution arrives.

There’s Nothing To Be Learned From Pokemon Go

Pokemon Go is a watershed moment in gaming. I’ve never seen a game have this much traction this fast. My neighborhood is filled with wandering players of all demographics, strolling around with phone in hand looking for Pokemon. Since the game’s launch, everyday has looked like Halloween without the costumes.

In general, the job of a venture capitalist is really easy. For most, you simply wait around for another firm to invest in something and then add to that round. Or, you can wait for something to be really successful and cultivate clones of it. I can guarantee there are now a few VCs with deals in motion to build a “fast follow” mimic of Pokemon Go.

Please don’t.

There is absolutely no way another developer can duplicate the success of this game. In fact, it remains to be seen if this game will be a success beyond its initial pop. No game has ever had an opening weekend of this scale–but still, remember Draw Something or maybe even Fallout Shelter? I’m enjoying Pokemon Go myself, but many of my colleagues are questioning whether it has legs. Regardless of that, any location-based game you may be thinking of making is probably missing a few key ingredients to Pokemon Go’s success.

IMG_2963

My pathetically low level character

Niantic has the Best Location Data in the Business

I’ve spent time building location-based service apps in the past. The biggest problem with making games that play over the real world is populating the map with interesting stuff to do. Firstly, there’s access to map data–on Pokemon Go’s scale, this is not cheap (although there are open source solutions). Simply having a map is one piece of the puzzle–you need to have information about how the locations are used. Which places are busiest? Where do players like to group up at?

Niantic has this data from years of running Ingress–pretty much the largest location-based game ever made. Over the years Ingress was running as a project fully funded and supported by Google, Niantic built an incredibly valuable data layer on top of the real world that has been repurposed for Pokemon Go.

You could possibly license similar information from other companies (Foursquare comes to mind), but Niantic’s data is probably more geared towards the activity patterns of mobile gamers than those who want to Instagram their lunch. (Granted, there’s a lot of overlap there)

Pokemon Is One of the Biggest IPs in the World

Previous to Pokemon Go, even prior to Ingress, there have been plenty of location-based games. Anyone remember Shadow Cities? Or Booyah? They may have just been too early–back then there weren’t enough smartphones to solve the density problem you have with location-based games. Now that smartphones are ubiquitous, how do you get enough players to fill up the world map? One way is to use one of the biggest video game IPs on the planet.

The demand for Nintendo IPs on other platforms is unprecedented.  The fervor for Pokemon in particular is huge–with lots of false Pokemon apps taken off Google Play and the App Store over the years. Investors have responded to this craze, with Nintendo’s stock jumping 25% since the release of Pokemon Go.

There really isn’t another IP as big as Pokemon that can be applied to a game of this scale. Sprinkle a little Pokemon on to a little Ingress and the results are explosive.

There’s nobody else on the planet that can do this. 

Kakao is the new Asian mobile gaming powerhouse

Kakao has leveraged their insanely popular mobile group messaging app, Kakao Talk, to launch a social gaming network called Game Center in Korea. This is similar to Tencent’s strategy of using their Chinese instant messaging network, QQ, to drive traffic to f2p PC games. Yet in the West, Apple’s iMessage seems to have killed off most group messaging startups, and the only one to pursue a gaming strategy released a tower defense game based on Shannon Tweed. No, really.

Kakao has successfully commissioned established game developers to produce high quality games for Game Center. The slickly produced Match-3 game, Anipang, recently rose to the top of the Korean App Store due to its use of Kakao Talk as a viral messaging channel. (It’s also available for Android) Encouraging users to message friends in order to get extra plays, Anipang uses social news update notifications similar to the early days of Facebook games to drive reach. Korean friends of mine have deleted the app from their iPhones to avoid the onslaught of game messages–reminiscent of the bad old days of “Lost Sheep” Farmville spam. Regardless, Kakao has mined a passionate and highly monetizable social network of gamers from Kakao Talk.

With over 55 million users, Kakao may eclipse the size of DeNA’s Mobage platform. It remains to be seen what their plans for the US are. Although Game Center will probably have to be renamed before it launches in the West. I hear Apple can be quite litigious when it comes to their intellectual property.

Come to my session next week at Austin GDC

By the way…On Wednesday, October 10th at 5:35 PM at Austin GDC come see my talk on cross-platform Unity3D mobile games development. Hey, it’s only 25 minutes long. How bad can it be?

Android Users Are Apathetic

I launched the virtual pets spoof, Brick Buddies, on Android and iOS last month with zero PR. It was a crazy idea I wanted to make for no particular reason. Since both versions were launched with the same minimal PR effort (a mere tweet and a Facebook post), I figured I’d use this as an opportunity to analyze both platforms from a new perspective.

The top line: Brick Buddies on iOS gets 10X the downloads of the Android version. I got more iOS downloads in the first day than I did in 3 weeks on Google Play. Although both versions have earned a pittance, iOS users spend more money too.

Brick Buddies’ iOS release got picked up as a news story on at least 3 websites, including The Guardian, with no PR. I saw far more Facebook likes, shares, and retweets about Brick Buddies on iOS through my social graph than the Android release.

But wait, isn’t Android beating iOS in market share?

Pondering this, I thought about everyone I know that has an Android phone. They are my friends least into mobile tech. When I get a peek at their home screen, they hardly have any apps installed. They are seemingly content to have a slick-looking phone with a giant screen that makes calls and sends messages. iPhone users (myself included) appear to be platform zealots and voracious consumers of apps.

I had a hunch that most Android users just aren’t into their phones–which makes sense. If you aren’t into mobile tech, you’d probably settle for an Android device. Let’s face it–Aside from the Google ‘pure’ handsets, most really aren’t so great.

I put a survey up on Mechanical Turk to unscientifically poll the public about the habits of iPhone and Android users. I wanted to see how they like their phones and how excited they are about apps. I only got about 200 responses, so this really isn’t statistically significant.

Hey–it’s good fodder for an inflammatory linkbait post about Android users! Here are some results:

WHO ARE ANDROID USERS?

Both Android and iOS had the same ratio of men to women users.

My results supported what I’ve heard from other studies–Android users trend younger than on the iPhone. 60.6% of Android users polled were under 29 as opposed to 47.3% of iPhone users.

Android users trend younger than iPhone

Android users also reported lower incomes and education levels than iPhone users. Not that this is relevant information–Unless you are an Instagram snob.

ANDROID USERS SEEM LESS SATISFIED WITH THEIR PHONE

When asked if they would buy the same kind of phone again, 89.3% of iPhone users said yes, while 78.9% of Android users did. Android owners also seem a little less satisfied with their phone when asked–89.2% of iPhone users were satisfied or extremely satisfied with their devices versus 81.7% for Android.

ANDROID USERS DON’T CARE ABOUT APPS

A chart showing the app recommendation habits of iPhone and Android users.

Android users seem less likely to blab about apps to their friends.

Android users seem slightly less likely to recommend apps to friends, with 37.5% of iPhone users recommending apps to friends often or extremely often and only 19.7% for Android. Hey, Android users just aren’t into apps–why talk about them?

Nearly 43% of iPhone users reported using apps extremely often, compared to 31% of Android owners.

ANDROID USERS ARE CHEAPSKATES

Android vs. iPhone users: Have you paid for an app?

iPhone users pay for apps, Android users don't.

Although the vast majority of Android and iPhone users have downloaded free apps, only 47.5% of Android users have ever paid to download an app vs. 80.4% of iPhone users. Also, 77.5% of Android users reported never having made an in app purchase in a freemium game versus 58.9% of iPhone owners. Hey, it’s been said before.

CONCLUSION

This data is based on too small a sample to really make a conclusion. I still think it’s decent data to expand on my hunch–Android users just aren’t into apps. This presents a marketing challenge. Android users are out there, but how do you get them excited about your content?

According to this final chart, users of both platforms look for information about apps in similar places.

Info sources for new apps ranked by popularity for Android and iPhone users

Android and iPhone users have similar tastes as far as info sources for new apps.

The audience exists. Perhaps you have to address them differently in the same channels.

Some developers seemed to have cracked this code. For most, monetizing apathetic Android users remains a challenge.