​ARKit, ARCore, Facebook and Snapchat or THE BATTLE FOR SMARTPHONE AR WORLD SUPREMACY

I haven’t written a blog post in awhile. Over the past 6 months, I’d try to pontificate on the topic of Augmented Reality but some major new development would always occur. I have a bunch of scrapped posts sitting in Google Drive that are now totally irrelevant. Cruising through December, I figured the coast was clear. I was considering writing a dull year in review post when the final paradigm shift occurred with Snap’s release of Lens Studio. So, let’s try and get this out before it’s obsolete!

The Return of Smartphone AR

Smartphone AR is definitely back.  After Apple’s announcement, everyone wanted to talk about ARKit. Despite developing the award-winning Holographic Easter Egg Hunt for HoloLens with Microsoft this past Spring, discussions with clients and investors became laser-focused on smartphone AR instead of mixed reality.

It looks like 2018 will be a big year for these platforms while mixed reality headset makers gear up for 2019 and beyond. Because of this renewed interest in smartphone AR, this is a good time to investigate your options if you’re looking to get into this platform.

ARKit and ARCore

Despite being announced after Facebook’s AR Camera Effects platform, it really was Apple’s ARKit’s announcement that set off this new hype cycle for smartphone AR. Google’s announcement of ARCore for Android was seemingly a me-too move, but also quite significant.

This isn’t about ARKit versus ARCore since there is no competition. They both do similar things on different platforms. ARCore and ARKit have a common set of features but implement them in ways that are subtly different from the user’s perspective. Because of this, it’s not super difficult to port applications between the two platforms if you are using Unity.

The biggest limitation of both ARKit and ARCore is that when you quit the application, it forgets where everything is. Although you can place anchors in the scene to position virtual objects in the real world, there is no persistence between sessions. I suspect ARCore might advance quicker in this department as Google’s ill-fated Tango technology had this in their SDK for years. I’m assuming we’ll see more and more Tango features merged into ARCore in 2018. Rumors suggest ARKit 2.0 will also see similar improvements.

ARKit does one up ARCore with the addition of face tracking for the iPhone X. This is the most advanced facial tracking system currently available on mobile phones. However, it’s only on one device–albeit a wildly popular one. ARKit’s facial tracking seems to produce results far beyond current mask filter SDKs as it builds a mesh out of your face using the TrueDepth camera. However, there doesn’t seem to be a reason why many of the basic facial tracking features can’t be brought over to phones with standard cameras. Maybe we’ll see a subset of these features trickle down into other iOS devices in the near future.

ARKit has far more penetration than ARCore. ARCore runs on a tiny fraction of Android devices, and this isn’t likely to improve. ARKit requires an iPhone 6S and above, but that’s still a large chunk of iOS devices. There probably is zero business case for focusing on ARCore first. If you truly need to develop a standalone AR app, your best bet is to target iOS primarily and Android second (if at all). If ARCore starts to get some of Tango’s features added to it ahead of ARKit, then there will be compelling use cases for ARCore exclusive apps.

Facebook Camera Effects Platform vs. Snapchat World Lens

When ARKit was first announced, I had a few meetings at large companies. They all thought it was cool, but didn’t want to develop standalone apps. Getting users to download yet another app is expensive and somewhat futile as most go unused after a few tries. There’s a lot more interest in distributing AR experiences inside apps people already have installed. Before Facebook Camera Effects was announced, the only option was Blippar. Which really isn’t an option since hardly anyone uses it.

I got access to Facebook Camera Effects early on and was really impressed with the tools. Leading up to the public release, Facebook has added a lot of features. I’ve seen everything from simple masks to full-blown multiplayer games built with Facebook’s AR Studio.

Screen Shot 2017-12-18 at 6.09.19 PM

Facebook’s AR Studio

Facebook developed an entire 3D engine inside the Facebook Camera. It has an impressive array of features such as a full-featured JavaScript API, facial tracking, SLAM/plane detection, bones (sadly only animated in code), 2D sprite animation, particles, shaders, UI, and advanced lighting and material options. You also can access part of the Facebook graph as well as any external URL you want. If you can fit it inside the filter’s size, poly count, and community guideline restrictions–you can make a fairly elaborate AR app far beyond simple masks.

The great thing about Camera Effects Platform is you are able to distribute an AR experience through an app that already has hundreds of millions of users. Because of this reach, a filter must be tested on a wide variety of phones to account for per-platform limitations and bugs. This is because Facebook AR filters run on a huge number of devices–whether they have native AR SDKs or not.

What’s tricky is after getting approval for distribution of your filter, you still have to somehow tell users to use it. Facebook provides a few options, such as attaching a filter to a promoted Facebook page, but discovery is still a challenge.

As Camera Effects Platform opened to all, Snap released Lens Studio for both Windows and Mac. This platform allows developers to create World Lens effects for Snapchat. I was really excited about this because a lot of clients were just not very enthusiastic about Facebook’s offering. I kept hearing that the valuable eyeballs are all on Snapchat and not Facebook, despite Snapchat’s flatlining growth. Brands and and marketers were chomping at the bit to produce content for Snapchat without navigating Snap’s opaque advertising platform.

Screen Shot 2017-12-18 at 6.07.56 PM

Snap’s Lens Studio

Lens Studio shares many similarities to Facebook’s AR Studio, including the use of JavaScript as a language. The big difference here is that Lens Studio does not expose Snapchat’s facial tracking features. You can only make World Lenses–basically placing animated 3D objects on a plane recognized by the rear camera.

World Lenses also have much tighter size and polycount restrictions than Facebook Camera Effects. However, Lens Studio supports the importing of FBX bone animations and morph targets, along with a JavaScript API to play and blend simultaneous animations. Lens Studio also supports Substance Designer for texturing and a lot of great material and rendering options that make it easier to build a nice looking World Lens despite having lower detail than Facebook.

As for distribution, you still have to go through an approval process which includes making sure your lens is performant on low-end devices as well as current phones. Once available you can link your lens to a Snapcode which you can distribute any way you want.

Which should you develop for? Unlike ARCore and ARKit, Facebook and Snapchat have wildly different feature sets. You could start with a Facebook Camera Effect and then produce a World Lens with a subset of features using detail reduced assets.

The easier path may be to port up. Start with a simple World Lens and then build a more elaborate Facebook AR filter with the same assets. Given how few people use Facebook’s stories feature, I feel that it may be smarter to target Snapchat first. Once Facebook’s Camera Effects Platform works on Instagram I’d probably target Facebook first. It really depends on what demographic you are trying to hit.

App vs. Filters

Should you develop a standalone AR app or a filter inside a social network platform? It really depends on what you’re trying to accomplish. If you want to monetize users, the only option is a standalone ARKit or ARCore app. You are free to add in-app purchases and ads in your experience as you would any other app. Facebook and Snap’s guidelines don’t allow this on their respective platforms. Are you using AR to create branded content? In the case of AR filters, they are usually ads in themselves. If you are trying to get as much reach as possible, a properly marketed and distributed AR filter is a no-brainer. A thorough mobile AR strategy may involve a combination of both native apps and filters–and in the case of Facebook’s Camera Effects Platform, they can even link to each other via REST calls.

spectrum

How each platform ranks sorted by feature complexity

2018 is going to be an exciting year for smartphone AR. With the explosive growth of AR apps on the AppStore and the floodgates opening for filters on social media platforms, you should be including smartphone AR into your mixed reality strategy. Give your users a taste of the real thing before the mixed reality revolution arrives.

Facebook: The Next Generation Game Publisher

Upon the eve of Casual Connect one of the big announcements was Facebook becoming a mobile games publisher. Much like how the launch of Facebook’s mobile ad network went largely unnoticed only to become a huge deal later, I suspect we may see this move in a similar light in the near future.

View from the podium before my Casual Connect event.

View from the podium before my Casual Connect event.

A social network like Facebook directly publishing apps isn’t without precedent. Facebook took their lead from Asian mobile social networks like Japan’s LINE and Korea’s Kakao Talk. Both extended their messaging services to include mobile games that use their respective social network for viral reach in ways similar to the bad old days of Farmville spam.

Both LINE and Kakao Talk have been able to send games to the top of the charts in their native countries netting big revenue. With increasing adoption of messaging applications in the West, this trend may continue here.

In the boxed software era you had few options other than to go through a publisher for distribution. Publishers had guaranteed shelf space at national retailers. Now that software doesn’t exist in boxes, there’s no need for shelves. As we’ve discussed before, users are the new shelf space.

Mobile publishers like GREE and DeNA pride themselves on having a huge audience to advertise games to. This usually involves blowing lots of money buying users through ads–many of which show up on Facebook. Pretty much this is the only service mobile publishers provide.

If user acquisition is all a mobile game publisher does, why not cut out the middleman?

Facebook can acquire users much cheaper than GREE or DeNA–they own the network. In fact, this is a major reason why GREE and DeNA make so much money in Japan. Especially in the feature phone era, they operated mobile social networks they also published their own games on.

Perhaps Facebook did the math and figured out that a cut of an app’s revenue in exchange for premium placement of ads is a profitable exchange. Instead of having an audience of 30-40 million users as DeNA’s mobage network does, Facebook has over 800 million mobile users.

In the past, Facebook has proven they can make a game popular–at least for a short time. It’s up for the game developer to create a game that lasts. Given the Chaotic Evil alignment of modern game publishers, I’d much rather make this deal with Facebook* than with one of them.

* Oh, I’d take this deal with Twitter too. They’ve been able to get Vine to the top of the App Store charts all year. Imagine what they could do if they published mobile games!

FarmVille 2 Illustrates Zynga and Facebook’s Desperation

The launch of FarmVille 2 last week was interesting to watch. Sequels to social games seem like a bad idea–why fragment the user base of a highly popular service for a numerical sequel? MMOs have had the same problem in the past and Zynga’s own Mafia Wars 2 is largely seen as a failure. I was curious to see what approach Zynga would take with their next attempt at a sequel to a flagship game.

The good news is FarmVille 2 is a gorgeous and even charming follow-up to the original game. The 3D graphics using Flash 11 are beautiful and the whole thing is ultra polished. This is one of the slickest Facebook games I’ve ever seen. The mega talented team that built this should follow all the ex-EA executive carpetbaggers and bail. I bet they could raise money and build a great company based on quality like this–for the right platform, of course.

As we’ve discussed here for years, and has become obvious in recent times, Facebook games are dead. Zynga’s and Facebook’s catastrophic stock declines are both linked to a collapse of the Facebook game ecosystem. Both Zynga and Facebook have failed to effectively monetize mobile users, especially in gaming.

Earlier this year, Zynga actually shut down the iOS version of FarmVille. They just couldn’t figure out how to make one of the most popular social games of all time work on mobile. Yet, Zynga doubles down on Facebook with an elaborate and exquisitely detailed sequel. The fact that the primary platform for FarmVille 2 is Facebook proves Zynga still has no effective mobile strategy. It could be contractual. If so, Zynga is tied to a sinking ship.

FarmVille 2 is also a sign of desperation. A few years back social game developers flooded Facebook with spam notifications from games as a so-called ‘viral expansion loop’. Facebook newsfeeds were cluttered with stories about lost sheep and found energy packs. Concerned with spam levels worsening the user experience, Facebook severely restricted access to viral channels.

FarmVille 2 seems to violate the spirit of Facebook’s carefully considered spam policies put in place after the viral channel crackdown. The biggest evidence being the opt-out newsfeed notification. If you look at almost every alert dialog in FarmVille 2, the only way to not have it not shared on your newsfeed is to click a tiny checkbox in the lower left corner. I can’t say I haven’t seen this tactic before, but it seems extra insidious in FarmVille 2’s case. The box is almost invisible and it has to be clicked every time, as the check box is reset on every alert. As a result, FarmVille 2 is bringing Facebook spam back to 2010 levels of annoyance.

Yes, FarmVille 2 uses sleazy tactics to gain free users. Even worse is the fact that Facebook enables this tactic. It’s not like Zynga is some rogue developer. Their special relationship with Facebook gives them access to a lot of early platform perks–including timeline stories, frictionless permissions, and now deceptive spam newsfeed posts. Both Facebook and Zynga are seemingly out of ideas and are going back to blatant spamming to reach and retain users.

Oh, and you can turn off newsfeed spam using this method.

Facebook’s Mobile Gaming Apocalypse

Crowdstar recently announced they are abandoning the Facebook platform to focus on mobile social games. After amazing success on iOS, they have discovered what many of us have known for years: Facebook games are dead.

In the wake of Farmville’s massive success in 2009, investment in social gaming hit a fever pitch. The Facebook audience grew to 500 million with the rising tide floating all boats. We went from “RIP Good Times” to a veritable all-you-can-snort coke buffet in the little over a year.

During this period, Facebook shut down viral channels making it more difficult to acquire ‘free’ customers and instituted a 30% tax on social gaming revenue in the form of Facebook Credits. By 2011, user growth flattened out and user acquisition costs skyrocketed as social gaming companies blew their war chests fighting over the same group of casual social gaming customers.

Apple, on the other hand, introduced In-App-Payments for free apps and created an entirely new genre of tablet games with the introduction of the iPad. News of new social gaming startups declined, but mobile gaming investments became white hot. The mobile social gaming gold rush was on.

Recent filings from Facebook show that Zynga, one of Facebook’s single biggest contributors of revenue, is now responsible for a shrinking portion of Facebook’s income. This may be due to a change of focus. New game releases on Facebook from Zynga have slowed to a trickle. Meanwhile, Zynga has been feverishly acquiring mobile startups and barking up their stock price with social gambling chatter. While some companies stubbornly cling to the Facebook platform, in most cases social gaming companies are evacuating Facebook for mobile.

Facebook can’t earn a dime off of mobile social games despite their usage of the Facebook API because mobile billing is all controlled by Apple or Google (and now Amazon). There is no place for Facebook credits in the mobile ecosystem. If you try to use any alternative payment system in an iOS app, Apple won’t approve it.

This is why Facebook is making carrier billing agreements and beefing up their HTML5 platform. They can’t get a cut of native app revenue, but can position themselves as a premier destination for HTML5 mobile browser games with Facebook Credits as the billing system.

Even if buying Facebook Credits can be made as seamless as iTunes billing, Facebook still has to fix the fact that HTML5 sucks. This is a problem that is somewhat out of their control, as HTML5 performance is affected by features in mobile browsers developed by Apple and Google.

Facebook is desperately trying to figure out mobile–spending $1 billion on Instagram is an example of this. The unstoppable shift to mobile media consumption threatens Facebook’s core revenue streams from all angles. Facebook Credits have no use in native mobile games and Facebook can’t generate much ad revenue as ads are largely non-existent in their own mobile apps. Facebook’s walled garden is under attack from another walled garden of closed mobile devices. I guess it’s karma.

My new conspiracy theory about Facebook wall posts

So, the Facebook API changes enacted a few days back further kill app virality by making wall posts only visible to users who already have the app installed. This is pretty much the last nail in the coffin for free user acquisition.

Why would Facebook do this? The current mantra is that Facebook is reducing app spam and making Facebook more usable and, uh, “meaningful” for you and your friends. And yeah, I guess that’s true. Newsfeed spam was OUT OF CONTROL last year.

But–that was last year. Since the last major TOS change, I haven’t seen too many news posts about my friends clobbering defenseless animals to death in Frontierville. But that could just be because my friends stopped playing Facebook games. After all, MAU numbers have plunged on most titles.

But here’s my conspiracy theory.

Zynga is making boatloads of money because they directly monetize Facebook users. It took Facebook WAY too long to wise up, but they now have created Facebook Credits to skim off of all the app monetization. This is smart.

They also killed most viral channels, forcing VC backed mega giant developers to blow millions of dollars per month on Facebook ads to acquire users. Smart, but killed the garage developer.

Wall posts were the last free way to acquire users. But why should there be any free way to acquire users? Especially if Facebook needs to show profit for a possible IPO?

I think that in the near future, Facebook will re-enable wall posts from apps to users that don’t have them installed. Except Facebook are going to charge per click, or per app install for these wall posts.

This will still cut out severe spam, because no spammer is going to want to pay for clicks. But it will also be essentially the same thing as a Cost-Per-Install promotion. Some Facebook developers are allegedly paying up to $3 or more to acquire a user. Facebook controls one of the most important channels for user acquisition and it’s time to pay up.

But it’s just a theory…