Web Augmented Reality in 2022
Have you ever wondered how to get Augmented Reality(AR) onto standard mobile phones? Today, I’ll show you how with a bit of HTML and Javascript.
A bit of history
Augmented Reality has been here with us for many years. In fact, Morton Heilig conducted the first experiment with AR in 1962. Of course, he did it without a computer, as you can see here:
Only in 1990 did Thomas P. Caudell coin the phrase “Augmented Reality”. Over the years, Augmented Reality has evolved alongside Virtual Reality(VR). AR/VR desktop apps have appeared in many forms and ultimately, AR/VR came into our browsers.
It was 2015 when I saw the first real demos of AR in browsers; in 2022, Web AR is still considered an “emerging technology”. That, unfortunately, means chaos across multiple formats and inconsistent behavior across devices. Moreover, AR itself is a complex topic, so I’ve divided this blog post into three parts:
- Types of AR
- 3D models
- How to implement Web AR
Types of AR
AR has many forms. Some are suitable for the web and some are fancier and not web-related. Examples of non web-related AR include this fantastic sandbox or future Mercedes navigation
Now, let’s see the types suitable for web browsers.
Location-Based
The mechanism of this AR type is straightforward. First, an application gets GPS data and compares it with the point of interest. Then, after users reach the correct location, the application finds a proper place to show a 3D model laid over our real world. Pokemon Go represents a great example of how location-based Augmented Reality works.
Marker Based
Marker-based AR requires a marker to show an augmented scene. Markers are graphic patterns that cameras recognize easily. For example, the following picture shows a famous Hiro marker. It is named after the pioneer of AR, Hirokazu Kato, who developed one of the first AR libraries for mobiles in 1999.
You can usually print the markers on paper or things in the real world. So when the camera sees one of those markers, it shows the 3D object over it, as you can see in the following image:
Image Tracking
Image Tracking makes it possible to scan a picture, a drawing, or any image and display content over it. It’s the same mechanism as AR markers, but Image Tracking needs more CPU power since the pictures are usually more complex than specialized markers like the Hiro Marker.
Markerless
Markerless AR doesn’t require a marker or location as it draws a 3D Model directly onto a surface in your environment. IKEA takes advantage of this AR type. Are you thinking of buying a new piece of furniture? Then, you can download their app to see how your new chair will fit into your home.
Face-based AR filters
This is a famous use case of AR. I think the example explains itself. :).
3D Models
Now that we’re finished with the previous brief and incomplete list of AR types, let’s look at what is required for a working AR.
First, you need a binary file describing your 3D model. Graphic designers use multiple 3D model formats, whereas on the web, there are only two competing ones: USDZ for Apple and GLB for Android.
USD stands for Universal Scene Description and USDZ is just a zipped USD. Pixar developed the USD format for their internal purposes, not for AR.
GLB format is a binary format of glTF. Unlike USD, The glTF was developed by the Khronos Group as an optimized format for Augmented Reality.
OK, so if they both do the same, then why do browsers require us to support two formats? It’s similar to the situation with different cables, like the USB-C vs. Lighting Cable, wherein Apple and Google can’t agree on one 3D model format for AR. gLFT is minimalistic, whereas USDZ contains an expressive but complex composition system. We can help ourselves understand differences with the following vague comparison:
gLFT to USD is like PNG to a Photoshop file.
The reason why Apple has chosen USDZ remains a mystery for me. I see it as just another unnecessary layer of complexity. Especially since some web AR libraries convert GLB to USDZ on the fly.
How to implement Web AR
Now let’s look at what tools you can use to get AR onto your phones. I’ve found three ways to bring Augmented Reality to mobile phones:
- Device built-in solutions
- JS libraries utilizing WebRTC and WebGL
- JS libraries using the WebXR API
Device built-in solutions
Apple
You must have an iPhone supporting AR Quick Look. The rest is easy. You only need one proper formatted link and the 3D model in USDZ format, as in the following example:
Android
Android offers a similar tool: Scene Viewer. You only need a properly formatted URL to open the Scene Viewer.
Despite the claim that it should work in Firefox, I haven’t managed to make it work. Nevertheless, it works fine in Chrome or Chromium-based browsers.
JS libraries utilizing WebRTC and WebGL
AR can be achieved by using two not-so-related APIs: WebGL and WebRTC.
The library’s authors use WebGL for drawing 3D models on HTML canvas, while WebRTC provides access to the device’s camera. So, with the combination of these two APIs, you can draw 3D models on the video stream from your device’s camera.
Following JS libraries take advantage of this synergy and offer nice AR that can be used on Android or iOS devices.
AR.JS
AR.js represents a higher-level library for Augmented Reality on the Web.
It uses internally lower-level libraries like Three.js, A-Frame, and ARToolKit.js. So you can provide a great AR experience with a small amount of code.
The authors claim their library is the only one that supports all of these types of AR:
- Image Tracking
- Location-Based AR
- Marker Tracking
In the following example, you can see the A-Frame markup enhanced with AR.JS functionality.
A-Frame vs. Three.js
AR.JS uses A-Frame and Three.js as rendering libraries instead of writing their own functionality for showing 3D models. You can choose which rendering library you want to use. I prefer A-Frame to Three.JS since the Three.JS code seems more complicated. Check the following two examples to see the difference:
MindAR
MindAR is an open-source javascript library. It supports Image Tracking and Face Tracking. MindAR, like with AR.JS, uses the A-Frame library. Plus recently, the authors have started working on direct integration with three.js
As you can see in the code, it’s similar to AR.JS, an A-Frame markup with MindAR features.
JS libraries using the WebXR API
Even though a combination of WebGL and WebRTC seems a handy approach, the authors of the browsers wanted something more native, so WebXR was born.
It’s a powerful API covering a lot of use cases, from AR to gaming. The API contains a rich functionality covering rendering 3D scenes and interacting with virtual objects via your screen. You can read more here in the explainer.
The “only” problem is that Firefox and Safari don’t support it at all, as you can see here:
It’s surprising because Mozilla has its own Firefox Reality browser dedicated to VR headsets. Apple, another big player, claims that AR/VR is one of its priorities, but we still need to wait for better support.
Model-viewer
Google offers an elegant library called Model-viewer to show a 3D model in your environment. It supports WebXR API and native solutions (Scene Viewer or Quick Look).
Example:
However, there is a catch in this claimed support of browsers. Library’s authors state that it should work on the last two major versions of all evergreen desktop and mobile browsers, plus the last two versions of Safari. However, I can’t get it to work on Firefox on Android.
As you can see in this issue, it doesn’t work even on some Android Chromes. In this case, the model-viewer is innocent since the problem is probably in the mobile device implementation.
A-Frame supports AR
As you’ve seen, the Javascript library A-Frame shines in viewing 3D models. Nevertheless, it also supports AR via WebXR API, which means it works only on Android Chrome and Edge.
You can see an example here.
Sidenotes
Relation between WebXR and JS libraries utilizing WebRTC and WebGL
As I’ve said, some JS libraries use a combination of WebGL and WebRTC to render the AR. But they claim they will convert their solution from WebGL&WebRTC to native WebXR when the support is better. That would be really cool!
WebXR polyfill
Regarding the missing WebXR in Safari and Firefox, I’ve found a polyfill. Unfortunately, the last version was published in 2020, plus there are a couple of old unanswered bug reports in their repository. So use it with caution.
Death of WebVR
No summary of Web AR would be complete without the mention of WebVR. WebVR API was first conceived in 2014 by the Mozilla team. Google took the API into account in 2016, refactored it a bit, and incorporated it into Chrome. However, in 2019, the new WebXR specification was published, and WebVR was deprecated.
The only browsers that support WebVR are Chromium-based browsers and Firefox, but reportedly its support will be removed in the future.
Real support
Because of numerous contradictive information, I decided to check the support for each mentioned library on my own.
I used those devices:
- iPhone 2020, iPhone 13
- Samsung Galaxy A52s, Motorola Moto E7 Plus
The results:
QuickLook — Apple native solution
It worked on all iOS browsers.
Scene Viewer — Google native solution
I tried it on Android’s Chrome & Edge, which works fine. However, I could not manage to run it on Firefox.
Model-viewer:
It worked on all iOS browsers and Android Chrome and Edge.
AR.JS & MindAR
It worked on all iOS or Android browsers!
A-Frame AR
It worked only on Android Chrome and Edge!
Conclusion
After a bit of a painful journey to learn the current status of Web AR, I can recommend only AR.JS or MindAR. Both seem to be excellently supported AR libraries with many features. Model-viewer works fine for markless AR, but I failed to make Model-viewer work on Android Firefox.
A last general tip: be careful, test everything, and don’t forget to provide your users with a proper fallback in case the user’s device is not AR compatible.
We are ACTUM Digital and this piece was written by Marek, Senior Front End Developer of Apollo Division.
If you seek help with your project or initiative, just drop us a line. 🚀
