Understanding XR, AR, VR, and MR
Extended reality (XR) is the umbrella term that’s used to explain technology that engages our senses. This includes providing information, recreating worlds, or enhancing the world in real time. It was developed to enable more immersive experiences using digital objects. When we look at how digital objects are used, it is often through a 2D experience. This experience can include animation, word processing, video games, and even training simulations. Incorporated digital content can include images, and 3D designs that are rendered on a screen. But why should we spend hours building 3D content only to experience it in 2D? XR provides a way out of this limitation by creating a pathway for viewing 3D content in a 3D space. If we think in 3D and build in 3D, then we must have a way to experience our content in 3D.
XR is an umbrella term for augmented reality (AR), virtual reality (VR), and mixed reality (MR) (Figure 1.2). On the surface, people often confuse the three, but it would be valid to say that even if they are different from each other, they all comprise XR.
Within XR, we can think of AR, VR, and MR on a spectrum just like immersion. On one side, you have a completely digital world with digital objects, and on the other side, you have a physical world with digital objects. The consistent component across each experience is the digital objects, but the difference is how connected to the physical world the experience is. There are specific hardware and sensors that contribute to attaining these experiences, but we will get to that a bit later.
Figure 1.2 – Overview of VR, AR, MR, and XR
AR is when you have an experience that places digital objects in a completely physical world. This experience is dependent on sensors from a device that can scan the surrounding area to create a believable experience for the user. You are usually adding digital elements to the screen of a live camera feed. The camera feed is most likely from a smartphone or webcam. There are AR headsets such as HoloLens that create more immersive experiences, but they place those experiences into another category (MR). AR can also incorporate audio such as Bose glasses, which infuse audio into your environment without the need for headphones. Some popular AR experiences are found on smartphones: Pokémon Go, Snapchat face filters, and IKEA Place.
VR is when you have an experience in a completely digital world. In VR, you are not tied to the physical world. You can think of it as being inside a computer, like in Tron, or inside your favorite game. You can walk, run, and jump as an avatar in the digital world. Compared to AR, VR is not sensor-heavy, but it does require specific hardware to get the most out of the experience. At most, you would require a headset such as the Oculus Quest, but you can also use Google Cardboard, where you can use your phone with a low-cost headset case to have bite-sized experiences.
MR lies somewhere in the middle of that spectrum. It combines both AR and VR elements, allowing real words and digital objects to interact seamlessly. Instead of removing yourself from the physical world to have more interactions with digital objects in the digital world, you are integrating more sensors to track your body so that you can interact with digital objects in the physical world. In this experience, you are combining camera sensors with HMDs to scan the world around you, scan your body, and build an immersive environment that combines the best of AR and VR. Devices such as HoloLens and Magic Leap allow you to do that.
Understanding the difference between AR, VR, and MR
AR uses interactions on a screen such as toggles, sliders, or buttons. You can think of this as playing a phone game. AR does not allow you to interact with digital objects outside of screen and button input. It is mainly used for rendering digital objects in a digital environment or adding digital elements such as animation and 3D models to print media. The major draw to AR is the fact that it doesn’t remove the user from the physical world they are in. It enhances the real environment with digital content.
VR uses interactions in a completely virtual way. Compared to AR, VR requires a headset and an adequate amount of space for most experiences. When you put the headset on, you are essentially leaving the physical world for the digital world. Rather than having button interactions on a screen, you are making gestures and movements instead. Most headsets have joysticks and handheld devices that give you more control over the interactions. VR experiences, because they are not tied to the physical world, will have larger-scale experiences that can last minutes to hours. Because you have a headset on, you are inside the game as if you were the real character. You don’t control the character. You are the character.
MR is often confused with AR because the core element of AR (placing digital objects in the real world) is wrapped within MR. The key differentiator is that in MR, your hands are usually free because you have an HMD rather than a smartphone, so sensors can track and occlude your hands, and you can interact with digital objects naturally with simple hand gestures. You can think of it as AR with hand tracking and object interactions, or VR in the real world. If we revisit the spectrum analogy, again, MR lies somewhere in the middle of AR and VR.
As a comic book artist and visual storyteller, I delve into the distinct attributes of each XR branch. My webcomic, Eyelnd Feevr, weaves immersive tales on paper and elevates them with the fusion of technology on digital platforms. These technologies breathe life into comics in unparalleled ways: AR animates characters off the pages, VR submerges readers into the comic’s universe, and MR intertwines the narrative with tangible reality. The depth and richness of Afro-centric narratives present vast possibilities to harness these innovations, enriching the storytelling journey.
With that, we have covered what XR is and the difference between AR, VR, and MR. You should now have a better understanding of which medium would be used for which experiences you want to pursue. If you want people to be in an experience for an extended period, go with VR. If you want people to have an experience that is accessible and affordable, go with AR. If you want people to experience the best of the real and digital world combined, go with MR. Since this is a book focused on VR experiences, let’s get a little more background on how VR came to be.
Brief history of VR
The exact moment when VR was developed is currently disputed. There were references to the idea of an artificial world noted back in the early 1900s. Thomas G. Zimmerman and Jaron Lanier developed the first commercial applications of virtual technology at the company VPL Research. The main purpose of VR was for flight simulations, automotive design, and military training. This usage lasted from the 1970s to the 1990s. By the early 90s, VR started to become more mainstream. Nintendo, SEGA, and Apple were developing products as line extensions of their already popular products. By the 21st century, we started to see different mobile form factors and more portable solutions. Since then, there has been an ever-growing industry and landscape of applications using VR to where it is now seen as a medium rather than an application. The shift from an application to a medium allows VR to be industry-agnostic. Whether it is print publishing, healthcare, or higher education, VR can be used to engage customers, clients, and students.
With our increased background on VR and how it can be used to build experiences, we will now explore how it works, and what is required to create the experiences we want.