Tracking a user's motion and ultimately their position in 2D and 3D space is fundamental to any AR application. ARCore allows us to track position changes by identifying and tracking visual feature points from the device's camera image. An example of how this works is shown in this figure:

Feature point tracking in ARCore

In the figure, we can see how the user's position is tracked in relation to the feature points identified on the real couch. Previously, in order to successfully track motion (position), we needed to pre-register or pre-train our feature points. If you have ever used the Vuforia AR tools, you will be very familiar with having to train images or target markers. Now, ARCore does all this automatically for us, in real time, without any training. However, this tracking technology is very new and has several limitations. In the later part of the book, and specifically in Chapter 5, Real-World Motion Tracking, we will add a feature to our AR assistant that allows us to track multiple objects' positions from multiple devices in real time using GPS. Then, in Chapter 10, Mixing in Mixed Reality, we will extend our tracking to include augmented maps.

The better an AR application understands the user's reality or the environment around them, the more successful the immersion. We already saw how ARCore uses feature identification in order to track a user's motion. Yet, tracking motion is only the first part. What we need is a way to identify physical objects or surfaces in the user's reality. ARCore does this using a technique called meshing.

We will cover more details about meshing in later chapters, but, for now, take a look at the following figure from Google that shows this meshing operation in action:

Google image showing meshing in action

What we see happening in the preceding image is an AR application that has identified a real-world surface through meshing. The plane is identified by the white dots. In the background, we can see how the user has already placed various virtual objects on the surface. Environmental understanding and meshing are essential for creating the illusion of blended realities. Where motion tracking uses identified features to track the user's position, environmental understanding uses meshing to track the virtual objects in the user's reality. In Chapter 8, Recognizing the Environment, we will look at how to train our own machine learning object identifier, which will allow us to extend our meshing to include automatically recognizable objects or areas of an environment.

Magicians work to be masters of trickery and visual illusion. They understand that perspective and good lighting are everything in a great illusion, and, with developing great AR apps, this is no exception. Take a second and flip back to the scene with the virtual lion. Note the lighting and detail in the shadows on the lion and ground. Did you note that the lion is casting a shadow on the ground, even though it's not really there? That extra level of lighting detail is only made possible by combining the tracking of the user's position with the environmental understanding of the virtual object's position and a way to read light levels. Fortunately, ARCore provides us with a way to read or estimate the light in a scene. We can then use this lighting information in order to light and shadow virtual AR objects. Here's an image of an ARCore demo app showing subdued lighting on an AR object:

Google image of demo ARCore app showing off subdued lighting

The effects of lighting, or lack thereof, will become more obvious as we start developing our startup applications. Later, in Chapter 9, Blending Light for Architectural Design, we will go into far more detail about 3D lighting and even build some simple shader effects.

In this chapter, we didn't go into any extensive details; we will get to that later, but you should now have a good grasp of the core elements ARCore was developed to address. In the next section, we will take a closer look at how best to use the material in this book.