In this chapter, we shall be detecting faces in images. In the context of artificial intelligence, the action of processing an image for the purpose of extracting information about the visual content of that image is called image processing.

Image processing is an emerging field, thanks to the surge in the number of better AI-powered cameras, medical imagery-based machine learning, self-driving vehicles, analysis of people's emotions from images, and many other applications.

Consider the use of image processing by a self-driving vehicle. The vehicle needs to make decisions in as close to real time as possible to ensure the best possible accident-free driving. A delay in the response of the AI model running the car could lead to catastrophic consequences. Several techniques and algorithms have been developed for fast and accurate image processing. One of the most famous algorithms in the domain of image processing is the convolutional neural network (CNN).

We will not be developing a complete CNN in this chapter, however, we have briefly discussed CNNs in Chapter 1, Introduction to Deep Learning for Mobile. Later, we shall build a face detection Flutter app using a pre-trained model that's present on the device.

Before we delve into the processing of images, let's discuss the anatomy of an image from the perspective of computer software. Consider the following simple image:

The preceding image is a 10 x 10 pixel image (zoomed in); the top two rows of pixels are purple, the next six rows of pixels are red and the last two rows of pixels are in yellow.

However, the computer does not see the colors in this image. The computer sees this image in the format of a matrix of pixel densities. We are dealing with an RGB image here. RGB images are composed of three layers of colors—namely red, green, and blue. Each of these layers is represented by a matrix in the image. The elements of each matrix correspond to the intensity of the color represented by that matrix in each pixel of the image.

Let's examine the preceding image in a program. One of the two rows of pixels that is purple is represented by the following array:

[[255, 0, 255],
[255, 0, 255],
[255, 0, 255],
[255, 0, 255],
[255, 0, 255],
[255, 0, 255],
[255, 0, 255],
[255, 0, 255],
[255, 0, 255],
[255, 0, 255]]

In the preceding matrix, the first column of 255 represents the color red. The second column represents green and the third column represents blue. Thus, the first pixel of the image on the top left corner is a combination of red, green, and blue. Both red and blue are at their full intensities, while green is entirely missing. Thus, as expected, the combined color produced is purple, which is essentially red and blue mixed in equal proportions. If we observe any pixel from the red region of the image, as expected, we get the following array:

[ 255, 0, 0 ]

Similarly, from the yellow region, since the color yellow is a combination of red and green in equal proportions, the pixel is represented by this:

[  255, 255, 0 ]

Now, if we turn off the red and green components of the image, keeping only the blue channel on, we get the following image:

This is very much according to our previous observation that only the top two rows of pixels contain the blue component and the rest of the image has no blue component, hence it is depicted in black, which indicates the absence of intensity, or 0 intensity of blue.

In this section, we shall be discussing how some common manipulations on images can be done to aid with image processing. Often, some simple manipulations on images can lead to quicker and better predictions.

Let's say we wish to rotate the image in our example by 90 degrees. If we examine the first row of pixels from the top after the rotation, we would expect that the first two pixels of the row would be purple, the middle six would be red, and the last two would be yellow. In the analogy of rotation of matrices, this might be seen as a transpose operation wherein the rows are converted to columns and vice versa. The image then looks like this:

And, as expected, the first row of pixels is represented by the following matrix:

[[255,   0, 255],
[255, 0, 255],
[255, 0,   0],
[255, 0,   0],
[255, 0,   0],
[255, 0,   0],
[255, 0,   0],
[255, 0,   0],
[255, 255,   0],
[255, 255,   0]]

In this matrix, the first two elements represent purple, followed by six red, and the last two yellow.

It is often useful to completely remove the color information from an image before performing machine learning on it. The reason is that color is sometimes not a contributing factor in the prediction being asked for. For example, in a system that detects digits in an image, the shape of the digit matters, whereas the color of the digit does not contribute to the solution.

Grayscale images, in simple terms, is a measure of how much light is visible in an area of the image. Usually, the most dominant light-colored elements are completely removed to display contrast with the less visible areas.

The formula for converting RGB to grayscale is as follows:

Y is the final value that the pixel being converted to grayscale would hold. The R, G, and B are the values of red, green, and blue for that particular pixel. The output produced is as follows:

Let's now dive into developing a face detection application!