2. MLP, CNN, and RNN
We've already mentioned that we'll be using three deep learning networks, they are:
- MLP: Multilayer Perceptron
- CNN: Convolutional Neural Network
- RNN: Recurrent Neural Network
These are the three networks that we will be using throughout this book. Later on, you'll find that they are often combined together in order to take advantage of the strength of each network.
In this chapter, we'll discuss these building blocks one by one in more detail. In the following sections, MLP is covered alongside other important topics such as loss functions, optimizers, and regularizers. Following this, we'll cover both CNNs and RNNs.
The differences between MLP, CNN, and RNN
An MLP is a fully connected (FC) network. You'll often find it referred to as either deep feed-forward network or feed-forward neural network in some literature. In this book, we will use the term MLP. Understanding this network in terms of known target applications will help us to get insights about the underlying reasons for the design of the advanced deep learning models.
MLPs are common in simple logistic and linear regression problems. However, MLPs are not optimal for processing sequential and multi-dimensional data patterns. By design, an MLP struggles to remember patterns in sequential data and requires a substantial number of parameters to process multi-dimensional data.
For sequential data input, RNNs are popular because the internal design allows the network to discover dependency in the history of the data, which is useful for prediction. For multi-dimensional data like images and videos, CNNs excel in extracting feature maps for classification, segmentation, generation, and other downstream tasks. In some cases, a CNN in the form of a 1D convolution is also used for networks with sequential input data. However, in most deep learning models, MLP and CNN or RNN are combined to make the most out of each network.
MLP, CNN, and RNN do not complete the whole picture of deep networks. There is a need to identify an objective or loss function, an optimizer, and a regularizer. The goal is to reduce the loss function value during training, since such a reduction is a good indicator that a model is learning.
To minimize this value, the model employs an optimizer. This is an algorithm that determines how weights and biases should be adjusted at each training step. A trained model must work not only on the training data but also on data outside of the training environment. The role of the regularizer is to ensure that the trained model generalizes to new data.
Now, let's get into the three networks – we'll begin by talking about the MLP network.