Deep Neural Networks

In this chapter, we'll be examining deep neural networks. These networks have shown excellent performance in terms of the accuracy of their classification on more challenging datasets like ImageNet, CIFAR10 (https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf), and CIFAR100. For conciseness, we'll only be focusing on two networks: ResNet [2][4] and DenseNet [5]. While we will go into much more detail, it's important to take a minute to introduce these networks.

ResNet introduced the concept of residual learning, which enabled it to build very deep networks by addressing the vanishing gradient problem (discussed in section 2) in deep convolutional networks.

DenseNet improved ResNet further by allowing every convolution to have direct access to inputs, and lower layer feature maps. It's also managed to keep the number of parameters low in deep networks by utilizing both the Bottleneck and Transition layers.

But why these two models, and not others? Well, since their introduction, there have been countless models such as ResNeXt [6] and WideResNet [7] which have been inspired by the technique used by these two networks. Likewise, with an understanding of both ResNet and DenseNet, we'll be able to use their design guidelines to build our own models. By using transfer learning, this will also allow us to take advantage of pretrained ResNet and DenseNet models for our own purposes such as for object detection and segmentation. These reasons alone, along with their compatibility with Keras, make the two models ideal for exploring and complimenting the advanced deep learning scope of this book.

While this chapter's focus is on deep neural networks; we'll begin this chapter by discussing an important feature of Keras called the Functional API. This API acts as an alternative method for building networks in tf.keras and enables us to build more complex networks that cannot be accomplished by the Sequential model API. The reason why we're focusing so much on this API is that it will become a very useful tool for building deep networks such as the two we're focusing on in this chapter. It's recommended that you've completed Chapter 1, Introducing Advanced Deep Learning with Keras, before moving onto this chapter as we'll refer to introductory level code and concepts explored in that chapter as we take them to an advanced level in this chapter.

The goals of this chapter are to introduce:

• The Functional API in Keras, as well as exploring examples of networks running it
• Deep Residual Networks (ResNet versions 1 and 2) implementation in tf.keras
• The implementation of Densely Connected Convolutional Networks (DenseNet) in tf.keras
• Explore the two popular deep learning models, ResNet and DenseNet

Let's begin by discussing the Functional API.