Optimization is a branch of applied mathematics that has applications in a multitude of fields, such as physics, engineering, economics, and so on, and is of vital importance in developing and training of deep neural networks. In this chapter, a lot of what we covered in previous chapters will be very relevant, particularly linear algebra and calculus.
As we know, deep neural networks are developed on computers and are, therefore, expressed mathematically. More often than not, training deep learning models comes down to finding the correct (or as close to the correct) set of parameters. We will learn more about this as we progress further through this book.
In this chapter, we'll mainly learn about two types of continuous optimization—constrained and unconstrained. However, we will also briefly touch on other forms of optimization, such as genetic algorithms...