Building a better optimizer

In this chapter we have so far discussed several examples in which better neural network architectures allowed for breakthroughs; however, just as (and perhaps even more) important is the optimization procedure used to minimize the error function in these problems, which "learns" the parameters of the network by selecting those that yield the lowest error. Referring to our discussion of backpropagation, this problem has two components:

• How to initialize the weights: In many applications historically, we see that the authors used random weights within some range, and hoped that the use of backpropagation would result in at least a locally minimal loss function from this random starting point.
• How to find the local minimum loss: In basic backpropagation, we used gradient descent using a fixed learning rate and a first derivative update to traverse the potential solution space of weight matrices; however, there is good reason...