So far, we have been using a learning rate of 0.01 while training our models. In Chapter 1, Artificial Neural Network Fundamentals, we learned that the learning rate plays a key role in attaining optimal weight values. Here, the weight values gradually move toward the optimal value when the learning rate is small, while the weight value oscillates at a non-optimal value when the learning rate is large. We worked on a toy dataset in Chapter 1, Artificial Neural Network Fundamentals, so we will work on a realistic scenario in this section.
To understand the impact of the varying learning rate, we'll go through the following scenario:
- Higher learning rate (0.1) on a scaled dataset
- Lower learning rate (0.00001) on a scaled dataset
- Lower learning rate (0.001) on a non-scaled dataset
- Higher learning rate (0.1) on a non-scaled dataset
Overall, in this section, we'll be learning about the impact that various learning rate values...