The most common way to train NNs these days is with a backward propagation of errors algorithm, or backpropagation (often backprop for short). As we have seen already, individual neurons remind us of linear or logistic regression a lot, so it should not come as a surprise that backpropagation usually comes together with our old friend the gradient descent algorithm. NN training works in the following way:
- Forward pass—input is presented to the layer and the transformations are applied to it layer by layer until the prediction is outputted on the last layer.
- Loss computation—the prediction is compared to the ground truth, and an error value is calculated for each neuron of the output layer using the loss function J.
- The errors are then propagated backward (backpropagation), such that each neuron has an error associated to it, proportional to its contribution to the output.
- Weights (w) are updated using one step of gradient descent. The gradient of the loss function is calculated...
