All we essentially do to our input data is compute a dot product, add a bias term, pass it through a non-linear equation, and then compare our prediction to the real output value, taking a step in the direction of the actual output. This is the general architecture of an artificial neuron. You will soon see how this structure, configured repetitively, gives rise to some of the more complex neural networks around.
Exactly how we converge to ideal parametric values by taking a step in the right direction is through a method known as the backward propagation of errors, or backpropagation for short. But to propagate errors backwards, we need a metric to assess how well we are doing with respect to our goal. We define this metric as a loss, and calculate it using a loss function. This function attempts to incorporate the residual difference between what our...