Considering that a biological neuron has an astonishingly complex structure (see Figure 8.1), how do we approach modeling it in our programs? Actually, most of this complexity is, so to say, at the hardware level. We can abstract it out and think of the neuron as a node in a graph, which takes one or more inputs and produces some output (sometimes called activation).
Wait, but doesn't that sound like something familiar? Yes, you are right: an artificial neuron is just a mathematical function.
The most common way to model the neuron is by using the weighted sum of inputs with the non-linearity function f:
Where w is a weights vector, x is an input vector, and b is a bias term. The y is a neuron's scalar output.
Figure 8.1: A typical motor neuron of a vertebrate. Public domain diagram from Wikimedia Commons
Figure 8.2: Artificial neuron diagram
A typical artificial neuron processes input in the following three steps, as demonstrated in the preceding diagram (Figure 8.2):
- Take...