Neural networks are a class of algorithm that was originally designed based on the way that human brains work. However, modern advances are generally based on mathematics rather than biological insights. A neural network is a collection of neurons that are connected together. Each neuron is a simple function of its inputs, which generates an output:

The functions that define a neuron's processing can be any standard function, such as a linear combination of the inputs, and are called the activation function. For the commonly used learning algorithms to work, we need the activation function to be derivable and smooth. A frequently used activation function is the logistic function, which is defined by the following equation (k is often simply 1, x is the inputs into the neuron, and L is normally 1, that is, the maximum value of the function):

The value of this graph, from -6 to +6, is shown as follows:

The red lines indicate that the value is 0.5 when x is zero.

Each...