Neural networks are a class of algorithm that was originally designed based on the way that human brains work. However, modern advances are generally based on mathematics rather than biological insights. A neural network is a collection of neurons that are connected together. Each neuron is a simple function of its inputs, which are combined using some function to generate an output:

The functions that define a neuron's processing can be any standard function, such as a linear combination of the inputs, and is called the activation function. For the commonly used learning algorithms to work, we need the activation function to be derivable and  smooth. A frequently used activation function is the logistic function, which is defined by the following equation (k is often simply 1, x is the inputs into the neuron, and L is normally 1, that is, the maximum value of the function):

The value of this graph, from -6 to +6, is shown below. The red lines indicate that the value...