Activation functions are so important to neural networks as they introduce non-linearity to a network. Deep learning consists of multiple non-linear transformations, and activation functions are the tools for non-linear transformation. Hence, activation functions are applied before sending an input signal to the next layer of neural networks. Due to activation functions, a neural network has the power to learn complex features.
Deep learning has many activation functions:
- The threshold function
- The sigmoid function
- The rectifier function
- The hyperbolic tangent function
- The cost function
In the next section, we will start with one of the most important activation functions, called the threshold activation function.