Neural networks, which were originally called artificial neural networks, are inspired by actual neurons found in animal's brains and other parts of the nervous system. Neurons are connected to each other and they receive and send impulses throughout the animal's body, or in the case of computing, the network.
The following diagram shows the components of a single neuron:
Components of a single neuron
The following graph shows how a neuron fires:
How a neuron fires
It is all or nothing, meaning, when the neuron gets enough input from its neighbors, it quickly fires and sends a signal down its axon to each forward-connected neuron.
Here, we can see actual neurons in a brain:
Actual neurons in a brain
A human brain has about 100 billion neurons all together, and has about 100 trillion connections. It is worth noting that the neural networks we create in software have at least 1 million times less complexity.