Deep neural networks that classify images and play Go better than we do create an impression of extremely complex models whose internals are inspired by our own brain's structure. In fact, the central ideas behind neural networks are easy to grasp. While first neural networks were indeed inspired by the physical structure of our brain, the analogy no longer holds and the relation to physical processes inside the human brain is mostly historical.
To demystify neural networks, we will start with the basic building blocks: artificial neurons. An artificial neuron is nothing more than two mathematical functions. The first takes a bunch of numbers as input and combines them by using its internal state—weights. The second, an activation function, takes the output of the first and applies special transformations. The activation function tells us how...