The previous section has been pretty diligent in the use of the term algorithm. Throughout this book, the term is liberally sprinkled, but is always used judiciously. But what is an algorithm?
To answer that, well, we must first ask, what is a program? A program is a series of steps to be performed by the computer. An algorithm is a set of rules that will solve the problem. A ML algorithm is hence a set of rules to solve a problem. They are implemented as a program on a computer.
One of the most eye-opening moments in truly and deeply understanding what exactly an algorithm is for me was an experience I had about 15 years ago. I was staying over at a friend's place. My friend had a seven year old child, and the friend was exasperated at trying to get her child to learn programming as her child had been too stubborn to learn the discipline of syntax. The root cause, I surmised, was that the child had not understood the idea of an algorithm. So the following morning, we tasked the child to make his own breakfast. Except he wasn't to make his own breakfast. He was to write down a series of steps that his mother was to follow to the letter.
The breakfast was simple—a bowl of cornflakes in milk. Nonetheless, it took the child some eleven attempts to get a bowl of cereal. It ended in tears and plenty of milk and cereal on the countertop, but it was a lesson well learned for the child.
This may seem like wanton child abuse. but it served me well too. In particular, the child said to his mother and me, in paraphrase, But you already know how to make cereal; why do you need instructions to do so? His mum responded, think of this as teaching me how to to make computer games. Here we have a meta notion of an algorithm. The child giving instructions on how to make cereal is teaching the child how to program; is itself an algorithm!
A ML algorithm can refer to the algorithm that is learned, or the algorithm that teaches the machine to use the correct algorithm. For the most part of this book, we shall refer to the latter, but it's quite useful to think of the former as well, if only as a mental exercise of sorts. For the most parts since Turing, we can substitute algorithm with machine.
Take some time to go through these sentences after reading the following section. It will help in clarifying what I mean upon a second read.