2.1 Refreshing our knowledge of Bayesian modeling

Bayesian modeling is concerned with understanding the probability of an event occurring given some prior assumptions and some observations. The prior assumptions describe our initial beliefs, or hypothesis, about the event. For example, let’s say we have two six-sided dice, and we want to predict the probability that the sum of the two dice is 5. First, we need to understand how many possible outcomes there are. Because each die has 6 sides, the number of possible outcomes is 6 × 6 = 36. To work out the possibility of rolling a 5, we need to work out how many combinations of values will sum to 5:

Figure 2.1: Illustration of all values summing to five when rolling two six-sided dice

As we can see here, there are 4 combinations that add up to 5, thus the probability of having two dice produce a sum of 5 is , or . We call this initial belief the prior. Now, what happens if we incorporate information from an observation? Let’s say we know what the value for one of the dice will be – let’s say 3. This shrinks our number of possible values down to 6, as we only have the remaining die to roll, and for the result to be 5, we’d need this value to be 2.

Figure 2.2: Illustration of remaining value, which sums to five after rolling the first die

Because we assume our die is fair, the probability of the sum of the dice being 5 is now . This probability, called the posterior, is obtained using information from our observation. At the core of Bayesian statistics is Bayes’ rule (hence ”Bayesian”), which we use to determine the posterior probability given some prior knowledge. Bayes’ rule is defined as:

Where we can define P(A|B) as P(d₁ + d₂ = 5|d₁ = 3), where d₁ and d₂ represent dice 1 and 2 respectively. We can see this in action using our previous example. Starting with the likelihood, that is, the term on the left of our numerator, we see that:

We can verify this by looking at our grid. Moving to the second part of the numerator – the prior – we see that:

On the denominator, we have our normalization constant (also referred to as the marginal likelihood), which is simply:

Putting this all together using Bayes’ theorem, we have:

What we have here is the probability of the outcome being 5 if we know one die’s value. However, in this book, we’ll often be referring to uncertainties rather than probabilities – and learning methods to obtain uncertainty estimates with DNNs. These methods belong to a broader class of uncertainty quantification, and aim to quantify the uncertainty in the predictions from an ML model. That is, we want to predict P(ŷ|𝜃), where ŷ is a prediction from a model, and 𝜃 represents the parameters of the model.

As we know from fundamental probability theory, probabilities are bound between 0 and 1. The closer we are to 1, the more likely – or probable – the event is. We can view our uncertainty as subtracting our probability from 1. In the context of the example here, the probability of the sum being 5 is P(d₁ + d₂ = 5|d₁ = 3) = = 0.166. So, our uncertainty is simply 1 − = = 0.833, meaning that there’s a > 80% chance that the outcome will not be 5. As we proceed through the book, we’ll learn about different sources of uncertainty, and how uncertainties can help us to develop more robust deep learning systems.

Let’s continue using our dice example to build a better understanding of for model uncertainty estimates. Many common machine learning models work on the basis of maximum likelihood estimation or MLE. That is, they look to predict the value that is most likely: tuning their parameters during training to produce the most likely outcome ŷ given some input x. As a simple illustration, let’s say we want to predict the value of d₁ + d₂ given a value of d₁. We can simply define this as the expectation of d₁ + d₂ conditioned on d₁:

That is, the mean of the possible values of d₁ + d₂.

Setting d₁ = 3, our possible values for d₁ + d₂ are {4,5,6,7,8,9} (as illustrated in Figure 2.2), making our mean:

This is the value we’d get from a simple linear model, such as a linear regression defined by:

In this case, the values of our intersection and bias are β = 1, ξ = 3.5. If we change our value of d₁ to 1, we see that this mean changes to 4.5 – the mean of the set of possible values of d₁ + d₂|d₁ = 1, in other words {2,3,4,5,6,7}. This perspective on our model predictions is important: while this example is very straightforward, the same principle applies to far more sophisticated models and data. The value we typically see with ML models is the expectation, otherwise known as the mean. As you are likely aware, the mean is often referred to as the first statistical moment – with the second statistical moment being the variance, and the variance allows us to quantify uncertainty.

The variance for our simple example is defined as follows:

These statistical moments should be familiar to you, as should the fact that the variance here is represented as the square of the standard deviation, σ. For our example here, for which we assume d₂ is a fair die, the variance will always be constant: σ² = 2.917. That is to say, given any value of d₁, we know that values of d₂ are all equally likely, so the uncertainty does not change. But what if we have an unfair die d₂, which has a 50% chance of landing on a 6, and a 10% chance of landing on each other number? This changes both our mean and our variance. We can see this by looking at how we would represent this as a set of possible values (in other words, a perfect sample of the die) – the set of possible values for d₁ + d₂|d₁ = 1 now becomes {2,3,4,5,6,7,7,7,7,7}. Our new model will now have a bias of ξ = 4.5, making our prediction:

We see that the expectation has increased due to the change in the underlying probability of the values of die d₁. However, the important difference here is in the change in the variance value:

Our variance has increased. As variance essentially gives us the average of the distance of each possible value from the mean, this shouldn’t be surprising: given the weighted die, it’s more likely that the outcome will be distant from the mean than with an unweighted die, and thus our variance increases. To summarize, in terms of uncertainty: the greater the likelihood that the outcome will be further from the mean, the greater the uncertainty.

This has important implications for how we interpret predictions from machine learning models (and statistical models more generally). If our predictions are an approximation of the mean, and our uncertainty quantifies how likely it is for an outcome to be distant from the mean, then our uncertainty tells us how likely it is that our model prediction is incorrect. Thus, model uncertainties allow us to decide when to trust the predictions, and when we should be more cautious.

The examples given here are very basic, but should help to give you an idea of what we’re looking to achieve with model uncertainty quantification. We will continue to explore these concepts as we learn about some of the benchmark methods for Bayesian inference, learning how these concepts apply to more complex, real-world problems. We’ll start with perhaps the most fundamental method of Bayesian inference: sampling.