We will begin by taking a moment to consider linear models in a very general context, since, many nonlinear models can be seen as a generalization of their linear counterparts.

We traditionally see the linear model expressed as an equation giving a variable Y in terms of a linear prediction terms BX, as follows:

The equation is the classic algebraic formula describing a line. However, we can describe regression, linear or otherwise, in a more general framework. What if we don't actually know the formula to relate Y and X to one another? We can still predict Y value for a given X value without any idea about the algebraic relationship between Y and X, so long as we simply rely on the observed X and Y values for predictions of Y given X, by taking the mean of observed Y values for any given X value:

The previous formula is a kind of point-wise regression, specifically point-wise mean regression. We define an function, which is equal to the expected value of Y...