Bias-variance decomposition

The bias-variance tradeoff arise from a frequentist viewpoint of the model complexity issue (as opposed to the Bayesian one). This decomposition helps to understand how complexity affects generalization. Suppose we have a model $\hat{y}(x)$ and a dataset composed of inputs $x_i$ and outputs $y_i$, $\mathcal{D} = \{(x_i, y_i)_{i=1}^N\}$. We would like to find a meaningful decomposition of the expectation of the sum of square loss function when we repeatedly train our model over different datasets $\mathcal{D}:$ $\mathbb{E}_{\mathcal{D}}\mathcal{L}(\hat{y}(x), y) =\mathbb{E}_{\mathcal{D}}\mathbb{E}_x((\hat{y}(x)- y)^2) =\mathbb{E}_{x}\mathbb{E}_{\mathcal{D}}((\hat{y}(x)- y)^2)$. We further assume that the observations $y = t + \epsilon$ are the sum of a real target and intrinsic noise $\epsilon$ having zero mean and variance $\sigma^2$.

Let’s start considering the standard variance decomposition of a random variable $x$:

where the expectation is taken over the probability distribution of x, p(x). Rearranging the terms, we have an expression for the sum of squares:

for instance, this can be view as a special case of Jensen’s inequality for convex functions ($\mathbb{E}f(x)\leq f(\mathbb{E}x)$). Now, we can replace $x$ with the model error $y-\hat{y} = t+\epsilon-\hat{y}$ and use the above expression to retrieve the e expectation over different datasets $\mathcal{D}$ of the sum of squares loss, while keeping $x$ fixed:

The last passage holds since $\epsilon$ has zero mean.

Now, applying (3) to the first term:

Putting the two equations together, we get:

The first term, called the squared bias, represents the extent to which the average prediction over all data sets differs from the desired regression function. The second term, called the variance, measures the extent to which the solutions for individual data sets vary around their average, and hence this measures the extent to which the function $\hat{y}(x_i)$ is sensitive to the particular choice of data set.