Week 7 — Linear Models

Adjust the sliders to see how each parameter shapes the mean trend (white line) and the observation ribbons. The ribbon shows where ~68 / 95 / 99.7% of observations are expected to fall.

σ controls the spread of observations around the mean trend. It is the standard deviation of the Normal likelihood. σ does not change the mean trend — it only changes how tightly observations cluster around it.

At each input value xₙ, the response follows a different Normal distribution:

Example from the lecture: at xₙ = −1.5, β₀=1, β₁=1, σ=0.5:

We want to find the values of β = (β₀, β₁) that best explain the data. With σ known, we maximize the likelihood (or equivalently, the log-likelihood) over β:

Since log p ∝ −SSE, maximizing the log-likelihood is equivalent to minimizing the SSE:

The residual εₙ is the difference between the observed response and the model's predicted mean:

# Generate data (true β₀=-0.25, β₁=1.15, σ=0.5)
set.seed(42)
x <- rnorm(n = 100)
mu_true <- -0.25 + 1.15 * x
y <- rnorm(n = length(x), mean = mu_true, sd = 0.5)

# Fit linear model (OLS = MLE under Gaussian likelihood)
my_df <- data.frame(x = x, y = y)
mod <- lm(y ~ x, data = my_df)
summary(mod)   # shows β̂₀, β̂₁ and their uncertainty
coef(mod)      # extract coefficient estimates

In the Bayesian framework we treat β = (β₀, β₁) as unknown random variables and place priors on them. The posterior combines the prior belief with the data via the likelihood:

We assume σ is known for now. The unknowns are only β₀ and β₁.

With only 1 observation and a diffuse prior, the posterior contours appear as diagonal parallel lines. Why?

Many different (β₀, β₁) pairs produce the same value for μₙ₌₁. Increasing the slope while decreasing the intercept by the right amount keeps μ fixed. The likelihood is constant along that diagonal.

The lecture walks through adding the first 10 observations one at a time, watching the joint posterior over (β₀, β₁) evolve:

The joint posterior over (β₀, β₁) induces a distribution over the mean trend μ(x) at every x. We can summarize it as a ribbon. The visualization below shows the key difference between the confidence interval (uncertainty about the mean line itself) and the observation noise bands from the explorer tab (where σ lives).

The informative prior (sd = 1) acts to constrain parameters away from extreme values. After just 1 observation:

With a single input the mean trend is μₙ = β₀ + β₁xₙ. But real problems have dozens or hundreds of inputs. We need a notation that scales cleanly to D inputs without writing out every term.

For D inputs, the mean trend is a sum of D coefficient–input products:

To pull the intercept inside the sum, introduce a fake variable x_n,0 ≡ 1:

Stack all N observations row-by-row. Each row is the n-th observation's input vector (including the fake intercept column):

Organise the D+1 coefficients into a column vector β. The mean trend for the n-th observation is a row–column inner product:

Stacking all N inner products gives the full N×1 vector of mean trends in one matrix multiplication:

The scalar SSE sum becomes a compact quadratic form:

So the full log-posterior (with diffuse prior) in matrix notation is:

Differentiate the log-likelihood with respect to β (using matrix calculus — chain rule analog for vectors):

Set the gradient to zero (g = 0) to find the mode β̂:

The Hessian is the matrix of second derivatives of L with respect to β. Take the derivative of g with respect to β:

This is why the Newton–Raphson method converges in exactly one step for the linear model:

Using the Laplace Approximation, the approximate posterior covariance is the negative inverse Hessian evaluated at the mode:

With a diffuse prior, the Laplace Approximation gives the exact posterior (because the log-likelihood is exactly quadratic):

This is an exact result for the linear model — not an approximation — because the log-posterior is a perfect quadratic in β.

Set D=0. The design matrix is just a column of ones: X = [1, 1, …, 1]ᵀ (N×1). Let's verify the general formula recovers what we already know: