← Back

Phys 1379 · Machine Learning in the Sciences · demo

Gaussian Processes

Fit a curve through a handful of noisy points and get, along with the best guess, an honest measure of how uncertain it is everywhere else.

Course demo · 7 November 2026

1A probability distribution over functions

The usual way to fit data is to pick a formula with a few parameters (a straight line, a polynomial) and adjust the parameters. A Gaussian process skips the formula. It puts a probability distribution directly on the space of functions, and lets the data decide which functions are likely.

The definition is short: \(f\) is a Gaussian process if the values \(f(x_1), \dots, f(x_n)\) at any set of points are jointly Gaussian. Taking the mean to be zero, the whole process is fixed by a kernel \(k(x, x')\), the covariance between the function values at two points:

\[ \begin{equation} \big\langle f(x)\, f(x') \big\rangle = k(x, x'). \tag{1}\label{eq:gp} \end{equation} \]

The kernel encodes everything we assume about the function before seeing data. If \(k(x,x')\) is large only when \(x\) and \(x'\) are close, nearby points have similar values and distant points are unrelated: the function is continuous but otherwise free.

2Kernels

The demo offers four kernels (Table 1). Three depend only on the distance \(r = |x - x'|\) and have an amplitude \(\sigma_f\) and a length scale \(\ell\): how far the function wanders, and how far apart two points must be before they become independent. They differ in how smooth the functions are.

Table 1. The kernels in the demo, with \(r = |x-x'|\). \(\sigma_f\) and \(\ell\) are fitted to the data; Brownian motion has no free parameters.
Kernel\(k(x, x')\)Typical function
RBF (squared exponential)\(\sigma_f^2\, e^{-r^2/2\ell^2}\)infinitely smooth
Matérn-5/2\(\sigma_f^2 \big(1 + \tfrac{\sqrt5 r}{\ell} + \tfrac{5r^2}{3\ell^2}\big) e^{-\sqrt5 r/\ell}\)twice differentiable
Matérn-3/2\(\sigma_f^2 \big(1 + \tfrac{\sqrt3 r}{\ell}\big) e^{-\sqrt3 r/\ell}\)once differentiable
Brownian motion (BM)\(\min(s, t)\), with \(s, t\) measured from the left edgecontinuous, nowhere differentiable: a random walk

Measurements carry noise of size \(\sigma_n\), which adds \(\sigma_n^2\) to the covariance of each data point with itself.

3Conditioning on the data

Given \(N\) measurements \(y_i\) at points \(x_i\), the rules for Gaussian distributions give the distribution of \(f\) at any new point \(x\) in closed form. Write \(K\) for the \(N\times N\) matrix \(K_{ij} = k(x_i, x_j) + \sigma_n^2\delta_{ij}\) and \(\mathbf{k}(x)\) for the vector \(k(x, x_i)\). The prediction has mean and variance

\[ \begin{equation} \bar f(x) = \mathbf{k}(x)^{T} K^{-1} \mathbf{y}, \tag{2}\label{eq:mean} \end{equation} \]
\[ \begin{equation} \sigma^2(x) = k(x, x) - \mathbf{k}(x)^{T} K^{-1}\, \mathbf{k}(x). \tag{3}\label{eq:var} \end{equation} \]

The mean, Eq. \eqref{eq:mean}, is a weighted sum of kernels centred on the data points. The variance, Eq. \eqref{eq:var}, starts at the prior value \(k(x,x)\) and is reduced by what the data tell us; it is small near data points and grows back far from them. It does not depend on the measured values at all, only on where the measurements were made.

4Choosing the hyperparameters

The kernel parameters \(\sigma_f\), \(\ell\) and \(\sigma_n\) are called hyperparameters. They are chosen by maximising the probability of the data, the log marginal likelihood

\[ \begin{equation} \begin{aligned} \ln p(\mathbf{y}) = {}& -\tfrac12\, \mathbf{y}^{T} K^{-1} \mathbf{y} - \tfrac12 \ln\det K \\ & - \tfrac{N}{2}\ln 2\pi . \end{aligned} \tag{4}\label{eq:lml} \end{equation} \]

The first term rewards fitting the data; the second penalises a kernel flexible enough to fit anything. Maximising the sum is a built-in Occam's razor: a very short length scale threads every point but pays for its flexibility. The noise \(\sigma_n\) is fitted too, so the model estimates how noisy the data are.

5Try it

Points are drawn from a known function (dashed) with Gaussian noise. The green curve is the posterior mean, Eq. \eqref{eq:mean}; the band is \(\pm 2\sigma\), Eq. \eqref{eq:var}; the lower panel shows \(\sigma(x)\) on its own. The hyperparameters are refitted every time something changes.

Function
Kernel

Things to try

6Code and references

← Back to the course page