Phys 1379 · Machine Learning in the Sciences · demo
Gaussian Processes
Fit a curve through a handful of noisy points and get, along with the best guess, an honest measure of how uncertain it is everywhere else.
1A probability distribution over functions
The usual way to fit data is to pick a formula with a few parameters (a straight line, a polynomial) and adjust the parameters. A Gaussian process skips the formula. It puts a probability distribution directly on the space of functions, and lets the data decide which functions are likely.
The definition is short: \(f\) is a Gaussian process if the values \(f(x_1), \dots, f(x_n)\) at any set of points are jointly Gaussian. Taking the mean to be zero, the whole process is fixed by a kernel \(k(x, x')\), the covariance between the function values at two points:
The kernel encodes everything we assume about the function before seeing data. If \(k(x,x')\) is large only when \(x\) and \(x'\) are close, nearby points have similar values and distant points are unrelated: the function is continuous but otherwise free.
2Kernels
The demo offers four kernels (Table 1). Three depend only on the distance \(r = |x - x'|\) and have an amplitude \(\sigma_f\) and a length scale \(\ell\): how far the function wanders, and how far apart two points must be before they become independent. They differ in how smooth the functions are.
| Kernel | \(k(x, x')\) | Typical function |
|---|---|---|
| RBF (squared exponential) | \(\sigma_f^2\, e^{-r^2/2\ell^2}\) | infinitely smooth |
| Matérn-5/2 | \(\sigma_f^2 \big(1 + \tfrac{\sqrt5 r}{\ell} + \tfrac{5r^2}{3\ell^2}\big) e^{-\sqrt5 r/\ell}\) | twice differentiable |
| Matérn-3/2 | \(\sigma_f^2 \big(1 + \tfrac{\sqrt3 r}{\ell}\big) e^{-\sqrt3 r/\ell}\) | once differentiable |
| Brownian motion (BM) | \(\min(s, t)\), with \(s, t\) measured from the left edge | continuous, nowhere differentiable: a random walk |
Measurements carry noise of size \(\sigma_n\), which adds \(\sigma_n^2\) to the covariance of each data point with itself.
3Conditioning on the data
Given \(N\) measurements \(y_i\) at points \(x_i\), the rules for Gaussian distributions give the distribution of \(f\) at any new point \(x\) in closed form. Write \(K\) for the \(N\times N\) matrix \(K_{ij} = k(x_i, x_j) + \sigma_n^2\delta_{ij}\) and \(\mathbf{k}(x)\) for the vector \(k(x, x_i)\). The prediction has mean and variance
The mean, Eq. \eqref{eq:mean}, is a weighted sum of kernels centred on the data points. The variance, Eq. \eqref{eq:var}, starts at the prior value \(k(x,x)\) and is reduced by what the data tell us; it is small near data points and grows back far from them. It does not depend on the measured values at all, only on where the measurements were made.
4Choosing the hyperparameters
The kernel parameters \(\sigma_f\), \(\ell\) and \(\sigma_n\) are called hyperparameters. They are chosen by maximising the probability of the data, the log marginal likelihood
The first term rewards fitting the data; the second penalises a kernel flexible enough to fit anything. Maximising the sum is a built-in Occam's razor: a very short length scale threads every point but pays for its flexibility. The noise \(\sigma_n\) is fitted too, so the model estimates how noisy the data are.
5Try it
Points are drawn from a known function (dashed) with Gaussian noise. The green curve is the posterior mean, Eq. \eqref{eq:mean}; the band is \(\pm 2\sigma\), Eq. \eqref{eq:var}; the lower panel shows \(\sigma(x)\) on its own. The hyperparameters are refitted every time something changes.
Things to try
- Choose step. None of these kernels can make a jump, so the fit compromises: the mean rounds off each step and the band widens around it. Compare RBF with Matérn-3/2 to see how differently they handle the corners.
- Choose BM. Between data points the band bulges like a random walk tied down at both ends (a Brownian bridge); beyond the last point it grows without limit.
- Reduce \(N\) to 4 or 5 and watch the band open up between the points. Far from all the data the mean returns to zero, the prior guess.
- Raise the noise. The fitted \(\sigma_n\) follows it, and the mean stops chasing individual points.
- Press New data a few times: the fitted length scale changes from one data set to the next.
6Code and references
- The course code,
gp_explorer.py(numpy, matplotlib, scikit-learn). This page re-implements it in JavaScript: the same functions, kernels, parameter bounds and marginal-likelihood fit, with a simplex search in place of scikit-learn's L-BFGS-B. The band shows the uncertainty of \(f\) itself, without the measurement noise. - C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning (MIT Press, 2006), free online. Chapter 2 covers Eqs. 2–4.
- scikit-learn, Gaussian processes user guide.