← Back

Phys 1379 · Machine Learning in the Sciences · demo

Variational Autoencoders

Squeeze each handwritten digit down to just two numbers, then turn any two numbers back into a digit. Drag a point around the map and watch the network draw.

Course demo · 7 November 2026

1Compressing a picture to two numbers

An MNIST digit is a 28 × 28 grey-scale image: 784 numbers. An autoencoder is a pair of neural networks. The encoder maps the 784 pixels to a short code, here just two numbers \(z = (z_1, z_2)\); the decoder maps the code back to 784 pixels. The two are trained together to reproduce their input, by minimising the squared difference between the image \(x\) and its reconstruction \(\hat x\):

\[ \begin{equation} \mathcal{L}_{\rm rec} = \sum_{\text{pixels}} (x - \hat x)^2 . \tag{1}\label{eq:rec} \end{equation} \]

Nobody tells the network what the two numbers should mean. Because similar images must get similar codes to be reconstructed well, the digits sort themselves into groups in the two-dimensional latent space: the 1s in one region, the 0s in another. No labels are used in training; the colours in the demo are added afterwards.

2The trouble with a plain autoencoder

A plain autoencoder only learns to decode the codes it has seen. Its latent space is spread out unevenly, with dense clusters, long arms and empty gaps. A point in a gap, or beyond the end of an arm, may decode to a distorted shape or one that is not a digit at all, so the decoder cannot be trusted to make new digits: there is no way to know in advance which codes are safe.

3The variational fix

A variational autoencoder (VAE) makes two changes. First, the encoder outputs a small cloud rather than a point: a centre \(\mu\) and a width \(\sigma\) for each latent coordinate. During training the code is drawn at random from that cloud,

\[ \begin{equation} z = \mu + \sigma\,\epsilon, \qquad \epsilon \sim \mathcal{N}(0, 1), \tag{2}\label{eq:reparam} \end{equation} \]

so the decoder has to produce a sensible digit from anywhere in the cloud, not just from its centre. Writing the noise as a separate \(\epsilon\) (the "reparameterisation trick") keeps \(z\) a smooth function of \(\mu\) and \(\sigma\), so backpropagation still works.

Second, the loss gains a penalty that pulls every cloud towards the standard normal distribution centred at the origin:

\[ \begin{equation} \mathcal{L} = \mathcal{L}_{\rm rec} + \sum_{\text{latent}} \Big[ \tfrac12 \big(\sigma^2 + \mu^2 - 1\big) - \ln\sigma \Big] . \tag{3}\label{eq:vae} \end{equation} \]

The second term is the Kullback–Leibler divergence between the cloud and the standard normal. It is zero when \(\mu = 0\) and \(\sigma = 1\), and it grows if a cloud wanders far from the origin or shrinks to a point. The two terms pull against each other: reconstruction wants the clouds small and well separated, the penalty wants them wide and packed around the origin. The compromise is a latent space with no holes, where every point decodes to something digit-like. Picking \(z\) at random from the standard normal then generates new digits.

4Try it

Both networks were trained with the course code on the 60,000 MNIST training digits. The map shows where 3,000 test digits land, coloured by the digit they are. The decoder runs in your browser.

Model
Click on the map to

Loading the networks (1.6 MB)…

Things to try

5Code and references

← Back to the course page