Phys 1379 · Machine Learning in the Sciences · demo
Variational Autoencoders
Squeeze each handwritten digit down to just two numbers, then turn any two numbers back into a digit. Drag a point around the map and watch the network draw.
1Compressing a picture to two numbers
An MNIST digit is a 28 × 28 grey-scale image: 784 numbers. An autoencoder is a pair of neural networks. The encoder maps the 784 pixels to a short code, here just two numbers \(z = (z_1, z_2)\); the decoder maps the code back to 784 pixels. The two are trained together to reproduce their input, by minimising the squared difference between the image \(x\) and its reconstruction \(\hat x\):
Nobody tells the network what the two numbers should mean. Because similar images must get similar codes to be reconstructed well, the digits sort themselves into groups in the two-dimensional latent space: the 1s in one region, the 0s in another. No labels are used in training; the colours in the demo are added afterwards.
2The trouble with a plain autoencoder
A plain autoencoder only learns to decode the codes it has seen. Its latent space is spread out unevenly, with dense clusters, long arms and empty gaps. A point in a gap, or beyond the end of an arm, may decode to a distorted shape or one that is not a digit at all, so the decoder cannot be trusted to make new digits: there is no way to know in advance which codes are safe.
3The variational fix
A variational autoencoder (VAE) makes two changes. First, the encoder outputs a small cloud rather than a point: a centre \(\mu\) and a width \(\sigma\) for each latent coordinate. During training the code is drawn at random from that cloud,
so the decoder has to produce a sensible digit from anywhere in the cloud, not just from its centre. Writing the noise as a separate \(\epsilon\) (the "reparameterisation trick") keeps \(z\) a smooth function of \(\mu\) and \(\sigma\), so backpropagation still works.
Second, the loss gains a penalty that pulls every cloud towards the standard normal distribution centred at the origin:
The second term is the Kullback–Leibler divergence between the cloud and the standard normal. It is zero when \(\mu = 0\) and \(\sigma = 1\), and it grows if a cloud wanders far from the origin or shrinks to a point. The two terms pull against each other: reconstruction wants the clouds small and well separated, the penalty wants them wide and packed around the origin. The compromise is a latent space with no holes, where every point decodes to something digit-like. Picking \(z\) at random from the standard normal then generates new digits.
4Try it
Both networks were trained with the course code on the 60,000 MNIST training digits. The map shows where 3,000 test digits land, coloured by the digit they are. The decoder runs in your browser.
Loading the networks (1.6 MB)…
Things to try
- With the VAE, drag slowly from the 1s to the 7s to the 9s. Each digit morphs smoothly into the next; between the clusters you find in-between shapes.
- Switch to the autoencoder and drag into the empty regions between and beyond the arms. The decoder still draws something, but often a distorted or ambiguous shape.
- Compare the two maps. The VAE's points fill a round region around the origin, about ±3 across; the autoencoder's sprawl over a much larger, uneven area.
- Choose interpolate, then click a 3 and a 7. The strip shows the digits decoded along the straight line between them, as in the course code.
- Notice the blur. A two-number code cannot hold the details of every stroke, and the squared-error loss rewards an average of the plausible shapes, which is blurry. Real models use tens or hundreds of latent dimensions.
5Code and references
- The course code,
VAE.pyandAE.py(PyTorch), after A. van de Kleut, avandekleut.github.io/vae. Both use a 784–512–2 encoder and a 2–512–784 decoder, trained for 20 epochs with Adam. For this page the trained decoders were saved and re-implemented in JavaScript; the encoder is not needed in the browser because the test digits were encoded in advance. - D. P. Kingma and M. Welling, "Auto-encoding variational Bayes", arXiv:1312.6114 (2013).
- C. Doersch, "Tutorial on variational autoencoders", arXiv:1606.05908 (2016).