← Back

Phys 1379 · Machine Learning in the Sciences · demo

Neural Style Transfer

Take a photograph and a painting, and make a new picture with the content of one and the style of the other.

Course demo · 6 November 2026

1The idea

A convolutional network trained to recognise objects builds up a description of an image layer by layer. Early layers respond to edges and colours, deeper layers to textures, shapes and whole objects. The demo uses VGG19, a 19-layer network trained on the ImageNet photographs, and never changes its weights. It is used only as a ruler for comparing images.

Two different measurements are taken from the network's feature maps \(F^\ell_{ik}\) (filter \(i\) of layer \(\ell\), at pixel position \(k\)):

2Making the mash-up

The new image \(x\) starts as random noise. Its pixels are the parameters, and they are adjusted by gradient descent (the Adam optimiser, 500 steps) to minimise

\[ \begin{equation} \mathcal{L}(x) = w_c\,\mathcal{L}_c(x) + w_s\,\mathcal{L}_s(x), \tag{2}\label{eq:loss} \end{equation} \]

with a content loss and a style loss

\[ \begin{equation} \mathcal{L}_c = \big\| F^{4}(x) - F^{4}(\text{base}) \big\|^2, \tag{3}\label{eq:lc} \end{equation} \]
\[ \begin{equation} \mathcal{L}_s = \frac{1}{5}\sum_{\ell=0}^{4} \big\| G^\ell(x) - G^\ell(\text{style}) \big\|^2 . \tag{4}\label{eq:ls} \end{equation} \]

Eq. \eqref{eq:lc} pulls the content of \(x\) toward the base photograph; Eq. \eqref{eq:ls} pulls its style toward the painting, in the first five convolutional layers. The gradient with respect to the pixels comes from backpropagation through the network, exactly as in training, except that the image changes and the weights do not.

The two terms compete. With the style weight fixed at \(w_s = 10^{-2}\), the content weight \(w_c\) sets the balance: small \(w_c\) gives an image that is mostly texture, large \(w_c\) keeps the photograph recognisable with the painting's colours and strokes laid over it.

3Try it

Pick a base picture and a style, and choose the content weight \(w_c\).

Base image
Base
Style image
Style
Style-transfer result
Mash-up
Content weight \(w_c\)

← more style · more content →

4Code and references

← Back to the course page