Phys 1379 · Machine Learning in the Sciences · demo
Neural Style Transfer
Take a photograph and a painting, and make a new picture with the content of one and the style of the other.
1The idea
A convolutional network trained to recognise objects builds up a description of an image layer by layer. Early layers respond to edges and colours, deeper layers to textures, shapes and whole objects. The demo uses VGG19, a 19-layer network trained on the ImageNet photographs, and never changes its weights. It is used only as a ruler for comparing images.
Two different measurements are taken from the network's feature maps \(F^\ell_{ik}\) (filter \(i\) of layer \(\ell\), at pixel position \(k\)):
- Content is the feature maps of a deep layer themselves. Two images have the same content if the same features light up in the same places.
- Style is the correlation between feature maps, summed over all positions: the Gram matrix
\[ \begin{equation} G^\ell_{ij} = \sum_k F^\ell_{ik}\,F^\ell_{jk}. \tag{1}\label{eq:gram} \end{equation} \]The sum over positions throws away where things are and keeps only which features occur together: brush strokes, colour combinations and textures, but not the layout of the picture.
2Making the mash-up
The new image \(x\) starts as random noise. Its pixels are the parameters, and they are adjusted by gradient descent (the Adam optimiser, 500 steps) to minimise
with a content loss and a style loss
Eq. \eqref{eq:lc} pulls the content of \(x\) toward the base photograph; Eq. \eqref{eq:ls} pulls its style toward the painting, in the first five convolutional layers. The gradient with respect to the pixels comes from backpropagation through the network, exactly as in training, except that the image changes and the weights do not.
The two terms compete. With the style weight fixed at \(w_s = 10^{-2}\), the content weight \(w_c\) sets the balance: small \(w_c\) gives an image that is mostly texture, large \(w_c\) keeps the photograph recognisable with the painting's colours and strokes laid over it.
3Try it
Pick a base picture and a style, and choose the content weight \(w_c\).
← more style · more content →
4Code and references
- The course code,
styleTransfer.py(PyTorch), adapted from A. van de Kleut, avandekleut.github.io/nst. - L. A. Gatys, A. S. Ecker and M. Bethge, "A neural algorithm of artistic style", arXiv:1508.06576 (2015).
- K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition" (VGG), arXiv:1409.1556 (2014).