Phys 1379 · Machine Learning in the Sciences · demo
Eigenfaces
A photograph of a face is 4,096 numbers, but faces differ from one another in far fewer ways than that. Principal component analysis finds those ways, and a few dozen numbers then rebuild a recognisable face.
1A face is a vector
The Olivetti collection holds 400 grey-scale photographs of 40 people, ten of each, taken in the early 1990s at AT&T Laboratories Cambridge. Each is 64 × 64 = 4,096 pixels. Writing the pixel brightnesses in one long list turns each photograph into a vector \(x\) with 4,096 components, a single point in a 4,096-dimensional space.
Most points in that space are television static. Faces occupy a tiny region of it: every face has two eyes above a nose above a mouth, lit from roughly the front. That suggests describing a face not by its pixels but by how it differs from a typical face, along a small number of well-chosen directions.
2Principal components
Start from the average face \(\bar x\), the pixel-by-pixel mean of all 400 photographs. Each face differs from it by \(x_n - \bar x\). The covariance matrix records how the pixels vary together across the collection:
It is a 4,096 × 4,096 symmetric matrix, so it has 4,096 orthogonal eigenvectors \(e_k\) with eigenvalues \(\lambda_k \ge 0\):
The eigenvalue \(\lambda_k\) is the variance of the collection along the direction \(e_k\). The first eigenvector is the single direction in which faces differ most; the second is the direction of largest variance at right angles to the first, and so on. These eigenvectors are the principal components. Each one is itself a 4,096-pixel image, an eigenface. With only 400 photographs, at most 399 eigenvalues are non-zero. In practice the eigenvectors are found from the singular value decomposition of the 400 × 4,096 data matrix rather than by building \(C\).
3Rebuilding a face
Because the eigenvectors are orthonormal, any face can be written as the average face plus a sum over eigenfaces, with coefficients found by projecting:
With all 399 components the sum is exact. Keeping only the first \(K\) gives the best possible approximation by \(K\) numbers, in the sense of the smallest squared error over the collection. The fraction of the variation it captures is \(\sum_{k \le K} \lambda_k / \sum_k \lambda_k\), shown in Table 1.
| \(K\) | 1 | 2 | 5 | 10 | 20 | 50 | 100 | 200 |
|---|---|---|---|---|---|---|---|---|
| captured | 24% | 38% | 54% | 66% | 76% | 87% | 94% | 98% |
Fifty numbers in place of 4,096 keep 87% of the variation, and the face is easily recognised. This is lossy compression, and the coefficients \(c_k\) are a compact description that can be compared between photographs: the basis of early face recognition.
4Try it
All computed in your browser from the average face and the first 100 eigenfaces of the Olivetti collection (0.9 MB).
Loading the eigenfaces…
Rebuild a face
The eigenfaces
The first twelve principal components. Grey is zero, light is positive, dark is negative. The early ones are mostly about lighting and the overall shape of the head; detail such as glasses, eyebrows and mouths comes in later.
Make a face
Set the first six coefficients \(c_k\), in units of their spread \(\sigma_k = \sqrt{\lambda_k}\) over the collection, and the face is \(\bar x + \sum_k c_k e_k\). The course script does the same with typed numbers.
Things to try
- Rebuild the first face, the man with glasses and a moustache, step by step. With one or two components it is a generic lit head; the glasses arrive at around ten, the moustache needs more.
- Compare people: some faces are rebuilt well with few components, others need many. A face unlike the rest of the collection needs more terms.
- In Make a face, move one control at a time from −3 to +3 and watch what that eigenface changes: lighting from one side, a longer or wider face, and so on.
- Copy the chosen face sets the six controls to that person's coefficients, the ones the course script prints. Six numbers give only a rough likeness.
5PCA as a linear autoencoder
Equation \eqref{eq:rebuild} is an encoder and a decoder. The encoder turns 4,096 pixels into \(K\) numbers, \(c = E^{\mathsf T}(x - \bar x)\), where the columns of \(E\) are the first \(K\) eigenfaces; the decoder turns them back, \(\hat x = \bar x + E c\). Of all encoders and decoders that are linear maps, this pair has the smallest squared reconstruction error. An autoencoder replaces both linear maps with neural networks, which can follow curved families of images that no flat subspace fits, at the price of a training run in place of one matrix decomposition.
6Code and references
- The course code,
faces.py, uses scikit-learn'sfetch_olivetti_facesandPCA. For this page the average face, the first 100 components and sixteen faces were exported and the sum in Eq. \eqref{eq:rebuild} re-implemented in JavaScript. - Data: F. S. Samaria and A. C. Harter, "Parameterisation of a stochastic model for human face identification", Proc. 2nd IEEE Workshop on Applications of Computer Vision (1994); the Olivetti (ORL) database of faces, AT&T Laboratories Cambridge.
- L. Sirovich and M. Kirby, "Low-dimensional procedure for the characterization of human faces", J. Opt. Soc. Am. A 4, 519 (1987).
- M. Turk and A. Pentland, "Eigenfaces for recognition", J. Cognitive Neuroscience 3, 71 (1991).