Scientists discover how the brain recognises faces – by reading monkeys' minds

Scientists have created eerily perfect replicas of human faces that were shown to monkeys just from recording the animal’s brain waves, in one of the most impressive feats of mind reading to date.

The demonstration solves one of the most intractable problems in neuroscience – how the brain recognises faces – and raises the prospect of our innermost thoughts becoming increasingly accessible to scientific exploration.

Doris Tsao, who led the work at the California Institute of Technology, believes the work may have practical consequences in the future. “A face is impossible to describe in words,” she said. “One can imagine applications in forensics where one could reconstruct the face of a criminal by analysing a witness’s brain activity ... you could imagine reading out what face a person is imagining.”

The findings represent the first time that scientists have gained such a complete understanding of one of the brain’s more sophisticated functions.

Professor Rodrigo Quian Quiroga, a neuroscientist at the University of Leicester who was not involved in the work, described it as “quite a revolution in neuroscience”. “It’s solving a decades-long mystery,” he added.

Almost algorithmic

The puzzle of how the brain identifies a familiar face dates back to the 1960s, when the US neuroscientist, Jerry Lettvin, suggested that people have hyper-specific neurons that respond to specific objects, a notion that became known as “grandmother cells”, based on the idea that you have a specific neuron that would fire on seeing your grandmother.

More recently scientists found “face patches”, clusters of neurons that respond almost exclusively to faces, but how recognition was achieved had remained a “black box” process. In the absence of proof otherwise, the grandmother model continued to appeal because it tallied with the subjective “ping” of recognition we experience on seeing a familiar face.

“This paper completely kills that,” said Quian Quiroga.

Instead, the new work shows our brains rely on the kind of maths that an algorithm might use to perform the task. In fact, Tsao and her colleague, Steven Le Chang, stumbled on their discovery while working on computer vision. The pair had initially set themselves the challenge of coming up with a way of reliably converting facial images into a numerical representation.

They started by marking up the recognisable landmarks on the face using dots, creating something like a dot-to-dot picture of a face. They then converted a database of thousands of faces into dot-to-dot versions and looked for which 25 measurements between dots best allowed the faces in the database to be reconstructed – essentially converting the shape information in the face into a matrix of numbers.

“You have 25 knobs that you can twiddle and they basically describe all the variations in the shape of the face,” said Tsao.

It turned out that the most useful measurements – the ones that gave the best coverage of the variety of face shapes in the database – were fairly abstract. One of the numbers might reflect a mixture of information about the distance between the eyes and the height of the forehead, for instance.

Tsao then came up with a second set of 25 numbers to explain the remaining characteristics – skin tone, eye colour, musculature – once the features had been stripped away.

Tsao had not expected the scheme she had devised – “we thought we had picked it out of the blue” – to mirror the responses of neurons, but incredibly there was an almost direct mapping between the two when the team carried out direct brain recordings from the cortex of two monkeys.

Tsao had inadvertently replicated almost the exact scheme that our brain uses to recognise faces in what initially seemed an unlikely stroke of luck.

It was only later that Tsao realised that the method was, mathematically, the most efficient way to convert faces into sets of numbers. “If you look at methods for modelling faces in computer vision, almost all of them ... separate out the shape and appearance,” said Tsao. “The mathematical elegance of the system is amazing.”

Reading minds

The researchers then developed an algorithm that could decode previously unseen faces from neural responses recorded from directly from the brains of two monkeys. Electrical recordings made from around 100 neurons in the face patch regions of each monkey were enough to recreate almost identical images to those seen by the animals.

“The predictions were so good, I was kind of amazed,” said Tsao, adding that when she first saw the results she asked her colleague if he had made a mistake.

The study, published in the journal Cell, also uncovered the series of mathematical transformations used by the brain to recognise faces no matter whether they are seen face on or in side-profile.

Tsao believes the findings could pave the way for similar breakthroughs in understanding the neural code of even more complex processes such as memory and imagination.

Might this prospect raise the more sinister possibility of people’s thoughts being read or controlled against their will? “I can’t really relate to that,” said Tsao. “For me it’s great that we can understand something that’s been so mysterious.”