Subscribe now

Health

Thousands of fMRI brain studies in doubt due to software flaws

Brain Scanner is Simon Oxenham's weekly column that sifts the pseudoscience from the neuroscience

By Simon Oxenham

18 July 2016

New Scientist Default Image

How sound are fMRI brain studies?

Zephyr/Science Photo Library

It’s another blow for neuroscience. The discovery of major software flaws could render thousands of fMRI brain studies inaccurate.

The use of fMRI is a common method for scanning the brain in neuroscience and psychology experiments. To make sense of the data produced, researchers sometimes use a technique called spatial autocorrelation to identify areas of the brain that appear to “light up” during particular tasks or experiences. But some software flaws in the popular fMRI data analysis packages SPM, FSL and AFNI meant this technique routinely produced false positives, resulting in errors 50 per cent of the time or more.

Anders Eklund and Hans Knutsson at Linköping University in Sweden and Thomas Nichols  at the University of Warwick, UK, calculated this by analysing brain data from a collaborative open fMRI project called 1000 Functional Connectomes. Most fMRI statistical methods have been developed using simulated data, but in this case the team was able to use real brain information to validate the techniques for identifying significant patterns of activity in fMRI scans.

This work enabled the researchers to confirm that a statistical software flaw they first identified in 2012 truly does produce false positives at an alarmingly high rate. Four years ago, they were not taken very seriously because their work at that time was based on data from a single person.

Now we know that this is a very real problem indeed. Although the software error has now been corrected, potentially thousands of fMRI studies are in doubt.

Dead fish

It isn’t the first time this field of research has taken a major hit. In 2009, a tongue-in-cheek study showed that some fMRI methods can detect statistically significant activity in a dead fish.

That study illustrated why it is essential to apply statistical corrections when an experiment involves multiple tests. If you ask enough questions, you will eventually detect an association that isn’t really there – unless you apply corrections that reduce the likelihood of such false positives.

The same problem is at the heart of the latest study. The software flaws identified by Eklund’s team involve the statistical corrections that were missing in the fish study, but sometimes the affected software packages did not apply them in the right way.

However, the elephant in the room is that when the team looked at 241 recent fMRI papers, it found that the researchers did not even ask their software to apply any kind of correction in 40 per cent of them. This means a large proportion of recent research probably contains the very same types of error highlighted by the dead fish study from seven years ago.

What’s the damage?

According to Eklund, it is difficult to determine which studies have been affected by the spatial autocorrelation flaw because raw data from past studies is rarely available.

Nichols estimates that around one in 10 fMRI studies may be affected. This is somewhat lower than the full “15 years of brain research” that some publications have suggested are now called into question.

However, when you take into account that, on top of the software flaw, a further 40 per cent of this type of study may be compromised by researchers failing to apply the right corrections in the first place, this hints that many studies indeed may have reported false positive results.

The types of study most likely to be affected are those that are usually reported with headlines along the lines of “X causes part Y of your brain to light up”, or “This is your brain on drug Z”. Those that show relatively weak statistical associations are the likeliest to be inaccurate – whereas studies reporting strongly significant findings may ultimately still be sound.

Read with caution

But not all fMRI researchers are worried. Jens Foell at Florida State University in Tallahassee likens fMRI to using a blurry magnifying glass to look at the most complex object in the universe. As a consequence, he says, we should always be sceptical of fMRI studies with results that have only borderline statistical significance.

In light of this growing evidence of the unreliability of some fMRI research, it is more important than ever to treat findings with caution. There is a large body of psychological evidence that shows people are particularly easily swayed by neuroscience. But although neuroscience is an important field, it is a young one that relies on inherently fuzzy data. Psychological studies that measure behaviour and actions directly may often be more useful, relevant and reliable.

“What I expect the fMRI research community to do now is neither to worry nor to be surprised, but to engage in rigorous discussion,” says Foell.

Journal reference: PNAS, DOI: 10.1073/pnas.1602413113

Read more: The trouble with neuroscience

Topics:

Sign up to our weekly newsletter

Receive a weekly dose of discovery in your inbox! We'll also keep you up to date with New Scientist events and special offers.

Sign up