This week’s blog is on a research paper named, “Puzzling high correlations in fMRI studies of emotion, personality, and social cognition” (Vul. Et al, 2009), previously titled ‘Voodoo Correlations.’
Vul et al observed that correlations reported between brain imaging (fMRI) and studies on emotion, personality, and social cognition was extremely high (approx. >.8) and wanted to work out why this was after a thorough analysis on the previous studies two major points came up
- The initial problem that they identified was that the results obtained throughout these different studies were “impossibly high”, Vul et al claimed that the reliability of both fMRI studies and personality measures are limited and thus provide an upper bound barrier on the possible correlation that can be observed between the two measures (fMRI and personality measures). Vul et al used a well-known equation to theoretically test their assumption that these statistics were impossibly high.
rObservedA, ObeservedB = rA,B × √ (reliabilityA × reliabilityB)
This basically means that the strength of the correlation observed between Measures A and B (rObservedA,ObservedB) reflects not only the strength of the relationship between the traits underlying A and B (rA,B) but also the reliability of the measures of A and B (reliabiltyA and reliabilityB). Vul et al estimated that the reliability of fMRI studies computed at the voxel level is no greater than .7. They also estimated that current measures of personality have a reliability of between .7 and .8. With these reliability estimates posing an upper-bound limit on the possible correlations >.8 does seem “impossibly high” as Vul suggests.
2. The second problem identified was the methodology that 53% of the studies analysed used. When analysing the fMRI’s these researchers selected the voxels that exceeded a certain threshold to the particular behaviour and then only correlated the data provided by these voxels. With only the statistical significant voxels being selected it led to impossibly high data.
The histogram shows the correlation values reported within the studies Vul et al were analysing. The red squares indicate where the methodology was used by selecting voxels when they exceed a certain threshold, orange squares are where the researchers declined to comment on their methodology and finally the green squares indicate where a different methodology is used that does not hand-pick the higher scoring voxels.
There have been many comments in reply to Vul et al’s paper;
- Lieberman (2009) said in response to the disagreement in methodology that the researchers were confused by the questions posed to the them by Vul regarding their methodology and that isn’t how they retrieve their data.
- Lieberman (2009) also commented that although Vul et al used the correct formula in determining whether these results were “impossibly high” he did not make appropriate estimates on the reliability of the measures and therefore his conclusions are false.
- Fiedler (2011) however defended Vul et al stating that Voodoo Correlations are everywhere not just in neuro research, researchers are always manipulating elements of their research to achieve the most visible results
There has been outrage over this publication, primarily due to the sinister connotations of its original title “Voodoo Correlations” insinuating that the studies involved are fraudulent and not scientific. Secondly the implications the paper has on the field, if the conclusions drawn from these studies are incorrect then where do we go with further studies in the area, the methodology will have to be completely overhauled and any progress that has seemingly happened will have to be discounted.
Whilst reading these papers I have to side with Vul et al and the conclusions they made. It is important as psychologists that any results we find are as unbiased as they possibly can be and if Vul reported on the correct methodology this isn’t the case. There needs to be a way to be able to understand the mind brain connection without manipulating the data collected.
What do you think, is Vul et al just releasing a personal attack on neuroscience to try and undermine its credibility or have they identified a true issue?
 Observed A = fMRI studies, Observed B = personality studies