Weird edge-case with correction for attenuation

TL;DR: Corrected correlations can sometimes exceed 1 if reliabilities are low. Which is weird.

Things in psychology are never measured with perfect accuracy. The accuracy of a measure is quantified in a reliability coefficient, which approximates the proportion of “signal” in measurement scores. When two such measures are correlated with each other, error dilutes their relationship: observed correlation becomes smaller than the “true correlation” which would have been observed if the things would have been measured with perfect accuracy.

This true correlation is sometimes approximated using correction-for-attenuation formula which “boosts” the correlation as a function of joint measurement inaccuracy. More error there is, the bigger boost it gets. Rationale here is that only very small correlations could be observed if the measures are very noisy, even if there were strong true relationships between the things measured. In its simplest form the formula is expressed as division of the observed correlation by the square root of the product of reliabilites.

\[r_{x'y'} = \frac{r_{xy}}{\sqrt{r_{xx}r_{yy}}}\] where \(r_{x'y'}\) is the estimated true correlation, \(r_{xy}\) is the observed correlation, and \(r_{xx}\), \(r_{yy}\) are reliabilities for X and Y.

In this post I want to draw attention to a weird edge-case which can happen when reliabilities are very low. The denominator consisting of reliabilities will always be less than 1, thus the correlation is always magnified by some amount. As the joint reliability approaches zero, the magnification approaches infinity. Thus even correlations that are bound to be [-1, 1] can become essentially any real number, and hence not be interpreted as correlations in the traditional sense anymore. Here I present a brief simulation example of this happening.

Simulation

Below I simulate 100000 “responses” for two uncorrelated “scales” with no inter-item correlations. Thus I know as a fact that the true correlation between the two measures is 0, and that all inter-item correlations are 0 as well. Thus also the reliabilities of both measures are 0 because there is no true shared variance between any items.

###  R code  ###
library(psych)

set.seed(0)

x1 <- rnorm(100000)
x2 <- rnorm(100000)
x3 <- rnorm(100000)
x4 <- rnorm(100000)

y1 <- rnorm(100000)
y2 <- rnorm(100000)
y3 <- rnorm(100000)
y4 <- rnorm(100000)

X_scale <- cbind(x1, x2, x3, x4)
Y_scale <- cbind(y1, y2, y3, y4)

Cronbach’s alpha is not ideal for quantifying reliability but I use it here for convenience’s sake as a rough estimate.

###  R code  ###
Rxx <- psych::alpha(X_scale)$total$std.alpha
Ryy <- psych::alpha(Y_scale)$total$std.alpha

We get \(r_{_{XX}}\) = 0.004 and \(r_{_{YY}}\) = 0.004 which are rather close to zero. Next I’ll compute sum scores and their correlation.

###  R code  ###
X <- rowSums(X_scale)
Y <- rowSums(Y_scale)

Rxy <- cor(X, Y)
Rxy

## [1] -0.005096129

With this seed and N, the correlation is zero up until the third decimal. Next I apply the correction for attenuation formula using the reliabilities computed above.

###  R code  ###  
Rx.y. <- Rxy / sqrt(Rxx*Ryy)
Rx.y.

## [1] -1.308922

The observed correlation of -0.005 went up (or down) to -1.309 even though it’s perfectly clear that there is zero correlation! Now I have to say that I parameter-hacked this simulation a bit: I chose a seed and a sample size that resulted in this absurd extreme result just to show that it is possible. Rerunning this with different parameters will result in the same general pattern: trivial correlation from sampling error becomes nontrivial or even huge after the correction.

I guess that the formula used was derived with some set of assumptions in mind, some of which might have been violated here. For example, applying the formula might assume (just waving my hands here) that any observed correlation between scales represents a real correlation rather than noise. If I would have substituted the observed correlation with the real zero correlation, I would have got the correct result: true correlation of 0. One might wonder what value a fringe toy example like this has, but the motivation for this post, however, came from observing this happen with real data, where it is very difficult to grasp how real the observed correlation is.

In the end I don’t really know what to make of this. What is certain is that the formula does not always approximate the real correlation between two imperfect measures, at least in a way that is interpreted as a correlation traditional sense. The bottom line is that this formula might work, but it might also not and knowing when the conditions are right might not be straightforward.

Acknowledgements: All the nice people who gave me some perspective on this on Twitter.