We need to talk about reliability: making better use of test-retest studies for study design and interpretation

Test–retest reliability

Test-retest reliability is typically estimated using the ICC. There exist several different forms of the ICC for different use cases (Shrout & Fleiss, 1979; McGraw & Wong, 1996), for which the two-way mixed effects, absolute agreement, single rater/measurement (the ICC(A,1) according to the definitions by McGraw & Wong (1996), and lacking a specific notation according to the definitions by Shrout & Fleiss (1979) is most appropriate for test-retest studies (Koo & Li, 2016). (2)${\displaystyle ICC=\frac{M{S}_R-M{S}_E}{M{S}_R+\left(k-1\right)M{S}_E+\frac{k}{n}\left(M{S}_C-M{S}_E\right)}}$ where MS refers to the mean sum of squares: MS_R for rows (also sometimes referred to as MS_B for between subjects), MS_E for error and MS_C for columns; and where k refers to the number of raters or observations per subject, and n refers to the number of subjects.

While many test-retest studies, at least in PET imaging, have traditionally been conducted using the one-way random effects model (the ICC(1,1)) according to the definitions by Shrout & Fleiss (1979), the estimates of these two models tend to be similar to one another in practice, especially relative to their confidence intervals. As such, this does not nullify the conclusions of previous studies; rather their outcomes can be interpreted retrospectively as approximately equal to the correct metric.

Importantly, the ICC is an approximation of the true population reliability: while true reliability can never be negative (Eq. 1), one can obtain negative ICC values, in which case the reliability can be regarded as zero (Bartko, 1976).

Measurement error

Each measurement is made with an associated error, which can be described by its standard error (σ_e). It can be estimated (denoted by a hat ∧) as the square root of the within subject mean sum of squares (MS_W), which is used in the calculation of the ICC above (Baumgartner et al., 2018). (3)${\displaystyle {\hat{\sigma }}_e^2=M{S}_W=\frac{1}{n\left(k-1\right)}\sum _{i=1}^n\sum _{j=1}^k{\left(y_{ij}-{\bar{y}}_i\right)}^2}$ where n represents the number of participants, i represents the subject number, j represents the measurement number, k represents the number of measurements per subject, y represents the outcome and ${\bar{y}}_i$ represents the mean outcome for that subject.

The standard error can also be estimated indirectly by rearranging Eq. (1), using the ICC as an estimate of reliability and using the squared sample standard deviation (s²) as an estimate of the total population variance (${\sigma }_{tot}^2$). This is often referred to as the standard error of measurement (SEM) (Weir, 2005; Harvill, 1991). (4)${\displaystyle SEM=s\sqrt{1-ICC}\approx {\sigma }_e={\sigma }_{tot}\sqrt{1-\rho }}$ in which s refers to the standard deviation of all measurements in the sample (both test and retest measurements for test-retest studies).

The standard error can be expressed either in the units of measurement, or relative to some property of the sample. It can be: (i) scaled to the variance of the sample as an estimate of the reliability (ICC, Cronbach’s α), (ii) scaled to the mean of the sample as an estimate of the relative uncertainty (the within-subject coefficient of variation, WSCV) (Baumgartner et al., 2018), or (iii) unscaled as an estimate of the absolute uncertainty ( ${\hat{\sigma }}_e$ or SEM).

The relative or absolute uncertainty can be used to calculate the smallest detectable difference (SDD) (also sometimes referred to as the minimum detectable difference) between two measurements in a given subject which could be considered sufficiently large that it is unlikely to have been due to chance alone (say, according to a 95% confidence interval, i.e., using z_(1−α∕2) = 1.96 below) (Weir, 2005; Baumgartner et al., 2018). This is calculated using the following equation. (5)${\displaystyle SDD=\sqrt{2}\times z_{\left(1-\alpha ∕2\right)}\times {\hat{\sigma }}_e.}$

It can also be extended to a group level examining differences in means. (6)${\displaystyle SDD=\sqrt{2}\times z_{\left(1-\alpha ∕2\right)}\times \frac{{\hat{\sigma }}_e}{\sqrt{n}}.}$

It is important to mention that this measure assumes that all measurements belonging to the original test-retest study and later application study exhibit exactly the same underlying standard error. Further, this measure is not a power analysis: the SDD simply infers that a change in outcome or mean outcome of a group of measurements before and after an experimental manipulation is larger than would be expected by chance. For performing a power analysis for the within-subject change for application to a new study, one must consider the expected effect size relative to the standard deviation of the within-individual changes in the outcome measure.

Standard thresholds

In psychometrics, reliability can be calculated with one scale by examining its internal consistency. Despite being calculated from the consistency of responding between items, reliability assessed by internal consistency amounts to the same fundamental definition of reliability, namely the ratio of the true to the total variance in the data. It is considered good practice in psychometric studies to calculate the reliability of a scale using measures of internal consistency in the study sample prior to performing statistical inference. The reliability can be used to confirm that the data is sufficiently capable of distinguishing between individuals, for which Nunnally (1978) recommended 0.7 as a default lowest acceptable standard of reliability for scales used in basic research, and 0.8 as adequate. For applied settings in which important decisions are made based on measured outcomes, he suggests a reliability of 0.9 as a minimum and 0.95 as adequate, since even with a reliability of 0.9, the standard error of measurement is almost a third the size of the standard deviation.

Test-retest reliability has traditionally been defined by more lenient standards. Fleiss (1986) defined ICC values between 0.4 and 0.75 as good, and above 0.75 as excellent. Cicchetti (1994) defined 0.4 to 0.59 as fair, 0.60 to 0.74 as good, and above 0.75 as excellent. These standards, however, were defined considering the test-retest (as opposed to internal consistency) reliability of psychometric questionnaires (Shrout & Fleiss, 1979; Cicchetti, 1994) for which changes from test to retest could be caused by measurement error, but could also be caused by actual changes in the underlying ‘true’ value. In PET measurement where protein concentrations are estimated, or indeed in many other physiological measures, these within-subject fluctuations can usually be assumed to be negligibly small over short time periods. For this reason, these standards cannot be considered to be directly applicable. More conservative standards have been proposed by Portney & Watkins (2015), who define values between 0.5 and 0.75 as “poor to moderate”, 0.75 to 0.9 as “good”, and above 0.9 as acceptable for “clinical measures”. These standards correspond more closely with those defined for the internal consistency of psychometric instruments, and can be considered to be more applicable for measures for which ‘true’ changes are thought to be negligible.

Relation to effect size estimates

It is important to consider that reliability is related to effect sizes, and more specifically effect size attenuation. This applies both to studies examining correlations with a continuous variable, and those making group comparisons. This relation is described by the following equation (Spearman, 1904; Nunnally, 1970). (7)${\displaystyle r_{ObsA,ObsB}=r_{A,B}\times \sqrt{{\rho }_A\times {\rho }_B}}$ where r_A,B is the true association between variables A and B, and r_ObsA,ObsB is the observed correlation between the measured values for A and B after taking into account their reliability (ρ). When comparing two or more measures, each of which are recorded with some degree of imprecision, the combined reliability is equal to the square root of their product. This defines the degree to which an association is attenuated by the imprecision of the recorded outcomes, and sets an upper bound on the strength of association between variables that can be expected (Vul et al., 2009). This attenuation is important when performing power analyses as it decreases the range of potential standardised effect sizes which can be expected.

Although smaller standardised effect sizes are to be expected with increasing measurement error, such decreases are only to be expected on average. Loken & Gelman (2017) point out that greater measurement error also leads to greater variation in the measured effect sizes which, particularly for smaller sample sizes, also means that some measured effect sizes overestimate the true effect size by chance. This is exacerbated by publication bias, i.e., the selection of tested effects or hypotheses which are found to be statistically significant. This can even lead to a higher proportion of studies being reported whose estimates exaggerate the true underlying effect size, despite the attenuation on average as a result of poor reliability, if only those results which reach statistical significance are selected.

When planning a study, sample size is often determined using power analysis: this means calculating the required sample size to detect a given effect size a given proportion of the time. The effect size is sometimes specified as equal to that reported by previous studies; however, this makes studies vulnerable to the accuracy by which the effect size is approximated (Simonsohn, 2015; Morey & Lakens, 2016). A better approach is to power studies for the smallest effect size of interest such that all interesting effects are likely to be detected. More conservative still is the suggestion to power studies according to the level at which meaningful differences between individual studies can be reliably detected (Morey & Lakens, 2016). These strategies would ensure that the scientific literature would produce robust findings, and that the effects in individual studies can be statistically compared, respectively. However, for costly and/or invasive methods, such ideals are not usually attainable, or ethical. Rather, for studies with these constraints, we require a lower standard of evidence, with strong conclusions perhaps left for meta-analyses (or better yet, for meta-analyses of pre-registered studies to avoid effect size inflation for statistically significant results due to low power (Cremers, Wager & Yarkoni, 2017) or publication bias (Ferguson & Heene, 2012) and not drawn from statistically significant findings from individual studies alone. To determine the feasibility of an individual study using these methods, it may be more useful not to consider the point at which an individual study is ideal for the scientific literature as described by Morey & Lakens (2016), but rather the point at which the study will simply not be bad, i.e., will at least be capable of answering the research question at hand given realistic assumptions.

For this latter aim, we can perform power analysis for study feasibility assessment in terms of a maximum realistic strength of association in biological terms. In this way, the corresponding effect size can be calculated after taking the effect size attenuation into account. If a study will be insufficiently powered even for such a strength of association, then the study is certainly miscalibrated for the detection of the effect of interest. In this way, this method is useful for deciding which research questions can be rejected out of hand given the relevant constraints on the sample size.

Assumption 1

The absolute measurement error (σ_e) will be either similar between groups and/or studies, or the extent to which it will be larger or smaller in a new sample can be approximated.

Assumption 2

The absolute measurement error (SEM) is likely to be relatively stable across different ranges of the same outcome measure (i.e., an individual with a low score will have the same SEM as an individual with a high score on the same outcome measure).

Assumption 3

The ICC, and its extension here, make the assumption of normality of the data.

Assumption summary

These assumptions are reasonable given that the goal is a rough approximation of the reliability in a new sample in order to assess feasibility in study design and interpretation, under the assumption of normality. The first two assumptions are primarily based on the fact that most previous and current test-retest studies are performed on young, healthy participants. For planning a new study for which large deviations are expected from either of these two assumptions, one approach could be to make approximations based on best- and worst-case scenarios. If such an approach is not possible for whichever reason, this speaks in favour of simply conducting a new test-retest study to directly evaluate the feasibility of such a study design in the specific context of the proposed study.

The extrapolated ICC

The ICC can be expressed using the SEM and the SD (s) of the original study from Eq. 4. (8)${\displaystyle ICC=1-\frac{SE{M}^2}{s^2}.}$

Given these two assumptions above, we can approximate the ICC for a new sample given only the standard deviation of the new sample (and an approximation of the inflation extent of the SEM): (9)${\displaystyle IC{C}_{NewStudy}=1-\frac{{\left(\tau SE{M}_{TRT}\right)}^2}{s_{NewStudy}^2}=1-\frac{{\tau }^2{s}_{TRT}^2\left(1-IC{C}_{TRT}\right)}{s_{NewStudy}^2}}$ where τ represents the error (SEM) inflation factor, representing the multiplicative increase in expected measurement error in the new study, for whichever reason. Using τ = 1 assumes that the SEM is the same between studies, while τ = 1.2 would assume a 20% increase in the measurement error. If τ is thought to be substantially greater than 1, then a new test-retest study is perhaps warranted. With τ = 1 or τ ≈ 1, the difference between the ICC of the original study and the extrapolated ICC of a new study will be determined primarily by the SD of the new study.

Studies making use of continuous independent variables (correlations)

For a new study which is going to examine a correlation, reliability is simply determined by the variance of the study group. With more variation in the study sample, the reliability is therefore higher, all else being equal. Increasing sample variability can be attained by implementing wider recruiting strategies, and not simply relying on convenience sampling (see Henrich, Heine & Norenzayan, 2010). This also has the advantage of increasing the external validity of findings.

It should be noted, however, that increasing sample variability by including individuals who differ on another variable which is known to be associated with the dependent variable (for example, age) would not necessarily increase the reliability despite increasing the variance of the study group. In these cases, this variable should be included in the statistical test of an applied study as a covariate, and hence the variance of the unstandardised residuals after accounting for the covariate would therefore be a better estimation of the total variance for reliability analysis in this test-retest sample.

Studies making use of binary independent variables (t-tests)

For a new study which is comparing two independent groups, one is fitting a binary regressor to the dependent outcome variable. The SD for the ICC is calculated based on the sample variability before fitting the regressor, and thus all individuals are included in the calculation of this value (Fig. 2). The total SD is therefore dependent on both the within-group standard deviation, as well as the degree of difference between the two groups, which is measured by the effect size. For independent sample t-tests, one can therefore calculate the effect size (Cohen’s d) for which the reliability of a measure would reach a certain desired threshold.

Thus, by estimating the within-group standard deviation of each of the two groups, one can calculate the required effect size to obtain a sufficient level of reliability. The total SD of the entire sample (including both groups) is described by the following equation: (10)${\displaystyle {\sigma }_{total}=\sqrt{\frac{\left(n_1-1\right){\sigma }_1^2+n_1{\mu }_1^2+\left(n_2-1\right){\sigma }_2^2+n_2{\mu }_2^2-\left(n_1+n_2\right){\mu }_{total}^2}{n_1+n_2-1}}}$ where n is the number of participants in each group, μ is the mean of each group, σ is the SD of each group, and the subscripts 1, 2 and total refer to group 1, group 2, and the combined sample. The total mean (μ_total) is calculated as follows. (11)${\displaystyle {\mu }_{total}=\frac{1}{n_{total}}\left(n_1{\mu }_1+n_2{\mu }_2\right).}$

We therefore only need to solve this equation for μ₂, the mean of the second (e.g., patient) group. This can be calculated as follows (separating the equation into three parts): (12)${\displaystyle A={\mu }_1^2{n}_1{n}_2+n_1^2{\sigma }_1^2-n_1^2{\sigma }_{total}^2+n_1{n}_2{\sigma }_1^2+n_1{n}_2{\sigma }_2^2-2n_1{n}_2{\sigma }_{total}^2-n_1{\sigma }_1^2}$ (13)${\displaystyle B=-n_1{\sigma }_2^2+n_1{\sigma }_{total}^2+n_2^2{\sigma }_2^2-n_2^2{\sigma }_{total}^2-n_2{\sigma }_1^2-n_2{\sigma }_2^2+n_2{\sigma }_{total}^2}$ (14)${\displaystyle {\mu }_2=\frac{1}{2n_1{n}_2}\times \left(\right.2{\mu }_1{n}_1{n}_2\pm \sqrt{4{\mu }_1^2{n}_1^2{n}_2^2-4n_1{n}_2\left(\right.A+B\left)\right.}\left)\right..}$

In this way, given the group sizes and estimates of the standard deviation within both groups, we can calculate the mean difference and effect size for which the degree of variability in the total sample is sufficient to reach a specified level of reliability.