Denouncing the use of field-specific effect size distributions to inform magnitude

Reporting effect size

An ES can offer information about the calculated value of effects, the direction of effects, the magnitude of effects, and relevant interpretations of effects (Nakagawa & Cuthill, 2007). Organizations such as The American Psychological Association (APA), Washington, DC, USA, The American Educational Research Association (AERA), Washington, DC, USA, The Society for Industrial and Organizational Psychology (SIOP), Bowling Green, OH, USA, and The National Centre for Education Statistics (NCES), Washington, DC, USA, have released official statements regarding the importance of incorporating ES values into research (Kelley & Preacher, 2011). In turn, almost all journal editors have required that an ES value, and its corresponding confidence interval, be reported alongside, or instead of, p-values and null hypothesis significance test results. A good ES value should be scaled accordingly given the question of interest, be accompanied by a confidence interval, and be asymptotically unbiased, consistent, and efficient (Kelley & Preacher, 2011).

An ES measure can be expressed in either unstandardized or standardized format, with the former representing magnitude in the units of the study variables and the latter representing magnitude in generic units. Examples of unstandardized ES measures includes mean differences and regression coefficients, whereas examples of standardized ES measures include standardized mean differences, correlation coefficients, standardized regression coefficients, and odds/risk ratios (Kelley & Preacher, 2011; Rosenthal, 1994). Standardized mean differences are often categorized as part of the d family of ES measures (e.g., Cohen’s d, Hedges’ g: a bias-corrected version of Cohen’s d, Glass’ Δ: a version of Cohen’s d which uses a single group’s standard deviation), whereas correlation coefficients are often categorized as part of the r family of ES measures (e.g., Pearson’s r, partial/semi-partial r, multiple R, biserial/point-biserial r). Although d was designed as a measure of standardized mean difference and r was designed as a measure of correlation, these measures can be converted between one another in order to facilitate comparisons across settings and designs (e.g., d to r, r to d, odds ratio to d, etc.).

As constructs become more abstract, if they are not measured reliably, standardized ES measures are at risk of increased measurement error (Hedges, 1981). Increased measurement error results in lower standardized ES values (Burchinal, 2008). Although there are adjustments that may be used for reliability that may produce potentially better estimates of ES (Baguley, 2009), there is no substitute for better measurement and ongoing validation of the construct itself (Flake, Pek & Hehman, 2017).

ES values results from the application of a relevant ES measure to given data (Kelley & Preacher, 2011). For example, the correlation between two measures can be expressed with an ES value of r = 0.78. Alternatively, an unstandardized mean difference between two groups on the number of cookies consumed can be expressed with an ES value of, for example, two cookies. However, interpreting an ES value’s magnitude involves combining the numeric value of an ES with the context in which it was computed. Although it is not difficult to report standardized and unstandardized values, the burden is placed on the research to explain their meaning given the context of the study by interpreting ES magnitude (Kelley & Preacher, 2011).

Meaningful contextual interpretations

Interpreting an ES within a research context can be difficult. Asking a statistician whether or not a given ES is important or meaningful, is, in all likelihood, much less useful than posing the same question to a subject matter expert (Anderson, 2019). As Anderson (2019) states, what implications an ES has to a field requires a thorough understanding of the utilized variables and their corresponding measures, as well as the extent to which the finding reflects what is expected by a theory. Other factors that can be valuable in providing meaningful interpretations include how the results situate relative to studies evaluating the same hypothesis(es), the potential impact of the results on individuals, families, policies, and resources (e.g., financial, labour, time) required for implementing recommendations. In other words, subject matter experts are in the best position to comment on extended implications of a finding in real life.

For example, consider a fictional example of interpreting an ES value within context in basic science for research on productivity and focus. Assume a researcher is judging the effect of a novel productivity exercise’s effect on focus when completing a task, and finds that the exercise resulted in an average increase of 15 min of focus per day per employee. Consider the following questions:

• Is an average increase of 15 min of focus per day after the exercise important?

• What are the real-life implications of 15 min of extra focus time?

• How long are people focused on average without any intervention?

• How much variability is there in the change in focus across employees?

• How does the focus increase from this particular exercise compare to other productivity exercises?

• Is the focus task used in the exercise relevant to the actual tasks of the employees?

• Is the sample used in this study representative of the population of interest?

• How does the population sampled from the study translate to other populations?

• Is 15 min of extra focus time large enough relative to the cost of introducing the productivity exercise?

These are just a small subset of questions that can be used to address an ES value’s magnitude within the context of the study; further, relevant questions can only be raised and addressed by those with knowledge of the theoretical content and the finding’s real world implications. In other words, these questions cannot readily be answered by a statistician removed from the research project, but instead should be illuminated upon by psychology researchers who specialize in focus and productivity, as well as other stakeholders and experts in education and workplace management.

It is worth noting that in this example, and extending to other areas of the paper, we are interpreting ES values at the population level. However, there could be important and meaningful differences at the individual level that are also worth exploring. For instance, perhaps in this example, an individual who was not able to focus on any task was suddenly productive for 15 min per day (as opposed to zero) following the intervention. This individual ES scenario might be more meaningful than increasing the productivity of a person who was focused for 5 h per day prior to the intervention, but focused for 5 h and 15 min afterwards.

A potential critique regarding in-context interpretations of an ES value may challenge whether a cumulative science can be built at all when this approach is used. Indeed, if interpretations apply to one unique setting only, it is important to speak to how different ES values may be combined and meta-analyzed within a broader setting. We argue that in-context interpretations of an ES value’s magnitude actually facilitates the building of a cumulative science much better than approaches that use benchmark or cutoff values. If researchers define ES magnitude using in-context interpretations, it is much easier for another researcher reviewing the literature to ascertain which studies may be combined and meta-analyzed because of explicitly shared contextual interpretations. Moreover, the resulting estimated effect from a meta-analysis can be interpreted, and therefore expand on in-context interpretations from studies addressing research questions that are identical in their intent and purpose.

In contrast, interpretations of an ES such as “large” and “small” are in danger of masking critical contextual and study-specific information (Pek & Flora, 2018). For example, a researcher combining studies which use the benchmark interpretation of an ES value may believe that they are combining information that is similar because it is “context free.” In actuality, it may be that contextual information could reveal that studies with similar magnitudes differ greatly in terms of their practical significance (highlighting again the limitations of using cut-off values).

Although we believe the contextual approach to be the most useful way to interpret an ES value, this is often not the approach taken by researchers (Manolov et al., 2016). The two approaches we describe below are the traditional cutoff approach and the observed ES distribution-based approach.

Traditional/Popular cutoffs for ESs

Cohen’s d and Pearson’s r are both well-known and popular ES measures among psychological researchers, as they are based on well-known statistics (i.e., standardized mean difference, correlation). Popular ES guidelines for Cohen’s d associate a small effect with an ES value that is between d = 0.2 and d = 0.5, a medium effect with an ES value that is between d = 0.5 and d = 0.8, and a large effect with an ES value that is greater than d = 0.8. (Cohen, 1988). Similarly, popular guidelines for Pearson’s r associate a small effect with an ES value that is between r = 0.1 and r = 0.3, a medium effect with an ES value that is between r = 0.3 and r = 0.5, and a large effect with an ES value that is greater than r = 0.5 (Cohen, 1988). While quantifying the magnitude of an effect is important, ES magnitude is recommended to be interpreted within the context of the specific study (Kelley & Preacher, 2011; Pek & Flora, 2018; Richard, Bond & Stokes-Zoota, 2003). Accordingly, when introducing popular cutoffs in his 1988 book, Cohen states that they are “recommended for use only when no better basis for estimating the index is available” (p. 25).

Interpreting effect size magnitude via field-specific effect size distributions

Recent methodological studies in psychology have expressed that the ES benchmark values for Cohen’s d and Pearson’s r are unrealistic based on the observed distribution of ES values in their specific field (e.g., Brydges, 2019). While some researchers convey that the current threshold values are set too low (Morris & Fritz, 2013; Welsh & Knight, 2014), other researchers claim that such values are set too high, making large ESs too difficult to achieve (Hemphill, 2003; Bosco et al., 2015).

The notion that ES benchmark values are unrealistic relative to observed ES distributions in a field was first explored in a study conducted by Hemphill (2003), who considered the distribution of Pearson’s r across 380 studies that discussed psychological assessment and treatment. Hemphill aimed to “extend Cohen’s benchmarks by deriving empirical guidelines concerning the magnitude of correlation coefficients found among psychological studies” (p. 78). Notably, the distribution of observed ES values explored by Hemphill was not synonymous to a theoretical probability distribution (e.g., t distribution). Instead, Hemphill obtained benchmark values by gathering observed ES values from different studies from psychological assessment and treatment, combining them into a single distribution.

All ES values reported were either presented as correlation coefficients or were converted from Cohen’s d to Pearson’s r to compare observed ES value distributions. Finally, values were divided equally into lower, middle, and upper thirds. Hemphill found that the lower third contained ESs between r = 0 and r = 0.2, the middle third contained ESs between r = 0.2 and r = 0.3, and the upper third contained ESs above r = 0.3. These results led Hemphill to conclude that correlation values less than 0.2 should be considered a small effect, values between 0.2 and 0.3 should be considered a medium effect, and values larger than 0.3 should be considered a large effect, to reflect such distributions.

Furthermore, if we convert these correlation values to a standardized mean difference metric, d values less than 0.4 should be considered a small effect, values between 0.4 and 0.6 should be considered a medium effect, and values larger than 0.6 should be considered a large effect.

Hemphill adopted the observed ES approach as an alternative method for interpreting ES values. For instance, Hemphill noted that his findings “suggest that the value Cohen used to represent a large correlation coefficient occurs somewhat infrequently in many key research studies in psychology and that a lower value might be warranted in some instances (p. 79)”. Accordingly, to interpret ES values using this approach and using Hemphill’s distribution as a reference, a researcher would compare their own observed ES values relative to the benchmarks specified from the 280 psychological assessment and treatment studies in order to obtain a “small”, “medium” or “large” label (to see a demonstration of how this method works with simulated data see https://mfr.ca-1.osf.io/render?url=https://osf.io/34gf9/?direct%26mode=render%26action=download%26mode=render).

This approach is extremely problematic because practical significance often becomes distorted (Pek & Flora, 2018). For example, if the benchmark value for a “large” effect was set to a relatively small value (such as r = 0.3 in the Hemphill study) in a specific field, there is the potential for researchers to claim meaningfulness even in situations where the effect has no practical significance (Pogrow, 2019). Further, working in a context-free setting, Beribisky, Davidson & Cribbie (2019) found that a correlation of r = 0.3 was the smallest association that was deemed meaningful when participants viewed scatterplots of context-free associations. Thus, there is simply no valid theoretical rationale for creating benchmarks of ES values for researchers in psychology (or any other discipline/sub-discipline) by considering the distribution of observed ES values within these disciplines (or sub-disciplines). Above all, encouraging researchers to compare ES values relative to the distribution of ES within a discipline (or sub-discipline) discourages the practice of interpreting ES magnitude within the context of a specific study and promotes reliance on context-free benchmark values to conclude practical significance. Exclusively relying on context-free benchmarks was strongly advised against by Cohen (1988) in his initial writings on the topic, and countless authors since. We expand on this point below by highlighting two specific problems with the field-specific observed ES distribution approach: (1) equating ES magnitude with importance, and (2) publication bias.

Part I: identifying primary articles

In addition to the Hemphill (2003) study discussed above, we identified four other studies from the field of psychology that proposed ES benchmark value recommendations based on the observed ES values distribution approach, namely Gignac & Szodorai (2016), Rubio-Aparicio et al. (2018), Lovakov & Agadullina (2017), and Brydges (2019) (Table 1). These are particularly influential because they have been published in prospering sub-fields of psychology, including personality and individual differences, clinical psychology, social psychology, and gerontology. Each paper also cites Hemphill’s publication in an attempt to justify their approaches. Further, most have been published recently, highlighting the current popularity of the method.

Gignac & Szodorai (2016) considered the distribution of Pearson’s r in personality and individual differences across 87 meta-analyses from six field-specific journals. Similar to Hemphill’s method, all ES values were divided equally into lower, middle, and upper thirds. The researchers found that the lower third contained ESs between r = 0 and r = 0.15, the middle third contained ESs between r = 0.15 and r = 0.25, and the upper third contained ESs between r = 0.25 and r = 0.35. Subsequently, r values less than 0.15 should be considered a small effect, values between 0.15 and 0.25 should be considered a medium effect, and values between 0.25 and 0.35 should be considered a large effect. The authors concluded that differential psychologists will “be arguably better served by applying the correlation guidelines reported in this investigation rather than those reported by Cohen (1988, 1992) or even Hemphill (2003), in order to obtain an accurate sense of the magnitude” (Gignac & Szodorai, 2016).

Rubio-Aparicio et al. (2018) considered the distribution of Cohen’s d in clinical psychology from 54 meta-analyses. The authors recommended benchmark values for researchers in clinical psychology such that between d = 0 and d = 0.249 should be considered a small effect, between d = 0.249 and d = 0.409 should be considered a medium effect, and between d = 0.409 and d = 0.695 should be considered a large effect (Rubio-Aparicio et al., 2018). They also state that “this classification should be used instead of Cohen (1988) proposal, for the interpretation of the standardized mean change values in the clinical psychological context” (Rubio-Aparicio et al., 2018).

Lovakov & Agadullina (2017) gathered 161 meta-analyses among 29 social psychology journals and reported the lower, middle, and upper thirds of Pearson’s r. They recommended ES values between r = 0 and r = 0.1 to be considered a small effect, between r = 0.1 and r = 0.25 to be considered a medium effect, and between r = 0.25 and r = 0.4 to be considered a large effect (Lovakov & Agadullina, 2017), highlighting that Cohen (1988) guidelines were too conservative.

Most recently, Brydges (2019) focused on the distribution of Pearson’s r in gerontology by collecting 88 meta-analyses from 10 field-specific journals. He proposed ES values less than r = 0.1 to be considered a small effect, between r = 0.1 and r = 0.2 to be considered a medium effect, and between r = 0.2 and r = 0.3 to be considered a large effect (Brydges, 2019). Again, Brydges (2019) discusses how Cohen (1988) ES benchmark values were set too high. Similar to the other authors, Brydges (2019) also ignores the fact that Cohen (1988) suggested only using such guidelines when no contextual information is available.

Part II: citations of primary articles

In order to determine if the recommendations of the primary articles found in Part I were being adopted by researchers, we coded articles that cited Hemphill (2003), Gignac & Szodorai (2016), or Rubio-Aparicio et al. (2018) as found on Google Scholar (2020). The research conducted by Lovakov & Agadullina (2017) and Brydges (2019) were excluded from the coding procedure, as they were published too recently to observe any trends within the citations.

First, the team met to derive a coding checklist and discuss the criteria for selecting specific classifications. In order to evaluate the reason behind the citation of one of the three selected primary articles (i.e., to verify that it was to interpret an ES value using field-specific observed ES distributions as opposed to another reason), six coding categories were developed. Publications that used the distribution of observed ES values approach proposed by Hemphill (2003), Gignac & Szodorai (2016), or Rubio-Aparicio et al. (2018) to interpret ES magnitude were coded as Direct Interpretations; papers that used the observed ES value distribution concept from Hemphill (2003), Gignac & Szodorai (2016), or Rubio-Aparicio et al. (2018), but different benchmark values to interpret ES magnitude were coded as Conceptual Interpretations; papers that were of a methodological or theoretical nature were coded as Methodological/Theoretical; and lastly papers that cited a primary article for a reason unrelated to the interpretation of an ES were coded as Irrelevant. Articles that used more than one method for interpreting ES magnitude (e.g., Hemphill’s (2003) approach and Cohen (1988) cutoffs) were coded as Multiple Methods, and articles that could not be accessed were coded as Inaccessible. Articles that used a direct or conceptual interpretation of ES values demonstrate the popularity of applying the field-specific observed ES distributions method to interpret ES values.

Books, dissertations, theses, and non-English articles with no available translations were excluded from the sample.

To establish interrater reliability, the team (the three authors plus five research assistants) used the checklist to code ten articles selected at random that cited any of the three primary articles. The group met to discuss their findings and found a total of 15 discrepancies across 320 items in total (10 articles $\times$ 8 coders $\times$ 4 items), offering 95.3% agreement among coders. The discrepancies were discussed in order to clarify any issues regarding the coding. Further, a decision regarding any controversial categorizations was made by the authors.