ERGs and Effect Sizes
The following in a fictional conversation between an author and an editor/reviewer/gatekeeper (an ERG). It may or may not be based on conversations that have actually occurred.
Author: Here is the revised version of my manuscript.
ERG: Great. One last thing.
Author: Yes?
ERG: The journal has recently adopted some very progressive guidelines regarding the reporting of results and statistical analyses. You may have heard that the field is dealing with a bit of a crisis and some believe it is related to statistics.
Author: Do you believe that?
ERG: . . .
Author: . . .
ERG: Anyway, the journal’s very progressive guidelines suggest reporting effect sizes.
Author: Oh. Ok.
ERG: So please do that and then I will be able to accept your submission.
Author: Suggest or require?
ERG: Suggest.
Author: So it’s optional.
ERG: Um. Strongly suggest.
Author: So . . . optional?
ERG: Do it or I can’t accept your manuscript.
Author: Got it. Is there any further guidance about what is required . . .
ERG: Suggested.
Author: . . . about what is suggested regarding these effect sizes?
ERG: To report them.
Author: An effect size?
ERG: Yes.
Author: Any effect size?
ERG: . . .
Author: Like . . . are there specific effects sizes that you require . . .
ERG: Suggest!
Author: Are there specific effects sizes that you suggest?
ERG: The policy just says effect sizes. Can’t be too careful these days, now can we?
Author: Ok. Well, as you know, the paper concerns the size (in square kilometers) of public parks in two different countries: Country A and Country B. I have reproduced the measurements and the country means for your convenience:
| Country A | Country B | |
|---|---|---|
| 17 | 18 | |
| 38 | 40 | |
| 25 | 60 | |
| 36 | 70 | |
| 66 | ||
| 40 | ||
| M | 29.0 | 49.0 |
| SD | 9.83 | 21.66 |
ERG: That’s convenient.
Author: Given our interest in comparing the sizes of the two countries, I can report that the effect size is estimated to be 20 square kilometers.
ERG: 20? That doesn’t sound right. I feel like there should be more of the alphabet included when you report an effect size. A, b, c, . . . something like that. Possibly some of the squiggly ones I can never quite make sense of.
Author: Letters from the Greek alphabet?
ERG: Those are the ones!
Author: Well this particular effect size doesn’t really have a concise name. It might be described as the unstandardized group mean difference.
ERG: Wait a second! Did you just subtract the average of Country A from the average of Country B?
Author: Yes.
ERG: No no no. That will not do. The journal requires that . . .
Author: Suggests?
ERG: The journal suggests that authors report effect sizes.
Author: The difference between the two group means in an effect size.
ERG: You’re not understanding me. We at this journal have proclaimed an ongoing commitment to scientific rigor and advocate for use of the new statistics (Cumming, 2013, 2014).
Author: And the new statistics suggests the use of effect sizes?
ERG: And confidence intervals!
Author: Well I provided an effect size. I am interested in the relationship between country and park size with a specific focus on how much larger the parks from Country B are than the parks in Country A. That effect is best thought of as a difference. The size of that difference in the parks I sampled is 20 square kilometers.
ERG: What you describe is far too simplistic to be considered consistent with the very progressive statistical approaches advocated by the journal.
Author: So there is some sort of complexity requirement.
ERG: Not a requirement!
Author: So there is . . . an expectation that effect sizes will be somewhat complex.
ERG: Well . . . yes! We’re talking about statistical rigor and very progressive statistical reforms.
Author: You mentioned that.
ERG: We obviously can’t permit authors to just do simple arithmetic.
Author: Obviously. What sorts of complexity are you . . . expecting?
ERG: You know . . .
Author: . . .
ERG: The type of complexity typically found when doing these sorts of things. You might have to . . . take the square root of something. Stats people always seem to be doing things like that. And there are usually technical terms and fancy-sounding jargon.
Author: Ok. Well what about one of the Common Language Effect Sizes (CLES, McGraw & Wong, 1992; Grissom & Kim, 2012)? Those have the advantage of being non-parameteric.
ERG: Ah ha! Finally. This is what I am talking about. I have heard the phrase non-parameteric spoken by people who seem to know things about statistics. I don’t know quite what it means and that makes it seem sophisticated.
Author: Um. Ok. So the probability of superiority is .875.
ERG: The probability of superiority? I like it. It sounds somewhat technical.
Author: It’s the probability that a randomly selected park from Country B will be larger than a randomly selected park from Country A.
ERG: What? No. That’s not complex at all. It’s much too easy to understand.
Author: Yes. Common Language Effects Sizes are intended to help non-technical audiences make sense of effect sizes.
ERG: No no no. Trying to communicate with unsophisticated audiences is not sophisticated. How can you not see that?
Author: So . . . you want audiences to not understand?
ERG: If everyone understood the statistics you were reporting, they obviously couldn’t be very progressive, could they? Your statistics should be appropriate for your data and your research question and they should be rigorous. But your statistics should also give off a whiff of . . . mystery. They should seem hard-fought. They should seem a bit exotic. They should have some . . . je ne sais quoi.
Author: I don’t know?
ERG: Now you’re getting it! The statistics you use should resist immediate comprehension.
Author: Uh huh. Ok, I think I see what you’re driving at. So what about this? What if I sprinkle my results with statistical approaches, including effect sizes, that seem wrong, but can be justified by appealing to arcane statistical knowledge.
ERG: Oh my. Yes. Yes! Now we are talking. That sounds exactly like what we are looking for.
Author: That would convey the sense of sophistication you mentioned and would imply that the approaches taken were very progressive.
ERG: Exactly!
Author: It would also create an opportunity for a bit of signaling. Anyone asking about the details or rationale of the incomprehensible statistical approaches clearly isn’t as sophisticated and progressive as we are.
ERG: Precisely!
Author: This would also make the approaches resistent to criticism. Advocates won’t really understand the approach they are arguing for but will believe they (mostly) do. Because they won’t really be able to engage with criticism, they will instead interpret critics as either simpletons or Luddites.
ERG: I think we are finally on the same page. I am glad that I have been able to bring you around to our way of thinking. Enlightened scholars, such as ourselves, are the only hope this field has for scientific progress.
Author: Obviously.
References
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7-29.
Cumming, G. (2013). Understanding the New Statistics: Effect sizes, Confidence Intervals, and Meta-analysis. Routledge.
Grissom, R. J., & Kim, J. J. (2012). Effect Sizes for Research: Univariate and Multivariate Applications. Routledge.
McGraw, K. O., & Wong, S. P. (1992). A common language effect size statistic. Psychological Bulletin, 111(2), 361.