1.96 is the magic number

Deirdre McCloskey has long emphasized the warping effects of the “statistical significance” hurdle to publication in economics. Alan Gerber and Neil Malhotra survey the top two journals in political science to produce this finding:

There are plenty of publications with findings that are barely statistically significant and a noticeable absence of papers that fall just short of the goalline. Figure 2a is more damning.

What’s the implication? Andrew Gelman thinks it shows why hypothesis testing is problematic. Kevin Drum says it demonstrates massaging of data. The authors say:

The goal of this paper is to raise awareness of publication bias in political science. We have found that many more results are published just over the p=.05 threshold than below it, implying a certain amount of bias in parameter estimates. Our results suggest that as reviewers, editors, and researchers, political scientists appear to be far too conscious of the .05 significance level, and that this might cause important distortions in how knowledge advances in political science.

Full paper here.

3 thoughts on “1.96 is the magic number

  1. Michael Allen

    I have to read through the paper, but it is not interesting that the APSR or the AJPS do not publish many statistically insignificant papers. If you fail to reject the null, then you don’t learn much. The test of significance is biased in favor of the null (95%) so that failing this test is almost meaningless (i.e. you do not go on to accept the null).

    Maybe I need to go through it, but why would you want to publish a paper that fails to reject the null hypothesis?

  2. Jonathan Dingel

    Michael,
    Do you learn much at p=0.04 but not much at p=0.07? Is that difference significant (statistically or otherwise)? In fact, the APSR and AJPS do appear to publish plenty of statistically insignificant findings, but there is a noticeable absence of papers falling just short of 1.96. Gelman, Drum, and the authors offer potential explanations of this distortion.

Comments are closed.