Matthew Hankins recently wrote a very nice post cataloging the ways that researchers try to indicate that some result is *this* close to being significant, but doesn’t quite make the cut. His point is a very good one: a result from a statistical test is either significant, it passes some rather arbitrary threshold (say, less than 0.05), or it isn’t. There’s no almost in significance, no trending toward significance, no flirting with significance. It’s significant or it’s insignificant, period.
I thought it would be useful to also catalog the flip side of this coin: what about when a result passes a significance test and keeps on going? These results are still “just significant”- not “ultra-incredibly significant”, since significance is a binary value. Accordingly, I have assembled a list of ways that authors have expressed that a result has a very low p value and thus is very significant. I also sampled real publications and found that the actual use of these phrases danced lightly about the verge of sending out tendrils to touch something that is close to significance. I feel that this result really wanted to be significant and was moving in that direction, toward significance. With a little more effort and if everyone believes, it can be significant. Say it with me, “I believe it’s significant. I believe it’s significant” (p value 0.99).
Disclaimer: I know that I have, on more than one occasion, been a perpetrator of each of these errors stating that a result is ‘close to significant’ or is ‘highly significant’. I’ll try to be better in the future.
Thanks to Shanon White for the idea for this post. This is an incomplete list. If you have other examples please add them to the comments or Tweet them with the hashtag #ohsosignificant.
- highly significant (p<1e-8)
- very significant (p<0.01)
- extremely significant (p<0.0001)
- whoah baby that’s significant (p<1e-6)
- I got your significance right here (p<0.0002)
- Holy Sh*t! In you FACE b*tches. This sh*t’s significant (p<1e-90)
- By the power of Greyskull, we have the significance! (p<1e-23)
- Say hello to my little significance (p<1e-14)
- You can’t HANDLE the significance (p<1e-30)
- BAM! There it is daawwg! That’s significance right there! (p<1e-20)
- You call THAT significant? That’s not significant. THIS is significant (p<1e-45)
- the mostest significant in the whole wide world (p<1e-29)
- Neener neener neener motherf**ker (p<1e-65)
- significance of the utmost elevated level (p<1e-9)
- Oh that’s good. Really good. Actually I’m thinking that might be Science or Nature good it’s that good. Holy crap, this is actually working. For once it’s working. Oh god I’m so excited, I’m going to totally rub it in the faces of my smug thesis committee. That’ll show them. Yeah. Oh god I hope it’s not wrong. Please let it be not wrong (p<1e-18)
- solidly, unequivocally significant (p<1e-12)
- Bonferroni? We don’t need no stinkin’ Bonferroni (p<1e-56)
- First, we brought you a significant result (p<0.05). Then we rolled out a very significant result (p<0.01). But can we go further? That’s just crazy, right? Nope. We did it, presenting our new ultra significant result (p<1e-20), now with smoother trending.
Here’s the serious part of this post
This is a semantic argument at its heart. It’s a valuable, important, and true fact that statistical significance does not come in shades of gray; it either is or it isn’t. However, we as intelligent, statistically savvy readers interpret these statements, or at least those that are on the border and not hyperbole, as meaning, “if we were to shift our arbitrary threshold we used for statistical significance to a more lenient/conservative value, then the result we talk about would now meet our new criterion for significance”. Yes, the authors should have just set that level of significance to start out with and not bothered to backtrack to make a point. And yes, many of the real statements on Matthew’s post and the (mostly) fake statements on mine are in the realm of the far out and are just silly (a p value of 0.3 being ‘nearly’ significant, really!?). But really the important thing is that you clearly and completely report your findings, the methods you used to arrive at those findings (and conclusions), and provide access to your data so that the interested reader can make their own judgement.