Add a Comment
Email This
Statistics
RE: A common error regarding the p-value
Yes, and no.
From a strict mathematical perspective, you're right, of course. I've always found the precise meaning of p to be a little hard to articulate in a way that really conveys the difference, though, and unless it's relevant, I don't bother.
In general, on the additional hypothesis that the experiment is correctly constructed, the p value does become the probability the tested effect occurred by chance. In a correctly constructed experiment, there are only two possible causes to which one can attribute an observed effect - to "chance" or to the treatment under test.
So, p (properly) is the probability the observed difference would occur if the treatment was ineffective. But, our experiment is constructed so that if the treatment is ineffective, then the observed outcome is due to chance. So, under those conditions, p is the probability that the observed difference would occur due to chance.
RSS
A common error regarding the p-value
Although you are to be commended for suggesting limitations regarding the applicability of Dr. Ioannidis's controversial hypothesis, you inadvertently perpetuated a common misunderstanding regarding the meaning of p-values. Whereas you stated that "[i]f p < 0.05, then there's less than a one in twenty probability the observed effect happened by chance." the p-value dose not comment upon the probability that the outcome was due to chance. Rather, the p-value is calculated after assuming that there is no real difference between the treatment arms and represents the probability of obtaining a distribution of data that is at least as extreme as that observed, again in the absence of any real difference. The p-value describes the data observed, not the probability of the hypothesis. Of course, a very small p-value suggests that the data are not consistent with the null hypothesis; one may conclude that the null hypothesis is improbable, presuming that the experimental hypothesis is plausible (if it weren't, I wonder why the experiment was conducted). To sum up, a p < 0.05 is equivalent to the statement "Even if there is no difference between the treatment arms, there is less than 5% chance of observing this much difference in outcome between them." All of which is consistent with your commentary as a whole, and with Ioannidis's article as well.
View Full Discussion