A common error regarding the p-value

Although you are to be commended for suggesting limitations regarding the applicability of Dr. Ioannidis’s controversial hypothesis, you inadvertently perpetuated a common misunderstanding regarding the meaning of p-values. Whereas you stated that "[i]f p < 0.05, then there’s less than a one in twenty probability the observed effect happened by chance." the p-value dose not comment upon the probability that the outcome was due to chance. Rather, the p-value is calculated after assuming that there is no real difference between the treatment arms and represents the probability of obtaining a distribution of data that is at least as extreme as that observed, again in the absence of any real difference. The p-value describes the data observed, not the probability of the hypothesis. Of course, a very small p-value suggests that the data are not consistent with the null hypothesis; one may conclude that the null hypothesis is improbable, presuming that the experimental hypothesis is plausible (if it weren’t, I wonder why the experiment was conducted). To sum up, a p < 0.05 is equivalent to the statement "Even if there is no difference between the treatment arms, there is less than 5% chance of observing this much difference in outcome between them." All of which is consistent with your commentary as a whole, and with Ioannidis’s article as well.