|
10
Nerd-Its
|
|
article
by Wall Street Journal coverage has revived interest in Dr. John Ioannidis' 2005 Public Library of Science article, "Why Most Published Research Findings Are False." This piece addresses flaws in Ioannidis' reasoning.
Add a Comment (4)
Email This
Message Author
Although you are to be commended for suggesting limitations regarding the applicability of Dr. Ioannidis's controversial hypothesis, you inadvertently perpetuated a common misunderstanding regarding the meaning of p-values. Whereas you stated that "[i]f p < 0.05, then there's less than a one in twenty probability the observed effect happened by chance." the p-value dose not comment upon the probability that the outcome was due to chance. Rather, the p-value is calculated after assuming that there is no real difference between the treatment arms and represents the probability of obtaining a distribution of data that is at least as extreme as that observed, again in the absence of any real difference. The p-value describes the data observed, not the probability of the hypothesis. Of course, a very small p-value suggests that the data are not consistent with the null hypothesis; one may conclude that the null hypothesis is improbable, presuming that the experimental hypothesis is plausible (if it weren't, I wonder why the experiment was conducted). To sum up, a p < 0.05 is equivalent to the statement "Even if there is no difference between the treatment arms, there is less than 5% chance of observing this much difference in outcome between them." All of which is consistent with your commentary as a whole, and with Ioannidis's article as well.
Yes, and no.
From a strict mathematical perspective, you're right, of course. I've always found the precise meaning of p to be a little hard to articulate in a way that really conveys the difference, though, and unless it's relevant, I don't bother.
In general, on the additional hypothesis that the experiment is correctly constructed, the p value does become the probability the tested effect occurred by chance. In a correctly constructed experiment, there are only two possible causes to which one can attribute an observed effect - to "chance" or to the treatment under test.
So, p (properly) is the probability the observed difference would occur if the treatment was ineffective. But, our experiment is constructed so that if the treatment is ineffective, then the observed outcome is due to chance. So, under those conditions, p is the probability that the observed difference would occur due to chance.
RSS
In fairness... by JSinger :: NR2 :: on 10 October 2007
Clinical trials are required to specify the details of their analysis in advance and correct for multiple testing, so while your explanation is correct, your choice of example is unfair. Ioannidis is talking about epidemiology, which is notorious for exactly the kind of fishing you describe, and as you say, that issue isn't especially applicable to more qualitative experiments.
RE: In fairness... by scottb :: NR7 :: on 10 October 2007
Since Ioannidis manages to come to the conclusion that "Most Published Research Findings are False", and since it's exactly that broad applicability that garnered the article to much attention, I'm not sure my choice of example is "unfair".
But the real meat of Ioannidis' argument is not about the bias introduced by "fishing" for correlation. He's arguing it's inherent in the process. It comes (in his model) from the false positives, and is magnified by an inadequate level of independent verification.
My point is that his model is fundamentally wrong. His R arguably not a well-defined random variable at all, and even if it were, it's absurd to treat its distribution the way he has. Doing so assumes that hypothesis selection is completely unbiased with respect to the truth of the hypothesis - that scientists have no "intuition" whatsoever about the likely outcomes of experiments.
That may be true in a few areas - the gene testing he describes in the paper plausibly suffers from it - but it's not at all true of the overwhelming majority of science.