RE: In fairness...

Since Ioannidis manages to come to the conclusion that "Most Published Research Findings are False", and since it's exactly that broad applicability that garnered the article to much attention, I'm not sure my choice of example is "unfair".

But the real meat of Ioannidis' argument is not about the bias introduced by "fishing" for correlation. He's arguing it's inherent in the process. It comes (in his model) from the false positives, and is magnified by an inadequate level of independent verification.

My point is that his model is fundamentally wrong. His R arguably not a well-defined random variable at all, and even if it were, it's absurd to treat its distribution the way he has. Doing so assumes that hypothesis selection is completely unbiased with respect to the truth of the hypothesis - that scientists have no "intuition" whatsoever about the likely outcomes of experiments.

That may be true in a few areas - the gene testing he describes in the paper plausibly suffers from it - but it's not at all true of the overwhelming majority of science.

Star This to Save in Your Profile