2 Nerd-Its - +

In fairness...

Comment a comment by JSinger, published on 10 October 2007
Navigate to the top level to view all replies to the article Science - Not So Sloppy
a lone nerd has left 1 comment below

Clinical trials are required to specify the details of their analysis in advance and correct for multiple testing, so while your explanation is correct, your choice of example is unfair. Ioannidis is talking about epidemiology, which is notorious for exactly the kind of fishing you describe, and as you say, that issue isn’t especially applicable to more qualitative experiments.

Thread parent sort order:
Thread verbosity:
1 Nerd-It - +
RE: In fairness... by scottb :: NR8

Since Ioannidis manages to come to the conclusion that "Most Published Research Findings are False", and since it’s exactly that broad applicability that garnered the article to much attention, I’m not sure my choice of example is "unfair".

But the real meat of Ioannidis’ argument is not about the bias introduced by "fishing" for correlation. He’s arguing it’s inherent in the process. It comes (in his model) from the false positives, and is magnified by an inadequate level of independent verification.

My point is that his model is fundamentally wrong. His R arguably not a well-defined random variable at all, and even if it were, it’s absurd to treat its distribution the way he has. Doing so assumes that hypothesis selection is completely unbiased with respect to the truth of the hypothesis – that scientists have no "intuition" whatsoever about the likely outcomes of experiments.

That may be true in a few areas – the gene testing he describes in the paper plausibly suffers from it – but it’s not at all true of the overwhelming majority of science.

The Showcase

Nerd-Its   Nerd Trends   Last Ten  

  1. RE: We can do better. in U.S. Healthcare: the Best, the Worst, and the Irrelevant
  2. The world could end, any moment, any second... in NASA: THE WORLD WILL NOT END IN 2012
  3. RE: We can do better. in U.S. Healthcare: the Best, the Worst, and the Irrelevant
  4. RE: Why wouldn't it be a religion? Yes, but .... in Scientology: We've had it with you
  5. RE: Why wouldn't it be a religion? Yes, but .... in Scientology: We've had it with you
  6. RE: Why wouldn't it be a religion? Yes, but .... in Scientology: We've had it with you
  7. RE: Why wouldn't it be a religion? Yes, but .... in Scientology: We've had it with you
  8. RE: Sick care in U.S. Healthcare: the Best, the Worst, and the Irrelevant
  9. RE: Why wouldn't it be a religion? Yes, but .... in Scientology: We've had it with you
  10. RE: Why wouldn't it be a religion? Yes, but .... in Scientology: We've had it with you

What is OmniNerd?

Omninerd_icon Welcome! OmniNerd's content is generated by nerds like you. Learn more.

Voting Booth

The Interstate Commerce Clause of the U.S. Constitution empowers Congress to regulate?

11 votes, 0 comments