What is OmniNerd?

Welcome! OmniNerd's content is generated by you, the reader. Through voting and moderation we strive to highlight the nerdiest of what's around and provide content that's a little more thought provoking than other sites.

Submit New Content

Voting Booth

Ever lie?

View Results, All Polls
16 votes, 1 comment
2
Nerd-Its
+ -

In fairness...

Comment comment by JSinger on 10 October 2007

Clinical trials are required to specify the details of their analysis in advance and correct for multiple testing, so while your explanation is correct, your choice of example is unfair. Ioannidis is talking about epidemiology, which is notorious for exactly the kind of fishing you describe, and as you say, that issue isn't especially applicable to more qualitative experiments.

You are a viewing a subset of a larger discussion.
View Full Discussion
Star This to Save in Your Profile Favorite
Thread parent sort order:
Highest Voted : Lowest Voted : Oldest : Newest
Thread verbosity:
Expand All : Minimize Replies to Comments
1 Nerd-It - +
RE: In fairness... by scottb :: NR7

Since Ioannidis manages to come to the conclusion that "Most Published Research Findings are False", and since it's exactly that broad applicability that garnered the article to much attention, I'm not sure my choice of example is "unfair".

But the real meat of Ioannidis' argument is not about the bias introduced by "fishing" for correlation. He's arguing it's inherent in the process. It comes (in his model) from the false positives, and is magnified by an inadequate level of independent verification.

My point is that his model is fundamentally wrong. His R arguably not a well-defined random variable at all, and even if it were, it's absurd to treat its distribution the way he has. Doing so assumes that hypothesis selection is completely unbiased with respect to the truth of the hypothesis - that scientists have no "intuition" whatsoever about the likely outcomes of experiments.

That may be true in a few areas - the gene testing he describes in the paper plausibly suffers from it - but it's not at all true of the overwhelming majority of science.

Posted 10 October 2007, Hide, Parent, Reply