Loading 1 Vote - +

Testing MythBusters' Yawn Contagiousness Data

MythBusters,1 a popular television program on the Discovery Channel, is co-hosted by a pair of geeky, middle-aged engineer types in Adam Savage and Jamie Hyneman.2 Building contraptions and using a loose interpretation of the experimental method, Adam and Jamie have addressed topics such as if bulletproof shields are really bullet proof,3 if one’s toes can really be amputated inside of a steel-toed boot,4 and most recently, if ninjas can run on water.5

Usually, these destroyers of wives’ tales are fairly thorough in their analysis, but there seem to be some instances where the busters are themselves busted. Some say their measurement of a daddy long legs’ fang is off,6 some say they use the wrong kinds of arrows,7 and others say their paper is much too thin.8 Granted, such scrutiny is to be expected from armchair critics everywhere, but few are able (or willing?) to design and carry out counter tests.

Occasionally, however, a blunder arises which can be exposed without expensive test fixture designs. Sometimes all you need is a simple understanding of statistics. MythBusters’ investigation into the suspected contagious nature of the everyday yawn9 is one such instance.

Experiment

The premise of the experiment was to invite a stranger to sit in a booth for a period of time sufficient to bore even the least ADD prone. Fifty subjects were said to be tested in total, of which two-thirds were "seeded" with a yawn by the experiment attendee. Those in the remaining third were given no yawn seed. Using a two-way mirror and a hidden camera, Kari, Scottie and Tory (Adam and Jamie’s young helpers) observed and recorded how many of the fifty yawned and how many did not.

MythBusters’ Conclusion

After the experiment coverage, the show documented Kari, Scottie and Tory approaching their elder mentors with the following results:

  • 25% yawned of those not given a yawn seed.
  • 29% yawned of those given a yawn seed.
    Faced with these numbers, the masters of determining truth from error cited the "large sample size" and the 4% difference in the results in confidently concluding the yawn seed had a significant effect on the subjects and, therefore, the yawn is decisively contagious.

Busting the Busters

EDITORIAL NOTE: See this comment thread for an important addendum to the statistical analysis.

Statistical Significance

The issue, of course, is significant results cannot be determined with one’s gut in many cases, as a cursory observation of sample size and resulting percentage can be misleading. There are, however, simple and straightforward statistical methods to determine whether or not input factors correlate to output results. When applied to the data in this particular MythBusters episode, it does not bode well for Adam and Jamie.

Sample Size Analysis

The first order of business is to extrude more details from the reported results. The available information includes:

  • 25% yawned of those not given a yawn seed.
  • 29% yawned of those given a yawn seed.
  • Approximately 50 subjects were tested.
  • Two-thirds were seeded (given a yawn seed).
  • One-third was not seeded.
    4_article_75_thumb_sample_analysis

    Table 1. Sample size analysis using the total sample size and the reported percentages


    Next, the following variables are assigned:
  • x – subjects not seeded with a yawn
  • y – subjects seeded with a yawn

Transferring these variables into the above information yields:

  • x + y ~ 50
  • .25_x_ = close to a whole number
  • .29_y_ = close to a whole number
  • x ~ 50 / 3
  • y ~ 50 * 2 / 3
    4_article_75_thumb_yawn_data

    Table 2. Layout of likely data gathered by MythBusters team


    Taking various values of x and y that might have been construed into these results, the most fitting set seems to confirm the sample size of 50, with 34 seeded subjects were given a yawn seeded.

Thus, the results are likely distributed as follows:

  • 4 yawned out of 16 subjects not given a yawn seed
  • 10 yawned out of 34 subjects given a yawn seed
  • 14 total yawns out of 50 subjects

Correlation Analysis

Correlation analysis is the perfect tool to determine if these results indicate a yawn seed would significantly alter the likelihood of a subject yawn. The indicated factor in this analysis is the correlation coefficient. Continuing with the previous denomination of those given the yawn seed as series A and those who yawned as series B, the following calculations apply:10

4_article_75_thumb_stat_calcs

where

  • Covariance(A,B) is the sample covariance between A and B,
  • Variance(A) is the sample variance of A, and
  • Variance(B) is the sample variance of B.

These calculations result in a correlation coefficient of .045835 between the two series, or between those given the yawn seed and those who actually yawned. Sorry Adam and Jamie, this indicates no correlation between the two series. (A value of at least 0.1 is needed for even a weak correlation.11)

Where MythBusters Went Wrong

If 29% is not considered beyond the reach of chance with respect to 25% in a sample set of 50, what is? While it may initially seem an increase in the sample size might help, if the percentages remain the same, the correlation coefficient does not move. The only change that would make Adam and Jamie’s assessment correct, then, would be if the percentages were further apart.

Assuming the sample size remained at 50 subjects, two-thirds of whom were seeded with yawns, adding one yawner in the seeded group raises the correlation coefficient to .074848. Another needs to be added before the coefficient breaks into the significant range, at .102941. On the other hand, if only one yawner is taken out of the non-seeded group, the figure jumps over the threshold to .113385. Thus, the following two conditions satisfy the requirements for statistically significant results:

  • 4/16 (25.00%) of those not seeded yawn, and 12/36 (33.33%) of those seeded yawn.
  • 3/16 (18.75%) of those not seeded yawn, and 10/36 (29.41%) of those seeded yawn.
    It appears a percentage difference at least in the range of 8-10% is required given MythBusters’ setup – double the 4% found in the actual experiment performed and so embarrassingly interpreted in front of millions of viewers.

MythBuster’s Reaction Since

It is difficult to see how these results could have been missed – if not at the time, then at least in hindsight. This is a big-time television program with a whole staff of editors. Surely _some_one in back raised their hand in protest _some_where along the way to point out _some_thing just did not seem right. Not that they would publish a recall or anything, but at least they could avoid the subject when in public

Quite to the contrary, calling on the results of this yawn contagiousness "proof," MythBusters launched a campaign in September 2006 to try and send a yawn "around the world."12 Not only that, they invited browsers to "catch" a yawn by viewing a video of Adam on YouTube,13 while claiming responsibility for discovering the science behind the alleged phenomenon:

The MythBusters identified that yawning is officially contagious, but we want to go one step further and involve as many people as possible in our biggest ever experiment. If only one per cent of the global population took part in the Yawn Around The World experiment then 65 million people would have yawned across the globe, which would be an amazing achievement.
The odd thing is, both may very well work. While the MythBusters should be ashamed to fall victim to such an obvious statistical blunder, it is still very possible that yawns are contagious. If they are, and if these experiments succeed, however, it most definitely will not be because MythBusters had anything to do with "officially" determining the contagiousness of a yawn – not with a correlation coefficient of .045835.

Notes

1 "MythBusters." Discovery Channel. Accessed March 2007 from http://dsc.discovery.com/fansites/mythbusters/mythbusters.html.

2 "MythBusters: Bios." Discovery Channel_. Accessed March 2007 from http://dsc.discovery.com/fansites/mythbusters/meet/meet_main.htmlmain.html.

3 "Episode 16: : Ancient Death Ray, Skunk Cleaning, What Is Bulletproof?" MythBusters_. Aired September 29, 2004. Accessed March 2007 from http://dsc.discovery.com/fansites/mythbusters/episode/00to49/episode_07.html07.html.

4 "Episode 42: Steel Toe-Cap Amputation." MythBusters_. Aired November 9, 2005. Accessed March 2007 from http://dsc.discovery.com/fansites/mythbusters/episode/00to49/episode_02.html02.html.

5 "Episode 78: Walking on Water." MythBusters. Aired April 25, 2007. Accessed April 2007 from http://dsc.discovery.com/fansites/mythbusters/episode/episode.html.

6 "Daddy Long Legs – Great Moments in Science – The Lab." ABC.net.au. Accessed March 2007 from http://www.abc.net.au/science/k2/moments/s1721788.htm.

7 "Mythbusters are dead wrong…" Sword Forum International. Accessed March 2007 from http://forums.swordforum.com/showthread.php?threadid=70841.

8 "Paper folding." Discovery Channel Fansite. Accessed March 2007 from http://community.discovery.com/eve/forums/a/tpc/f/9801967776/m/9041918678.

9 "Episode 28: Is Yawning Contagious?" MythBusters_. Aired March 9, 2005. Accessed March 2007 from http://dsc.discovery.com/fansites/mythbusters/episode/00to49/episode_05.html05.html.

10 Weiss, Neil A. "Descriptive Methods in Regression Correlation." Elementary Statistics, 4th ed. Addison Wesley Longman, Inc., 1999. p.195-246. See also: "Correlation Coefficient." WolframMathWorld. Accessed April 2007 from http://mathworld.wolfram.com/CorrelationCoefficient.html.

11 Cohen, J. Statistical power analysis for the behavioral sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates. 1988.

12 "A new yawn for MythBusters." News.com.au. Accessed April 2007 from http://www.news.com.au/entertainment/story/0,23663,20471697-10388,00.html.

13 “Yawning is contagious.” YouTube.com. Accessed April 2007 from http://www.youtube.com/watch?v=Cy-Pf6oJNRo.

Similarly tagged OmniNerd content:

Information This article was edited after publication by the author on 21 Dec 2008. View changes.
Thread parent sort order:
Thread verbosity:

O.K., here’s a question about correlation coefficients in general.
Say you somehow repeated this same experiment 50 times, and 45 out of 50 times the percentages came out with a correlation coefficient of something below the .10 needed for a weak correlation, but at the same time in favor of the yawns being contagious. In other words the correlation coefficient of whether each experiment’s results were majority or minority would show a high correlation. What would this say about yawn contagiousness then?
Or is this considered statistically impossible to actually happen, given the CC’s obtained from the first experiment?

This article is being challenged on Slashdot.

0 Votes  - +
Completely Flawed. by Anonymous

You do NOT use descriptive statistics to study a sample, you need a completely different way of approaching things, namely, statistical analysis.

What you really need to do is Hypothesis testing (http://en.wikipedia.org/wiki/Statistical_hypothesis_testing), and test whether the hypothesis that more people yawn when seeded than when not seeded.

Your analysis is completely flawed, ask any statistician.

Well, it is flawed, but not totally wrong. I coded the data set that was used on the web page and ran it through a chi-square/McNemar+Risk Estimate test (appropriate tests for dichotomous treatment variables + dichotomous outcome variables). No significant difference alpha=.744. But shame on you for using a straight up correlation. —chris

I received a number of emails concerning the statistical method I used (Pearson’s correlation coefficient), which provided some insight but does not sufficiently address the issue of causation in the results. Personally, I don’t understand how there can be so obviously not a correlation between two variables and there still be a chance there is causation involved, but with the aim of statistical appropriateness, I have included a number of alternative statistical methods below.

Association Test
An association test such as Fisher’s Exact Test is appropriate. This method is specifically for determining any non-random association between two categorical (discrete) variables – which is exactly what we have in this instance. Its use, then, removes any issues there may have been in the Pearson analysis having to do with the data set not being continuous.

For those interested, the calculations are described in the link above. The results are easy to come by, however, using online tools such as this calculator at Matforsk.com. Inserting the MythBuster’s data results in the following:


TABLE = [ 4 , 10 , 12 , 24 ]
Left   : p-value = 0.5127817757319189
Right  : p-value = 0.7416878304307283
2-Tail : p-value = 1

This corresponds to there being 4 non-seeded subjects who yawned, 10 seeded who yawned, 12 non-seeded to didn’t yawn, and 24 seeded who didn’t yawn. The resulting p-values are all well above the commonly accepted limit of .05 for significance.

Confidence Interval for the Difference in Rates
This method was recommended via email by Max Kuhn, a "Ph.D. statistician who works in industry." Max provided a very thorough and helpful analysis of the data, which I’ve included below:

Here is what I would do: create a confidence interval for the difference in rates. I would do this because 1) it can be used to evaluate whether the difference in rates is equal to zero, 2) the width of the interval helps characterize the uncertainty in the data, which is directly related to sample size and 3) p-values alone do not provide people with enough information. Here is a link to a summary of the calculations.

I use the R statistical language a lot, and here is how I got the result:


          p1 <- 10/34
          p2 <- 4/16
          n1 <- 34
          n2 <- 16
          q1 <- 1-p1
          q2 <- 1-p2
          p1 - p2
          [1] 0.04411765
          
          sqrt((p1*q1/n1) + (p2*q2/n2))
          [1] 0.1335103
          
          p1 - p2 + (qnorm(0.05) * sqrt((p1*q1/n1) + (p2*q2/n2)))
          [1] -0.1754872

This means that if they were to repeat the same experiment a large number of times, we could feel confident that the difference might be as low as 18% in the other direction. This doesn’t give us a good feeling that the seed did anything.

One caveat: the number of events is somewhat low here and many people would tell you that the statistical theory may not be valid for our data. To investigate this a little more, I bootstrapped the simple difference in rates 5,000 times. This lets us estimate the empirical distribution of the difference in rates for our data instead of relying on distributional assumptions and approximations to give us a confidence bound. The empirical distribution of the difference in rates looks fairly symmetric and Gaussian. Using a "bootstrap-t" interval, the lower 95% confidence bound was -0.28, which gives s more reason to doubt that there is a difference. Code for this analysis is below. …


          testStat <- function(index, data)
          {
             bootSample <- data[index,]
             p1 <- mean(bootSample[bootSample$group == "withSeed", "outcome"] == "yawn")
             p2 <- mean(bootSample[bootSample$group == "noSeed", "outcome"] == "yawn")
             p1 - p2
          }
 
          noSeed <- rep(c("yawn", "none"), times = c(4, 12))
          withSeed <- rep(c("yawn", "none"), times = c(10, 24))
 
          mythBusters <- data.frame(
             outcome = factor(c(noSeed, withSeed), levels = c("yawn", "none")),
             group = factor(rep(c("noSeed", "withSeed"), times = c(16, 34))))
 
          testStat(1:50, mythBusters)
 
          library(bootstrap)
          set.seed(1)
          results <- boott(1:50, theta = testStat, nboott = 5000, data = mythBusters, perc = 0.05)

Linear Regression to Show Sample Size Needed for Significance
I received yet another very friendly and helpful email from Zinj Boisei who pointed out I was too hasty in dismissing the use of an increased sample size. By using a more appropriate analysis, linear regression in this case, Zinj confirmed there was little significance at the sample size of 50 – and even went on to find out large a sample size of the same makeup would need to be for the results to be significant:

[T]he low correlation [found in the article] indicates that only a small amount of the variability may explained by "yawn seed", but the real question is "is the effect real, whatever the size?" A larger sample size would certainly answer this question.

I took the data you prepared, and constructed a linear model for a logistic regression analysis which could be used to test such a question. As expected, I found that at n=50 the results were nowhere near significant (probability of .75 that we would see these results by chance alone). I then searched for the multiple of the sample size that would reject a null hypothesis of no effect at 95% confidence. I found that this occurred at 37 times the original sample size, or n=1850.

Conclusion Addendum
While the statistical method used in the article was sufficient to show the yawn seed was responsible for a negligible amount of the variance, methods such as association tests, confidence interval analysis and linear regression provide more appropriate insight into the causation involved. In this case, all tests lend credence to the original conclusion: the results of the MythBuster’s yawn experiment did not support their conclusion.

I preformed a Chi-Square Test on a TI-84 Silver Edition calculator, so any error in rounding or anything is not my fault, it is the fault of the statistics package. With that out of the way….

I decided to use the data the Myth Buster’s collected on yawning as follows:

I choose to take their percents and apply them to two groups out of 100, one seeded, and one not. I performed the following tests assuming their data is correct, but there is always the chance that it is not. There is also another facter that I will discuss at the bottom.

I took the expected number of people who yawned without being “seeded” to be 25 and the people in the same group who didn’t yawn to be 75.

In the group who were “seeded”, I used the 29% the Myth Buster’s percieved and ended up with 29 yawners, andn 71 not.

H0= A group of people who are seeded WILL NOT yawn more that a group who of people who aren’t seeded,

Ha= A group of people who are seeded WILL yawn more that a group who of people who aren’t seeded

I performed a Chi-Square Goodness of Fit Test on the data with one degree of freedom and if gave me the following:

Chi-Square Value: 0.853333 (three repeating)

p=0.3556110613

df=1

A p-value of 0.356 (rounded) is MUCH too high to reject the null hypothesis. I generally use an alpha-level of 0.05, (alpha levels are generally somewhere from 0.10 to 0.01), but no proper statistitian would ever reject the null hypothesis on a p-value thats as high as 0.356.

In colloquial terms, there is NOT enough evidence to conclude that when a group of people are seeded with a yawner, they are more likely to yawn than otherwise.

Now, here is that other factor I promised to get to:

If the two groups are seeded and unseeded, that means in one group, a person yawns, attempting to get others to yawn, and in another group, they are left on their own as a control group. Here is the problem with that logic.

As soon as one person yawns in the control group (or unseeded group), the rest of the people in that group are now seeded, which makes the group no longer valid as a control group. This is a difficult obstacle to overcome and I will not say that I have the answer, but I will say that it distorts the data.

In the end, my conclusion is thus:

With the data the Myth Busters collected, there is not enough evidence to say that yawns are contagious. However, since their “control” group really isn’t entirely unseeded (only the first yawn is), it makes the rest of the yawns seeded. It is entirely possible that more people yawned after the first person yawned because they were seeded, which means yawns are contagious, but the data (which I have just proven is unreliable) shows that there is no proof that yawns are contagious. I am not saying that yawns aren’t contagious, I am simply saying there is no proof that they are. A good metaphor is like a court. I am not saying they are innocent (or that yawns aren’t conagious), I am simply stating that there is not enough evidence to consider them guilty (or that yawns are contagious). To continue the analogy, the suspect may have actually committed the crime, but without the proper evidence, we cannot convict. So, to sum things up in one sentence:

The data found by the Myth Busters is flawed, but if one truely still believed it to be true, there is still not enough evidence to support their claim that yawns are contagious, so we must stay in the mindset that they are not contagious until unflawed data is produced that proves otherwise.

2 Votes  - +
Yawning Babies by VnutZ

So I’ve noticed … my kid does not yawn after seeing either of her parents yawn. But we certainly do after seeing her yawn. My scientifically sound sample of one therefore leads me to conclude it’s a learned behavior. :-)

Share & Socialize

What is OmniNerd?

Omninerd_icon Welcome! OmniNerd's content is generated by nerds like you. Learn more.

Voting Booth

What if a spouse cheats?

23 votes, 4 comments