Loading 3 Votes - +

Internet Weather Forecast Accuracy

Weather forecasting is a secure and popular online presence, which is understandable. The weather affects most everyone’s life, and the Internet can provide information on just about any location at any hour of the day or night. But how accurate is this information? How much can we trust it? Perhaps it is just my skeptical nature (or maybe the seeming unpredictability of nature), but I’ve never put much weight into weather forecasts – especially those made more than three days in advance. That skepticism progressed to a new high in the Summer of 2004, but I have only now done the research necessary to test the accuracy of online weather forecasts. First the story, then the data.

An Internet Weather Forecast Gone Terribly Awry

It was the Summer of 2004 and my wife and I were gearing up for a trip with another couple to Schlitterbahn in New Braunfels – one of the (if not the) best waterparks ever created.1 As a matter of course when embarking on a 2.5-hour drive to spend the day in a swimsuit, and given the tendency of the area for natural disasters,2 we checked the weather. The temperatures looked ideal and, most importantly, the chance of rain was a nice round goose egg.

A couple of hours into our Schlitterbahn experience, we got on a bus to leave the "old section" for the "new section." Along the way, clouds gathered and multiple claps of thunder sounded. "So much for the 0% chance of rain," I commented. By the time we got to our destination, lightning sightings had led to the slides and pools being evacuated and soon the rain began coming down in torrents – accompanied by voluminous lightning flashes. After at least a half an hour the downpour had subsided, but the lightning showed no sign of letting up, so we began heading back to our vehicles. A hundred yards into the parking lot, we passing a tree that had apparently been split in two during the storm (whether by lightning or wind, I’m not sure). Not but a few yards later, there was a distinct thud and the husband of the couple accompanying us cried out as a near racquetball sized hunk of ice rebounded off of his head and onto the concrete. Soon, similarly sized hail was falling all around us as everyone scampered for cover. Some cowered under overturned trashcans while others were more fortunate and made it indoors.

The hail, rain and lightning eventually subsided, but the most alarming news was waiting on cell phone voicemail. A friend who lived in the area had called frantically, knowing we were at the park, as the local news was reporting multiple people had been by struck by lightning at Schlitterbahn during the storm.

"So much for the 0% chance of rain," I repeated.

Testing the Skepticism

After that experience, I gave up using online weather forecasts (actually any weather forecast) for more than getting a reasonable idea of the "temperature decade" for the next day. I’ve recently begun to be a little skeptical of my own skepticism, however. What if I was the victim of a freak waterpark occurrence and was missing out on the typically reliable weather information online? Using a spreadsheet, observed data and straightforward statistics, I was set to find out.

My plan was to record the weather forecasts of some of the most popular Internet weather sites as well as actual temperatures and then to analyze the data to determine each site’s accuracy. I would then be able to draw supported conclusions to apply to future use of Internet weather forecasts (if any).

Data Mining Internet Weather Forecasts

Doing an Internet search for various weather related keywords, and then cross-referencing to avoid duplication,3 I selected the top ten weather forecast sites to be included in my survey using their Google Toolbar4 PageRank (PR).5 Additionally, I selected Houston, Texas as the location and The Weather Channel as my "actual temperature" source.6

  • The National Weather Service7 – PR9
  • BBC Weather8 – PR9
  • The Weather Channel9 – PR8
  • The Weather Underground10 – PR8
  • IntelliCast11 – PR8
  • CNN Weather12 – PR8
  • MSN Weather13 – PR8
  • The Weather Network14 – PR7
  • Unisys15 – PR7
  • AccuWeather16 – PR6
  • Actual (as reported on weather.com)17

Then, on a daily basis I recorded the predicted low and high temperatures on each weather forecast site going back as far as was made available. This varied greatly from site to site, with CNN Weather, BBC Weather, The Weather Underground and The Weather Network providing only the current day and four days into the future, and Accuweather providing the current day and four_teen_ days into the future. I usually logged the data at 12pm CST, but occasionally as late as 5pm CST, which resulted in some high temperature predictions for the current day not being available, as well as (oddly enough) the low temperature not being available in a few cases. I also recorded average and record temperatures for all days considered.

4_article_69_thumb_weather_data

Table 1: Sample portion of data sheet.

Calculations on Forecast Accuracy and Consistency

In order to assess the accuracy and consistency of each weather forecast site, I first found the absolute values of the differences between the predicted and actual temperatures. For example, considering the data presented in Table 2 above, the actual high temperature in Houston, TX on Thursday, December 21st was 70° F. At noon on Thursday, December 14th, The Weather Channel online predicted the high on that day would be 60° F, 10° off of the actual and yielding an "accuracy value" of 10. On the same day, MSN Weather online predicted a high of 45° F, corresponding to a value of 25 – the higher number indicating a poorer performance. The tables turned somewhat two days later when The Weather Channel predicted 66° and MSN Weather predicted 68°, resulting in accuracy values of 4 and 2, respectively.

Next, I calculated the mean and standard deviation of these accuracy values for each weather forecast site and predictive period (e.g., Accuweather two days in advance, The Weather Network four days in advance, etc.). The mean value representing the average accuracy and the standard deviation representing the consistency, or "spread," of the accuracy values.18

The following tables and graphs summarize the gathered weather forecast accuracy and consistency data by organizing it into columns by the number of days previous. Note than in both cases a lower number represents a better performance.

4_article_69_thumb_accuracy

Table 2: Average accuracy of each weather forecast site by the number of days previous (lower is better).

4_article_69_thumb_accuracy_high_bar

Figure 1. Average accuracy of the high temperature forecasts of each weather forecast site by the number of days previous (lower is better).

4_article_69_thumb_accuracy_low_bar

Figure 2. Average accuracy of the low temperature forecasts of each weather forecast site by the number of days previous (lower is better).

4_article_69_thumb_consistency

Table 3: Consistency of each weather forecast site by the number of days previous (lower is better).

4_article_69_thumb_consistency_high_bar

Figure 3. Consistency of the high temperature forecasts of each weather forecast site by the number of days previous (lower is better).

4_article_69_thumb_consistency_low_bar

Figure 4. Consistency of the low temperature forecasts of each weather forecast site by the number of days previous (lower is better).

Ranking Forecasts by Accuracy and Consistency

I then ranked the accuracy and consistency of each weather forecast site as compared to the competing sites (i.e., the other sites providing forecasts). Note that days 10 through 14 were omitted as Accuweather was the only site providing a weather forecast.

4_article_69_thumb_accuracy_rank

Table 4: Accuracy rank of each weather forecast site by the number of days previous (higher is better).

4_article_69_thumb_consistency_rank

Table 5: Consistency rank of each weather forecast site by the number of days previous (lower is better).

Additionally, I organized the accuracy and consistency rankings with respect to short, mid and longterm weather forecasts as dictated by the data groupings. I scored each weather forecast site with points corresponding to each ranking it received within the specified time period. For example, in order to rank weather forecast sites in the short term grouping (0-4 days in advance), I multiplied the number of first place ranks by 10, added the number of second place ranks multiplied by 9, and continued in this manner through adding the number of tenth place rankings multiplied by 1. The higher the score, the higher the ranking. Mid and long term group rankings were similarly determined with the calculations modified to fit the number of participating weather forecast sites.

4_article_69_thumb_accuracy_rank_group

Table 6: Accuracy rank groupings for short (0-4 days previous), mid (5-6 days previous), and long term (7-9 days previous) weather forecasts.

4_article_69_thumb_consistency_rank_group

Table 7: Consistency rank groupings for short (0-4 days previous), mid (5-6 days previous), and long term (7-9 days previous) weather forecasts.

Correlation of Variables to Weather Forecast Accuracy and Consistency

I also ran correlation analysis on various factors to see if they explained any of the accuracy differences observed.19 Specifically, I analyzed the following variables in order to check for the listed corresponding correlation trends:

  • Number – trends over time. I numbered the days for which the temperature was being forecast from 1 to 62, starting with December 1, 2006 (#1) and ending with January 31, 2007 (#62).
  • Day – trends with the day of the week.
  • Hi/Lo – trends between high and low forecasts. If the forecast was for a daytime high, this value was 0; if it was for an overnight low, it was 1.
  • Site – trends between different sites. Again a column for each site was used with a 1 value for when the particular site was making the prediction and a 0 value when it was another site.
  • Previous – trends between the number of days ahead the forecast was predicting. Starting with predictions made on the same day (i.e., the forecast for today’s high or tonight’s low), this value ran from 0 to 14.
4_article_69_thumb_regression_data

Table 8. Accuracy data arranged for correlation analysis.

I compared the resulting correlation values with some standard values20 to determine if there were small, medium, large or no trends correlating with the weather forecast accuracy numbers.

4_article_69_thumb_correlation_interpretation

Table 9. Standard limits for interpreting correlation values.

4_article_69_thumb_correlation_all

Table 10. Correlation values between various variables and weather forecast accuracy.

These results indicated a trend for more accurate weather forecasts closer to the temperature in question and when a high temperature is being predicted. No weather forecast site was shown to be significantly more accurate than another, though – something that does not seem to jive with the previously generated tables and graphs. It is important to note, however, that these values are generalized over all forecasting periods and for both high and low temperatures. I ran another correlation analysis to remove these variables, just the high temperature forecasts published 0 days previous in this case.

4_article_69_thumb_correlation_one_type

Table 11. Correlation values by weather forecast site for high temperature forecasts made 0 days previous.

These numbers again showed no significant correlation between the weather forecast site and the accuracy of the weather forecast. Looking back at Figure 1 and Table 2, however, this wasn’t too surprising. The forecast accuracies in this selection were reasonably tightly grouped, with the exception of BBC, and BBC was on the verge of having a small correlation. I made another selection and ran a third correlation analysis, this time on the more loosely grouped accuracy values for low temperature forecasts published 3 days previous. These numbers showed small correlations for MSN and Unisys, which are reflected by the relatively large separation from the pack in Figure 2 and Table 2.

4_article_69_thumb_correlation_two_type

Table 12. Correlation values by weather forecast site for low temperature forecasts 3 days previous.

Conclusion

While the tabled rankings brought out the competitor in me, it was obvious from the correlation analysis that only the numbers clearly "separate from the pack" in Figures 1-4 are better or worse enough as to be statistically significant. Thus, there were a few shiners and a few duds, but the variation among the rest can be explained away by chance. The trends I observed included:

  • In seeking high temperature forecasts, it looked best to use IntelliCast or The Weather Channel in the long term, but there wasn’t a clear leader in the short to mid term. BBC seemed unreliable in all cases, as well as MSN in the long term. The Weather Network, CNN and Unisys all had blemishes (3, 4 and 0 days in advance, respectively), but were generally in with the pack.
  • In seeking low temperature forecasts, IntelliCast and The Weather Channel were again the choice in the long term, joined by Unisys in the short term. BBC was still a dud in anything but the very short term, and MSN performed horribly in nearly all cases, as well as Accuweather in the long term.
  • Accuweather was the clear leader in anything greater than 10 days in advance, being the only site providing a weather forecast.

In addition to the above observations/recommendations, it was clear from the correlation analysis that the further removed a weather forecast is, the less accurate it will likely be. Much more unexpectedly, however, it was also clear that predictions of the overnight low temperature are less accurate than those of the daytime high.

Overall, the accuracy and consistency values prescribed caution – even when considering the most accurate and consistent weather forecasts. For example, if I wanted to know the high temperature for tomorrow, the numbers showed CNN Weather to be the most accurate Internet weather resource. Its weather forecast, however, comes with an average accuracy value of over 3° and a consistency value of over 2°. Thus, the conscientious browser would need to mentally append "with an accuracy of 3°±2°" to the temperature prediction and realize this results in a two degree span at best and a ten degree span at worst. This means a pessimist would be justified in reading a prediction of "75°" for tomorrow’s high as nothing more than "70°-80°" – and this using the best online resource available! Granted, the optimist would also be justified in reading the same prediction as "74°-76°," but it’s always best to plan for the worst case – especially when going to Schlitterbahn.

Many of the other less accurate weather forecasts, then, seem to be practically worthless for all but the most optimistic. Take, for example, the best option for determining the overnight low temperature a week from today, The Weather Channel. The appropriate accuracy baggage on this Internet weather forecast site would be ~5.6°±4.4°, pessimistically reducing a forecast of "50°" to "40°-60°" (!!). Perhaps this explains why only four sites ventured to provide weather forecasts more than a week in advance, and four others didn’t even push beyond four days.

So, what of my skepticism? I’d say it’s going strong. While the difference between online weather forecast sites was less than I expected, the accuracy and consistency results support a strong dose of skepticism anytime you lookup the weather on the Internet.21

Notes

1 "Schlitterbahn Waterpark Resort." Schlitterbahn.com. Accessed January 2007 from http://www.schlitterbahn.com/nb/. According to this site, " Schlitterbahn Waterpark Resort® received top awards in the World’s Best Waterpark, World’s Best Waterpark Landscaping and the World’s Best Waterpark Ride categories during the 2006 Golden Ticket Award ceremony at Holiday World amusement park."

2 "‘Devastating’ Texas floods kill 9." CNN.com. Accessed January 2007 from http://archives.cnn.com/2002/WEATHER/07/05/texas.flooding/index.html. I personally participated in the cleanup efforts following this flooding as a member of a group of about 15 people that spent an entire day tearing apart a house that had been picked up in this flood and dropped on it’s site. You bet we were going to check the weather.

3 One example of such duplication is that both Yahoo Weather (accessed January 2007 from http://weather.yahoo.com/) and USAToday Weather (accessed January 2007 from http://asp.usatoday.com/weather/weatherfront.aspx) use The Weather Channel [[accessed January 2007 from http://www.weather.com/) as their source.

4 "Google Toolbar Features." Google_. Accessed December 2007 from http://toolbar.google.com/button_help.htmlhelp.html.

5 "Our Search: Google Technology." Google. Accessed December 2007 from http://www.google.com/technology/index.html. According to Google, "PageRank relies on the uniquely democratic nature of the web by using its vast link structure as an indicator of an individual page’s value."

6 While some may suspect biased results due to selecting one of the weather forecast sites included in my survey for the actual temperature comparison, these values are reported from third-party measuring stations such as airports without regard to the reporting site.

7 The National Weather Service. Main URL: http://www.nws.noaa.gov/. Data gathered from: http://www.srh.noaa.gov/forecast/MapClick.php?CityName=Houston&state=TX&site=HGX. Accessed January 2007.

8 BBC Weather_. Main URL: http://www.bbc.co.uk/weather/. Data gathered from: http://www.bbc.co.uk/weather/5day_f.shtml?world=0268f.shtml?world=0268. Accessed January 2007.

9 The Weather Channel_. Main URL: http://www.weather.com/. Data gathered from: http://www.weather.com/weather/tenday/USTX0617?from=month_topnav_undeclaredundeclared. Accessed January 2007.

10 The Weather Underground. Main URL: http://www.wunderground.com/. Data gathered from: http://www.wunderground.com/cgi-bin/findweather/getForecast?query=houston%2C+tx. Accessed January 2007.

11 IntelliCast. Main URL: http://www.intellicast.com/IcastPage/LoadPage.aspx. Data gathered from: http://www.intellicast.com/IcastPage/LoadPage.aspx?seg=LocalWeather& SearchResults=True&loc=kiah&product=Forecast&prodgrp=Forecasts&prodnav=none. Accessed January 2007.

12 CNN Weather. Main URL: http://www.cnn.com/WEATHER/. Data gathered from: http://weather.cnn.com/weather/forecast.jsp?locCode=HOU. Accessed January 2007.

13 MSN Weather. Main URL: http://weather.msn.com/. Data gathered from: http://weather.msn.com/tenday.aspx?wealocations=wc:USTX0617. Accessed January 2007.

14 The Weather Network. Main URL: http://www.theweathernetwork.com/. Data gathered from: http://www.theweathernetwork.com/weather/cities/usa/Pages/USTX0617.htm#longTerm. Accessed January 2007.

15 Unisys. Main URL: http://weather.unisys.com/. Data gathered from: http://weather.unisys.com/forecast.cgi?Name=houston%2C+tx&Go.x=0&Go.y=0. Accessed January 2007.

16 AccuWeather. Main URL: http://home.accuweather.com/index.asp?partner=accuweather. Data gathered from: http://wwwa.accuweather.com/forecast-15day.asp?partner=accuweather&traveler=0&zipChg=1&zipcode=77001&metric=0. Accessed January 2007.

17 The Weather Channel_. Main URL: http://www.weather.com/. Data gathered from: http://www.weather.com/weather/pastweather/USTX0617?from=36hr_topnav_undeclaredundeclared. Accessed January 2007.

18 It may be best to picture these values as if you were at a shooting range. Someone who shoots close to the center is accurate, while someone who shoots with a tight grouping at any location is consistent.

19 Weiss, Neil A. "Elementary Statistics: Descriptive Methods in Regression and Correlation." Addison Wesley Longman, Inc. 1999. More information on the methods and calculations involved in correlation analysis.

20 Cohen, J. Statistical power analysis for the behavioral sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates. 1988.

21 I’d be willing to bet skepticism would be warranted for weather forecasts on television, also, but that’s another article for another time.

Similarly tagged OmniNerd content:

Information This article was edited after publication by the author on 10 Dec 2008. View changes.
Thread parent sort order:
Thread verbosity:

I live in Europe and most of the sites you studied are terrible in just giving current conditions in foreign cities. They will tell me it is raining when the sun is shining and that it is several degrees warmer or cooler than it actually is. I am not speaking of forecasts, but only giving current conditions.

This is a very nice study. I commend you on your dedication and analysis!

My company basicly does the same thing, ongoing, and for about 800 locations within the U.S. You can check out some basic statistics at ForecastAdvisor or look at ForecastWatch which is used by professional meteorological companies like Accuweather, CustomWeather, and The Weather Channel.

You might also like some additional analysis. Take a look at how forecast accuracy varies over time, how forecast accuracy varies over the number of days out (much like your study, and even how accuracy varies by how far away from normal the actual temperature was.

Enjoy!
Ace

0 Votes  - +
Pedantic by Anonymous

Just a quick note: some of your references are dated December 2007 – which obviously hasn’t happened yet ;-)

0 Votes  - +
Save some time by Anonymous

Or if you wanted to not waste your life doing statistics, you could just look out outside.

0 Votes  - +
don't lie by Anonymous

This thing starts out with an obvious lie. No one who analyzes weather data in their free time is married. No one!

0 Votes  - +
Rankings Backwards by Anonymous

It appears as if the ranking description is backward (when comparing to the figures). You either need to say that "lower is better" for the rankings (Tables 4 & 5) or you need to invert your ranking system (assigning a high rank, i.e. 10, to the best, and a low rank to the worst).

Otherwise, very interesting story.

Dear Brandon,

You’ve done a great job is showing some of the limitations of the current batch of weather forecasts. I think the next step for you, or some other enterprising soul, is to improve the forecasts! Using purely statistical techniques, it seems like a feasible and straight-forward project. Here are some promising leads for this:

1) Do weather forecasts weigh current temperature too much or too little? Let’s say on Monday there is a high of 30 degrees and that the Monday forecast says that Thursday’s high temperature will be 40. Is Thursday’s temperature more likely to be under or over 40? If there is a systematic bias either way (across many cities and days), then this is something that you could take into account to improve the Monday forecast. For example, it may be that 3-day temperatures are typically closer to the current temperature than forecasts predict. If so, then this could be incorporated by factoring in a weighted variable for "today’s temperature" to improve forecasts.

2. Do forecast stick too close to original anchor temperatures, or overinterpret new information? Continuing with the above example, on Tuesday, the new forecast for Thursday may be 32 degrees. If you have enough data, it would be easy to get a good estimate on whether this new forecast for Tuesday is too closely tied to the original Monday forecast, or not (in the example, it seems to be). If it turns out that Day X+1’s prediction for Day X+2 tends to fall between Day X’s prediction for Day X+2 and Day X+2’s actual temperature, then the difference between Day X+1’s prediction and Day X’s prediction can be used as a positive variable in a regression for prediction Day X+2’s temperature. Concretely put, because there is a 2 degree increase in the prediction for Thursday’s temperature going from Monday to Tuesday, this could be evidence that the actual temperature will be even higher than this. Of course, it could work out the opposite at this, but either way, it would be another predictive factor.

3) Similarly, do weather forecasts weight historic temperatures too much or too little? Another predictive variable is simply the historic temperature over many years for a city for the day in question. When this variable is added to a regression predicting (predicted temperature for Day X – actual temperature for Day X), does it receive a positive or negative weight? Concretely, if the prediction for January 10 Cleveland for a given year is 20 degrees, the actual temperature was 15 degrees, and the historic temperature is 10 degrees for this day, then the historic temperature would receive a positive weight for prediction when added to the forecast’s prediction.

There are other possibilities along these lines, but I’ll bet that some of these factors could easily improve weather forecasts, provided that you have enough forecast data to get reliable estimates for them. People tend not to go back and compare the actual temperatures to the forecasted temperatures, but this is just the sort of data needed for improving forecasts!

Best,
Robert Goldstone

Brandon,

I enjoyed your study. A while ago, and for similar reasons, I did a smaller study using only The Weather Channel’s 10-day temperature forecast. I might suggest conducting one additional analysis with your data: Use regression analysis to see whether the forecast temperatures correlate with the actual temperatures after controlling for the average high/low for the day.

The reason for this is that the average high and low are the best estimates of any given day’s temperatures in the absence of any additional useful information. I found that, ten days in advance, the Weather channel’s data did not add any useful info (i.e. you’d be better off just assuming the average high/low for the day). But from nine days in advance on, the forecast, while not always accurate, did provide some useful guidance. I’d be interested to see how the different services stack up in this regard.

Thanks for the interesting analysis. -Nick

I’de love to see the same analysis applied to forecasting the probability and quantity of percipitation.

Any takers?

I received the following inquiry via email:

How certain were you of the accuracy of the "actual" value you took from weather.com? It would be interesting to compare the actual values reported by the various sites to see how much they disagreed. This might be less of an issue in Houston than in coastal areas where the location of the measurement can make a difference. I have noticed substantial differences where I live in Baltimore. -Michael

When I first looked into gathering the actual temperatures, weather.com had two values – one for each major airport in Houston. By the time I started collecting official data points, however, there was just one and the source was omitted. I wonder what means they know use to obtain a single value. Could it be they are averaging the two together? or maybe found a new source?

In any case, I think you have the premise to write an article of your own. Let me know if my data or spreadsheet setup can be of any help and I’ll email it to you.

I received the following comments via email:

>I would like to mention that forecast meteorologists tend to take forecast verification very seriously, and that there’s a body of knowledge about verification and the statistics. Here’s a google search for your amusement, should you like some further reading.
>Your NWS office should have a SOO (Science Operations Officer), and he/she should be able provide details on how they do their verifications. -James

I found a very interesting article in the results of the Internet search: Forecast Verification – Methods & FAQ. It describes the method I used, as well as many others – I think you’ve helped me find a statistical basis for many, many potential follow-up articles. :)

0 Votes  - +
Two questions by Brandon

I received the following via email:

> Congratulations on an outstanding job of analyzing weather webpages — really enjoyed the time and effort you spent to provide great (useful) numbers.
>Your original question was prompted by an unpredicted storm. While I understand that monitoring high/low temperature predictions has value, I agree that a slightly warmer or slightly cooler day at Schlitterbahn is less significant than a hail and lightning storm (whose impact may be fatal!)
>I also loved the first-person description of the storm — the split tree, frequent lightning flashes, and hail "bombs"; and I appreciate your not bringing in second-hand accounts of the details of the effects of the storm on others. You kept it wonderfully personal.
>One piece of detail was slightly vague, which is quite understandable for an event almost 3 years old, but I wonder if you recall how far in advance you checked the weather? I’m only curious because your detailed analysis demonstrates that longer-term forecasts have lower reliability.
>In any case, my questions would be: 1) Would you considering comparing the accuracy of precipitation forecasts? 2) For storm predictions, is there a way of measuring the significance of the difference between the predicted and actual severity?
>I’m thinking the new Yahoo Pipes may allow some type of automation of this type of data gathering, so you could automate the analysis (less effort).
>-Robert

If I recall correctly, we checked the day before.

As for your questions:

  • Yes, I’m considering writing a follow-up article on the accuracy of precipitation forecasts.
  • The best way I can think of at the moment would be to use the predicted chance of rain vs the actual amount of precipitation (in inches). As far as I know, those are the only two data points available. (Does anyone know otherwise?)

Lastly, I hadn’t yet heard of Yahoo Pipes. I tinkered with it a little, and it looks like it may be very useful – if not in this application, at least in a couple of others (where I track the number of results to certain search engine queries, for example).

A very interesting article, but while I can’t claim to understand the rationale of your conclusions, I do use various weather forecasts for farming predictions, where chances of rain are critical. I think it unreasonable to expect forecasts to be accurate in terms of temperature, while one hopes that the variation in actual and predicted numbers to be a little as possible. But where the forecasts are extremely useful is predicting precipitation, perhaps not in how much rain in centimetres or inches, but whether it will rain or not and a lot of people would share my view. I find a daily check of 3 forecasters provides an aprox. 80% accuracy over 3-4 days. I don’t expect better odds. Also your storm experience is something I have had to cope with several times in summer. Local thunderstorms are unpredictable and can affect one place and pass by another only kilometres away of course with sometimes serious consequences. And they are virtually never predicted by meteorologists.

0 Votes  - +
Waste of time by Brandon

Another comment via email:

> Wow, very impressive website. Whoever made that wasted WAY too much time. I think everyone already knows that a weather forecast is a FORECAST.
> Forecast – to calculate or predict (some future event or condition) usually as a result of study and analysis of available pertinent data.
> The fact that weather forecasts are inaccurate are because of the pertinent data. Meteorology is an inexact science, so there is no true way to accurately predict the weather. Until we understand weather patterns completely, there will always be inaccuracies to complain about.
> I do appreciate you noting which websites are more accurate at describing temperatures, although the reason I read most of the article was because of the story about Schlitterbaun at the beginning (which was not a problem with temperature, but precipitation).
> Well I just wanted to give some feedback. Thanks for wasting so much of your time. -Paul

The point of the article wasn’t to determine if forecasts were exact, but to analyze which, if any, of the sites were the most accurate and to get a grip on a typical +/- implicit in forecasts given at various predictive intervals.

Meteorology is undoubtedly complicated, and no one expects a forecast to be spot on all of the time, but it is nevertheless useful to know where to get the most accurate forecast available and just how that forecast should be interpreted. I don’t think figuring that out is a waste of time.

Brandon -

The site you actually wanted to check to see if you’d get rained on at Schlitterbahn was the NOAA’s Storm Prediction Center, or SPC. It’s website is http://spc.noaa.gov … check the Convective Outlooks to see where they think that convection (i.e. thunderstorms) will hit.

-Karl Katzke, College Station TX

0 Votes  - +
Crowd Wisdom? by Anonymous

Thanks—-great post. Out of curiosity, what happens if you add another forecast that is equal to the average of the other forecasts? Does the average do particularly well or badly compared to individual forecasts?

Emailed inquiry:

Last fall I embarked upon a slightly similar quest as part of a contest. I was initially drawn to http://forecastadvisor.com/, which records accuracy of internet weather for locations all through out the country. It proved insufficient for my needs, which left me to gather several weeks worth of forecasts on my own. The most interesting thing I noticed is that some forecasts in some locations seemed to have a steady-state error, causing the high temperature forecasts to be consistently high or low. I was wondering if you ran across anything similar? Cheers, Kerry

I actually didn’t test for this, as I used the absolute value of the difference between the forecasted and actual temperature to assess accuracy. To investigate, I copied my data spreadsheet and removed the absolute value from the equations. This yielded some interesting results.

In order to display the results, I will dubbed the new averages with the following symbols:

  • -1 to +1 : no trend (0)
  • /-1.01 to +/-3 : slightly high/low (/-)
  • /-3.01 to +/-5 : high/low (+/—)
  • >/< /-5.01 : very high/low (!+/-!-)
    The different Internet weather forecast sites broke down as follows according to the above, starting with 0 days previous and moving back:
    High Temperatures
  • The National Weather Service: 0, 0, 0, 0, 0, 0, 0
  • BBC Weather: , -, -, 0, +
  • The Weather Channel: 0, 0, 0, 0, 0, 0, +, +, +, 0
  • The Weather Underground: -, -, -, -, -
  • IntelliCast: 0, 0, 0, 0, 0, +, +, +, +, 0
  • CNN Weather: 0, 0, 0, 0, +
  • MSN Weather: 0, , 0, 0, 0, 0, 0, -!, !, !
  • The Weather Network: -, -, 0, -, 0
  • Unisys: -, -, 0, 0, 0, 0, -
  • AccuWeather: 0, 0, 0, 0, 0, 0, 0, 0, , -, -, 0, -, -, -

Low Temperatures

  • The National Weather Service: -, 0, 0, -, -, -, X
  • BBC Weather: 0, , +, +, +
  • The Weather Channel: 0, 0, 0, 0, 0, 0, 0, 0, -, 0
  • The Weather Underground: -, -, -, -, -
  • IntelliCast: 0, 0, 0, 0, 0, 0, 0, -, -, 0
  • CNN Weather: 0, +, +, 0, +
  • MSN Weather: !, -, -, -, -, -, -, !, !, !
  • The Weather Network: , -, -, -, -
  • Unisys: X, -, -, -, -, -, -
  • AccuWeather: , 0, -, -, -, -, -, -!, !, !, !, !, !, !, !
    As you can see, many sites show definite biases for either a high or low forecast. The Weather Underground and The Weather Network, for example, look to always be slightly low. MSN Weather usually faults low for the low temperatures in short to mid range, and very low for both high and low temperatures in the long range. BBC Weather habitually forecasts too high for the overnight lows, CNN Weather does sometimes, but the rest don’t have even a single fault on the high side. (I’ll stop the list here, although there are more apparent trends.)
0 Votes  - +
Apple Weather Widget by VnutZ

Curious – does anybody know where Apple’s Weather Widget draws its data from? I’ve noticed it tends to differ significantly from the reports on weather.com and tends to only be "accurate" in the morning.

I hate when the forecasters say "low temperature tonight will be…. " or "the overnight low will be…. " !! And the Weather.com forecasts always list the "Low for Tonight".

It’s NOT the low for tonght ! It’s the low for TOMORROW ! It normally occurs just before sunrise, which is TOMORROW morning !!!

So listen up, you weather forecasters !! Quit lying about "tonight’s low", and please please PLEASE beging referring to it as TOMORROW’S LOW !!!!

THE PUBLIC WILL LEARN !!!

Coincidentally, I live in New Braunfels and believe you are referring to Memorial weekend a few years back. Talk about a freak storm! I was outside of the tube chute on the Comal when the hail started (out of nowhere). Stayed in the river with my beer of course…and just covered up with a trash can lid.

The missing variable is of course, “if you don’t like the weather in Texas, wait a minute…”

0 Votes  - +
absolute value by Anonymous

What exactly is an absolute value? What did it represent or mean in your paragraph titled “Calculations on Forecast Accuracy and Consistency”?

0 Votes  - +
Wow by Anonymous

I know you did a lot of work, and it’s all very impressive. And there’s probably a clear conclusion buried in there somewhere. I just didn’t have the patience to wade through all the fascinating muck to find it. Thanks, though. I’m sure this is very interesting for someone.

0 Votes  - +
weatherist by Anonymous

I like www.weatherist.com. They study the accuracy of other forecasts then create their own based off of that data. Pretty interesting.

Share & Socialize

What is OmniNerd?

Omninerd_icon Welcome! OmniNerd's content is generated by nerds like you. Learn more.

Voting Booth

The most important factor in buying my next car is?

7 votes, 1 comment