Internet Weather Forecast Accuracy

I live in Europe and most of the sites you studied are terrible in just giving current conditions in foreign cities. They will tell me it is raining when the sun is shining and that it is several degrees warmer or cooler than it actually is. I am not speaking of forecasts, but only giving current conditions.

This is a very nice study. I commend you on your dedication and analysis!

My company basicly does the same thing, ongoing, and for about 800 locations within the U.S. You can check out some basic statistics at ForecastAdvisor or look at ForecastWatch which is used by professional meteorological companies like Accuweather, CustomWeather, and The Weather Channel.

You might also like some additional analysis. Take a look at how forecast accuracy varies over time, how forecast accuracy varies over the number of days out (much like your study, and even how accuracy varies by how far away from normal the actual temperature was.

Enjoy!

Ace

Just a quick note: some of your references are dated December 2007 - which obviously hasn't happened yet ;-)

Or if you wanted to not waste your life doing statistics, you could just look out outside.

This thing starts out with an obvious lie. No one who analyzes weather data in their free time is married. No one!

It appears as if the ranking description is backward (when comparing to the figures). You either need to say that "lower is better" for the rankings (Tables 4 & 5) or you need to invert your ranking system (assigning a high rank, i.e. 10, to the best, and a low rank to the worst).

Otherwise, very interesting story.

Dear Brandon,

You've done a great job is showing some of the limitations of the current batch of weather forecasts. I think the next step for you, or some other enterprising soul, is to improve the forecasts! Using purely statistical techniques, it seems like a feasible and straight-forward project. Here are some promising leads for this:

1) Do weather forecasts weigh current temperature too much or too little? Let's say on Monday there is a high of 30 degrees and that the Monday forecast says that Thursday's high temperature will be 40. Is Thursday's temperature more likely to be under or over 40? If there is a systematic bias either way (across many cities and days), then this is something that you could take into account to improve the Monday forecast. For example, it may be that 3-day temperatures are typically closer to the current temperature than forecasts predict. If so, then this could be incorporated by factoring in a weighted variable for "today's temperature" to improve forecasts.

2. Do forecast stick too close to original anchor temperatures, or overinterpret new information? Continuing with the above example, on Tuesday, the new forecast for Thursday may be 32 degrees. If you have enough data, it would be easy to get a good estimate on whether this new forecast for Tuesday is too closely tied to the original Monday forecast, or not (in the example, it seems to be). If it turns out that Day X+1's prediction for Day X+2 tends to fall between Day X's prediction for Day X+2 and Day X+2's actual temperature, then the difference between Day X+1's prediction and Day X's prediction can be used as a positive variable in a regression for prediction Day X+2's temperature. Concretely put, because there is a 2 degree increase in the prediction for Thursday's temperature going from Monday to Tuesday, this could be evidence that the actual temperature will be even higher than this. Of course, it could work out the opposite at this, but either way, it would be another predictive factor.

3) Similarly, do weather forecasts weight historic temperatures too much or too little? Another predictive variable is simply the historic temperature over many years for a city for the day in question. When this variable is added to a regression predicting (predicted temperature for Day X - actual temperature for Day X), does it receive a positive or negative weight? Concretely, if the prediction for January 10 Cleveland for a given year is 20 degrees, the actual temperature was 15 degrees, and the historic temperature is 10 degrees for this day, then the historic temperature would receive a positive weight for prediction when added to the forecast's prediction.

There are other possibilities along these lines, but I'll bet that some of these factors could easily improve weather forecasts, provided that you have enough forecast data to get reliable estimates for them. People tend not to go back and compare the actual temperatures to the forecasted temperatures, but this is just the sort of data needed for improving forecasts!

Best,

Robert Goldstone

Brandon,

I enjoyed your study. A while ago, and for similar reasons, I did a smaller study using only The Weather Channel's 10-day temperature forecast. I might suggest conducting one additional analysis with your data: Use regression analysis to see whether the forecast temperatures correlate with the actual temperatures after controlling for the average high/low for the day.

The reason for this is that the average high and low are the best estimates of any given day's temperatures in the absence of any additional useful information. I found that, ten days in advance, the Weather channel's data did not add any useful info (i.e. you'd be better off just assuming the average high/low for the day). But from nine days in advance on, the forecast, while not always accurate, did provide some useful guidance. I'd be interested to see how the different services stack up in this regard.

Thanks for the interesting analysis. -Nick

I'de love to see the same analysis applied to forecasting the probability and quantity of percipitation.

Any takers?

I received the following inquiry via email:

When I first looked into gathering the actual temperatures, weather.com had two values - one for each major airport in Houston. By the time I started collecting official data points, however, there was just one and the source was omitted. I wonder what means they know use to obtain a single value. Could it be they are averaging the two together? or maybe found a new source?

In any case, I think you have the premise to write an article of your own. Let me know if my data or spreadsheet setup can be of any help and I'll email it to you.

I received the following comments via email:

I found a very interesting article in the results of the Internet search: Forecast Verification - Methods & FAQ. It describes the method I used, as well as many others - I think you've helped me find a statistical basis for many, many potential follow-up articles. :)

I received the following via email:

If I recall correctly, we checked the day before.

As for your questions:

1. Yes, I'm considering writing a follow-up article on the accuracy of precipitation forecasts.
2. The best way I can think of at the moment would be to use the predicted chance of rain vs the actual amount of precipitation (in inches). As far as I know, those are the only two data points available. (Does anyone know otherwise?)

Lastly, I hadn't yet heard of Yahoo Pipes. I tinkered with it a little, and it looks like it may be very useful - if not in this application, at least in a couple of others (where I track the number of results to certain search engine queries, for example).

A very interesting article, but while I can't claim to understand the rationale of your conclusions, I do use various weather forecasts for farming predictions, where chances of rain are critical. I think it unreasonable to expect forecasts to be accurate in terms of temperature, while one hopes that the variation in actual and predicted numbers to be a little as possible. But where the forecasts are extremely useful is predicting precipitation, perhaps not in how much rain in centimetres or inches, but whether it will rain or not and a lot of people would share my view. I find a daily check of 3 forecasters provides an aprox. 80% accuracy over 3-4 days. I don't expect better odds. Also your storm experience is something I have had to cope with several times in summer. Local thunderstorms are unpredictable and can affect one place and pass by another only kilometres away of course with sometimes serious consequences. And they are virtually never predicted by meteorologists.

Another comment via email:

The point of the article wasn't to determine if forecasts were exact, but to analyze which, if any, of the sites were the most accurate and to get a grip on a typical +/- implicit in forecasts given at various predictive intervals.

Meteorology is undoubtedly complicated, and no one expects a forecast to be spot on all of the time, but it is nevertheless useful to know where to get the most accurate forecast available and just how that forecast should be interpreted. I don't think figuring that out is a waste of time.

Brandon -

The site you *actually* wanted to check to see if you'd get rained on at Schlitterbahn was the NOAA's Storm Prediction Center, or SPC. It's website is http://spc.noaa.gov ... check the Convective Outlooks to see where they think that convection (i.e. thunderstorms) will hit.

-Karl Katzke, College Station TX

Thanks---great post. Out of curiosity, what happens if you add another forecast that is equal to the average of the other forecasts? Does the average do particularly well or badly compared to individual forecasts?

Emailed inquiry:

I actually didn't test for this, as I used the absolute value of the difference between the forecasted and actual temperature to assess accuracy. To investigate, I copied my data spreadsheet and removed the absolute value from the equations. This yielded some interesting results.

In order to display the results, I will dubbed the new averages with the following symbols:

• -1 to +1 : no trend (0)
• +/-1.01 to +/-3 : slightly high/low (+/-)
• +/-3.01 to +/-5 : high/low (++/--)
• >/< +/-5.01 : very high/low (+!+/-!-)

The different Internet weather forecast sites broke down as follows according to the above, starting with 0 days previous and moving back:

High Temperatures

• The National Weather Service: 0, 0, 0, 0, 0, 0, 0
• BBC Weather: -, --, -, 0, +
• The Weather Channel: 0, 0, 0, 0, 0, 0, +, +, +, 0
• The Weather Underground: -, -, -, -, -
• IntelliCast: 0, 0, 0, 0, 0, +, +, +, +, 0
• CNN Weather: 0, 0, 0, 0, +
• MSN Weather: 0, -, 0, 0, 0, 0, 0, -!-, -!-, -!-
• The Weather Network: -, -, 0, -, 0
• Unisys: -, -, 0, 0, 0, 0, -
• AccuWeather: 0, 0, 0, 0, 0, 0, 0, 0, -, -, -, 0, -, -, --

Low Temperatures

• The National Weather Service: -, 0, 0, -, -, -, X
• BBC Weather: 0, +, +, +, ++
• The Weather Channel: 0, 0, 0, 0, 0, 0, 0, 0, -, 0
• The Weather Underground: -, -, -, -, -
• IntelliCast: 0, 0, 0, 0, 0, 0, 0, -, -, 0
• CNN Weather: 0, +, +, 0, +
• MSN Weather: -!-, --, --, --, --, --, --, -!-, -!-, -!-
• The Weather Network: -, -, -, --, -
• Unisys: X, -, -, -, -, -, -
• AccuWeather: -, 0, -, -, -, -, -, -!-, -!-, -!-, -!-, -!-, -!-, -!-, -!-

As you can see, many sites show definite biases for either a high or low forecast. The Weather Underground and The Weather Network, for example, look to always be slightly low. MSN Weather usually faults low for the low temperatures in short to mid range, and very low for both high and low temperatures in the long range. BBC Weather habitually forecasts too high for the overnight lows, CNN Weather does sometimes, but the rest don't have even a single fault on the high side. (I'll stop the list here, although there are more apparent trends.)

Curious - does anybody know where Apple's Weather Widget draws its data from? I've noticed it tends to differ significantly from the reports on weather.com and tends to only be "accurate" in the morning.

I hate when the forecasters say "low temperature tonight will be.... " or "the overnight low will be.... " !! And the Weather.com forecasts always list the "Low for Tonight".

It's NOT the low for tonght ! It's the low for TOMORROW ! It normally occurs just before sunrise, which is TOMORROW morning !!!

So listen up, you weather forecasters !! Quit lying about "tonight's low", and please please PLEASE beging referring to it as TOMORROW'S LOW !!!!

THE PUBLIC WILL LEARN !!!