"What is a Good Weather Forecast?"
in the eyes of a forecaster

Brad Colman
Science and Operations Officer
National Weather Service
Seattle, WA
colman@seawfo.noaa.gov

Background

This question was presented to me as my discussion topic by the Workshop organizers. It initially seemed like a simple question with a simple answer, however, I quickly realized it is a multifaceted question with rather complicated answers. The intent of including this topic in the Workshop was to bring the perspective of the forecaster into the discussions of public and private sector decision making.

The core mission of the National Weather Service is the protection of life and property through its Watch and Warning program. This note will concentrate on public forecasts since these are integral to this program. The public is also one of the more difficult user groups to work with given their diverse character and needs. Forecasts for the public contrast sharply with forecasts prepared for specialized groups including agriculture, aviation, and commerce. In addition, many of these forecast responsibilities are being transferred to the private sector through the public-private partnership.

The Modernization of the National Weather Service is nearing completion at a price tag of over $4 billion. It has been an overwhelming technological success. In addition, dramatic improvements in several forecast categories have been documented, e.g., lead time for tornado warnings. On the other hand, it may be more debatable as to whether the promise of much improved public forecasts and warnings has been met. So, in the forecasters' opinion, what makes a good public forecast?

Methodology

A short survey was conducted as a means to gain insight into how a small group of forecasters viewed their public forecasts and what criteria they might use to measure the "goodness" of a forecast. All forecasters from the Seattle forecast office were surveyed. The group of approximately 20 includes both interns that are relatively inexperienced and senior forecasters with over 20 years of experience. Participants were asked not to discuss the questions with each other prior to completing the survey. The survey was very simple and included three questions:

1. What makes a good forecast? Please explain.

2. Give an example of a good forecast. Please explain.

3. Give an example of a bad forecast. Please explain.

Results

A great deal of work has already been done to evaluate or verify forecasts, both within the NWS and the academic community. Typically public forecasts, are verified against another forecast (e.g., MOS or climatology) to demonstrate some improvement over the other forecast, or skill. These are typically single point forecasts. It therefore seems logical that some of the forecasters would feel a good forecast was one that showed skill over a competing scheme. A definition might include some specific ranges of tolerance for various parameters. For example, a 24 hour maximum temperature forecast that is within two degrees of what occurs might be considered a good forecast.

Of the group of forecasters surveyed, only four focused their answer on measures of skill. Since these are region specific and have been documented objectively in many rigorous objective studies, I will not present any details of these responses. Nonetheless, there is clearly some weight given to the accuracy of the forecast in measuring its goodness.

The remainder of the forecasters placed their measure of a "good" forecast in the accuracy of its perception by the public or its societal impact. Specific remarks given by the forecasters included:

Although these are not dramatic or earth shattering they do show that the forecasters are concerned about public perception and the action the forecast instills.

When the examples of good forecasts and bad forecasts were compared, there were a few surprises. For example, one forecaster felt that good forecasts were "very short forecasts." The argument was that detail is lost to most people and by keeping the forecast simple you minimize possible confusion. The forecaster's examples were:

GOOD FORECAST: "Scattered showers and sunbreaks."

BAD FORECAST: "Rain likely, turning to scattered showers, mainly near the foothills this afternoon with isolated thunderstorms. Possible small hail and gusty winds with the thunderstorms."

In the good forecast case, the forecaster suggested most people would experience benign weather so the forecast would be accurate short and sweet for them. It would only be the few individuals who experienced a thunderstorm who would think it was a bad forecast, and to include the additional material for those few was not worth the risk of confusion.

This set of forecasts contrasts sharply with another forecaster's examples. These were:

GOOD FORECAST: "Rain ending and becoming partly sunny during the afternoon. Except mostly cloudy with occasional showers continuing north of Seattle through early afternoon."

BAD FORECAST: "Mostly cloudy with scattered showers."

It seems this contrast in forecaster opinion might be linked to the Modernization, which promised highly detailed and accurate mesoscale forecasts. The technology is in place to make those forecasts but the means to communicate that information is still under development. At this time it seems the text product is a synoptic-scale forecast system and does not do as well at communicating mesoscale information to the general public.

Additional discussion from the forecasters included external factors determining whether a forecast was considered a good forecast or a bad forecast. Two forecasters mentioned the role of credibility. If there have been a few bad forecasts then credibility goes down -- less and less attention is paid to the forecasts by the user. The forecasters also mentioned other external factors that play a role, including: several sources of forecasts resulting in confusion, paraphrasing by radio disk jockeys and/or the use of old forecasts, access to weather information, and a lack of attention by the news media and emergency managers.

I have attached statements from two windstorms that struck western Washington. The statement for each storm is the one that told of the issuance of a high wind warning. Although it is difficult to compare forecasts for an entire event (since literally dozens of products are issued) these seem as good as any. What is rather dramatic about these two storms is that even though both forecasts verified well and had about the same lead time, user perception and forecast impact were dramatically different. For case 1, the windstorm was perceived as a "surprise." Even an editorial in the Seattle Times chastised the forecasters for not getting the warning out. (The Times wrote a follow-on editorial after the time line of watches and warnings was brought to their attention acknowledging the forecasts had been good). For case 2, public reaction was dramatic. Schools were dismissed early, businesses closed early, and grocery stores were emptied. Why was one of these forecasts a good forecast and one a bad forecast?

Case 1 happened to be inauguration day and there was a lot of news. The forecast office had also issued several false-alarm forecasts during the previous month. It appears that these facts conspired against the forecast and the word never got out. For Case 2 it is a little difficult to pinpoint what made the people react. There was something that evidently was very compelling to the public and to the decision makers. One possibility is that in several statements issued prior to the storm references were made to the storm -- an historical event for the Pacific Northwest. Perhaps this personalized the storm for those who had been in Seattle for that storm and they chose to react.

Discussion

From this short analysis it is clear that several factors are viewed by forecasters as important measures of a forecast's goodness. Among these factors are accuracy and user perception, with user perception getting a greater emphasis. This is encouraging since the general public can't rely on cost benefit analyses. Rather the general public reacts only according to their individual perception of risk. To be effective the event must be personalized by the forecast, emergency managers, and news media. The forecast needs to be compelling enough to trigger the desired action.

This gets to a final point. If we assume a forecast is only good if it gets the general public to react according to the impending risk, then we need to protect against those things that might interfere with this process. There are probably many and these need to be explored. However, one that was discussed in this survey is an increasing trend for the issuance of warnings by more than one group. Contradictory statements and forecasts often result in a decrease in user confidence and thus little action. It is critical that a single voice have the sole responsibility to communicate watches and warnings to the general public. At this point it is the National Weather Service and this role needs to be protected.

Case 1

High wind warning
National Weather Service Seattle WA
1000 PM PST Tue Jan 19 1993

...A high wind warning is in effect for the Washington coast for late tonight and Wednesday...
...A high wind warning has been issued for the interior of western Washington for Wednesday...

South winds will increase along the Washington coast tonight with sustained winds to 40 MPH and gusts to 70 MPH by late tonight. The strong winds will continue through Wednesday morning...then shift to the southwest and decrease to 20 to 30 MPH by afternoon.

In the western Washington interior...south winds will begin increasing Wednesday morning. In the southwest interior and the Puget sound area...including Seattle...Tacoma and Everett...Sustained winds will reach 30 to 40 MPH with gusts to 50 MPH by late morning... then decrease by mid afternoon. South winds over the northwest interior of Washington will be slightly stronger...with sustained winds of 40 MPH and gusts to 70 MPH. The strongest winds will likely be felt on higher hills...headlands...and near terrain constricted areas.

If you live or are traveling in any of these areas...prepare for strong...possibly damaging winds. You may have downed tree limbs and power lines...flying debris...and power outages. Damage to structures is also a possibility.

A rapidly deepening storm system is developing several hundred miles offshore. It will track northeastward across the northern part of Washington late Wednesday morning.

KAM/ART

Case 2

High wind warning/watch
National Weather Service Seattle WA
330 AM PST Tue dec 12 1995

...A high wind warning is now in effect for all of western Washington today through this evening...
...A high wind watch is now in effect for all of eastern Washington for late this afternoon and evening...

The National Weather Service has issued a high wind warning for western Washington today with strong south winds of 50 to 60 MPH and gusts to 90 MPH developing along the coast by late morning. In the interior of western Washington strong south winds of 50 MPH and gust as high as 80 MPH are expected during the late afternoon and evening.

In eastern Washington a high wind watch is in effect for late this afternoon and evening where south winds of 40 MPH and gusts to 55 MPH are possible.

Satellite pictures and ocean body reports indicate an intense low pressure center 300 miles off the south Oregon coast. The current track of the low moves it along the Washington coast early this afternoon and to just off the Tatoosh Island by late afternoon.

Because this low pressure system has a very low central pressure and is moving very close to the Washington coast this storm has the potential for being one of the strongest wind storms in recent history.

People in the warning and watch areas should be prepared for strong damaging winds with this storm...which could result in widespread power outages due to downed trees and power lines.

Stay tuned to NOAA weather radio or your favorite radio and television stations for more information about this storm.

END/ART


Societal Aspects of Weather

Workshop's Main Page

Table of Contents