|
Modernization: The Challenge Continues Thomas R. Stewart
Modernization: The Challenge ContinuesThomas R. Stewart The multi-billion dollar, decade-long modernization of the National Weather Service is essentially completed. Weather Service Offices now have advanced instruments and information systems in place. As a result, forecasters have access to more information and model guidance than ever before. But one component of the weather forecasting system remains unaltered by the Modernization. Indeed, it has changed little since the beginning of weather forecasting. That component is the forecaster's brain. Although the challenges offered by our complex environment and extensive education may make us smarter than our ancestors, we are still equipped with essentially the same cognitive processes they had. As experience has shown repeatedly, those processes can be incredibly powerful on one occasion and hopelessly inadequate on another. Obviously, technological modernization of the weather service is pointless unless forecasters can use its improvements to issue better forecasts and warnings. After years of studying human judgment, I have high regard for weather forecasters. We have found higher accuracy among weather forecasters predicting temperature and precipitation than any other group of experts we have studied (see additional readings below). But I also believe that there is room for improvement in making use of improved information, particularly when issuing forecasts and warnings of severe weather. Joe Golden recently wrote that lead times and probability of detection "have improved steadily since 1990. However, the [false alarm rates] have increased slightly over the past few years . . . the reasons for more NWS false alarms are not clear." But an increase in false alarms should not be a surprise. For any fixed (less than perfect) level of forecast accuracy there are two types of errors — false alarms and misses (or surprises). These two errors are inevitably linked. Efforts to improve one (for example, increasing lead time to reduce surprises) inevitably make the other worse (more false alarms). This is a well known trade-off. The only way to reduce both false alarms and surprises is to increase the accuracy of forecasts. The NWS is currently looking for ways to reduce both false alarms and surprises. Reducing false alarms begins with knowing why they occur. Here is a recipe for increasing false alarms that uses ingredients available in any NWS Forecasting Office:
Simmer; serves 274 million. The first ingredient-uncertainty-creates the inevitable tradeoff between surprises and false alarms. The third-institutional pressure to increase lead times-encourages the forecaster to issue warnings sooner and based on less certainty than he or she might otherwise do. This strategy avoids surprises, but increases the number of false alarms. Overt pressure is not needed. It is sufficient if the forecaster knows that an important organizational goal is increased lead times. The second ingredient-information-is supposed to reduce both surprises and false alarms by improving the accuracy of forecasts, but it may have the opposite effect. Here's why: Good judgment, and therefore good forecasting, require four things. First, the forecaster must pay attention to relevant information. Second, the forecaster must ignore irrelevant information. Third, the forecaster must understand the uncertainty that he or she faces, that is, the forecasts must not be influenced by over- or under-confidence. Fourth, the forecaster must be consistent. Given identical information, the forecaster should produce identical forecasts. These are the keys to good forecasts. What happens when you give a forecaster more information? Inevitably, much of the information will be irrelevant in any given situation. The relevant information is then mired in irrelevant information, so there is a greater chance that the irrelevant will distract attention from the relevant. Furthermore, research suggests that people become overconfident when they get more information. Finally, judgments tend to become less consistent when information increases. Conclusion: More information can actually reduce forecast accuracy. This is important: More information is not necessarily better. In practical terms, less accurate forecasts mean we must pay a greater cost in false alarms to gain an increase in lead time. The modernization of the National Weather Service is a triumph of ingenuity, science, technology, and vision. It is time now to build on that achievement by addressing the human element in forecasting. Further improvement in forecast accuracy (resulting in the desired decrease in both false alarms and surprises) is possible by addressing the problems that modernization can create for the human component of the system. That requires a two-pronged effort. First, an obvious way to improve human forecasting is through training and selection of forecasters. Some forecasters are better than others. Presumably, some people are, by nature or experience, better able to cope with the information burden presented in the modern forecast office. If we better understood the reasons for individual differences in forecasting skill, we could select and train forecasters for more accurate forecasts. But training and selection are not enough. In general, forecasters are already well trained and highly competent. It's a common mistake to look inside the forecaster's head for ways to improve forecasts. Part of the problem, probably the largest part (and the part that can be most effectively addressed) is outside the forecaster. It's the surroundings, the context, the circumstances under which the forecast is made. How much information must the forecaster absorb in the time allotted? How is that information arranged? How reliable is the information? Does the format of the information display induce intuitive or analytic cognition (people are capable of both; there are advantages and disadvantages to each; and certain characteristics of the situation determines which will be used)? The second prong requires systematic study of the forecasting environment. The four keys to good forecasting (Pay attention to relevant information — Ignore irrelevant information — Assess uncertainty accurately — Be consistent) are simple to understand but difficult to implement. Information displays and forecasting procedures can help or hurt. The only way to make sure they help is by a program of empirical study. The NWS deserves congratulations on the completion of its technological modernization effort. With congratulations comes a challenge to take the next step and to invest in the application of what we in the decision sciences know about judgment and decision making to address the most important component of the weather forecasting system: the human forecaster. — Thomas R. Stewart
Comments? thunder@ucar.edu
|
