Decision Analysis Applied to Weather-Related Warning Systems

Gary McClelland
Dept. of Psychology, CB 345
University of Colorado
Boulder, CO 80309-0345; 303-492-8617
Albany, NY 12222
gary.mcclelland@colorado.edu

Methodology

Decision analysis is a well-developed tool for aiding decision making in uncertain environments. The familiar 2x2 decision table summarizes the decisions and outcomes for warnings about hazardous events. The decision maker has two choices - warn or not warn - and the hazardous event either occurs or not. This leads to four possible outcomes -

	Event Occurs	No Event Occurs
Warning Issued	Correct Warning	False Alarm
No Warning	Miss	Correctly Quiet

two correct decisions (correct warning and correct quiet) and two errors (false alarm and miss). If the goal is to minimize the costs due to errors, then the optimal decision is the one that minimizes the probability-weighted average of the costs of the two kinds of errors. Key technical difficulties in implementing the method for warning systems is modeling the event probabilities conditional on information such as forecasts and measuring the costs (which may involve difficult to measure and even difficult to discuss costs such as the value of a life).

Illustrative Case Study

Telemetered system of rain gauges for early warning of possible dam failure.
Original Problem: There are several hundred earthen dams in the western United States. An earthen dam, unlike a concrete dam, is likely to fail catastrophically if it is overtopped. Many people live downstream from many of these dams, but the dams themselves are often in remote locations which are difficult to monitor. Most of these dams are highly unlikely to be at risk in "100-year" storm events, but might be at risk for, say, a "500-year" storm event.
Engineering Solution: Place a system of rain guages above the dams, using telemetry to send data to a central site.
New Problem: After how much rain is on the ground should an evacuation warning be issued?

Decision Analysis Solution

We algebraically rearranged the standard decision analysis equation to answer the question: At what probability of dam failure should an evacuation warning be issued? Using reasonable assumptions, the optimal threshold probability can be expressed in this equation, where TPF is the time prior to [dam] failure that the warning is issued. This expression effectively separates the problem into a value or policy component - the right-hand side of this equation - and an expert or scientific estimate component - determining the actual values of P as a function of environmental conditions and rain gauge information. If the scientifically estimated value P exceeds the threshold value calculated above, then a warning should be issued.

On the right-hand side of the equation for the threshold probability, the difficult to measure costs were, of course, the value of life and the expected number of lives saved as a function of warning time prior to failure. The latter turned out to be reasonably simple to estimate by building a statistical model of deaths resulting from actual dam failures throughout the United States and Europe (see DeKay & McClelland, 1993). Valuing lives is more controversial but the results were not especially sensitive to differences in the standard values commonly used in such analyses. Also, for decision makers we reversed the calculations to show them the value of life implied by various threshold probabilities. For example, for one of our study dams, waiting until one is 50% certain the dam will fail before issuing a warning implies a value of life of ranging only from about $300 to $3000, clearly unreasonable values.

Across several dams studied, the optimal threshold probability for evacuation ranged from .01 to .0001, depending on dam characteristics, especially how many people live various distances in canyons below the dam.

The scientific estimation problem involves predicting reservoir hydrographs with standard errors, on the basis of initial reservoir level, degree of ground saturation, and amount of rain in the basin. Developing this model was not our task, but was instead the responsibility of the agency implementing the rain gauge system. Sensitivity analysis indicated that the accuracy of this system was a much more important determinant of overall system performance than was the accuracy of the cost estimates.

Implementation Difficulties

Evacuation Authority: The agency responsible for operating the dams has no authority to order evacuations. Instead, the agency can only recommend to the local sheriff that he or she order an evacuation. Given that warnings will be very rare (even a false alarm is expected to occur no more than once every 30 years at any given dam), it will be difficult to have in a place a system for contacting the sheriff and providing credible information that an evacuation is needed.

Policy Maker Conservatism: Policy makers and engineers developing systems such as the rain gauges prefer conservative strategies for dealing with risk. For example, engineers "overbuild" so that structures can withstand several times more than the greatest anticipated stress. However, when there are two kinds of errors, as in the case of warning systems, such conservatism is impossible because conservatism with respect to one kind of error necessarily increases the risk of the other kind of error. Policy makers do not like confronting such tradeoffs.

Fear of False Alarms: Extremely rare events, such as dam failures, are inherently difficult to predict. Even the best warning system will necessarily have far more false alarms than correct warnings. Public officials often receive severe criticism for false alarms (cf., swine flu inoculation decision) and they often fear that false alarms will diminish subsequent response when the threat is real. In this case, the probability of a false alarm at a given dam is very low (probably less than one false alarm in 50 years) and the probability of a false alarm at a given dam followed soon by a real threat is even lower. So even if there is a diminishing effect caused a false alarm, it is unlikely to have negative consequences because there will be little memory of the prior false alarm when the true warning occurs. However, system-wide across many dams, there is a good chance of making a false alarm every couple of years. Thus, the policy maker's experience with false alarms may be different than someone living below a dam. If the policy maker becomes more conservative in issuing warnings in order to avoid false alarms, then residents below dams will be at greater risk.

Conclusion

Decision analysis is an effective tool for informing decisions to issue weather-related warnings and evacuations. However, its successful implementation requires an undestanding of the social and political environment in which the warning system will be used.

References

DeKay, M.L., & McClelland, G.H. (1991). Setting decision thresholds for dam failure warnings: A Practical theory-based approach. CRJP Technical Report No. 328, Center for Research on Judgment and Policy, Univ. of Colorado, Boulder, CO 80309-0344.

DeKay, M.L., & McClelland, G.H. (1993). Predicting loss of life in cases of dam failure and flash flood. Risk Analysis, 13, 193-205.

Societal Aspects of Weather
Workshop's Main Page
Table of Contents