« Risk & Uncertainty »

Catastrophe Models: Boon or Bane?

I just returned from a meeting where I had a chance to discuss the role of "catastrophe models" in insurance and reinsurance. Upon returning I thought that it might be worth revisiting an essay I wrote six years ago for a newsletter that we used to publish called the WeatherZine. Here is the essay in full:

WeatherZine, February, 1999

The 1990s have seen the rise of the catastrophe modeling industry in response to demand, primarily from the insurance industry, for quantification of risk. Decision makers seek from catastrophe models some estimate of the risk that they face due to extreme events like hurricanes or earthquakes. A typical model will incorporate information on weather (e.g., hurricane landfall and wind speed probabilities), insurance (e.g., the value of exposed property), and damage potential (e.g., engineering, building materials, construction, codes). The model uses these data to calculate things like probable maximum loss, annual expected loss, and losses due to a specific event. Insured losses are typically much smaller than total economic losses in a catastrophe. Catastrophe models have become fundamental to the existence of financial products such as catastrophe bonds and futures. Even the United States government has begun to develop its own catastrophe models to aid its Federal Emergency Management Agency response to disasters. Clearly, with so many decision makers wanting to understand risk, the rise of the catastrophe model industry should be applauded. But there is reason for hesitation: No one knows how well the models actually perform.

Evaluation of predictions of any sort can be tricky. It involves more than just comparing the prediction with what actually unfolds. For example, in the late 1800s a scientist predicted days on which tornadoes would or would not occur with 96% accuracy. This seemed like a truly remarkable feat until someone noticed that predicting "no tornadoes" every day would have given a 98.5% accuracy rate! For a prediction to show "skill" it must outperform a simple prediction based on persistence. In weather forecasting the simple prediction is climatology; in economics it is called the nave forecast; mutual fund managers use the performance of the S&P 500 as a benchmark. While some in the insurance industry have sought to evaluate models against actual events and historical losses, there exists no community-wide benchmark for evaluation, leaving most users in the dark as to how well the models actually predict catastrophe losses. The State of Florida and particular companies have invested significant effort to evaluate the models, but for the most part these evaluations are based on qualitative criteria such as the credentials of the modelers and whether or not the results "look realistic."

Historically, catastrophe losses have not been particularly amenable to the development of such a benchmark because there is such dramatic change over time in the context in which losses occur. This means that one cannot generate a simple estimate of expected losses based on what has occurred in the past, as actuaries typically do for the insurance industry. Consider that the great Miami hurricane of 1926 caused an inflation-adjusted $100 million in losses. But Miami had only about 100,000 residents at the time. Comparing the losses of 1926 with potential losses of today is like comparing apples and oranges. Even comparing Andrew's losses in 1992 with today's potential losses can mislead. Indeed, underestimates of risk based on improperly aggregating losses over time is one factor that stimulated the rise of the catastrophe model industry.

But even with the difficulties associated with placing catastrophe losses on an actuarial basis, it has been done. Traveler's Insurance Company for many years adjusted catastrophe losses for changing societal conditions as part of an in-house research capability. More recently, work by Changnon et al. (1996) and Pielke and Landsea (1998) have sought to respectively adjust crop/property insurance losses and hurricane losses for changes in society. Such adjustments, properly done for the insurance industry, could form the basis of a community (i.e., public) benchmark against which to evaluate catastrophe models. A catastrophe model would have skill if it were shown to outperform the benchmark. The degree to which the model outperforms the benchmark would determine its relative skill as compared to other models.

On the one hand, it seems logical that evaluation of catastrophe models would be in the interests of the users of the models, but it would also benefit the developers of the models. Public information on relative skill of the models would aid in marketing and pricing of their services. On the other hand, it is also important to recognize that for a subset of users of catastrophe models, the performance of the models is less important than their mere existence. Because the models exist, they allow for the quantification of risk. Because risk can be quantified, financial instruments like bonds and futures can be created and traded in the financial markets. Significant financial returns result to those companies that create and manage these financial instruments made possible by the existence of catastrophe models. And for the most part, these are not the same companies that bear the risk of a catastrophic loss. In the war against catastrophe losses, they are making the bullets, so to speak. This is perhaps one reason why there has not been a greater push to evaluate catastrophe models in a public forum.

Given experience with multi-tens-of-billions of dollars in losses in Hurricane Andrew in Miami and the Great Hanshin Earthquake in Kobi, it is only prudent to ask about the consequences of once again failing to properly calculate the risks of catastrophic losses. Catastrophe models have provided decision makers with a means to better estimate risk, but at the same time in catastrophe bonds and other instruments decision makers have created products that depend upon greater accuracy in awareness of risk. Catastrophe models are here to stay and will likely be used to develop ever more precise predictions of risk (e.g., at the zip code or even household level). Because most everyone pays taxes or has insurance, it would seem to be in the common interest to know how well the models predict by developing a public approach to the evaluation of catastrophe models - before events show us that we waited too long.

For further reading see the publications of Rade Muslin at: http://www.ffbic.com/actuary/papers/index.html; and particularly his paper on "Issues in the Regulatory Acceptance of Computer Modeling for Property Insurance Ratemaking", Journal of Insurance Regulation, Spring, 1997, pp. 342-359. (You need Adobe Acrobat Reader to open.) Posted on February 24, 2005 12:03 AM

Comments