
Risk Managment Solutions (RMS) is a leading company that provides catastrophe models which are used to asses risk in the insurance and reinsurance industries. RMS plays a very important role in enabling companies and regulatiors to quantify risks and uncertainties associated with extreme events such as floods, hurricanes and terrorism.
I have in the past been somewhat critical of RMS for issuing short-term hurricane predictions (e.g., see here and here and here). I don’t believe that the science has demonstrated that such predictions can be made with any skill, and further, by issuing predictions, RMS creates at least the appearance of a conflict of interest as many of its customers will benefit (or lose) according to how these predictions are made. So RMS has an financial interest in the predictions that it issues. The issues faced by catastrophe modeling firms are similar to those faced by ratings agencies in the financial services sector.
A key part of RMS short-term prediction of hurricane activity is an expert elicitation. The elicitation is described in the paper by Jewsen et al. from RMS (PDF). I participated in the 2008 RMS expert elicitation held last October. The elicitation allows experts to allocate weights to different models of hurricane activity and then these weights are used to integrate the models for each expert, thus providing a view of that particular expert for the coming five years. For instance, if there were two models with annual landfall rates of 0.4 and 0.8 Catergory 3-4-5 hurricanes, and I decided to give the first 20% weight and the second 80%, then the result would be 0.4 * 20% + 0.8 * 80% = 0.72 landfalls per year. The views of the individual experts are combined to provide the results of the group elicitation. RMS conducts its elicitation with the experts blind to the results of the models, focusing instead on the models themselves.
The RMS elicitation has been controversial because it has resulted in a short-term prediction of activity higher than the climatological average, which means higher risk, and for coastal property owners the result is higher insurance rates. In 2006 for example the elicitation by the RMS panel of experts resulted in a prediction of 0.92 Category 3-4-5 storms making landfall every year for the period 2007 to 2011, which is considerably higher than the 1950-2006 average of 0.64. At the time, loss estimates increased by about 40%, with consequences for insurance and reinsurance rates.
Because Jewsen et al. conveniently provide the results for future storm activity from the 20 models used in the 2006 elicitation, I recently decided to do my own inexpert elicitation. I created a panel of 5 “monkeys” by allocating weights randomly across the 20 models for each of my participating monkeys. My panel of 5 monkeys came up with an estimated 0.91 storms per year for 2007 to 2011. I did this three more times, and these three different panels of “monkeys” projected 0.90, 0.93, 0.92 landfalling Category 3-4-5 storms per year. The RMS experts came up with 0.92.
In short, my panels of “monkeys” produced the exact same results as those produced by the RMS expert panels comprised of world leading scientists on hurricanes and climate. How can this be? The reason for this outcome is that the 20 models used in the expert elicitation (i.e., from Jewsen et al.) provide results that range from 0.63 on the low side to 1.21 at the high end, with an average of 0.90. Thus, if the elicitiors spread their weights across models the process will always result in values considerably higher than the historical average. The more spreading of weights and the more particpants will necessarily mean that the results will gravitate to the average across the models. So with apologies to my colleagues, we seem to be of no greater intellectual value to RMS than a bunch of monkeys. Though we do have credentials, which monkeys do not.
The stablity of results for the first 3 times that RMS ran its elicitation provided a hint that something in the process constrained possible results. In 2008 there was a downward change in the results as compared to the first three years. Based on the monkey exercise, I’d guess the change is largely due to the fact that the number of models used in the elicitation expanded to 39 from 20, and while I don’t have the output from the 39 models, I’d bet a beer that the average is lower than that of the previously used 20 models.
What does this mean? It means that the RMS elicitation process is biased to give results above the historical average. Of course, it may in fact be the case (and probably is) that many of the elicited experts actually do believe that storm activity will be enhanced during this period. However the fact that the RMS elicitation methodology does not distinguish between the views of these experts and a bunch of monkeys should give some concern about the fidelity of the process. Another way to look at it is that RMS is providing guidance on the future based on what RMS believes to be the case and the experts are added in the mix to provide some legitimacy to the process. I’m not a big fan of experts being used as props in the marketing of false certainties.
The RMS expert elicitation process is based on questionable atmospheric science and plain old bad social science. This alone should lead RMS to get out of the near-term prediction business. Adding in the appearance of a conflict of interest from clients who benefit when forecasts are made to emphasize risk above the historical average makes a stronger case for RMS to abandon this particular practice. RMS is a leading company with an important role in a major financial industry. It should let its users determine what information on possible futures they want to incorporate when using a catastrophe model. RMS should abandon its expert elicitation and its effort to predict future hurricane landfalls for the good of the industray, but also in service of its own reputation.