In today’s Financial Times columnist John Kay has a very insightful piece on the limits of models in decision making. He discusses the downfall of Amaranth, a hedge fund, which lost billions of dollars, in part, because its investors did not fully understand the full scope of uncertainties associated with their investment strategies. Kay highlights an important distinction between what he calls “in model” risk and “off model” risk. In model risk refers to the uncertainties that are associated with the design of the model, in data inputs, randomness, and so on. Modelers use techniques such as Monte Carlo analysis to get a quantitative sense of model uncertainties. Off model risk refers to the degree of conformance between a model and the real world. Models by their nature are always simplifications of the real world. As in the case of Amaranth, often hard lessons of experience remind us that as powerful as models are, they can also reinforce bad decisions. As Kay writes,

When someone does attach a probability to a forecast, they have – implicitly or explicitly – used a model of the problems. The model they have used accounts for in-model risk but ignores off-model risk. Their forecasts are therefore too confident and neither you nor they have much idea how over-confident they are. That is why mathematical modeling of risk can be an aid to sound judgment, but never a complete substitute.

The 2006 hurricane season is not yet in the books, and there is plenty of time remaining for additional storms. However, if we consider the damage that has occurred this year thus far in historical perspective, how does it rank?

Let’s assume that this season has $250 million in damage. For the following data, I am using what we call “normalized” hurricane damage which adjusts past losses to current values. The 2006 season thus far ranks 73 out of 106 seasons since 1900, and 41 out of 56 seasons since 1950.

We hope to have the completely updated normalized loss analysis and data available soon. Stay tuned.

Let’s be blunt. The phrase “climate change denier” is meant to be evocative of the phrase “holocaust denier“. As such the phrase conjurs up a symbolic allusion fully intended to equate questioning of climate change with questioning of the Holocaust.

Let’s be blunt. This allusion is an affront to those who suffered and died in the Holocaust. Let those who would make such an allusion instead be absolutely explicit about their assertion of moral equivalency between Holocaust deniers and those that they criticize.

This allusion has no place in the discourse on climate change. I say this as someone fully convinced of a significant human role in the behavior of the climate system.

Let’s declare a moratorium on the phrases “climate change denier” and “climate change denial.” Let’s invoke the equivalent of Godwin’s Law in discourse on climate policy. Maybe call it the Prometheus Principle.

No more invocation of “climate change deniers.”

In various comment threads I have sought to identify clear criteria that the Royal Society applied when deciding to target Exxon and its funding of advocacy groups. I have asserted that this decision was political. Several readers and Bob Ward have suggested that the decision was based solely an effort to police misrepresentations of science by Exxon and groups that it funds. In this lengthy comment I explore this issue a bit further. Please read on if you are interested.

(more…)

This report from the BBC on the latest international climate negotiations:

One delegate told me he thought the pace of political ambition on emissions was so slow that we had a 1,000-1 chance of avoiding dangerous climate change.

He later sent me a text message to assert that he had been overly pessimistic. The odds, he said, were only 100-1.

So when is it time to re-open for negotiation FCCC Article 2? For those wanting a bit more background on this cryptic post, please see this paper in PDF.

Thanks very much to those who sent me the “Dear Colleague” letter from NOAA Administrator Conrad Lautenbacher discussing the now-released NOAA fact sheet on hurricanes and climate change (here in PDF). The full letter can be seen below.

(more…)

{I am very pleased that Bob Ward, formerly of the Royal Society, has sent in the following comment which we are happy to post. Thanks very much! RP]

I’ve enjoyed reading this exchange of views, particularly the discussion over the Royal Society’s contribution to the debate. I thought it might help to set out some of my views, although rather belatedly. I should explain that my employment at the Royal Society ceased on 22 September – not, as some have suggested, because I was sacked but because I am moving on to a new job and had agreed my departure date about three months ago.

I’d like to give a bit of background about the ExxonMobil sag, but start with an explanation of how the Royal Society sees its role (writing as an ex-employee).

[continued]

(more…)

Here is an interesting article in The Scientist on the Bush Administration.

Across a number of threads comments have arisen about the role of forecasting in decision making. Questions that have come up include:

What is a good forecast?
When should research forecasts transition to operational forecasts?
What sorts of decisions require quantitative probabilities?
In what contexts can good decisions result without accurate predictions?

It was questions like these that motivated Rad Byerly, Dan Sarewitz, and I to work on a project in the late 1990s focused on prediction. the results of this work were published in a book by Island Press in 2000, titled “Prediction.”

With this post I’d like to motivate discussion on this subject, and to point to our book’s concluding chapter, which may provide a useful point of departure:

Pielke Jr., R. A., D. Sarewitz and R. Byerly Jr., 2000: Decision Making and the Future of Nature: Understanding and Using Predictions. Chapter 18 in Sarewitz, D., R. A. Pielke Jr., and R. Byerly Jr., (eds.), Prediction: Science Decision Making and the Future of Nature. Island press: Washington, DC. (PDF)

See in particular Table 18.1 on p. 383 which summarizes the criteria we developed in the form of questions which might be used to “question predictions.”

Comments welcomed on any of the questions raised above, and others as appropriate as well.