« Prediction and Forecasting »

Guest Comment: Sharon Friedman, USDA Forest Service - Change Changes Everything
   in Author: Others | Climate Change | Environment | Prediction and Forecasting | Science + Politics February 01, 2008

Updated IPCC Forecasts vs. Observations
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 26, 2008

Temperature Trends 1990-2007: Hansen, IPCC, Obs
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 18, 2008

UKMET Short Term Global Temperature Forecast
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 16, 2008

Verification of IPCC Sea Level Rise Forecasts 1990, 1995, 2001
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 15, 2008

James Hansen on One Year's Temperature
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 14, 2008

Updated Chart: IPCC Temperature Verification
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 14, 2008

Pachauri on Recent Climate Trends
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 14, 2008

Verification of IPCC Temperature Forecasts 1990, 1995, 2001, and 2007
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 14, 2008

Real Climate's Two Voices on Short-Term Climate Fluctuations
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 11, 2008

Verification of 1990 IPCC Temperature Predictions
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 10, 2008

Forecast Verification for Climate Science, Part 3
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 09, 2008

Forecast Verification for Climate Science, Part 2
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 08, 2008

Forecast Verification for Climate Science
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting | Scientific Assessments January 07, 2008

A Second Reponse from RMS
   in Author: Pielke Jr., R. | Disasters | Prediction and Forecasting | Scientific Assessments December 17, 2007

RMS Response to Forecast Evaluation
   in Author: Others | Disasters | Prediction and Forecasting | Scientific Assessments December 07, 2007

Revisiting The 2006-2010 RMS Hurricane Damage Prediction
   in Author: Pielke Jr., R. | Disasters | Prediction and Forecasting | Risk & Uncertainty | Scientific Assessments December 06, 2007

State of Florida Rejects RMS Cat Model Approach
   in Author: Pielke Jr., R. | Disasters | Prediction and Forecasting | Risk & Uncertainty May 11, 2007

Review of Useless Arithmetic
   in Author: Pielke Jr., R. | Prediction and Forecasting May 04, 2007

Now I've Seen Everything
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting March 29, 2007

Cashing In
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting March 29, 2007

Prediction in Science and Policy
   in Author: Pielke Jr., R. | Prediction and Forecasting February 20, 2007

Ryan Meyer in Ogmius
   in Author: Pielke Jr., R. | Climate Change | Prediction and Forecasting December 19, 2006

Limits of Models in Decision
   in Author: Pielke Jr., R. | Prediction and Forecasting October 10, 2006

Prediction and Decision
   in Author: Pielke Jr., R. | Prediction and Forecasting October 02, 2006

Guest Comment: Sharon Friedman, USDA Forest Service - Change Changes Everything

It is true that the calculus of environmental tradeoffs will be inevitably and irretrievably changed due to consideration of climate change. Ideas that were convenient (convenient untruths) like “the world worked fine without humans, if we remove their influence it will go back to what it should be” have continued to provide the implicit underpinning for much scientific effort. In short, people gravitated to the concept that "if we studied how things used to be" (pre- European settlement) we would know how they "should" be, with no need for discussions of values or involving non-scientists. This despite excellent work such as the book Discordant Harmonies by Dan Botkin, that displayed the scientific flaws in this reasoning (in 1992).

What's interesting to me in the recent article, "The Preservation Predicament", by Cornelia Dean in The New York Times
is the implicit assumption that conservationists and biologists will be the ones who determine whether investing in conservation in the Everglades compared to somewhere else, given climate change, is a good idea - perhaps implying that sciences like decision science or economics have little to contribute to the dialog. Not to speak of communities and their elected officials.

I like to quote the IUCN (The World Conservation Union) governance principles:

Indigenous and local communities are rightful primary partners in the development and implementation of conservation strategies that affect their lands, waters, and other resources, and in particular in the establishment and management of protected areas.

Is it more important for scientists to "devise theoretical frameworks for deciding when, how or whether to act" (sounds like decision science) or for folks in a given community, or interested in a given species, to talk about what they think needs to be done and why? There are implicit assumptions about what sciences are the relevant ones and the relationship between science and democracy, which in my opinion need to be debated in the light of day rather than assumed.

Sharon Friedman
Director, Strategic Planning
Rocky Mountain Region
USDA Forest Service

Updated IPCC Forecasts vs. Observations

Carl Mears from Remote Sensing Systems, Inc. was kind enough to email me to point out that the RSS data that I had shared with our readers a few weeks ago contained an error that RSS has since corrected. The summary figure above is re-plotted with the corrected data (RSS is the red curve). At the time I wrote:

Something fishy is going on. The IPCC and CCSP recently argued that the surface and satellite records are reconciled. This might be the case from the standpoint of long-term linear trends. But the data here suggest that there is some work left to do. The UAH and NASA curves are remarkably consistent. But RSS dramatically contradicts both. UKMET shows 2007 as the coolest year since 2001, whereas NASA has 2007 as the second warmest. In particular estimates for 2007 seem to diverge in unique ways. It'd be nice to see the scientific community explain all of this.

For those interested in the specifics, Carl explained in his email:

The error was simple -- I made a small change in the code ~ 1 year ago that resulted in a ~0.1K decrease in the absolute value of AMSU TLTs, but neglected to reprocess data from 1998-2006, instead only using it for the new (Jan 2007 onward) data. Since the AMSU TLTs are forced to match the MSU TLTs (on average) during the overlap period, this resulted in an apparent drop in TLT for 2007. Reprocessing the earlier AMSU data, thus lowering AMSU TLT by 0.1 from 1998-2006, resulted in small changes in the parameters that are added to the AMSU temperatures to make them match MSU temperatures, and thus the 2007 data is increased by ~0.1K. My colleagues at UAH (Christy and Spencer) were both very helpful in diagnosing the problem.

It is important to note that the RSS correction does not alter my earlier analysis of the IPCC predictions (made in 1990, 1995, 2001, 2007) and various observations. Thanks again to Carl for alerting me to the error and giving me a chance to update the figures with the new information!

Temperature Trends 1990-2007: Hansen, IPCC, Obs

The figure below shows linear trends in temperature for Jim Hansen's three 1988 scenarios (in shades of blue), for the IPCC predictions issued in 1990, 1995, 2001, 2007 (in shades of green), and for four sets of observations (in shades of brown). I choose the period 1990-2007 because this is the period of overlap for all of the predictions (except IPCC 2007, which starts in 2000).

Looking just at these measures of central tendency (i.e., no formal consideration of uncertainties) it seems clear that:

1. Trends in all of Hansen's scenarios are above IPCC 1995, 2001, and 2007, as well as three of the four surface observations.

2. The outlier on surface observations, and the one consistent with Hansen's Scenarios A and B is the NASA dataset overseen by Jim Hansen. Whatever the explanation for this, good scientific practice would have forecasting and data collection used to verify those forecasts conducted by completely separate groups.

3. Hansen's Scenario A is very similar to IPCC 1990, which makes sense given their closeness in time, and assumptions of forcings at the time (i.e., thoughts on business-as-usual did not change much over that time).

The data for the Hansen scenarios was obtained at Climate Audit from the ongoing discussion there, and the IPCC and observational data is as described on this site over the past week or so in the forecast verification exercise that I have conducted. This is an ongoing exercise, as part of a conversation across the web, so if you have questions or comments, please share them, either here, or if our comment interface is driving you nuts (as it is with me), then comment over at Climate Audit where I'll participate in the discussions.

UKMET Short Term Global Temperature Forecast

This figure shows a short-term forecast of global average temperature issued by the UK Meteorological Service, with some annotations that I've added and described below. The forecast is discussed in this PDF where you can find the original figure. This sort of forecast should be applauded, because it allows for learning based on experience. Such forecasts, whether eventually shown to be wrong or right, can serve as powerful tests of knowledge and predictive skill. The UK Met Service is to be applauded. Now on to the figure itself.

The figure is accompanied by this caption:

Observations of global average temperature (black line) compared with decadal ‘hindcasts’ (10-year model simulations of the past, white lines and red shading), plus the first decadal prediction for the 10 years from 2005. Temperatures are plotted as anomalies (relative to 1979–2001). As with short-term weather forecasts there remains some uncertainty in our predictions of temperature over a decade. The red shading shows our confidence in predictions of temperature in any given year. If there are no volcanic eruptions during the forecast period, there is a 90% likelihood of the temperature being within the shaded area.

The figure shows both hindcasts and a forecast. I've shaded the hindcasts in grey. I've added the green curve which is my replication of the global temperature anomalies from the UKMET HADCRUT3 dataset extended to 2007. I've also plotted as a blue dot the prediction issued by UKMET for 2008, which is expected to be indistinguishable from the temperature of years 2001 to 2007 (which were indistinguishable from each other). The magnitude of the UKMET forecast over the next decade is almost exactly identical to the IPCC AR4 prediction over the same time period, which I discussed last week.

I have added the pink star at 1995 to highlight the advantages offered by hindcasting. Imagine if the model realization begun in 1985 had been continued beyond 1995, rather than being re-run after 1995. Clearly, all subsequent observed temperatures would have been well below that 1985 curve. One important reason for this is of course the eruption of Mt. Pinatubo, which was not predicted. And that is precisely the point -- prediction is really hard, especially when conducted in the context of open systems, and as is often said, especially about the the future. Our ability to explain why a prediction was wrong does not make that prediction right, and this is a point often lost in debate about climate change.

Again, kudos to the UK Met Service. They've had the fortitude to issue a short term prediction related to climate change. Other scientific bodies should follow this lead. It is good for science, and good for the use of science in decision making.

Verification of IPCC Sea Level Rise Forecasts 1990, 1995, 2001

Here is a graph showing IPCC sea level rise forecasts from the FAR (1990), SAR (1995), and TAR (2001).

And here are the sources:

Observational data can be found here. Thanks to my colleague Steve Nerem.

Unlike temperature forecasts by the IPCC, sea level rise shows no indication that scientists have a handle on the issue. As with temperature the IPCC dramatically decreased its predictions of sea level rise in between its first (1990) and second (1995) assessment reports. It then nudged down its prediction a very small amount in its 2001 report. The observational data falls in the middle of the 1990 and 1995/2001 assessments.

Last year Rahmstorf et al. published a short paper in Science comparing observations of temperature with IPCC 2001 predictions (Aside: it is remarkable that Science allowed them to ignore IPCC 1990 and 1995). Their analysis is completely consistent with the temperature and sea level rise verifications that I have shown. On sea level rise they concluded:

Previous projections, as summarized by IPCC, have not exaggerated but may in some respects even have underestimated the change, in particular for sea level.

This statement is only true if one ignores the 1990 IPCC report which overestimated both sea level rise and temperature. Rahmstorf et al. interpretation of the results is little more than spin, as it would have been equally valid to conclude based on the 1990 report:

Previous projections, as summarized by IPCC, have not underestimated but may in some respects even have exaggerated the change, both for sea level and temperature.

Rather than spin the results, I conclude that the ongoing debate about future sea level rise is entirely appropriate. The fact that the IPCC has been unsuccessful in predicting sea level rise, does not mean that things are worse or better, but simply that scientists clearly do not have a handle on this issue and are unable to predict sea level changes on a decadal scale. The lack of predictive accuracy does not lend optimism about the prospects for accuracy on the multi-decadal scale. Consider that the 2007 IPCC took a pass on predicting near term sea level rise, choosing instead to focus 90 years out (as far as I am aware, anyone who knows differently, please let me know).

This state of affairs should give no comfort to anyone: over the 21st century sea level is expected to rise, anywhere from an unnoticeable amount to the catastrophic, and scientists have essentially no ability to predict this rise, much less the effects of various climate policies on that rise. As we've said here before, this is a cherrypickers delight, and a policy makers nightmare. It'd be nice to see the scientific community engaged in a bit less spin, and a bit more comprehensive analysis.

James Hansen on One Year's Temperature

NASA's James Hansen just sent around a commentary (in PDF here) on the significance of the 2007 global temperature in the context of the long-term temperature record that he compiles for NASA. After Real Climate went nuts over how misguided it is to engage in a discussion of eight years worth of temperature records, I can''t wait to see them lay into Jim Hansen for asserting that one year's data is of particular significance (and also for not graphing uncertainty ranges):

The Southern Oscillation and the solar cycle have significant effects on year-to-year global temperature change. Because both of these natural effects were in their cool phases in 2007, the unusual warmth of 2007 is all the more notable.

But maybe it is that data that confirms previously held beliefs is acceptable no matter how short the record, and data that does not is not acceptable, no matter how long the record. But that would be confirmation bias, wouldn't it?

Anyway, Dr. Hansen does not explain why the 2007 NASA data runs counter to that of UKMET, UAH or RSS, but does manage to note the "incorrect" 2007 UKMET prediction of a record warm year. Dr. Hansen issues his own prediction:

. . . it is unlikely that 2008 will be a year with an unusual global temperature change, i.e., it is likely to remain close to the range of (high) values exhibited in 2002-2007. On the other hand, when the next El Nino occurs it is likely to carry global temperature to a significantly higher level than has occurred in recent centuries, probably higher than any year in recent millennia. Thus we suggest that, barring the unlikely event of a large volcanic eruption, a record global temperature clearly exceeding that of 2005 can be expected within the next 2-3 years.

I wonder if this holds just for the NASA dataset put together by Dr. Hansen or for all of the temperature datasets.

Updated Chart: IPCC Temperature Verification

I've received some email comments suggesting that my use of the 1992 IPCC Supplement as the basis for IPCC 1990 temperature predictions was "too fair" to the IPCC because the IPCC actually reduced its temperature projections from 1990 to 1992. In addition, Gavin Schmidt and a commenter over at Climate Audit also did not like my use of the 1992 report. So I am going to take full advantage of the rapid feedback of the web to provide an updated figure, based on IPCC 1990, specifically, Figure A.9, p. 336. In other words, I no longer rely on the 1992 supplement, and have simply gone back to the original IPCC 1990 FAR. Here then is that updated Figure:

Thanks all for the feedback!

Pachauri on Recent Climate Trends

Last week scientists at the Real Climate blog gave their confirmation bias synapses a workout by explaining that eight years of climate data is meaningless, and people who pay any attention to recent climate trends are "misguided." I certainly agree that we should exhibit cautiousness in interpreting short-duration observations, nonetheless we should always be trying to explain (rather than simply discount) observational evidence to avoid the trap of confirmation bias.

So it was interesting to see IPCC Chairman Rajendra Pachauri exhibit "misguided" behavior when he expressed some surprise about recent climate trends in The Guardian:

Rajendra Pachauri, the head of the U.N. Panel that shared the 2007 Nobel Peace Prize with former U.S. Vice President Al Gore, said he would look into the apparent temperature plateau so far this century.

"One would really have to see on the basis of some analysis what this really represents," he told Reuters, adding "are there natural factors compensating?" for increases in greenhouse gases from human activities.

He added that sceptics about a human role in climate change delighted in hints that temperatures might not be rising. "There are some people who would want to find every single excuse to say that this is all hogwash," he said.

Ironically, by suggesting that their might be some significance to recent climate trends, Dr. Pachauri has provided ammunition to those very same skeptics that he disparages. Perhaps Real Climate will explain how misguided he is, but somehow I doubt it.

For the record, I accept the conclusions of IPCC Working Group I. I don't know how to interpret climate observations of the early 21st century, but believe that there are currently multiple valid hypotheses. I also think that we can best avoid confirmation bias, and other cognitive traps, by making explicit predictions of the future and testing them against experience. The climate community, or at least its activist wing, studiously avoids forecast verification. It just goes to show, confirmation bias is more a more comfortable state than dissonance -- and that goes for people on all sides of the climate debate.

Verification of IPCC Temperature Forecasts 1990, 1995, 2001, and 2007

Last week I began an exercise in which I sought to compare global average temperature predictions with the actual observed temperature record. With this post I'll share my complete results.

Last week I showed a comparison of the 2007 IPCC temperature forecasts (which actually began in 2000, so they were really forecasts of data that had already been observed). Here is that figure.

Then I showed a figure with a comparison of the 1990 predictions made by the IPCC in 1992 with actual temperature data. Some folks misinterpreted the three curves that I showed from the IPCC to be an uncertainty bound. They were not. Instead, they were forecasts conditional on different assumptions about climate sensitivity, with the middle curve showing the prediction for a 2.5 degree climate sensitivity, which is lower than scientists currently believe to the most likely value. So I have reproduced that graph below without the 1.5 and 4.5 degree climate sensitivity curves.

Now here is a similar figure for the 1995 forecast. The IPCC in 1995 dramatically lowered its global temperature predictions, primarily due to the inclusion of consideration of atmospheric aerosols, which have a cooling effect. You can see the 1995 IPCC predictions on pp. 322-323 of its Second Assessment Report. Figure 6.20 shows the dramatic reduction of temperature predictions through the inclusion of aerosols. The predictions themselves can be found in Figure 6.22, and are the values that I use in the figure below, which also use a 2.5 degree climate sensitivity, and are also based on the IS92e or IS92f scenarios.

In contrast to the 1990 prediction, the 1995 prediction looks spot on. It is worth noting that the 1995 prediction began in 1990, and so includes observations that were known at the time of the prediction.

In 2001, the IPCC nudged its predictions up a small amount. The prediction is also based on a 1990 start, and can be found in the Third Assessment Report here. The most relevant scenario is A1FI, and the average climate sensitivity of the models used to generate these predictions is 2.8 degrees, which may be large enough to account for the difference between the 1995 and 2001 predictions. Here is a figure showing the 2001 forecast verification.

Like 1995, the 2001 figure looks quite good in comparison to the actual data.

Now we can compare all four predictions with the data, but first here are all four IPCC temperature predictions (1990, 1995, 2001, 2007) on one graph.

IPCC issued its first temperature prediction in 1990 (I actually use the prediction from the supplement to the 1990 report issued in 1992). Its 1995 report dramatically lowered this prediction. 2001 nudged this up a bit, and 2001 elevated the entire curve another small increment, keeping the slope the same. My hypothesis for what is going on here is that the various changes over time to the IPCC predictions reflect incrementally improved fits to observed temperature data, as more observations have come in since 1990.

In other words, the early 1990s showed how important aerosols were in the form of dramatically lowered temperatures (after Mt. Pinatubo), and immediately put the 1990 predictions well off track. So the IPCC recognized the importance of aerosols and lowered its predictions, putting the 1995 IPCC back on track with what had happened with the real climate since its earlier report. With the higher observed temperatures in the late 1990s and early 2000s the slightly increased predictions of temperature in 2001 and 2007 represented better fits with observations since 1995 (for the 2001 report) and 2001 (for the 2007 report).

Imagine if your were asked to issue a prediction for the temperature trend over next week, and you are allowed to update that prediction every 2nd day. Regardless of where you think things will eventually end up, you'd be foolish not to include what you've observed in producing your mid-week updates. Was this behavior by the IPCC intentional or simply the inevitable result of using a prediction start-date years before the forecast was being issued? I have no idea. But the lesson for the IPCC should be quite clear: All predictions (and projections) that it issues should begin no earlier than the year that the prediction is being made.

And now the graph that you have all been waiting for. Here is a figure showing all four IPCC predictions with the surface (NASA, UKMET) and satellite (UAH, RSS) temperature record.

You can see on this graph that the 1990 prediction was obviously much higher than the other three, and you can also clearly see how the IPCC temperature predictions have creeped up as observations showed increasing temperatures from 1995-2005. A simple test of my hypothesis is as follows: In the next IPCC, if temperatures from 2005 to the next report fall below the 2007 IPCC prediction, then the next IPCC will lower its predictions. Similarly, if values fall above that level, then the IPCC will increase its predictions.

What to take from this exercise?

1. The IPCC does not make forecast verification an easy task. The IPCC does not clearly identify what exactly it is predicting nor the variables that can be used to verify those predictions. Like so much else in climate science this leaves evaluations of predictions subject to much ambiguity, cherrypicking, and seeing what one wants to see.

2. The IPCC actually has a pretty good track record in its predictions, especially after it dramatically reduced its 1990 prediction. This record is clouded by an appearance of post-hoc curve fitting. In each of 1995, 2001, and 2007 the changes to the IPCC predictions had the net result of improving predictive performance with observations that had already been made. This is a bit like predicting today's weather at 6PM.

3. Because the IPCC clears the slate every 5-7 years with a new assessment report, it is guarantees that its most recent predictions can never be rigorously verified, because, as climate scientists will tell you, 5-7 years is far too short to say anything about climate predictions. Consequently, the IPCC should not predict and then move on, but pay close attention to its past predictions and examine why the succeed or fail. As new reports are issued the IPCC should go to great lengths to place its new predictions on an apples-to-apples basis with earlier predictions. The SAR did a nice job of this, more recent reports have not. A good example of how not to update predictions is the predictions of sea level rise between the TAR and AR4 which are not at all apples-to-apples.

4. Finally, and I repeat myself, the IPCC should issue predictions for the future, not the recent past.

Appendix: Checking My Work

The IPCC AR4 Technical Summary includes a figure (Figure TS.26) that shows a verification of sorts. I use that figure as a comparison to what I've done. Here is that figure, with a number of my annotations superimposed, and explained below.

Let me first say that the IPCC probably could not have produced a more difficult-to-interpret figure (I see Gavin Schmidt at Real Climate has put out a call for help in understanding it). I have annotated it with letters and some lines and I explain them below.

A. I added this thick horizontal blue line to indicate the 1990 baseline. This line crosses a thin blue line that I placed to represent 2007.

B. This thin blue line crosses the vertical axis where my 1995 verification value lies, represented by the large purple dot.

C. This thin blue line crosses the vertical axis where my 1990 verification value lies, represented by the large green dot. (My 2001 verification is represented by the large light blue dot.)

D. You can see that my 1990 verification value falls exactly on a line extended from the upper bound of the IPCC curve. I have also extended the IPCC mid-range curve as well (note that my extension superimposed falls a tiny bit higher than it should). Why is this? I'm not sure, but one answer is that the uncertainty range presented by the IPCC represents the scenario range, but of course in the past there is no scenario uncertainty. Since emissions have fallen at the high end of the scenario space, if my interpretation is correct, then my verification is consistent with that of the IPCC.

E. For the 1995 verification, you can see that similarly my value falls exactly on a line extended from the upper end of the IPCC range. This would also be consistent with the IPCC presenting the uncertainty range as representing alternative scenarios. The light blue dot is similarly at the upper end of the blue range. What should not be missed is that the relative difference between my verifications and those of the IPCCs are just about identical.

A few commenters over at Real Climate, including Gavin Schmidt, have suggested that such figures need uncertainty bounds on them. In general, I agree, but I'd note that none of the model predictions presented by the IPCC (B1, A1B, A2, Commitment -- note that all of these understate reality since emissions are following A1FI, the highest, most closely) show any model uncertainty whatsoever (nor any observational uncertainty, nor multiple measures of temperature). Surely with the vast resources available to the IPCC, they could have done a much more rigorous job of verification.

In closing, I guess I'd suggest to the IPCC that this sort of exercise should be taken up as a formal part of its work. There are many, many other variables (and relationships between variables) that might be examined in this way. And they should be.

Real Climate's Two Voices on Short-Term Climate Fluctuations

Real Climate has been speaking with two voices on how to compare observations of climate with models. Last August they asserted that one-year's sea ice extent could be compared with models:

A few people have already remarked on some pretty surprising numbers in Arctic sea ice extent this year (the New York Times has also noticed). The minimum extent is usually in early to mid September, but this year, conditions by Aug 9 had already beaten all previous record minima. Given that there is at least a few more weeks of melting to go, it looks like the record set in 2005 will be unequivocally surpassed. It could be interesting to follow especially in light of model predictions discussed previously.

Today, they say that looking at 8 years of temperature records is misguided:

John Tierney and Roger Pielke Jr. have recently discussed attempts to validate (or falsify) IPCC projections of global temperature change over the period 2000-2007. Others have attempted to show that last year's numbers imply that 'Global Warming has stopped' or that it is 'taking a break' (Uli Kulke, Die Welt)). However, as most of our readers will realise, these comparisons are flawed since they basically compare long term climate change to short term weather variability.

So according to Real Climate one-year's ice extent data can be compared to climate models, but 8 years of temperature data cannot.

Right. This is why I believe that whatever one's position of climate change is, everyone should agree that rigorous forecast verification is needed.

Post Script. I see at Real Climate commenters are already calling me a "skeptic" for even discussing forecast verification. For the record I accept the consensus of the IPCC WGI. If asking questions about forecast verification is to be tabooo, then climate science is in worse shape than I thought.

Verification of 1990 IPCC Temperature Predictions

I continue to receive good suggestions and positive feedback on the verification exercise that I have been playing around with this week. Several readers have suggested that a longer view might be more appropriate. So I took a look at the IPCC's First Assessment Report that had been sitting on my shelf, and tried to find its temperature prediction starting in 1990. I actually found what I was looking for in a follow up document: Climate Change 1992: The Supplementary Report to the IPCC Scientific Assessment (not online that I am aware of).

In conducting this type of forecast verification, one of the first things to do is to specify which emissions scenario most closely approximated what has actually happened since 1990. As we have discussed here before, emissions have been occurring at the high end of the various scenarios used by the IPCC. So in this case I have used IS92e or IS92f (the differences are too small to be relevant to this analysis), which are discussed beginning on p. 69.

With the relevant emissions scenario, I then went to the section that projected future temperatures, and found this in Figure Ax.3 on p. 174. From that I took from the graph the 100-year temperature change and converted it into an annual rate. At the time the IPCC presented estimates for climate sensitivities of 1.5 degree, 2.5 degrees, and 4.5 degrees, with 2.5 degrees identified as a "best estimate." In the figure above I have estimated the 1.5 and 4.5 degree values based on the ratios taken from graph Ax.2, but I make no claim that they are precise. My understanding is that climate scientists today think that climate sensitivity is around 3.0 degrees, so if one were to re-do the 1990 prediction with a climate sensitivity of 3.0 the resulting curve would be a bit above the 2.5 degree curve shown above.

On the graph you will also see the now familiar temperature records from two satellite and two surface analyses. It seems pretty clear that the IPCC in 1990 over-forecast temperature increases, and this is confirmed by the most recent IPCC report (Figure TS.26), so it is not surprising.

I'll move on to the predictions of the Second Assessment Report in a follow up.

Forecast Verification for Climate Science, Part 3

By popular demand, here is a graph showing the two main analyses of global temperatures from satellite, from RSS and UAH, as well as the two main analyses of global temperatures from the surface record, UKMET and NASA, plotted with the temperature predictions reported in IPCC AR4, as described in Part 1 of this series.

Some things to note:

1) I have not graphed observational uncertainties, but I'd guess that they are about +/-0.05 (and someone please correct me if this is wildly off), and their inclusion would not alter the discussion here.

2) A feast for cherrypickers. One can arrive at whatever conclusion one wants with respect to the IPCC predictions. Want the temperature record to be consistent with IPCC? OK, then you like NASA. How about inconsistent? Well, then you are a fan of RSS. On the fence? Well, UAH and UKMET serve that purpose pretty well.

3) Something fishy is going on. The IPCC and CCSP recently argued that the surface and satellite records are reconciled. This might be the case from the standpoint of long-term liner trends. But the data here suggest that there is some work left to do. The UAH and NASA curves are remarkably consistent. But RSS dramatically contradicts both. UKMET shows 2007 as the coolest year since 2001, whereas NASA has 2007 as the second warmest. In particular estimates for 2007 seem to diverge in unique ways. It'd be nice to see the scientific community explain all of this.

4) All show continued warming since 2000!

5) From the standpoint of forecast verification, which is where all of this began, the climate community really needs to construct a verification dataset for global temperature and other variables that will be (a) the focus of predictions, and (b) the ground truth against which those predictions will be verified.

Absent an ability to rigorously evaluate forecasts, in the presence of multiple valid approaches to observational data we run the risk of engaging in all sorts of cognitive traps -- such as availability bias and confirmation bias. So here is a plea to the climate community: when you say that you are predicting something like global temperature or sea ice extent or hurricanes -- tell us is specific detail what those variables are, who is measuring them, and where to look in the future to verify the predictions. If weather forecasters, stock brokers, and gamblers can do it, then you can too.

Forecast Verification for Climate Science, Part 2

Yesterday I posted a figure showing how surface temperatures compare with IPCC model predictions. I chose to use the RSS satellite record under the assumption that the recent IPCC and CCSP reports were both correct in their conclusions that the surface and satellite records have been reconciled. It turns out that my reliance of the IPCC and CCSP may have been mistaken.

I received a few comments from people suggesting that I had selectively used the RSS data because it showed different results than other global temperature datasets. My first reaction to this was to wonder how the different datasets could show different results if the IPCC was correct when it stated (PDF):

New analyses of balloon-borne and satellite measurements of lower- and mid-tropospheric temperature show warming rates that are similar to those of the surface temperature record and are consistent within their respective uncertainties, largely reconciling a discrepancy noted in the TAR.

But I decided to check for myself. I went to the NASA GISS and downloaded its temperature data and scaled to a 1980-1999 mean. I then plotted it on the same scale as the RSS data that I shared yesterday. Here is what the curves look like on the same scale.

Well, I'm no climate scientist, but they sure don't look reconciled to me, especially 2007. (Any suggestions on the marked divergence in 2007?)

What does this mean for the comparison with IPCC predictions? I have overlaid the GISS data on the graph I prepared yesterday.

So using the NASA GISS global temperature data for 2000-2007 results in observations that are consistent with the IPCC predictions, but contradict the IPCC's conclusion that the surface and satellite temperature records are reconciled. Using the RSS data results in observations that are (apparently) inconsistent with the IPCC predictions.

I am sure that in conducting such a verification some will indeed favor the dataset that best confirms their desired conclusions. But, it would be ironic indeed to see scientists now abandon RSS after championing it in the CCSP and IPCC reports. So, I'm not sure what to think.

Is it really the case that the surface and satellite records are again at odds? What dataset should be used to verify climate forecasts of the IPCC?

Answers welcomed.

Forecast Verification for Climate Science

Last week I asked a question:

What behavior of the climate system could hypothetically be observed over the next 1, 5, 10 years that would be inconsistent with the current consensus on climate change?

We didn’t have much discussion on our blog, perhaps in part due to our ongoing technical difficulties (which I am assured will be cleared up soon). But John Tierney at the New York Times sure received an avalanche of responses, many of which seemed to excoriate him simply for asking the question, and none that really engaged the question.

I did receive a few interesting replies by email from climate scientists. Here is one of the most interesting:

The IPCC reports, both AR4 (see Chapter 10) and TAR, are full of predictions made starting in 2000 for the evolution of surface temperature, precipitation, precipitation intensity, sea ice extent, and on and on. It would be a relatively easy task for someone to begin tracking the evolution of these variables and compare them to the IPCC’s forecasts. I am not aware of anyone actually engaged in this kind of climate forecast verification with respect to the IPCC, but it is worth doing.

So I have decided to take him up on this and present an example of what such a verification might look like. I have heard some claims lately that global warming has stopped, based on temperature trends over the past decade. So global average temperature seems like a as good a place as any to provide an example.

I begin with the temperature trends. I have decided to use the satellite record provided by Remote Sensing Systems, mainly because of the easy access of its data. But the choice of satellite versus surface global temperature dataset should not matter, since these have been reconciled according to the IPCC AR4. Here is a look at the satellite data starting in 1998 through 2007.

This dataset starts with the record 1997/1998 ENSO event which boosted temperatures a good deal. It is interesting to look at, but probably not the best place to start for this analysis. A better place to start is with 2000, but not because of what the climate has done, but because this is the baseline used for many of the IPCC AR4 predictions.

Before proceeding, a clarification must be made between a prediction and a projection. Some have claimed that the IPCC doesn’t make predictions, it only makes projections across a wide range of emissions scenarios. This is just a fancy way of saying that the IPCC doesn’t predict future emissions. But make no mistake, it does make conditional predictions for each scenario. Enough years have passed for us to be able to say that global emissions have been increasing at the very high end of the family of scenarios used by the IPCC (closest to A1F1 for those scoring at home). This means that we can zero in on what the IPCC predicted (yes, predicted) for the A1F1 scenario, which has best matched actual emissions.

So how has global temperature changed since 2000? Here is a figure showing the monthly values, indicating that while there has been a decrease in average global temperature of late, the linear trend since 2000 is still positive.

But monthly values are noisy, and not comparable with anything produced by the IPCC, so let’s take a look at annual values.

The annual values result in a curve that looks a bit like an upwards sloping letter M.

The model results produced by the IPCC are not readily available, so I will work from their figures. In the IPCC AR4 report Figure 10.26 on p. 803 of Chapter 10 of the Working Group I report (here in PDF) provides predictions of future temperature as a function of emissions scenario. The one relevant for my purposes can be found in the bottom row (degrees C above 1980-2000 mean) and second column (A1F1).

I have zoomed in on that figure, and overlaid the RSS temperature trends 2000-2007 which you can see below.

Now a few things to note:

1. The IPCC temperature increase is relative to a 1980 to 2000 mean, whereas the RSS anomalies are off of a 1979 to 1998 mean. I don’t expect the differences to be that important in this analysis, particularly given the blunt approach to the graph, but if someone wants to show otherwise, I’m all ears.

2. It should be expected that the curves are not equal in 2000. The anomaly for 2000 according to RSS is 0.08, hence the red curve begins at that value. Figure 10.26 on p. 803 of Chapter 10 of the Working Group I report actually shows observed temperatures for a few years beyond 2000, and by zooming in on the graph in the lower left hand corner of the figure one can see that 2000 was in fact below the A1B curve.

So it appears that temperature trends since 2000 are not closely following the most relevant prediction of the IPCC. Does this make recent temperature trends inconsistent with the IPCC? I have no idea, and that is not the point of this post. I'll leave it to climate scientists to tell us the significance. I assume that many climate scientists will say that there is no significance to what has happened since 2000, and perhaps emphasize that predictions of global temperature are more certain in the longer term than shorter term. But that is not what the IPCC figure indicates. In any case, 2000-2007 may not be sufficient time for climate scientists to become concerned that their predictions are off, but I’d guess that at some point, if observations don’t match predictions they might be of some concern. Alternatively, if observations square with predictions, then this would add confidence.

Before one dismisses this exercise as an exercise in randomness, it should be observed that in other contexts scientists associated short term trends with longer-term predictions. In fact, one need look no further than the record 2007 summer melt in the Arctic which was way beyond anything predicted by the IPCC, reaching close to 3 million square miles less than the 1978-2000 mean. The summer anomaly was much greater than any of the IPCC predictions on this time scale (which can be seen in IPCC AR4 Chapter 10 Figure 10.13 on p. 771). This led many scientists to claim that because the observations were inconsistent with the models, that there should be heightened concern about climate change. Maybe so. But if one variable can be examined for its significance with respect to long-term projections, then surely others can as well.

What I’d love to see is a place where the IPCC predictions for a whole range of relevant variables are provided in quantitative fashion, and as corresponding observations come in, they can be compared with the predictions. This would allow for rigorous evaluations of both the predictions and the actual uncertainties associated with those predictions. Noted atmospheric scientist Roger Pielke, Sr. (my father, of course) has suggested that three variables be looked at: lower tropospheric warming, atmospheric water vapor content, and oceanic heat content. And I am sure there are many other variables worth looking at.

Forecast evaluations also confer another advantage – they would help to move beyond the incessant arguing about this or that latest research paper and focus on true tests of the fidelity of our ability to forecast future states of the climate system. Making predictions and them comparing them to actual events is central to the scientific method. So everyone in the climate debate, whether skeptical or certain, should welcome a focus on verification of climate forecasts. If the IPCC is indeed settled science, then forecast verifications will do nothing but reinforce that conclusion.

For further reading:

Pielke, Jr., R.A., 2003: The role of models in prediction for decision, Chapter 7, pp. 113-137 in C. Canham and W. Lauenroth (eds.), Understanding Ecosystems: The Role of Quantitative Models in Observations, Synthesis, and Prediction, Princeton University Press, Princeton, N.J. (PDF)

Sarewitz, D., R.A. Pielke, Jr., and R. Byerly, Jr., (eds.) 2000: Prediction: Science, decision making and the future of nature, Island Press, Washington, DC. (link) and final chapter (PDF).

A Second Reponse from RMS

A few weeks ago I provided a midterm evaluation of the RMS 2006-2010 US hurricane damage prediction. RMS (and specifically Steve Jewson) responded and has subsequently (and graciously) sent in a further response to a question that I posed:

Does RMS stand by its spring 2006 forecast that the period 2006-2010 would see total insured losses 40% above the historical average?

The RMS response appears below, and I'll respond in the comments:

Yes, we do stand by that forecast, although I should point out that we update the forecast every year, so the 2005 forecast (for 2006-2010) is now 2 years out of date. Apart from questions of forecast accuracy, there's no particular reason for any of our users to use the 2005 forecast at this point (that would be like using a weather forecast from last week). It is, of course, important to understand the correct mathematical interpretation of the forecast. In your original post you interpreted the forecast incorrectly in a couple of ways. Over the last 2-3 years we've issued this forecast to hundreds of insurance companies, and discussed it with dozens of scientists around the world, and none of them have misinterpreted it, so I don't think our communication of the intended meaning of the forecast is unclear. However, some explanation is required and I realise that you probably haven't had the benefit of hearing one of the many presentations we've given on this subject. The two things that need clarifying are: 1) This forecast is a best estimate of the mean of a very wide distribution of possible losses. Because of this no-one should expect to be able to verify or falsify the forecast in a short period of time.

This is a typical property of forecasts in situations with high levels of uncertainty. I think it's pretty well understood by the users of the forecast.

One curious property of the loss distribution is that it is very skewed. As a result the real losses would be expected to fall below the mean in most years. This is compensated for in the average by occasional years with very high losses.

In fact the forecast that we give to the insurance industry is a completely probabilistic forecast, that estimates the entire distribution of possible losses, but it's a bit difficult to put that
kind of information into a press release, or on a blog.

2) Your conditional interpretation of the forecast is not mathematically correct. Neither RMS, nor our clients, expect the losses to increase in 2008-2010 in the way you suggest just because they were low in 2006-2007. I can't think of any reason why that would be the case. To get the (roughly) correct interpretation for 2008-2010 you have to multiply the original 5 year mean values by 0.6. That's what the users of our forecast do when they want that number.

I hope that clarifies the issues a bit.

RMS Response to Forecast Evaluation

Robert Muir-Woods of RMS has graciously provided for posting a response to the thoughts on forecast verification that I posted earlier this week. Here are his comments:

Scientifically it is of course not possible to draw any conclusion from the occurrence of two years without hurricane losses in the US, in particular following two years with the highest level of hurricane losses ever recorded and the highest ever number of severe hurricanes making landfall in a two year period. Even including 2006 and 2007, average annualized losses for the past five years are significantly higher than the long term historical average (and maybe you should also show this five year average on your plot?)

The basis for catastrophe loss modeling is that one can separate out the question of activity rate from the question as to the magnitude of losses that will be generated by the occurrence of hurricane events. In generating average annualized losses we need to explore the full 'virtual spectrum' of all the possible events that can occur. The question about current activity rates is a difficult one, which is why we continue to involve some of the leading hurricane climatologists, and a very wide range of forecasting methodologies, in our annual hurricane activity rate update procedure. In October 2007 an independent expert panel concluded that activity rates are forecasted to remain elevated for the next five years. While this perspective was announced and articulated by RMS, we did not originate it. Each year we undertake this exercise, we ensure that the forecasting models used to estimate activity over the next five years also reflect any additional learning from the forecasting of previous years, including the low activity experienced in 2006 and 2007. We don't 'declare success' that the activity rate estimate that has emerged from this procedure over the past three years (using different forecast models and different climatologists) has scarcely changed, but the consistency in the three 5 year projections is interesting nonetheless.

You may also be surprised to learn that our five-year forward-looking perspective on hurricane risk does not inevitably produce higher losses than all other models, which use the extrapolation of the simple long-term average to estimate future activity. This is as shown in a comparison published in a report prepared by the Florida Commission on Hurricane Loss Projection Methodology for the Florida House of Representatives (see the Table 1 on page 25 of the report, which can be downloaded from here: http://www.sbafla.com/methodology/announcements.asp?FormMode=Call&LinkType=Section&Section=0)

Robert Muir-Wood
RMS

Revisiting The 2006-2010 RMS Hurricane Damage Prediction

In the spring of 2006, a company called Risk Management Solutions (RMS) issued a five year forecast of hurricane activity (for 2006-2010) predicting U.S. insured losses to be 40% higher than average. RMS is an important company because their loss models are used by insurance companies to set rates charged to homeowners, by reinsurance companies to set rates they charge to insurers, by ratings agencies for evaluating risks, and others.

We are now two years into the RMS forecast period and can thus say something preliminary about their forecast based on actual hurricane damage from 2006 and 2007, which was minimal. In short, the forecast doesn't look too good. For 2006 and 2007, the following figure shows average annual insured historical losses (for 2005 and earlier) in blue (based on Pielke et al. 2008, adjusted up by 4% from 2006 to 2007 to account for changing exposure), the RMS prediction of 40% more losses above the average in pink, and the actual losses in red.

The RMS prediction obviously did not improve upon a naive forecast of average losses in either year.

What are the chances for the 5-year forecast yet to verify?

Average U.S. insured losses according to Pielke et al. (2008) are about $5.2 billion per year. Over 5 years this is $26 billion, and 40% higher than this is $36 billion. A $36 billion dollar insured loss is about $72 billion in total damage, and $26 billion insured is about $52 billion. For the RMS forecast to do better than the naive baseline of Pielke et al. (2008) total damage in 2008-2010 will have to be higher than $62 billion ($31 billion insured). That is, losses higher than $62B are closer to the RMS forecast than to the naive baseline.

The NHC official estimate for Katrina is $81 billion. So for the 2006-2010 RMS forecast to verify will require close to another Katrina-like event to occur in the next 3 years, or several large events. This is of course possible, but I doubt that there is a hurricane expert out there willing to put forward a combination of event probability and loss magnitude that will lead to an expected $62 billion total loss over the next 3 years. Consider that a 50% chance of $124 billion in losses results in an expected $62 billion. Is there any scientific basis to expect a 50% chance of $124 billion in losses? Or perhaps a 100% chance of $62 billion in total losses? Anyone wanting to make claims of this sort, please let us know!

From Pielke et al. (2008) the annual chances of a >$10B event (i.e., $5B insured) during 1900-2005 about 25%, and the annual chances of a >$50 billion ($25 billion insured) are just under 5%. There were 7 unique three-year periods with >$62B (>$31B insured) in total losses, or about a 7% chance. So RMS prediction of 40% higher than average losses for 2006-2010 has about a 7% chance of being more accurate than a naive baseline. It could happen, of course, but I wouldn't bet on it without good odds!

So what has RMS done is the face of evidence that its first 5-year forecast was not so accurate? Well, they have declared success and issued another 5-year forecast of 40% higher losses for the period 2008-2012.

Risk Management Solutions (RMS) has confirmed its modeled hurricane activity rates for 2008 to 2012 following an elicitation with a group of the world's leading hurricane researchers. . . . The current activity rates lead to estimates of average annual insured losses that will be 40% higher than those predicted by the long-term mean of hurricane activity for the Gulf Coast, Florida, and the Southeast, and 25-30% higher for the Mid-Atlantic and Northeast coastal regions.

For further reading:

Pielke, R. A., Jr., Gratz, J., Landsea, C. W., Collins, D., Saunders, M. A., and Musulin, R. (2008). "Normalized Hurricane Damages in the United States: 1900-2005." Natural Hazards Review, in press, February. (PDF, prepublication version)

State of Florida Rejects RMS Cat Model Approach

According to a press release from RMS, Inc. the state of Florida has rejected their risk assessment methodology based on using an expert elicitation to predict hurricane risk for the next five years. Regular readers may recall that we discussed this issue in depth not long ago. Here is an excerpt from the press release:

During the week of April 23, the Professional Team of the Florida Commission on Hurricane Loss Projection Methodology (FCHLPM) visited the RMS offices to assess the v6.0 RMS U.S. Hurricane Model. The model submitted for review incorporates our standard forward-looking estimates of medium-term hurricane activity over the next five years, which reflect the current prolonged period of increased hurricane frequency in the Atlantic basin. This model, released by RMS in May 2006, is already being used by insurance and reinsurance companies to manage the risk of losses from hurricanes in the United States.

Over the past year, RMS has been in discussions with the FCHLPM regarding use of a new method of estimating future hurricane activity over the next five years, drawing upon the expert opinion of the hurricane research community, rather than relying on a simplistic long-term historical average which does not distinguish between periods of higher and lower hurricane frequency. RMS was optimistic that the certification process would accommodate a more robust approach, so it was disappointed that the Professional Team was "unable to verify" that the company had met certain FCHLPM model standards relating to the use of long-term data for landfalling hurricanes since 1900.

As a result of the Professional Team’s decision, RMS has elected this year to submit a revised version of the model that is based on the long-term average, to satisfy the needs of the FCHLPM.

This is of course the exact same issue that we highlighted over at Climate Feedback, where I wrote, "Effective planning depends on knowing what range of possibilities to expect in the immediate and longer-term future. Use too long a record from the past and you may underestimate trends. Use too short a record and you miss out on longer time-scale variability."

In their press release, RMS complains correctly that the state of Florida is now likely to underestimate risk:

The long-term historical average significantly underestimates the level of hurricane hazard along the U.S. coast, and there is a consensus among expert hurricane researchers that we will continue to experience elevated frequency for at least the next 10 years. The current standards make it more difficult for insurers and their policy-holders to understand, manage, and reduce hurricane risk effectively.

In its complaint, RMS is absolutely correct. However, the presence of increased risk does not justify using an untested, unproven, and problematic methodology for assessing risk, even if it seems to give the "right" answer.

The state of Florida would be wise to err in the decision making on the side of recognizing that the long-term record of hurricane landfalls and impacts is likely to dramatically understate their current risk and exposure. From all accounts, the state of Florida appears to be gambling with its hurricane future rather than engaging in robust risk management. For their part, RMS, the rest of the cat model industry, and insurance and reinsurance companies should together carefully consider how best to incorporate rapidly evolving and still-uncertain science into scientifically robust and politically legitimate tools for risk management, and this cannot happen quickly enough.

Review of Useless Arithmetic

In the current issue of Nature I review Useless Arithmetic: Why Environmental Scientists Can’t Predict the Future by Orrin Pilkey & Linda Pilkey-Jarvis. Here is my review in PDF. The book's home page can be found here.

Now I've Seen Everything

NASA's Jim Hansen has discovered STS (science and technology studies, i.e., social scientists who study science), and he is using it to justify why the IPCC is wrong and he, and he alone, is correct on predictions of future sea level rise and as well on calls for certain political actions, like campaign finance reform.

In a new paper posted online (here in PDF) Dr. Hansen conveniently selects a notable 1961 paper on the sociology of scientific discovery from Science to suggest that scientific reticence can be used to predict where future research results will lead. And he finds, interestingly enough, that they lead exactly to where his views are today.

What evidence does Dr. Hansen provide to indicate that his views on sea level rise are correct and those presented by the IPCC, which he openly disagrees with, are wrong? Well, for one he explains that no glaciologist agrees with his views (as they are apparently reticent), suggesting that in fact his views must be correct (a creative use of STS if I've ever seen one;-). If holding a minority view is a standard for predicting future scientific understandings then we should therefore apparently pay more attention to all those lonely skeptics crying out in the wilderness, no?

I find it simply amazing that Dr. Hansen has the moxie to invoke the STS literature to support his scientific arguments when that literature, had he looked at maybe one more paper, indicates that Bernard Barber's 1961 essay, while provocative is not widely accepted (see, e.g., this book or this paper). And even if one accepts Barber's article at face value which argues that scientists resist new discoveries (Thomas Kuhn, hello?), what Dr. Hansen doesn't explain (as he is throwing out the IPCC model of scientific consensus) is why his views are those that will prove to be proven correct in the future rather than those other scientific perspectives that are not endorsed by the IPCC. (Dr. Hansen appears to ignore Barber's argument in the same paper suggesting that older scientists are more likely to be captured by political or other interests when presenting their science.)

If we can use the sociology of science to foretell where science is headed, we could save a lot of money not having to in fact do the research. The climate issue is full of surprises and this one just about takes the cake for me. Now I've seen everything!

Cashing In

At least one IPCC lead author appears to be trying to cash in on concern over climate change. With the help of several University of Arizona faculty members, including one prominent IPCC contributor, a company called Climate Appraisal, LLC is selling address specific climate predictions looking out as far as the next 100 years. Call me a skeptic or a cynic but I'm pretty sure that the science of climate change hasn't advanced to the point of providing such place-specific information. In fact, I'd go so far as to suggest that if such information were credible and available, it'd already be in the IPCC. The path from global consensus to snake oil seems pretty short. I wouldn't deny anyone the chance to make a buck, but can this be good for the credibility of the IPCC?

Prediction in Science and Policy

In the New York Times today Corneila Dean has an article about a new book by Orrin Pilkey and Linda Pilkey-Jarvis on the role of predictions in decision making. The book is titled Useless Arithmetic: Why Environmental Scientists Can’t Predict the Future.

Here is an excerpt from the book’s description at Columbia University Press:

Writing for the general, nonmathematician reader and using examples from throughout the environmental sciences, Pilkey and Pilkey-Jarvis show how unquestioned faith in mathematical models can blind us to the hard data and sound judgment of experienced scientific fieldwork. They begin with a riveting account of the extinction of the North Atlantic cod on the Grand Banks of Canada. Next they engage in a general discussion of the limitations of many models across a broad array of crucial environmental subjects.

The book offers fascinating case studies depicting how the seductiveness of quantitative models has led to unmanageable nuclear waste disposal practices, poisoned mining sites, unjustifiable faith in predicted sea level rise rates, bad predictions of future shoreline erosion rates, overoptimistic cost estimates of artificial beaches, and a host of other thorny problems. The authors demonstrate how many modelers have been reckless, employing fudge factors to assure "correct" answers and caring little if their models actually worked.

A timely and urgent book written in an engaging style, Useless Arithmetic evaluates the assumptions behind models, the nature of the field data, and the dialogue between modelers and their "customers."

Naomi Oreskes offers the following praise quote:

Orrin H. Pilkey and Linda Pilkey-Jarvis argue that many models are worse than useless, providing a false sense of security and an unwarranted confidence in our scientific expertise. Regardless of how one responds to their views, they can't be ignored. A must-read for anyone seriously interested in the role of models in contemporary science and policy.

In an interview the authors comment:

The problem is not the math itself, but the blind acceptance and even idolatry we have applied to the quantitative models. These predictive models leave citizens befuddled and unable to defend or criticize model-based decisions. We argue that we should accept the fact that we live in a qualitative world when it comes to natural processes. We must rely on qualitative models that predict only direction, trends, or magnitudes of natural phenomena, and accept the possibility of being imprecise or wrong to some degree. We should demand that when models are used, the assumptions and model simplifications are clearly stated. A better method in many cases will be adaptive management, where a flexible approach is used, where we admit there are uncertainties down the road and we watch and adapt as nature rolls on.

I have not yet read the book, but I will.

Orrin participated in our project on Prediction in the Earth Sciences in the late 1990s, contributing a chapter on beach nourishment. The project resulted in this book:

Sarewitz, D., R.A. Pielke, Jr., and R. Byerly, Jr., (eds.) 2000: Prediction: Science, decision making and the future of nature, Island Press, Washington, DC.

Our last chapter can be found here in PDF.

Ryan Meyer in Ogmius

Ryan Meyer, whose letter to Science we highlighted a few days ago, also has the cover story in our Center's latest newsletter which has just been put online. Ryan's article is titled, "Arbitrary Impacts and Unknown Futures: The shortcomings of climate impact models" and be found here.

The newsletter, called Ogmius, can be found here in html and here in PDF. Have a look!

Limits of Models in Decision

In today’s Financial Times columnist John Kay has a very insightful piece on the limits of models in decision making. He discusses the downfall of Amaranth, a hedge fund, which lost billions of dollars, in part, because its investors did not fully understand the full scope of uncertainties associated with their investment strategies. Kay highlights an important distinction between what he calls “in model” risk and “off model” risk. In model risk refers to the uncertainties that are associated with the design of the model, in data inputs, randomness, and so on. Modelers use techniques such as Monte Carlo analysis to get a quantitative sense of model uncertainties. Off model risk refers to the degree of conformance between a model and the real world. Models by their nature are always simplifications of the real world. As in the case of Amaranth, often hard lessons of experience remind us that as powerful as models are, they can also reinforce bad decisions. As Kay writes,

When someone does attach a probability to a forecast, they have – implicitly or explicitly – used a model of the problems. The model they have used accounts for in-model risk but ignores off-model risk. Their forecasts are therefore too confident and neither you nor they have much idea how over-confident they are. That is why mathematical modeling of risk can be an aid to sound judgment, but never a complete substitute.

Prediction and Decision

Across a number of threads comments have arisen about the role of forecasting in decision making. Questions that have come up include:

What is a good forecast?
When should research forecasts transition to operational forecasts?
What sorts of decisions require quantitative probabilities?
In what contexts can good decisions result without accurate predictions?

It was questions like these that motivated Rad Byerly, Dan Sarewitz, and I to work on a project in the late 1990s focused on prediction. the results of this work were published in a book by Island Press in 2000, titled "Prediction."

With this post I'd like to motivate discussion on this subject, and to point to our book's concluding chapter, which may provide a useful point of departure:

Pielke Jr., R. A., D. Sarewitz and R. Byerly Jr., 2000: Decision Making and the Future of Nature: Understanding and Using Predictions. Chapter 18 in Sarewitz, D., R. A. Pielke Jr., and R. Byerly Jr., (eds.), Prediction: Science Decision Making and the Future of Nature. Island press: Washington, DC. (PDF)

See in particular Table 18.1 on p. 383 which summarizes the criteria we developed in the form of questions which might be used to "question predictions."

Comments welcomed on any of the questions raised above, and others as appropriate as well.