Real Climate’s Two Voices on Short-Term Climate Fluctuations
January 11th, 2008Posted by: Roger Pielke, Jr.
Real Climate has been speaking with two voices on how to compare observations of climate with models. Last August they asserted that one-year’s sea ice extent could be compared with models:
A few people have already remarked on some pretty surprising numbers in Arctic sea ice extent this year (the New York Times has also noticed). The minimum extent is usually in early to mid September, but this year, conditions by Aug 9 had already beaten all previous record minima. Given that there is at least a few more weeks of melting to go, it looks like the record set in 2005 will be unequivocally surpassed. It could be interesting to follow especially in light of model predictions discussed previously.
Today, they say that looking at 8 years of temperature records is misguided:
John Tierney and Roger Pielke Jr. have recently discussed attempts to validate (or falsify) IPCC projections of global temperature change over the period 2000-2007. Others have attempted to show that last year’s numbers imply that ‘Global Warming has stopped’ or that it is ‘taking a break’ (Uli Kulke, Die Welt)). However, as most of our readers will realise, these comparisons are flawed since they basically compare long term climate change to short term weather variability.
So according to Real Climate one-year’s ice extent data can be compared to climate models, but 8 years of temperature data cannot.
Right. This is why I believe that whatever one’s position of climate change is, everyone should agree that rigorous forecast verification is needed.
Post Script. I see at Real Climate commenters are already calling me a “skeptic” for even discussing forecast verification. For the record I accept the consensus of the IPCC WGI. If asking questions about forecast verification is to be tabooo, then climate science is in worse shape than I thought.
January 11th, 2008 at 4:39 am
Just documenting my exchange with Gavin Schmidt at Real Climate:
Gavin’s response to my first comment:
————————————–
Response: Roger, How can you read the post above and claim that we’re avoiding forecast verification altogether? This is just perverse. I spend almost all of my time comparing observations with predicted variables, so I don’t see what you are getting at in point 4 at all. The point being made is that each comparison has varying degrees of usefulness. 8 year trends in the global mean temperatures are not very useful. 20 year trends are more so. Better still are superposed averages of the response to volcanoes or ENSO variability etc. In each case, models predict a mean response and an estimate of the noise.
Dealing with your individual points though, IPCC may have been published in 2007, but the model runs reported were made in 2004 in most cases, similarly, Hansen’s paper was published in 1988, but the model runs were started in 1985, thus forecast validation can usefully be made from when the runs started, rather than when they were published. Secondly, the pictures you and John showed were not correct if you wanted to do a short term forecast simulation. The line you took from IPCC was not the envelope, or even a fair distribution of the simulations over that short period. You would find that the actual simulations would have a substantially greater error bar (as in the distribution of 8 year trends we mentioned above). Instead you used the IPCC estimate of the long term trend (which has a much smaller uncertainty). This data is easily available, and if you want to do forecast validation properly you should use it. The response you’ve got from other scientists is precisely because they are aware of this.
I’m not really sure what was being forecast in 1992, and I’ll have to look it up before responding.
You talk about the absence of short-term validation as being unscientific. This is simply ridiculous. It is easy to see that unpredictable weather noise dominates short term variability. It is well known that this is unpredictable more than a short time ahead. Claiming that the forced climate response must be larger than the weather noise for climate prediction on all time scales is just silly. There are examples where it is – for instance in the response to Pinatubo (for which validated climate model predictions were made ahead of time – Hansen et al 1992) – but this is not in general going to be true. A bigger point is that ‘predictions’ from climate models do not just mean predicting what is going to happen next year or the next decade. They also predict variables and relationships between variables that haven’t yet been measured or analysed – that is just as valid a falsifiability criteria. They can test hypotheses for climate changes in the past and so on. The statistics of the weather make short term climate prediction very difficult – particularly for climate models that are not run with any kind of initialization for observations – this has been said over and over. Why is this hard to understand? – gavin
—————–
My reply:
—————–
# Roger Pielke. Jr. Says: Your comment is awaiting moderation.
11 January 2008 at 12:08 PM
Gavin (7)-
Thanks, a few replies:
You ask “How can you read the post above and claim that we’re avoiding forecast verification altogether?”
Well, my first clue is when you misrepresented why I did, and what John Tierney reported. You characterized the effort as “attempts to validate (or falsify) IPCC projections of global temperature change over the period 2000-2007.” No such claims were made by me, and I don’t think by Tierney.
In my post I was careful to note the following: “I assume that many climate scientists will say that there is no significance to what has happened since 2000, and perhaps emphasize that predictions of global temperature are more certain in the longer term than shorter term.”
And John Tierney wrote: “you can’t draw any firm conclusions about the IPCC’s projections — a few years does not a trend make”
So why misrepresent what we said? Models of open systems cannot in principle be “validated” (see Oreskes et al 1994).
I simply compared IPCC predictions with observations as an example of how to do a verification, which is standard practice in the atmospheric sciences, but much less so in the climate modeling community (and yes, I think this is indeed the case). Instead of telling your readers all of the reasons that a verification exercise is “misguided” you might have instead constructively pointed to the relevant forecasts with proper uncertainty bars (please do post up the link), or better yet, simply shown how an analysis comparing 2000-2007 with relevant predictions would have been done to your satisfaction.
Given that you point to the IPCC AR4 Figure 1.1 in positive fashion, I remain confused about your complaint about what I did — I don’t recall you complaining about IPCC efforts in verification previously.
How about this: We agree that rigorous forecast verification is important. There also does not a clear agreement among researchers as to (a) what variables are most important to verify (b) Over what times scales, (c) what actual constitutes the relevant forecasts, and (d) what actually constitutes the relevant observational verification databases. Then this is a subject to work through collegially, rather than try to discredit, dismiss, or suppress.
Thanks.
PS. As I stated on my blg. If discussing forecast verification in the context of climate model predictions is to be a sign of “skepticism,” then climate science is in bad shape. For the record I accept the consensus of IPCC WG I.
January 11th, 2008 at 5:43 am
As is typical at Real Climate, rather than engaging substance they direct me to download some data to see what I can learn. I have no clue as to why they are complaining. Anyway, here is my last response to Gavin:
————
Gavin-
Thanks but this is a pretty lame response: “In a collegial spirit, I suggest you download the model data directly from PCMDI and really look at what you can learn from it.” You are the climate scientist no? If you are unwilling to explain what is substantively wrong is my efforts to provide an example of forecast verification, then so be it.
I am quite confident in my conclusions from this exercise as summarized from my blog Prometheus, and nothing that you say here contradicts those conclusions whatsoever:
1) Nothing really can now be said on the skill of 2007 IPCC predictions.
2) By contrast IPCC dramatically over-predicted temperature increases in its 1990 report.
For 1995, 2001 (and some interesting surprises) please tune in next week.
Gavin, if you do decide to provide substantive critiques of the two conclusions above please do share them, as I still have absolutely no idea what your complaint about this exercise actually is, other than the fact that it took place.
——————–
January 11th, 2008 at 7:27 am
Readers interested in the full exchange that I have had with Real Climate can head over there and have a look.
Once the conversation starts to turn to talk of “deniers” (as all RC discussions eventually wind up it seems) it is time for me to check out! I have no clue why they reacted so strongly in attack mode. For a long time I have thought that RC has simply been a front for politicized science, now I think that it has become more of a platform for shallow point scoring and targeted criticism. Too bad.
Anyway, I’ll continue this exercise next week, please tune in Monday for a discussion of the IPCC 1995 and 2001 predictions!
January 11th, 2008 at 9:34 pm
Realclimate is indeed a far cry from a voice of reason. Anyone who censors the hard questions as they do is not doing civilization any favor by their existence.
Those who don’t believe that are either unable or unwilling to ask those hard questions. For example, ask them this question: “Is it true that random numbers put into the MBH98 procedure usually generate a hockey stick shape?”
If you get it posted you will be the first.