Open Season on Hockey and Peer Review

February 18th, 2005

Posted by: admin

Author: Kevin Vranes (website. email)

The recent 2/14 WSJ article (“Global Warring…” by Antonio Regaldo) addresses the debate that most readers of this site are well familiar with: the Mann et al. hockey stick. The WSJ is still asking – and trying to answer – the basic questions: hockey stick or no hockey stick? But the background premise of the article, stated explicitly and implicitly throughout, is that it was the hockey stick that led to Kyoto and other climate policy. Is it?

I think it’s fair to say that to all of us in the field of climatology, the notion that Kyoto is based on the Mann curve is utter nonsense. If a climatologist, or a policy advisor charged with knowing the science well enough to make astute recommendations to his/her boss, relied solely on the Mann curve to prove definitively the existence of anthropogenic warming, then we’re in deeper trouble than anybody realizes. (This is essentially what Stephan Ramstorf writes in a 1/27 RealClimate post.) And although it’s easy to believe that national and international policy can hinge on single graphs, I hope we give policy makers more credit than that.

But maybe we are in that much trouble. The WSJ highlights what Regaldo and McIntyre says is Mann’s resistance or outright refusal to provide to inquiring minds his data, all details of his statistical analysis, and his code. The WSJ’s anecdotal treatment of the subject goes toward confirming what I’ve been hearing for years in climatology circles about not just Mann, but others collecting original climate data.


As concerns Mann himself, this is especially curious in light of the recent RealClimate posts (link and link) in which Mann and Gavin Schmidt warn us about peer review and the limits therein. Their point is essentially that peer review is limited and can be much less than thorough. One assumes that they are talking about their own work as well as McIntyre’s, although they never state this. Mann and Schmidt go to great lengths in their post to single out Geophysical Research Letters. Their post then seems a bit ironic, as GRL is the journal in which the original Mann curve was published (1999, vol 26., issue 6, p. 759), an article which is now receiving much attention as being flawed and under-reviewed. (For that matter, why does Table 1 in Mann et al. (1999) list many chronologies in the Southern Hemisphere while the rest of the paper promotes a Northern Hemisphere reconstruction? Legit or not, it’s a confusing aspect of the paper that should never have made it past peer review.)

Of their take on peer review, I couldn’t agree more. In my experience, peer review is often cursory at best. So this is what I say to Dr. Mann and others expressing deep concern over peer review: give up your data, methods and code freely and with a smile on your face. That is real peer review. A 12 year-old hacker prodigy in her grandparents’ basement should have as much opportunity to check your work as a “semi-retired Toronto minerals consultant.” Those without three letters after their name can be every bit as intellectually qualified, and will likely have the time for careful review that typical academic reviewers find lacking.

Specious analysis of your work will be borne out by your colleagues, and will enter the debate with every other original work. Your job is not to prevent your critics from checking your work and potentially distorting it; your job is to continue to publish insightful, detailed analyses of the data and let the community decide. You can be part of the debate without seeming to hinder access to it.

3 Responses to “Open Season on Hockey and Peer Review”

    1
  1. Lars Says:

    A couple of considerations wrt to on-demand sharing of data:

    1. Who paid for it? If a granting agency, what do they have to say (probably not much, but still…)? And if it is a grant that you worked off, to whom do you owe first consideration when it comes to dissemination of the data? If the taxpayer is the ultimate bagman, then perhaps your fellow citizens deserve the first chance to examine it. I only mention this last point because Mr. McIntyre is a Canadian, and perhaps ought not to have expected immediate access to data collected and techniques developed at the expense of the American taxpayer.

    2. What are your plans for further analysis? If the data are squeezed dry, as far as your forseeable research is concerned, there should be no objections to disseminating them far and wide. On the other hand, if the data set is part of an ongoing research program, it’s a bit of a strain on collegiality to be expected to yield it up to, well, anybody at all – you have no way of knowing who is going to use it to pre-empt your further research opportunities. This should apply even if the data are collected using government grant money.

  2. 2
  3. Jack Says:

    To Lars;

    1. If the data is propriatary and cannot be widely published, then neither should the research.

    2. If one is deterred by fear of piracy, then wait until the fear subsides.

  4. 3
  5. John A., Says:

    Quoting the article:
    The WSJ’s anecdotal treatment of the subject goes toward confirming what I’ve been hearing for years in climatology circles about not just Mann, but others collecting original climate data.

    As concerns Mann himself, this is especially curious in light of the recent RealClimate posts (link and link) in which Mann and Gavin Schmidt warn us about peer review and the limits therein. Their point is essentially that peer review is limited and can be much less than thorough. One assumes that they are talking about their own work as well as McIntyre’s, although they never state this. Mann and Schmidt go to great lengths in their post to single out Geophysical Research Letters. Their post then seems a bit ironic, as GRL is the journal in which the original Mann curve was published (1999, vol 26., issue 6, p. 759), an article which is now receiving much attention as being flawed and under-reviewed.

    Actually the word I would use is not “ironic”. It is “chutzpah”.

    It is chutzpah to claim that GRL’s peer review process is flawed when Mann’s original papers were published there and from which Mann and his acolytes started claiming their work was “peer reviewed” and represented the “scientific consensus”.

    It was not chutzpah when Mann called up the editor of GRL and tried to get him to stop the Mcintyre and McKitrick analysis published. That word is “censorship”, and is to my mind, a revealing act of desperation. If the M&M analysis was indeed flawed, then publication would be the best way to reveal it in a way that could not be denied.

    That’s a big “if”

    John A.