Hypotheses about IPCC and Peer Review

January 27th, 2006

Posted by: Roger Pielke, Jr.

The IPCC is the 800 pound gorilla in the climate debate. It has been the locus of legitimate and credible climate science (salience is another matter, but I digress). It is increasingly coming under criticism in a number of dimensions for some very good reasons. In this post I’d like to suggest a few hypotheses about how the IPCC has indirectly contributed to the politicization of climate science in ways we’ve not discussed here. These are for discussion, and I’d welcome evidence for/against and other sorts of examples.


Laundering Grey Literature

The IPCC has a requirement that its assessments be based on peer reviewed literature. It has not always held itself to this standard, particularly in its Working Groups II and III. I have noticed recently a number of peer-reviewed papers that reference so-called “grey literature” (e.g., agency, company, NGO reports) which hasn’t itself been peer reviewed. Then the peer-reviewed study that cites the grey literature is subsequently cited in another publication to refer to the information in the original non-peer reviewed source. This is a way to give the veneer of peer review to a non-peer-reviewed study. Here is an example of this dynamic.

Fun with Deadlines

The IPCC sets a deadline for when papers must be accepted in order to be considered in a particular assessment report. This guarantees that the assessment won’t have to be continually updated, but it also means that the assessment is automatically out-of-date in some case where new findings have been released. Because editors and journals have considerable discretion in when they publish what papers, the IPCC’s deadline can set the stage for some mischief in the publication process as papers with a particular slant are published before the deadline and other published after. I don’t have any data on this, but it’d be interesting to compare the time-to-publication of key papers cited in IPCC reports with a journal’s standard practices. This issue came to mind as I read this comment from RealClimate,

There are several more papers “in the mill” which we are not at liberty to discuss right now [Ed.- Embargoed, see below], which insure that the weight of peer-reviewed studies available for consideration in the next IPCC report will point towards a strengthening, not a weakening, of the IPCC ‘01 conclusions regarding the anomalous nature of recent hemispheric and global warmth in a long-term context.

Maybe it is just inartful language, but claims to “insure” previously found results do not make me comfortable about the agendas of climate scientists.

Embargoes as Silencers

This one is not about the IPCC, but Science and Nature. I was recently at a science talk at NCAR and a number of leading scientists refused to discuss their work because it would potentially be under “embargo” with Science or Nature, if accepted. My understanding is that embargoes refer to releasing papers accepted for publication to the media in advance of the artificial deadlines set by Science and Nature to generate news-worthiness. They do not apply to scientists talking among themselves in scientific settings. So when scientists use potential embargoes as a way to silence discussion and debate on their work, it reduces the internal vetting of scientific ideas and makes the leading journals the only place where debate can occur. Since Science and Nature are highly selective is what they allow as far as intellectual exchanges following up papers they publish, the entire process of scientific debate and learning is arguably slowed down. Meantime, this allows findings supporting one view or another to gain much greater standing in political debate than they might otherwise have.

Comments? Other observations?

12 Responses to “Hypotheses about IPCC and Peer Review”

    1
  1. Harold Brooks Says:

    I know that the Journal of Climate received something on the order of 300 submissions in May, the last month that papers could be submitted and be considered for inclusion in IPCC WG I, when there normal submission rate is (I think) ~25 per month. Given that the paper had to be accepted and have page numbers by the end of December, there’s no way that a significant number of those could make it through the review process at an AMS journal and get page nubmers in time. I’m not even sure how editors could find enough reviewers to review that load. The editors were looking to send whatever was remotely appropriate over to over AMS journals, but it was a situation that overwhelmed the system.

  2. 2
  3. Luke Lea Says:

    Judging by your examples there is something rotten in the state of Denmark. Why then do you say, “I am a strong advocate for policy action on climate change.”

    Where can general readers like myself — not expert, but science oriented, and following the debate closely — find evidence that realistic policy initiatives can be expected to have a significant impact on the future of global warming?

    I can’t even find articles that address the issue — as opposed to, say, articles that argue the relative merits alternative mechanisms for reducing carbon dioxide emissions around the world without reference to the issues of affordability and ultimate impact.

    Ozone depletion passed this test, why not global warming?

    Put another way: do the guys up at NCAR really think there are things we can do that will make a real difference, or are they just whistling Dixie?

    To me this looks like the 800 pound gorilla in the room.

    thanks for any help you can give

  4. 3
  5. William Connolley Says:

    Embargoed wok? Hot stuff!

  6. 4
  7. James Annan Says:

    Roger,

    While I agree that these issues are interesting ones, the real question is, how could it be done better? Given that an assessment report on this scale is desired, the authors need a timetable to work to, and some parameters to work with.

    I have commented previously on how disproportionate I think the influence of the IPCC is on the science that is performed, especially since they don’t actually fund it. But on the plus side, the timetable and pull for contributions has surely pushed forward the rate of scientific research beyond that which would have otherwise existed.

    And it’s simply not fair of you to bring up the Science/Nature embargo in the same article – this is an entirely separate business.

  8. 5
  9. Roger Pielke Jr. Says:

    James-

    Thanks for these comments.

    There are a lot of potential ways that things could be done differently, and maybe these would be better, maybe not. But every organization should routinely be evaluated and compared to possible alternatives. We should not fall into the status quo trap — that things are the way they are because they represent the best possible outcome.

    As examples of questions that might be raised:

    *Why have working groups?
    *Why these three?
    *Why not focus on policy recommendations?
    *Why focus on massive assessments?
    *Why not a “Wiki” model for assessment, updated continuously?
    *Why not start fresh each time with questions solicited from policy makers, and then issue smaller reports as appropriate?
    *Why not use expert elicitation as an alternative/complement?
    *Why not have people uninvolved in producing the research charged with summarizing it?
    *Why not oganize assessments under policy bodies?
    *Why not use assessments to do research not just summarize it?
    *Why not use the NOAA RISA model? US NACC model? NRC model? Others?

    In short there are many, many questions that might be asked about the IPCC, but aren’t mostly because everyone who cares has either a vested interested in preserving the status quo or attacking it. The answers to the questions above implicate a whole spectrum of different approaches to the time from/peer review issue.

    As far as you last point, the Science/Nature embargo is relevant to the IPCC because scientists use it as an excuse not to engage in scientific debate that might result in perspectives other than their being aired, and thus not represented in the peer reviewed liertature and not available on the time schedule of the IPCC. But you are ight that it is a much broader issue than IPCC or climate science.

  10. 6
  11. Rabett Says:

    AFAIK Science and Nature consider seeing it in the press as “prior”. They extend this to press releases and maybe even blogs, and reporters do walk around AGU meetings, believe it or not, and university press officers are always looking for stuff to brag on.

    Roger, your questions are shallow. Let us look at the first three:

    *Why have working groups?
    *Why these three?
    *Why not focus on policy recommendations?

    Have you perhaps read:

    Structure and Agent in the Scientific Diplomacy of Climate Change: An Empirical Case Study…
    by Tora Skodvin -

    or perhaps

    Confronting Climate Change: Risks, Implications and Responses
    by Irving Mintzer

    and any number of article in various journals or are you simply trying to blow us off by generating an endless list?

  12. 7
  13. Roger Pielke Jr. Says:

    Eli- Thanks for your comments. I’m sure our readers would welcome your summaries of this literature and your views of its relevance to this issue.

  14. 8
  15. Jeff Norman Says:

    Roger,

    An addition to your list might be recommendations from the IPCC to policy makers regarding climate research in their countries and/or regions that could be funded or otherwise facilitated to promote a better understanding of climate systems. This would help fill in some of the holes in our understanding.

    It is apparent to me that numerous weather station records ended or became incomplete since the start of the recent warm period. At the very least the IPCC could be recommending to policy makers that these records be maintained.

    Jeff

  16. 9
  17. Roger Pielke, Jr. Says:

    Jeff- Excellent point!

  18. 10
  19. Rabett Says:

    Roger, for a guy who posts weekly reading lists, you sure do play a mean game of you do it. Come on, you know that the questions you asked have been pretty much addressed, but you tried to put James off by giving him a list to wade through.

    The one about doing a Wiki for assessment was particularly amusing in a “is he silly enough to think that anyone would fall for that” sort of way.

  20. 11
  21. James Annan Says:

    Actually I think a (controlled) wiki between the relevant people could possibly be a very much more efficient and effective method than the current one. But OTOH I’m going to play my standard card that it’s not reasonable to blame the “at the coal-face” scientists for merely getting on with the task at hand, and deadlines and peer-reviewed “quality control”, although not perfect, are not obviously wrong in that respect.

  22. 12
  23. kevin Says:

    The grey lit issue was one of my strongest criticisms in the two AR4 WGII chapters I reviewed. Perhaps ‘appalled’ is too strong of a word, but I was unimpressed to the point of annoyance by the reliance on NGO reports to make some points and a conspicuous lack of citation in making others. The message I have been taking from this (although I am waiting on the revised chapters) is that the knowledge base just isn’t there on some issues that were tagged to be covered in AR4. But rather than report, “We have no idea,” the lead authors must feel that they have to give *something*.