The Case for Scientific Assessments

October 20th, 2005

Posted by: admin

**Post by Andrew Dessler

It has been argued on this web site that it is impossible to receive advice on science independent of political considerations. I disagree and suggest in this post a process for how it might be achieved. The process relies on scientific assessments: summaries of the scientific literature that are produced by expert scientists. Assessments connect the domains of science and democratic politics, but are distinct from both. They differ from science because rather than advancing the active, contested margin of knowledge on questions that are important for their intrinsic intellectual interest, they seek to make consensus statements of present knowledge and uncertainty on questions that are important because of their implications for decisions. They differ from democratic policy debate because they reflect deliberation over questions among scientific experts based on their specialized knowledge, not among all citizens or their representatives over what is to be done.

Here’s how the process would optimally work. Policymakers would determine the positive (scientific) questions of importance to them on some issue. This would likely be an iterative process, where scientists and policymakers together identify which scientific issues are most important for a particular issue. For the climate debate, the important questions might be: 1) is the Earth warming? 2) are human activities to blame? 3) what kind of warming do we expect over the next century? These questions would then be passed to the assessment body, which would use the existing peer-reviewed literature to determine the scientific consensus on those issues, and produce from that a report that is itself peer reviewed by outside experts. A good example of this process in action is the National Academy review of the IPCC Working Group I report initiated by the Bush Administration in 2001. The White House provided a list of questions, and the NAS panel responded to them.

Under this model, the questions investigated by the scientific assessment might have an obvious political agenda. This would not invalidate the non-political “honest broker” status of the assessment, however, since the political agenda is that of the policymakers who determined the questions, not the assessment scientists.

In one of Pielke’s “honest broker” posts, he posed a question about giving advice on where to eat. According to the assessment model described here, if a policymaker asks “where should I eat?”, an expert advisor should balk. Answering this question clearly requires the expert make some value judgments about what kind of dinner the policymaker wants. Rather, the expert should insist that the policymaker reconfigure the request into a series of specific positive questions that can be addressed by facts alone, without any value judgments by the expert: “Where is the nearest vegetarian restaurant?” “Which restaurants have entrees under $20?” “Where can I get a dinner that follows the Atkin’s diet?” etc. Such questions can be answered without the expert assuming any normative values because all necessary value choices were made by the policymaker in formulating the questions (e.g., decisions such as whether the dinner should be vegetarian or not).

One key assumption underlying this model is that policymakers are able to decide on a set of specific, well-phrased, positive questions for the assessment body to evaluate. For the climate change problem, this would require a values debate between and within national governments, which I believe would be extremely beneficial for the overall debate. It might be particularly beneficial to entrain skeptics like Joe Barton into the process of defining questions. This would allow them to pose their favorite questions to the assessment panel — e.g., Does the disagreement between the surface and satellite record invalidate the surface record? Is there any merit to Lindzen’s “iris” hypothesis? etc., etc., etc. (FYI, the answers are no and no).

For climate change policy, in the end I believe that it is possible to identify a set of pertinent scientific questions that most (but not all) policymakers would agree are the important questions for determining a policy — although this is supposition on my part. I should also note that defining a question is no guarantee that the scientific community can provide a good answer. Policymakers need to realize that policy still needs to be made, even if science cannot tell them everything they want to know. Uncertainty is a fact of life — deal with it!

Once one decides on a set of scientific questions, the IPCC Working Group I provides a good example of how the scientific community answers specific scientific questions without regard to the scientists’ normative values. [note: I’m not arguing that the IPCC TAR does or does not follow this model, just that the questions addressed in the Working Group I report are all answerable entirely from scientific facts.]

To summarize, if one could get all policymakers to agree to use assessments derived according to this model, then I believe that many of the problems of the politicization of science would disappear. I would be incredibly surprised, however, if it ever happened. There are simply too many incentives for policymakers to argue about science for them to willingly give up the opportunity.

7 Responses to “The Case for Scientific Assessments”

  1. Dylan Otto Krider Says:

    I like it. Though, I assume the political wrangling would be over what questions you sent over. And it would be interesting to see the political breakdown on who would oppose such an idea, since as you say, many people have an incentive to argue the science. Whether or not you could get the votes, it would be a valuable debate to have.

  2. 2
  3. Eben Polk Says:

    This approach may be represented well I think by the new Nicholas Institute for Environmental Policy Solutions. Time will tell. Interested folks should check out

  4. 3
  5. Mark Bahner Says:

    “For the climate debate, the important questions might be: 1) is the Earth warming? 2) are human activities to blame? 3) what kind of warming do we expect over the next century?”


    1) Yes.

    2) Yes…but to what extent is debatable. (This is not a “yes/no” question, since human influence on the degree to which the earth is warming can range zero percent to 100 percent.) (And this does not even address WHICH human activities might be responsible for the warming, e.g. what is the role of black carbon, or methane, or land use changes, versus CO2 emissions?)

    3) THERE is your whole enchilada! THAT is the question–”what will happen in the future?” (especially “with or without government intervention”)–that’s the most important question of all.

    And a panel of experts, followed by a review panel of MORE experts, doesn’t get away from the fundamental conflict of interest that exists. The conflict of interest is that the experts can accrue more money and fame to their area of expertise by exaggerating both the warming that will occur, and the magnitude of the ill effects that will occur. (Not to mention their conflict of interest to also ignore any benefits that might occur from a warming world.)

    This would be much better than a panel of experts, followed by another panel of experts:

    1) Get 30 scientists. Require that every single one of the scientists provide “50 percent probability” predictions for each of the parameters of interest (e.g., methane atmospheric concentrations, CO2 emissions and atmospheric concentrations, and resultant temperature increases).

    2) Pay the experts in direct relationship to the accuracy of their predictions. For example, let’s say each of the experts has provided a prediction for a) methane atmospheric concentration, b) CO2 emissions, c) CO2 atmospheric concentrations, d) world surface temperature, and e) world lower tropospheric temperature, every year from 2005 to 2030. Each year, pay the one who is closest on each parameter $40,000, pay the one who is second $20,000, pay the one who is third $10,000, and pay the rest nothing.

    Assuming we use the 5 parameters that I mentioned above, that would be a total reward of $350,000 per year, for the next 25 years. That represents a total payout of $8.75 million. (That’s probably less than 1/50th spent by the IPCC to date.)

    I guarantee that such a system would produce more accurate predictions than the IPCC has produced to date! (That’s partially due to the fact that, as of the IPCC TAR, the IPCC has produced NO “predictions”…only pseudoscientific “projections.”)

    The only problem with my method (from the standpoint of the “climate change community”!) is that the resultant predictions would probably be so low that fear of global warming would be considerably diminished.

    P.S. An even better way to do it would be to permanently drop the 15 least accurate scientists from the program each year, and to replace them with 15 more. Then allow all 30 scientists to re-predict for the remaining years.

    (Heh, heh, heh! Imagine that…climate scientists actually being rewarded for accurate predictions, and punished for inaccurate predictions!)

  6. 4
  7. Mark Bahner Says:

    Another thought:

    1) Start the awards much lower, but end the awards much higher. For example, for the first 10 years, award only $20,000, $10,000 and $5,000 for each parameter, but for the last 10 years award $60,000, $30,000, and $15,000 for each parameter.

    2) Replace only the worst 10 every 5 years, bringing on 10 more. And allow everyone to change their predictions every 5 years. However, for the new people, and the people who change their bets, they don’t get the big bucks at the end, if they’re closest. Instead, they get the little bucks like is awarded for the early years.

    But all these changes are really just gilding the lilly. The main problem is addressed either way. Either way repairs the problem caused by the conflict of interest, wherein ridiculously high predictions (or “projections”) are rewarded with more money, and accurate predictions aren’t rewarded at all.

  8. 5
  9. Andrew Dessler Says:


    I’m sure I’ll regret asking this, but what evidence do you have that climate scientists are intentionally making bad predictions to increase their funding?


  10. 6
  11. Daniel Collins Says:

    The scientific method, and all its rewards, would be the worse for wear with such a scheme.

  12. 7
  13. Rabett Says:

    Mark’s #1 is silly, because not all inputs are equally important. 2 fails for the same reason.