The assessors assessing the assessments
March 6th, 2007Posted by: admin
Fresh out of the National Academies, commissioned by the CCSP, is a fabulous new climate-related assessment: Analysis of Global Change Assessments: Lessons Learned. The report
identifies for the U.S. Climate Change Science Program the essential elements of effective global change assessments, including strategic framing, engagement of stakeholders, credible treatment of uncertainties, and a transparent interface between policymakers and scientists. The report reviews lessons learned from past assessments, which are intended to inform policymakers about the scientific underpinnings of critical environmental issues such as climate change, loss of biodiversity, and ozone depletion.
Which would be great, but. But for two things we can identify right off the bat:
1- The most identifiable end user of a climate change assessment is the federal-level (and perhaps state-level) policy maker. You’d think that if you want to assess assessments and make sweeping recommendations on how in the future they can best be presented and utilized, you’d involve the very end users that the assessments target. But the participant list is a roster dominated by the very people who produce the information, not the people who consume it. The report did hear from a few end users in a couple of sessions, but the list (pg. ix-x of the full report) is very thin. So how are the producers to know what the consumers really need if the consumers were not intimately involved in the project? Which leads us to …
2- Many of the people assessing the assessments are themselves involved in the original assessments. Further, the reviewers of the report (pg, ix) are all themselves information producers (save one). This is fine, but it leads me to wonder about the ultimate usefulness of the report. The point of this exercise should be to ensure that the information produced in assessments is useful to the end users. (That is mentioned as a goal, but as one goal in a list.) I’m not sure that scientists, essentially auditing themselves, are the best judge of whether their information is maximally useful to non-technical decision makers.
Nevertheless, it is important that we assess how we are giving technical information to decision makers. My argument is that we need to very closely involve decision makers themselves in that process. From what I can tell in the report’s preface and other front matter, this was mostly a scientist-driven process.
March 6th, 2007 at 2:15 pm
Quis custodiet ipsos custodes?
March 7th, 2007 at 8:55 am
This is the perennial challenge with all assessments. In 20 years, I can’t point to any major assessment that was truly end-user driven and/or particularly useful to end-users. I work a lot with end-users on a daily basis and its challenging. Each one has specific needs, many of which aren’t or can’t be addressed by GCMs. For example, corn producers have about a two-week window in heat stress that is very critical, right at pollination. That window depends on corn variety and when it was planted. So you have a moving target and a focus on extreme events during a short time frame.
A few years ago Stan Changnon did a study with a local utility to understand how they might use seasonal forecasting. You had to identify the decision-makers in the company, then you had to understand their operational needs, and then you had to try to match what you had available to their needs (and there is no guarantee that there is a match). It was very time consuming and that was only one company.
So, it takes a lot of work and requires continuous two-way communication to keep the end-users involved. I don’t think that happens too often.
Jim
March 8th, 2007 at 6:42 am
I don’t think it is controversial to say that this problem will always arise where there are two or more spheres of action (in this case, the scientific, academic, political, media and public), which are distinguished by both the methods and criteria of evaluation and by the language registers they use.
What constitutes a successful communication act to a scientist is fundamentally different to the same for a policy maker, or almost any non-scientist. Likewise, a report writer might be satisfied that the report expressed what was intended, even when the consumer has misunderstood or misinterpreted the material.
But all of the players in the science/policy/media debate are in a double-bind: make it too detailed, and your consumer may well lose the ability to make meaning; make it too simple, and you stand open to challenges of incompleteness, inaccuracy or unfairness, as well as the use of linguistic ‘tricks’ to take advatage of the different usages and meanings of language in different registers. And a clever ‘player’ in any one sphere of action can take advantage of the uncertainties inherent in language use difference to cast doubt on the veracity of statements where doubt probably doesn’t really exist.
if this is so, then the burden of responsibility – and the moral burden – for effective communication passes to the intermediary, the fortunate individual who can straddle the different spheres of action, manage the various language registers, and operate outside the confinement of evaluation by any one peer-group. For those whose place it is to make the meanings for the consumers, the imperative thus exists to offer up such interpretations as honestly, fairly and with as little prejudice as is possible within the bounds of their own personalities. You must say it as you see it and not be afraid to be challenged.
Regards,
March 12th, 2007 at 5:39 pm
I generally refer to this as a problem of “getting the right science.” Unfortunately, many of the end users of national and global level assessments still think it is just about “getting the science right.” And they are the ones that order up the science. The NAS is one of the few institutions that can produce a study that says the wrong questions are being asked, and still get paid for it. Ultimately it is an institutional problem, of getting science to support a place-based approach – an area where the natural sciences could learn a few lessons from the social sciences about dealing with complexity and still assuring quality of information. I will write more on this eventually but, I always wonder what the Millennium Ecosystem Assessment might have looked like had it been initiated by geographers, like Gilbert White…