Comments on: Even More: Mr. Issa’s Confusion and a Comment on Budget Politics http://cstpr.colorado.edu/prometheus/?p=4085 Wed, 29 Jul 2009 22:36:51 -0600 http://wordpress.org/?v=2.9.1 hourly 1 By: Richard Belzer http://cstpr.colorado.edu/prometheus/?p=4085&cpage=1#comment-7956 Richard Belzer Thu, 01 Feb 2007 12:57:53 +0000 http://sciencepolicy.colorado.edu/prometheusreborn/?p=4085#comment-7956 I'm not a fan of judging scientific work based on source of funding; I have stated elsewhere that it should be judged based on scientific quality. That said, the financial, commercial, policy, or ideological interests of sponsor ought to be taken into account because it provides clues concerning how it might have rigged the study. On top of that, polls and surveys are notoriously susceptible to sponsor manipulation. For that reason alone, it is absurd to take the UCS survey results at face value, as you have done. After all, you have shown that UCS conflated disputes about budget priorities with policy interference. But (perhaps only strategically) you have refrained from stating the obvious: if budget disputes are construed as policy interference, every university administration is guilty of it. Where is the outrage? With regard to the exchange with Rep. Issa, I agree with you that statistical power is not the issue. However, I think you have dismissed (or ignored) a genuine issue. The problem is that the results are representative of nothing. They cannot be generalized to any population because the prospect of nonresponse bias is overwhelming (279 responses out of 1,630 surveys distributed). If UCS was actually a scientific organization, it would of course know this and would not interpret the survey results as it has done. The most UCS can credibly say is that it has uncovered isolated and anonymous allegations made by individuals with a variety of possible motives, including incentives to lie. So it matters greatly whether the allegations are valid. But UCS treats all allegations are presumptively true; why a genuinely scientific organization would do this is hard to fathom. Surely it would not be hard to find a handful of government scientists who hold strong policy views willing to falsely claim that their scientific work had been subject to interference simply because their policy views did not prevail. In short, responses to the UCS survey may well be contaminated by political interference -- by the respondents! Taking a look at the UCS survey itself ("Atmosphere of Pressure: Appendix A"), it asks only for opinions -- not evidence. Worse, it asks for opinions about respondents *perceptions*: "I have perceived in others and/or personally experienced the following types of activities affecting climate science:" It's hard to define political interference any more broadly than that, which is exactly what UCS should be expected to do given its ideology. Given these embedded biases in the instrument, what's remarkable about the UCS survey results is that so many respondents DENIED experiencing or even perceiving "interference." Across each of the questions 19 through 31, a majority of respondents said that they neither perceived nor experienced the type of "interference" specified. These majorities ranged from a low of 54% ("Pressure to eliminate the word(s) “climate change” and/ or “global warming,” and/or similar terms"; "New or unusual administrative requirements or procedures that impair climate-related work") to a high of 89% ("Requests by officials for scientists to provide incomplete, inaccurate, or misleading information to the public"). That's pretty remarkable, actually. I’m not a fan of judging scientific work based on source of funding; I have stated elsewhere that it should be judged based on scientific quality.

That said, the financial, commercial, policy, or ideological interests of sponsor ought to be taken into account because it provides clues concerning how it might have rigged the study. On top of that, polls and surveys are notoriously susceptible to sponsor manipulation.

For that reason alone, it is absurd to take the UCS survey results at face value, as you have done. After all, you have shown that UCS conflated disputes about budget priorities with policy interference. But (perhaps only strategically) you have refrained from stating the obvious: if budget disputes are construed as policy interference, every university administration is guilty of it. Where is the outrage?

With regard to the exchange with Rep. Issa, I agree with you that statistical power is not the issue. However, I think you have dismissed (or ignored) a genuine issue. The problem is that the results are representative of nothing. They cannot be generalized to any population because the prospect of nonresponse bias is overwhelming (279 responses out of 1,630 surveys distributed). If UCS was actually a scientific organization, it would of course know this and would not interpret the survey results as it has done.

The most UCS can credibly say is that it has uncovered isolated and anonymous allegations made by individuals with a variety of possible motives, including incentives to lie. So it matters greatly whether the allegations are valid. But UCS treats all allegations are presumptively true; why a genuinely scientific organization would do this is hard to fathom. Surely it would not be hard to find a handful of government scientists who hold strong policy views willing to falsely claim that their scientific work had been subject to interference simply because their policy views did not prevail. In short, responses to the UCS survey may well be contaminated by political interference — by the respondents!

Taking a look at the UCS survey itself (“Atmosphere of Pressure: Appendix A”), it asks only for opinions — not evidence. Worse, it asks for opinions about respondents *perceptions*:

“I have perceived in others and/or personally experienced the following types of activities affecting climate science:”

It’s hard to define political interference any more broadly than that, which is exactly what UCS should be expected to do given its ideology.

Given these embedded biases in the instrument, what’s remarkable about the UCS survey results is that so many respondents DENIED experiencing or even perceiving “interference.” Across each of the questions 19 through 31, a majority of respondents said that they neither perceived nor experienced the type of “interference” specified. These majorities ranged from a low of 54% (“Pressure to eliminate the word(s) “climate change” and/ or “global warming,” and/or similar terms”; “New or unusual administrative requirements or procedures that impair climate-related work”) to a high of 89% (“Requests by officials for scientists to provide incomplete, inaccurate, or misleading information to the public”). That’s pretty remarkable, actually.

]]>
By: Margo http://cstpr.colorado.edu/prometheus/?p=4085&cpage=1#comment-7955 Margo Wed, 31 Jan 2007 17:35:15 +0000 http://sciencepolicy.colorado.edu/prometheusreborn/?p=4085#comment-7955 I also noticed a size-able amount of the proof of interference described in the UCS document was related to "not enough funding". (Links here: http://oversight.house.gov/story.asp?ID=1162&Issue=Environment ) I also noticed some of the "objectionable practices" affecting the integrity of climate science are actually rather routine practices at national labs, the practices aren't secret, they aren't specific to climate change and they pre-exist the current administration. ( I suspect many are older than dirt.) For example, on page 29 of "atmosphere of pressure", the authors mention federal agencies require pre-approval of media interviews between scientists and the media. Of course many do. DOE labs generally require pre-approval when any employee -- whether secretary of scientist-- talks to the media; this has gone on for a long time particular with regard to work affecting the public safety or defense. Are climate scientists under the impression that scientists and engineers working in areas related to stored radioactive waste, military defense projects, or public health issues are told they may just schedule their own interviews, write their own press releases and publicize to their hearts desires? In reality, employees at some labs are frequently sent emails reminding them that lab policy requires them to direct questions to public relations specialists; employees at other labs are rarely reminded. Generally, reminders precede periods when management anticipates a sizable amount of media scrutiny on a particular topic. The UCS document also complains that the government agencies "route" media contacts to specific scientists who are then interviewed. Of course this type of routing occurs-- much is entirely innocent. Generally, the media contact the publicity relations managers, ask for recommendations, and interview the person they public relations manager recommends. Is this new? Generally speaking, do media sources mind? Or do they consider the practice a time saving convenience? (If the media thought the stories were distorted, wouldn't the media try to get around the federal agency handlers?) Can there be problems with this system? Sure. Do some findings get preferential treatment -- particularly with regard to being publicized in agency press releases? Sure. Does it give those in the "upper tiers" the ability to spin the findings to their own liking? Of course. Is it new? Absolutely not. Most importantly, do climate scientists have a way around this problem: yes. They can form coalitions write things like the UCS, write things like "Atmosphere of Pressure", and communicate their concerns to congressman. That's happened. Now there are hearings to investigate whether the more specific things going on now fall in the category of "routine practice" or "abnormal supression". I suspect we'll here more, but right now, the UCS document doesn't make a convincing case for "abnormal supression". I also noticed a size-able amount of the proof of interference described in the UCS document was related to “not enough funding”. (Links here: http://oversight.house.gov/story.asp?ID=1162&Issue=Environment )

I also noticed some of the “objectionable practices” affecting the integrity of climate science are actually rather routine practices at national labs, the practices aren’t secret, they aren’t specific to climate change and they pre-exist the current administration. ( I suspect many are older than dirt.)

For example, on page 29 of “atmosphere of pressure”, the authors mention federal agencies require pre-approval of media interviews between scientists and the media. Of course many do.

DOE labs generally require pre-approval when any employee — whether secretary of scientist– talks to the media; this has gone on for a long time particular with regard to work affecting the public safety or defense. Are climate scientists under the impression that scientists and engineers working in areas related to stored radioactive waste, military defense projects, or public health issues are told they may just schedule their own interviews, write their own press releases and publicize to their hearts desires?

In reality, employees at some labs are frequently sent emails reminding them that lab policy requires them to direct questions to public relations specialists; employees at other labs are rarely reminded. Generally, reminders precede periods when management anticipates a sizable amount of media scrutiny on a particular topic.

The UCS document also complains that the government agencies “route” media contacts to specific scientists who are then interviewed.

Of course this type of routing occurs– much is entirely innocent. Generally, the media contact the publicity relations managers, ask for recommendations, and interview the person they public relations manager recommends. Is this new? Generally speaking, do media sources mind? Or do they consider the practice a time saving convenience? (If the media thought the stories were distorted, wouldn’t the media try to get around the federal agency handlers?)

Can there be problems with this system? Sure. Do some findings get preferential treatment — particularly with regard to being publicized in agency press releases? Sure. Does it give those in the “upper tiers” the ability to spin the findings to their own liking? Of course.

Is it new? Absolutely not.

Most importantly, do climate scientists have a way around this problem: yes. They can form coalitions write things like the UCS, write things like “Atmosphere of Pressure”, and communicate their concerns to congressman. That’s happened.

Now there are hearings to investigate whether the more specific things going on now fall in the category of “routine practice” or “abnormal supression”. I suspect we’ll here more, but right now, the UCS document doesn’t make a convincing case for “abnormal supression”.

]]>