I criticized the Information Technology and Innovation Foundation earlier for framing an event strictly around faulting the science behind a competitiveness report released by RAND.  The event in question took place earlier today in Washington, and I attended to follow up on my criticism.  The ITIF rebuttal report to the RAND competitiveness report is also available online, and you’re free to review both documents and respond to my analysis.  (Roger is quoted from Prometheus deep in the document – page 16 – about connections between student achievement and economic competitiveness.)

My initial concern – that the event would be used to try and close out a policy debate by appealing to science (or criticizing ‘bad’ science) – did not materialize, or at least in the way I initially suspected.  What happened is 90 minutes of presentation and discussion that had absolutely nothing to do with policy recommendations.  This is what I think Al Gore – circa 2000 – would pejoratively describe as an “academic exercise.”

The bulk of the presentation (and the report) is a listing of various historical and economic data points that the authors contend were not considered or effectively addressed in the RAND report.  However, the list ITIF has of the report’s ‘key limitations’ (page 2) indicates that the major disagreements with the RAND document are in framing and not really in science.  After all, ITIF agrees with the RAND report conclusion that the U.S. currently leads the world in science and technology, and RAND does make specific policy recommendations to maintain or improve the nation’s scientific and engineering enterprise – something I don’t think ITIF would object to, but they don’t even mention RAND’s recommendations in the report.

I’ll touch on some specific criticisms in a bit, but I fail to see the benefit – aside from filling in a conference room of people sympathetic to boosting S&T funding – in a think tank arguing interpretations of economic information with absolutely no connection to specific policy actions.  An economics or business department can certainly make the arguments over what the data tell us, but when the audience is policymakers and those with the ear of policymakers, the lack of a policy discussion is a collective waste of time.  ITIF wasted an opportunity to make the case that due to the overly optimistic (in their opinion) interpretation of RAND, the associated policy recommendations are wrong/inadequate/(insert proper descriptive term here).

Back to the paper, which is a long series of economic discussions without policy recommendations.

Here’s the list of limitations noted by ITIF:

• Framing the wrong fundamental question regarding the S&T competitiveness debate;
• Providing an incomplete historiography of U.S. science and policy development, particularly policies developed in response to previous challenges to U.S. S&T competitiveness;
• Using inappropriate or incomplete benchmark metrics to assess U.S. S&T and economic competitiveness;
• Under-emphasizing within the report a number of indicators that clearly demonstrate trends of weakening U.S. S&T competitiveness;
• Failing to include certain key measures needed to deliver a true assessment of U.S. S&T competitiveness; and
• Using available time-series data sets – ending by 2003 at the latest in most cases – that are not reflective of the competitive challenge that has emerged since 2000 and do not adequately reflect the competitive landscape of mid-2008.

The issue with the first point – that the RAND report asked the wrong question – is not scientific.  As the authors state toward the end of the report (page 17)

“The real question is, ‘Is the United States acting sufficiently to maintain its lead in science and technology in the face of trends that show a clear deterioration of its lead in key metrics and of competitors that are becoming increasingly sophisticated with policies to enhance their S&T enterprise and economic competitiveness?’”

Besides needing an editor (try and diagram that beast of a sentence), there are several policy assumptions underlying the question.  There’s an assumption that comparative advantage or being number one is the nec plus ultra of science and technology competitiveness.  There’s an assumption (perhaps true, but not effectively addressed) that the U.S. is falling, rather than the rest of the world is catching up.  There’s an assumption that the key metrics for ITIF are the same as for RAND, and another assumption that those key metrics (which include the usual suspects of patents and publications) effectively measure the phenomena at issue.

Most of the other limitations are connected to the differences between ITIF and RAND over the important question.  However, the use of historiography – how history is researched and written – is wrong.  They mean to say that there was an incomplete history of science and technology development, and each time they repeated the error and used historiography, it became that much harder to take the report seriously.  However, I don’t think a regular policymaker will read this report past the first page or two, given the lack of recommendations and the laundry list of economic data that won’t be new to anyone familiar with the issues.

Again, I’m complaining about the tactics of ITIF at least as much, if not more, than trying to defend the RAND report or criticize the ITIF rebuttal.  Anything coming out of a think tank needs to be filtered through the ideological and policy objectives of the organization.  Unlike most food sold in this country, these reports don’t come with an information label breaking down the contents.