CASE STUDIES
IN ASSESSING DEMAND

1. PRIORITY-SETTING WORKSHOPS

The Agricultural Research Service’s (ARS) Global Change research program convenes periodic workshops with scientists and users including those from federal agencies, agricultural nonprofits, and the agricultural producer community.

These workshops help the USDA set research priorities for the next planning cycle based in part on what customers want from research. Workshops of this kind not only directly inform priorities, but can also help to establish enduring lines of communication with potential end users, and move science closer to meeting demand. These workshops are typical of each of the ARS’s National Research Programs. Usually the Program Leaders, along with the rest of the program team consisting of three to four ARS scientists, are in charge of the workshops. The process runs on a five year cycle and features both backward-looking and forward-looking assessments of programmatic research. A common process will include a survey of the research that the program supplies and is proposing to supply, a needs assessment, during which the ARS invites users to discuss their informational requirements, and a discussion between users and suppliers on how researchers can work to meet those requirements.

2. TEST RUNS AND FEEDBACK

The Naval Research Laboratory’s Meteorology Division works to enhance the value of meteorological tools that the Navy uses. These tools can range from short term regional weather forecasts to more specified applications, such as those that let officers in the Middle East track dust storms. When making a more advanced model, or when adding a new feature intended to aid decision makers, NRL staff will often test an early version of the new technology with users in the Naval fleet by simply e-mailing them a link to the new model and asking for feedback to help assess the added value of new developments.

Agricultural Research Service scientists also work directly with users to test and refine products such as new agricultural management strategies or decision tools for farmers. Through test runs coordinated with users, researchers thus learn about the value of their work in a real world environment from actual users.

A mandate from the Bush Administration charged the ARS with developing mechanisms for accounting for agricultural carbon sequestration. In recent years, scientists in the ARS Global Change National Program have developed CQESTR, a computer model with the purpose of predicting carbon dioxide sequestration in agricultural land under different soil types, crops, and management regimes. In order for this model to actually help farmers in making decisions, however, they will need to see it as useful and understand how they will benefit from using it.

3. REVIEW PROTOCOLS FOR ASSESSING THE IMPACT OF RESEARCH

As a director of the National Institute of Standards and Technology’s (NIST) Material Science and Engineering Laboratory (MSEL), Richard Kayser worked at the helm of a research venture with the mission of meeting the materials science needs of industry. For the MSEL lab, and specifically, for the director, a significant challenge lies in creating an actual impact for those in the materials industry.

Kayser’s protocol for assessing proposed research projects considers both the impact of the research and the risk of failure. Assessment of impact includes both the promised level of impact as well as how well that impact is articulated. Risk includes both risk of technical failure and risk of failed technology transfer. The highest ranked proposals have a low risk and high impact that is convincingly articulated. Assessing the risk of technology transfer reflects NIST’s commitment to getting its products into the hands of industrial and academic users. Research is only usable to the extent that potential users are capable of adopting it.

Other decision makers have implemented or considered similar strategies. A consideration of impact during prioritization and evaluation could encourage projects that are more likely to address need. Thus, those proposals that are either better able to articulate eventual usability, or that might lead to a high impact (even if risks are moderate) become more likely to go forward. Scientific or technical excellence cannot stand alone as a criterion of usability or value.

4. PROBING THE OPTIONS: HEILMEIER QUESTIONS

A series of appropriate questions can also be used as a general guide for making sure science policy makers are addressing the need to reconcile supply and demand. One example is the Heilmeier Questions, originated by George Heilmeier, a former director of DARPA and vice president at Texas Instruments:

• What is the problem?
  Why is it hard?

• How is it solved today?
  By whom?

• What is the new technical idea?
  Why can we succeed now?

• Why should your institution do this?

• What is the impact if successful?
  Who would care?

• How will you measure progress?

• How much money?
  How long will it take?

The questions offer a way to address potential impacts of research. While the first three address the technical problem, and technical impact, the other four speak to issues of demand and fit to broader institutional goals. Finally, questions 6 and 7 address logistical research issues that affect implementation.

BACK TO PART 3: SUPPLY OF AND DEMAND FOR SCIENCE