July 1997 Workshop
10-12 July 1997, Boulder, Colorado, USA

Lessons Learned: A First Cut at Organization

Collated by Broad Questions

How should we evaluate the effectiveness of scientific predictions in the policy process?

I. What is the policy goal(s) (i.e., outcome) that prediction is intended to achieve?

  • Specify the purpose(s) [policy goal(s)] of the prediction.

II. How does the process of developing predictions influence the policy process (and vice-versa)?

  • Consider alternatives to prediction for achieving the purpose. Maintain flexibility of the system as work on predictions proceeds.

  • Recognize that a choice to focus on prediction (as well as the choice of specific predictive technique) will constrain future policy alternatives.

III. What are the direct societal impacts of the prediction?

  • Consider alternative societal impacts that might result from the prediction (including the different roles played by prediction).

  • Evaluate past predictions in terms of a) impacts on society.

  • Recognize that the prediction itself can be a significant event.

  • If possible, [Subtract the costs/Assess the impacts] of inadequate predictions [from the benefits/relative to the impacts] of successful ones.

IV. What are the scientific limitations and uncertainties of the prediction?

  • Evaluate past predictions in terms of b) scientific validity.

  • Recognize that different approaches can yield equally valid predictions.

  • Recognize that prediction is not a substitute for data collection, analysis, experience, or reality.

  • Recognize that predictions are always uncertain; assess the level of uncertainty acceptable in the particular context.

  • Beware of precision without accuracy.

  • Recognize that quantification and prediction are not a) accuracy; b) certainty; c) relevance; d) reality.

  • Computers hide assumptions. Computers don't kill predictions, assumptions do.

  • Recognize that the science base may be inadequate for a given type of prediction.

V. What factors can influence how a prediction is used by society?

  • Recognize that prediction may be more effective at bringing problems to attention than forcing them to effective solution.

  • Recognize that perceptions of predictions may differ from what predictors intend and may lead to unexpected responses.

  • Recognize that the societal benefits of a prediction are not necessarily a function of its accuracy.

  • Recognize that there are many types of prediction, and their potential uses in society are diverse.

VI. What political and ethical considerations are raised by the generation and dissemination of a prediction?

  • Pay attention to conflicts of interest [among those making predictions].

  • Understand who becomes empowered when the prediction is made. Who are the winners and losers?

  • Pay attention to the ethical issues raised by the release of predictions.

VII. How should predictions be communicated in society?

  • Make the prediction methodology as transparent as possible.

  • Predictions should be communicated a) in terms of their implications for societal response and b) in terms of their uncertainties.

VIII. Is the value of a prediction contingent on international context?

  • Predictions should be developed with an awareness of international context.

Back to Workshop Home Page
Back to Predictions Home Page