Collated by Broad Questions
How should we evaluate the effectiveness of scientific predictions in the policy process?
I. What is the policy goal(s) (i.e., outcome) that prediction is intended to achieve?
- Specify the purpose(s) [policy goal(s)] of the prediction.
II. How does the process of developing predictions influence the policy process (and
vice-versa)?
- Consider alternatives to prediction for achieving the purpose. Maintain flexibility of
the system as work on predictions proceeds.
- Recognize that a choice to focus on prediction (as well as the choice of specific
predictive technique) will constrain future policy alternatives.
III. What are the direct societal impacts of the prediction?
- Consider alternative societal impacts that might result from the prediction (including
the different roles played by prediction).
- Evaluate past predictions in terms of a) impacts on society.
- Recognize that the prediction itself can be a significant event.
- If possible, [Subtract the costs/Assess the impacts] of inadequate predictions [from the
benefits/relative to the impacts] of successful ones.
IV. What are the scientific limitations and uncertainties of the prediction?
- Evaluate past predictions in terms of b) scientific validity.
- Recognize that different approaches can yield equally valid predictions.
- Recognize that prediction is not a substitute for data collection, analysis, experience,
or reality.
- Recognize that predictions are always uncertain; assess the level of uncertainty
acceptable in the particular context.
- Beware of precision without accuracy.
- Recognize that quantification and prediction are not a) accuracy; b) certainty; c) relevance; d) reality.
- Computers hide assumptions. Computers don't kill predictions, assumptions do.
- Recognize that the science base may be inadequate for a given type of prediction.
V. What factors can influence how a prediction is used by society?
- Recognize that prediction may be more effective at bringing problems to attention
than forcing them to effective solution.
- Recognize that perceptions of predictions may differ from what predictors intend and
may lead to unexpected responses.
- Recognize that the societal benefits of a prediction are not necessarily a function of
its accuracy.
- Recognize that there are many types of prediction, and their potential uses in society
are diverse.
VI. What political and ethical considerations are raised by the generation and
dissemination of a prediction?
- Pay attention to conflicts of interest [among those making predictions].
- Understand who becomes empowered when the prediction is made. Who are the
winners and losers?
- Pay attention to the ethical issues raised by the release of predictions.
VII. How should predictions be communicated in society?
- Make the prediction methodology as transparent as possible.
- Predictions should be communicated a) in terms of their implications for societal response and b) in terms of their uncertainties.
VIII. Is the value of a prediction contingent on international context?
- Predictions should be developed with an awareness of international context.
