It is a canon of the academic enterprise that more and better knowledge is always a good thing. However, when it comes to actual processes of decision making, more knowledge does not always lead to better outcomes, and in fact may lead to worse outcomes. In a thoughtful column in a recent edition of the Wall Street Journal, David Wessel takes on this interesting subject. Here is an excerpt:

“How about the service offered by LegalMetric LLC, a start-up founded by patent lawyer Greg Upchurch? Contemplating a patent-infringement case in Delaware? For $795, Mr. Upchurch will tell you which judges rule most swiftly and which tend to favor patent holders. Making a motion for summary judgment? Mr. Upchurch can tell you how the judge has ruled on similar motions versus his peers. These data always have been available in court files, but putting the pieces together was so expensive no one did it. Now, it’s on the U.S. federal judiciary’s Web site. Mr. Upchurch and his two employees download dockets, key information into a database and push a button so their software generates detailed reports. For lawyer and client, this knowledge can be very valuable. But does it increase the chances that the judge will come to a just decision? It is the sort of information that Nobel laureate Kenneth Arrow labeled “socially useless but privately valuable.” It doesn’t help the economy produce more goods or services. It creates nothing of beauty or pleasure. It simply helps someone get a bigger slice of the pie. Sure, if the product helps win cases, then both sides will buy it — just as both sides in high-stakes product-liability cases invest in jury-selection experts and software — and neither will have an unfair advantage. But does that make the society better off? The same question arises in the sophisticated software used to draw the boundaries of U.S. congressional districts so precisely that Republicans and Democrats know which party is almost certain to win. This has enhanced the power of incumbency: In 2004, 401 of the 435 members of the U.S. House of Representatives sought re-election; all but seven won. It also has polarized the U.S. Congress, and made compromises scarce, because with safe districts, legislators have little reason to court the voters in the center. The advantage to individual lawmakers is clear; the value to society is not.”

In important respects uncertainty about future outcomes is what allows for risks to be shared equitably. Wessel asks us to imagine what would happen if we had perfect knowledge of the future:

“Imagine a place with uncertain weather where food is plentiful in rainy spots, but not in others. Residents, in essence, buy insurance. The lucky feed the unlucky. No one starves. Then it becomes possible to buy accurate weather forecasts. One who buys the forecast knows whether he needs insurance or not; he profits. But the total amount of food available is unchanged. And if everyone buys the weather forecast, the insurance market becomes impossible. “There is a double social loss — the resources used unnecessarily in acquiring information and the destruction of a market for risk sharing,” Mr. Arrow said when he posed this example in 1973. Eliminating uncertainty makes insurance impossible. That’s no small matter: If deciphering the human genome allows each of us to know the precise odds of contracting a dread disease, life and health insurance will be very tricky.”

For scientists seeking to justify investments in their area of research, findings such as these make it untenable to simply assert that society will inevitably be better off with more knowledge. Indeed the current debate over stem cell research indicates that some people in society are more than willing to forgo the potential fruits of science in service of other valued outcomes. The value structure of a scientific endeavor seeking to advance knowledge, well-described by Michael Polanyi thirty years ago, is not always the same value structure that underlies the needs of pragmatic decision making or political battles among people who hold different conceptions of how the world should look.