Peer Review – Great Idea, but the Execution Needs Work

March 5th, 2009

Posted by: admin

From the folks at Scientific Blogging I read a synopsis of a Policy Forum in the March 5 issue of Science (subscription required) on what happens to articles that appear similar to previously published work.  According to the work of a team at the Southwestern Medical Center of the University of Texas, a data analysis (conducted with software the team developed) of two years worth of abstracts in the Medline database showed nearly 70,000 citations that appeared pretty similar.  After a deeper analysis of a sample of those citations, the team found 207 pairs of articles that suggested possible plagiarism.

Before you go tsk-tsking about the gap between scientific ideals and scientific practice, I do want to point out that with an ever increasing amount of journals and research that conducting a thorough literature review is getting harder to do.  That said, researchers, editors and reviewers need to do a better job of reviewing extant literature.  Perhaps tools similar to the one developed by the UT Medical Center researchers are needed to make sure plagiarism – both intentional and inadvertant – is confronted.

The numbers are disappointing.

The research team sent out 162 sets of questionnaires, and received replies in 143 cases.  83 internal investigations took place, and there were 46 retractions.  12 cases will not be reviewed.

Before receiving the questionnaires, 93 percent of original authors were not aware of the duplication.  Of the duplicate authors that responded, 28 percent denied any wrongdoing; 35 percent admitted to having borrowed previously published material; and 22 percent were from co-authors claiming no involvement in the writing of the manuscript. An additional 17 percent said they were unaware their names appeared on the article in question.  The last figure is tough for me to understand.

Like the researchers, I do hope that tools like the one they developed for this study can be used to make peer review and literature review more effective in eliminating possible duplications.  I also think that such tools allow for peer review to become more transparent (while remaining anonymous for reviewers and authors).  The reputation of science depends in large part on the integrity and consistency of its processes.  Peer review probably could fit the Churchill quote about democracy, in that it’s the worst system of review with the exception of all others having been tried.  We can do better.

One Response to “Peer Review – Great Idea, but the Execution Needs Work”

    1
  1. rephelan Says:

    I agree with the last sentence: “We can do better.” I am becomingly less convinced that “peer review” is the path to “better”. The “critical methodology” approach adopted by the social sciences has come to permeate scientific discourse in general: “all issues are political”. Today’s topic du jour in the climate science blogs, the rejection of the Paltridge et al paper, is one case in point. My old mentor, the late Nicholas C. Mullins, made me read Kuhn’s “Structure of Scientific Revolutions”, which suggests that science is not free of bias and agendas or even the cultural context in which science is “done”.

    Max Weber, the great German Sociologist, gave an address called “Science as a Profession” – almost a century in the past but well worth the reading today. A dialogue between Weber and Crichton would be absolutely fascinating.

    In the 60’s and 70’s researching the latest information, thumbing through card catalogs, was a task of weeks. We may have more journals today, but the task of research can often be reduced to hours. (OK, there is a legititimate niche industry for researching precedents, but the author still has to review them!) There is less excuse in the 21st century for not knowing what has already been published and discussed.

    One of the fundamental bedrock values of science is transparency. Can it be replicated, are there alternative approaches? Some science blogs give us the impression that free-wheeling discussion of science is possible, but the “trolls” arguing for agendas (either political or paradigmatic) obscure the real issues. I would suggest that the peer-review system (along with the pay-to-view approach of most journals) is no longer functional. Too many scientists have adopted paradigmatic or political positions and will not be budged by evidence from them. Or have gotten too lazy to keep abreast of their literature (e.g.: “93 percent of original authors were not aware of the duplication”).

    The current peer-review process is too prone to orthodoxy. The current cultural climate is too prone to instrumentalism (“gee, professor, I took this course because I needed three social science credits at a convenient time … why don’t I have an “A”?)… some things are more important than fundamentals. We’ve been letting this go on for a generation.

    OK, I’ve just wiped the foam from my mouth and had my wife force water down my throat… the animal control officer may not need to put me down after all… science needs to be made more public, with tools to assist that, and a re-emphasis by the academy on just what academic integrity means… Michael Crichton would almost certainly agree with me that private resarch, proprietary knowledge, is not science.

    The issue is not how difficult it is to monitor one’s field, it is a question of character and ethics. period. ’nuff said.