Michael Kranish at Boston Globe considers a new study by John Ioannidis, which finds that

almost one-third of the top papers that appeared in top journals over a 13-year period from 1990 to 2003, had been either contradicted or found to have potentially exaggerated results. All the articles had undergone vigorous peer review, leading to questions about whether problems should have been caught by reviewers.

The author of that study, Dr. John Ioannidis, an adjunct professor at the Tufts University School of Medicine, said that flaws in the system were not solely responsible for the problems with the initial studies, but he said that they may be ”part of the puzzle” that should be examined to improve research. Ioannidis couldn’t actually study the peer reviews, of course, since they are confidential (and of course anonymous), which in his view and that of many at JAMA and elsewhere is part of the problem.

At AJOB, we’re working on this problem quite visibly, since our articles not only undergo peer review but are (in the case of Target Articles) then subject to open peer commentary – which is published right alongside the articles. Our theory has been that if the editors, reviewers and commentators don’t catch it, we’ve at least tried as hard as we can. But more than once an article has made it through peer review and editorial review only to get gored by open peer commentary that amounts to a “gotcha” peer review. That happens to lots of journals – but usually it isn’t caught at all:

PLOS Medicine also encourages peer reviewers to reveal their identity, but it does not demand it.

The journal’s senior editor, Barbara Cohen, said some reviewers want anonymity out of concern about retribution, which she described as ”you trashed my paper at Nature, now I’m trashing yours at Science,” referring to two leading journals.

Cohen also said she is sympathetic to younger peer reviewers who fear that providing criticism of a senior person in the field will hurt their career. This is a common complaint among reviewers.

But given the high number of studies that end up either wrong or deeply flawed, much of the medical profession is looking for new ways to examine research.

Armstrong, the professor who has read dozens of studies on peer review, cited numerous embarrassing incidents that he said had called the peer review process into question.

In one study, for example, researchers submitted a plagiarized paper to 110 journals, but only two publications recognized the problem.
In another study, researchers examined 18 papers that had been published in peer-reviewed journals by a person who later admitted scientific fraud; they found that 16 of the papers had an average of 12 errors each.

[thanks Udo]