Media Releases

Negative citations important to scientific progress, should be tracked, says new study

February 23, 2016

Toronto, ON – Having your work mentioned in somebody else’s research is an important way for scholars to build their academic reputations. Citing others’ work is also important to the credibility of new research.

But what happens if a researcher makes a citation in order to point out flaws or weaknesses in a previous study?

Negative citations are not necessarily a bad thing, says Nicola Lacetera, an associate professor of strategic management at the University of Toronto Mississauga who is also cross appointed to U of T’s Rotman School of Management. Tracking those citations can reveal where there is particular “vitality” in a research area, especially when there is controversy among scientists active in it.

“On the one hand, it might be that a particular study is just of poor quality and reliability,” says Prof. Lacetera, whose findings have been published in the Proceedings of the National Academy of Sciences. “On the other hand, maybe it is good science that can be improved and so criticisms are more constructive. That’s the way science evolves and becomes better.”

The effect and frequency of citations have been studied before but rarely has research looked at them in detail or attempted to classify them.

In this study, advancements in the computer science field of natural language processing allowed the researchers to target negative citations in more than 15,000 articles published over a decade in the Journal of Immunology, including the incidence of those citations and details about their patterns.

The researchers, including Prof. Christian Catalini of MIT’s Sloan School of Management and Prof. Alexander Oettl of Georgia Tech, who are both graduates of the Rotman PhD program, found that negatively-cited studies are in the minority, are usually high quality — cited both positively and negatively — and receive more attention than the average, showing only a slight drop-off later in the life of the paper.

Their negative citations tend to come in the first few years after the original study has been published. They are often from other scientists working in a similar research area who may even have worked with the cited author on other studies. However they don’t tend to come from colleagues who may be in the same building or down the hall, suggesting that perhaps those people share their criticisms informally with the original author rather than risking the social awkwardness of putting their critique in print.

Besides revealing patterns behind knowledge production, the method the researchers developed for their study holds promise for further exploration into the ways scientists build on each other’s work and the improvement of scientific paper search algorithms. Meanwhile, the study should be consolation to those researchers targeted for criticism — it’s the stuff great science is sometimes built on, says Prof. Lacetera.

“No paper is perfect, including ours,” he says.

For the latest thinking on business, management and economics from the Rotman School of Management, visit:

The Rotman School of Management is located in the heart of Canada’s commercial and cultural capital and is part of the University of Toronto, one of the world’s top 20 research universities. The Rotman School fosters a new way to think that enables our graduates to tackle today’s global business challenges.  For more information, visit


For more information:

Ken McGuffin
Manager, Media Relations
Rotman School of Management
University of Toronto
Voice 416.946.3818
Follow Rotman on Twitter @rotmanschool
Watch Rotman on You Tube