“Academic careers are based on reputation,” he says Ryan HillKellogg assistant professor of strategy who studies the motivations that drive scientific innovation. “If I want credit that I can convert into a salary from a university, people need to recognize that I made new discoveries.” Being the first to publish a finding is an important way for scientists to establish this recognition.
However, little is known about the effects these “priority races” have on scientists’ careers — and on the quality of the science itself. To find out, Hill and Carolyn Stein of the University of California, Berkeley investigated this topic in the field of structural biology, where researchers compete to discover the three-dimensional shapes of individual proteins.
Hill and Stein found that failure to be first has a measurable cost: second-place papers are nearly 20% less likely to be published in top journals, and papers receive 21% fewer citations than future papers.
Additionally, Hill and Stein found that the more competitive a particular structure-solving race was, the more researchers rushed their work—resulting in lower-quality findings.
“As economists, we like competition because it can encourage effort and the early disclosure of discoveries,” says Hill. “But it can also create unintended consequences that could have negative effects on science. It is very useful to understand how scientists behave, based on the motivations they face.”
Shaping science
To understand how competition affects scientific research, Hill and Stein needed an environment where they could not only watch multiple teams competing to solve the same scientific problem, but also watch their work after the competition was over. .
Hill and Stein found such an environment the Protein Data Bank (PDB). It is a repository of findings from structural biologists, who aim to describe the thousands of precise shapes that proteins fold into inside cells. A better understanding of these structures can lead to new medical and pharmaceutical discoveries, so many structural biology groups often compete to “solve” the same proteins.
“Once researchers have a model of how a protein conforms, they need to upload it to this database,” Hill explains. “This allows us to observe cases where two researchers report the same protein discovery at the same time, unbeknownst to them. Then we observe which paper is published first and see what happens to the group that got the scoop.”
The PSB also gave Hill and Stein a view of how these struggles affect the research process itself. Some proteins are known to be more biologically important than others—”perhaps they’re associated with a gene associated with a disease of interest,” says Hill. These “high-potential” proteins naturally attract more scientific competition to solve them.
Hill and Stein wrote an economic model to describe how this competition would affect the time scientists spend on a protein structure problem before uploading their findings to the database. In their model, investing more time in research would ensure higher-quality results, but also increase the risk of being distracted—especially in projects involving high-potential proteins.
The model suggested that scientists struggling to solve these important proteins would be more likely to engage in rushed, lower-quality research. Hill and Stein then compared their model’s predictions to the scientists’ real-world performance. They did this by analyzing quality metrics reported by structural biologists and validated by the WFP. These include the “analysis” of data generated in X-ray crystallography experiments and the goodness-of-fit between structural models and experimental data.
“We really wanted to look at the execution or reliability of the survey,” explains Hill. “If you open the PDB, the quality metrics are right there on every protein page. So even if a project is collected and a paper is not published, we can see the quality of their experimental results.”
Excessive fears, hasty results
After analyzing more than 1,600 priority matches in the PDB between 1999 and 2017, Hill and Stein found that scientists are right to be concerned about being swept away. Research that comes in second place is almost 20 percent less likely to appear in a top scientific journal and 24 percent less likely to become a “hit” paper that is highly cited by other scientists in the year of its publication. Scooped papers aren’t aging either: they receive 21 percent fewer citations than their top-ranked counterparts in the first five years after publication.
“These are significant results,” says Hill.
But they don’t imply that scientific discovery is a winner-takes-all game: scoop papers are only 2.6 percent less likely to be published. In other words, second place doesn’t mean you go into the scientific dustbin. “You might still get into a good magazine, just not a blockbuster” like Cell
or Naturesays Hill. “This represents real disappointment, but there is still a fair amount of credit allocation.”
Indeed, Hill and Stein surveyed 877 structural biologists and found that they significantly overestimated both the likelihood and the cost of engagement. Respondents assumed they had a 27 percent chance of being captured by a competitor, when the actual chance was only 3 percent. They also estimated that a scooped research project would receive 59 percent fewer citations, when, in fact, the penalty was nearly three times less.
“They think the cost is catastrophic when it’s probably not,” says Hill. “But these results speak to how important reputation is in science.”
This reputational pressure also affects the way the research itself is done—at least in structural biology. Hill and Stein’s economic model suggested that in a competitive race, scientists would offset their fears of being blinded by speeding up their research. When Hill and Stein analyzed real-life projects on the PDB, they found that scientists in the most competitive protein races uploaded their findings to the PDB two months faster than scientists on lower-priority proteins, resulting in significantly lower quality work .
“The results of these projects are a little more sketchy than we would like,” says Hill. “It’s a little bit flipped from how we tend to think about competition, which is, ‘This is important, so we better get it right.’ Here, the motivation might be to get something out there first.”
Maintaining balance
Does this mean that competition has corrupted the scientific method?
Not so fast, says Hill. Although the first solutions to these matches may not be the best, scientists often conduct additional work to improve the results afterwards. “For high-potential proteins, which are important to science, there seems to be enough incentive for others to come in and improve the quality,” he says.
But these self-corrections come at a price. Hill and Stein estimate that two to six billion dollars have been spent since 1971 in efforts to improve upon initial results in structural biology. “In some ways, we’re probably happy as a society to take on that cost,” Hill says. “But you could also consider changing incentives so you don’t have to.”
According to Hill, the key is to strike a better balance between incentives that encourage competition and those that discourage it. For example, from 2000 to 2015, the US government funded it Protein Structure Initiative in order to solve protein folding problems in a systematic way similar to decoding the human genome.
“They funded a number of labs to work through lists of proteins without being encouraged to write many papers,” says Hill. “This is a more laborious approach to knowledge creation. It balances that need to find answers first to get credit.”
As new technologies like artificial intelligence reshape the way science is done, Hill says the time is right to reexamine the institutional motivations that drive these discoveries. “We want scientists to be motivated to publish their results,” he says. “But if everyone lowers the quality of their science a little bit, it becomes very difficult to fix it later.”
