This is no surprise to me, and relates to the piece I started the day with: the rate at which scientists produce groundbreaking research has slowed dramatically in recent years.

This phenomenon has been quietly discussed among both scientists and social science researchers in recent years, and there has been a great deal of speculation about the reasons for the decline. To me, the reasons are pretty obvious.

First, the data. Carlson School of Management Associate Professor Russell Funk, doctoral student Michael Park and Professor Erin Leahey of the University of Arizona collaborated in a project to analyze the rate at which “disruptive” scientific research is produced.

“Disruptive” research significantly changes the way researchers look at a problem. It need not quite be an “aha!” discovery or theory, but it must at least be something close. It is the kind of research or discovery that changes subsequent research, rather than just being a continuation of a research path followed by many.

Funk et. al. found a significant decline in the production of such research over recent decades, despite a dramatic increase in the amount of scientific research. In fact, there is a high degree of correlation between the increase in the number of researchers and research studies and the decline in disruptive results.

Despite exponential growth in recent decades of research papers and patents, a new University of Minnesota study published in Nature suggests science and technology are becoming less disruptive.

Carlson School of Management Associate Professor Russell Funk, doctoral student Michael Park and Professor Erin Leahey of the University of Arizona analyzed data from 45 million papers and 3.9 million patents across six decades for their research. They used a “disruptiveness score,” which is based on the patterns of citations five years after publication, to assess the extent to which papers and patents push ideas toward new trajectories. They determined:

• Papers and patents are less likely to be disruptive, or make previous findings obsolete and push science and technology in a new direction, such as the discovery of the DNA double helix structure.
• Instead, papers and patents are more likely to be consolidating, or further developing previous work — e.g., the Kohn-Sham equation which improved upon existing equations about electron particles.
• Scientists and inventors are increasingly using narrower slices of knowledge to develop their new work.
• This pattern holds across all major fields of science, including technology, medicine and social sciences.

Simply put, most research done today is like a drop of rain into a pool. It gets lost in the great mass of water. It may, or may not, advance the science in some small way, but it doesn’t change things in any significant way.

Such science can be valuable. Not everything has to be a Nobel Prize-winning discovery or insight, just as Apple’s success was not due to solely Steve Jobs’ genius. Toyota’s Camry has spent decades as a marketplace winner, incrementally improved without any major disruption.

But it is disruptions which ultimately drive progress forward. The assembly line, Relativity, Quantum Mechanics, the Big Bang theory, transistors, the Green revolution, etc. Such things change the world. And the rate at which such things occur has slowed appreciably.

“A healthy scientific ecosystem is one where there’s a mix of disruptive discoveries and consolidating improvements, but the nature of research is shifting,” said Funk. “With incremental innovations being more common, it may take longer to make those key breakthroughs that push science forward more dramatically.”

For papers, the decrease in the disruptiveness score between 1945 and 2010 ranges from 91.9% for the social sciences to 100% for the physical sciences. For patents, the decrease between 1980 and 2010 ranges from 78.7% for computers and communications to 91.5% for drugs and medical.

One theory for the current trend is that all the “low-hanging fruit” of disruptive innovations have already occurred. The researchers also point to the growing burden of knowledge that scientists are required to learn, which means more time spent training rather than pushing the boundaries of science.

The findings call for a need to reimagine how science is conducted. Scholars sometimes face a “publish or perish” research culture, in which their success is based on the number of papers they publish or patents they develop. The researchers suggest federal agencies could implement funding changes to better support scholars’ long-term careers.

“A lot of innovation comes from trying new things or taking ideas from different fields and seeing what happens,” said Park. “But if you are worried about publishing paper after paper as quickly as you can, that leaves a lot less time to read deeply and to think about some of the big problems that might lead to these disruptive breakthroughs.”

This is exactly the phenomenon I discussed earlier today, where the process by which science is funded and rewards are distributed can impede progress. It’s not that there isn’t much to discover; it’s that the sociology of science impedes innovation. If your career path is built upon reinforcing the prestige of your seniors and confirming their priors, you are very likely to reinforce the prestige and confirm the priors of your seniors.

Another factor is the vast expense involved in doing groundbreaking research, making science a much more collective enterprise than in the past. Science is done by teams, and the process through which projects get funded tend to incentivize consensus. I have long believed that fusion research is hindered by the fact that each research project is hugely expensive, takes many years to implement, and the money for such experiments goes through a peer process that tends to focus research into a single path.

That’s great if the path is the best one, but devastating if billions of dollars and decades of work go into a dead end. By definition no truly disruptive path is taken because consensus is what created the path to begin with.

There are geniuses in science still; but genius is rarely cultivated. And not just because other scientists don’t want to reward it; many do or would. But genius rarely gets through the selection process.

Despite the current trend, the researchers say it’s important to note this doesn’t mean there are fewer technological advancements to discover.

“There’s a huge need for innovation to find answers to today’s most pressing challenges—from climate change to space exploration,” said Funk. “It’s clear there are still huge opportunities for disruptive innovations to happen and to make improvements for humanity.”

Add this all up and you get what is essentially a productivity crisis. More and more bodies and dollars are thrown into the research process, and yet the results are often more meagre than in the past.

There is no one solution to this quandary, but as is so often the case in modern society paring down the bureaucratic nature of the enterprise would be a great start. Famously productive scientific institutions like Bell Labs were based upon funding people, not projects in the main. Identify truly creative individuals and set them free with access to resources.

You will fund lots of dead ends, but also unleash creative energies that might get stifled in a traditional academic setting.

Albert Einstein’s great insights were developed not as an academic, but as a low-level staffer at the Swiss Patent Office. Why? Because no academic institution would hire him until he became famous.

There is a lesson to be learned there.