Meta-researcher Marcel van Assen: ‘Current science is far from functioning optimally’

For years, science has been plagued by what is known as the “replication crisis.” Results of original, sometimes renowned studies often do not hold up when repeated by other scientists. What is the state of this crisis anno 2022, and how is it being addressed? Marcel van Assen, co-founder of the Meta-Research Center at Tilburg University, takes us through.

For four years, Van Assen and two then PhD candidates participated in the Reproducibility Project, a large-scale 2015 study that tested 100 different psychology studies for repeatability. The result: only 37% of the cases found the same outcome as the original study. After researchers in other fields of science such as biology and economics drew similar conclusions, the scientific community could no longer close its eyes; the birth of the replication crisis was a fact.

But, it had been a crisis for some time, says Van Assen. Only the realization was lacking. “For as long as we have been doing research, it has been the case that results in replication studies are often different from original studies. Only recently have many scientists become aware that this is so.”

Furthermore, he would reserve the word “crisis” not so much for the fact that something different often comes out when you repeat a study, but rather for the status of science as a whole. “Because of nontransparent research methods, we only see part of what scientists do. This makes it difficult to repeat research at all because we don’t know exactly how the research was conducted. There is also a tendency to publish only research with a positive effect, which means that replication studies in most cases produce a less positive effect.”

The consequence of these ingrained research practices is that findings from existing literature are difficult to interpret and value.

The replication crisis paved the way for you to set up a research team with Jelte Wicherts a few years ago to do “meta-research.” What exactly does that entail?

“In a nutshell, we do research on research. That is, we look at how scientific research is done and how it could possibly be done better. We do this by analyzing both bad practices and good practices and what effect they have on scientific outcomes. We also try to promote the good practices. One example is the introduction of “statcheck,” a kind of spelling checker for statistical errors.

“And indeed, in a way, we as a research group benefit from the replication crisis because it allows us to publish about it as a meta-research center. In the eyes of some, therefore, we are parasites living like dung beetles off a pile of shit—the failing science in this case. Others, in turn, see us as pioneers of meta-science. There’s some truth in both, I think.”

Why is it that so few scientific studies can be replicated?

“It’s not simple to answer this. There are so many factors involved, and the extent to which they do is still quite unclear. One common argument for different results is that there is something in the new study that differs from the original study and explains the difference. This is often raised by the researchers of the original study and is sometimes called the “hidden moderator argument.” However, this rarely seems to be a valid explanation. A more likely cause for anomalous outcomes is simply chance. Indeed, to a large extent, doing research is simply rolling the dice; there is always a degree of randomness in it. In psychology, this is because your outcome is determined by the people who participate. Because each person is different, the sample average almost always varies.

“In addition, what we see is that the outcome of replication studies is systematically lower on average—that is, no effect or a smaller effect—than the outcome of the original study. This is partly explained by publication bias, the tendency of researchers and publishers to publish only positive results in scientific journals. Studies in which no effect was found do not see the light of day. One reason scientists hope for (as many) positive outcomes as possible is that it is good for their careers. Moreover, they are also just people who prefer to see new things, a nice story, and positive results. Unfortunately, this is not in line with the ideal image of a good scientist who has no regard for this.

“The fact that scientists benefit from positive results leads them to (unknowingly) engage in questionable research practices. An example is “massaging” the data in such a way that a correlation or positive effect becomes more likely. Research shows that such practices are frequent.”

What is the role of open science in combating the replication crisis?

“Regarding open science, I find open access less interesting because it doesn’t improve research per se. Of course, it is important that everyone has access to articles, but you get there indirectly anyway. Instead of focusing on that, the government should insist more on open research; open data, open research materials, open statistical codes, and pre-registration.

“Open research involves making all your research materials, data, and analysis codes available, as well as truthful reporting of both statistical analyses that were already planned (pre-registration), and analyses that were only conceived during and after data collection. This way of doing research takes more time than the old way, which is more about collecting data quickly, writing down some analyses, and done. Also, if you do it right, you are less likely to have positive findings. That together makes it difficult for the academic community to make the transition to open research.

“Applying open research ensures that questionable research practices have much less opportunity, and thus, we will see more valid published research results. The shared data, research methods, and code can also be evaluated by other scientists. They can question them and make adjustments as needed. Moreover, it helps replicate and build on research. This collective review also prevents frauds and makes them easier to detect.

“Other causes of the replication crisis such as chance and hidden moderators are not expected to be eliminated by open research. However, this does not alter the fact that open research does make it possible to identify the cause of a difference in outcome between the original and replication study. In short, it is necessary to efficiently collect and share good knowledge. That’s why I think open science is the future.”

Just last year, “superstar of behavioral science” Dan Ariely was caught by fellow researchers using fake data. This suggests that we are far from overcoming the replication crisis, of which frauds are an inseparable part. How would you describe the state of the replication crisis anno 2022?

“We live in an extraordinary time with interesting developments. Chief among these is the rise of “many lab studies.” These are studies in which one research question is carried out in dozens of labs simultaneously. That way you can properly analyze which factors cause the differences in results. There is also increasing attention to replication studies and open science seems to be increasing in popularity. So, there is progress.

“At the same time, we see that the quality of new initiatives still leaves quite a lot to be desired. This also applies to their dissemination. It is difficult to assess the extent to which scientists have significantly changed their behavior. I myself deal mostly with people who think the same way about science as I do, so you quickly tend to think that science is doing better. But how big this bubble is compared to the rest of academia; I dare not say. I fear that the vast majority of scientists are just carrying on as before.

“It is fortunate that the number of young researchers using these new research methods is proportionately greater than the number of older researchers doing so. This is good news because at some point the older scientists—those who are in charge now—will disappear from science and the younger generation will take over. This will hopefully bring about a “paradigm shift,” in the words of science philosopher Thomas Kuhn.

“But even if every scientist acted ‘perfectly’ and open research becomes mainstream, replication studies will continue to give different results compared to original studies. This is because chance and a changing context will always continue to have an effect on the outcome. At best, open science can make it easier to identify those causes.

“Finally, it is worth remembering that scientifically speaking, we are only at the caveman stage. If we equate humanity’s existence with the twenty-four hours that are in a day, we are only engaged in scientific research for the last thirteen seconds. Thus, we have only just begun and cannot know everything. Humility is, therefore, in order.”

Translated by Language Center, Riet Bettonviel

_Advertentie.