Researchers are getting better at communicating science in a rigorous and reproducible way, according to a text-mining analysis of around 1.6 million papers. But the findings have also sparked fears that progress is too slow.

The study used a software tool called SciScore, which gives studies a mark out of ten for ‘rigour and transparency’. SciScore searches the text in papers’ methods sections for around 20 pieces of key information, which act as proxies for how rigorous the experiments are, and how easy it would be for other researchers to reproduce them. The software can flag where authors have specifically identified the reagents they use, such as antibodies, software tools, cell lines or transgenic organisms they use, for example. It also checks whether they have discussed factors such as sample sizes, how tests have been blinded or the sex of animals used.

The researchers who created SciScore — led by Anita Bandrowski, an information scientist at the University of California, San Diego — analysed 1.58 million freely available life-sciences papers indexed in the PubMed Central database.

They found that between 1997 and 2019, the average score across all papers more than doubled, from 2 out of 10 to 4.2.

The study, posted on the preprint server bioRxiv on 18 January ¹ , says this rise shows that scientists are increasingly including fine detail about their experiments. This might be because many journals have actively tried to improve reporting standards, they suggest. “The scores are rising all the time,” says Bandrowski.

The analysis also showed that individual measures of rigour are on the rise. For example, less than 10% of papers published in 1997 discussed randomization in the methods; this had risen to around 30% in 2019 (see ‘Rigorous research’).

But the numbers overall haven’t increased as much as some researchers would like. One problem area is the poor identification of antibodies. The SciScore analysis finds that, despite an increase in the past few years, more than 50% of papers that mention antibodies still don’t include enough information to pinpoint them precisely. In recent years, problems with identifying the antibodies used in research have been highlighted as a factor in the lack of reproducibility in the academic literature.

“It is shocking that so many researchers are still referring to reagents ambiguously,” says Joshua LaBaer, executive director of the Biodesign Institute at Arizona State University in Tempe. “There are no simple solutions here, and we are all responsible.”

He adds that although SciScore is “a step in the right direction”, it doesn’t measure everything necessary to ensure that a paper is reliable and reproducible. For instance, the software doesn’t take into account whether an antibody actually binds to its intended target, or if it was appropriate to use for the study in the first place.

By calculating the average SciScore rating for all the papers in a given journal, Bandrowski and her colleagues created a metric they dubbed the Rigor and Transparency Index. Although the study finds that all journals’ average scores have increased since 1997, no title among those analysed has an index of more than than five out of ten. This suggests that “less than half of the rigor and reproducibility criteria are routinely addressed by authors”, the study says.

“Overall, as a field, we seem to be doing better,” says Bandrowski. “But we have quite a way to go.”