A GitHub review of three leading COVID-19 models has found that “Imperial CovidSim has the best historical fit, and the most accurate projections."

Since March, more than 60 GitHub and Microsoft software engineers have provided assistance to epidemiologists and public health experts as they work to model and understand the novel coronavirus.

Imperial CovidSim does a good job predicting the future. GitHub

They helped turn code open source and develop user interfaces to allow policymakers and non-experts to better understand the spread of the disease and the potential impact of mitigation measures.

Despite this work, the authors note, “because of the economic implications of the decisions made by governments, the models and their authors have been vehemently attacked in the press, on social media and even on GitHub itself. Some critics even suggested that unreliable projections from a single model were solely to be blamed for the economic pain that followed.”

Those criticisms, primarily from commentators and activists, have failed to stand-up. A recent Cambridge-led Codecheck into Professor Neil Ferguson and team’s high-profile Report 9 – which has drawn attacks from self-described ‘lockdown sceptics’ – confirmed the reproducibility and quality of its underlying code , which is publicly available as part of Imperial’s CovidSIM.

Three months on from the launch of several prominent COVID-19 models, “even non-experts can assess the accuracy of the projections, simply by running the models as black boxes and checking how the projections match up with reality. This note reports the results of one such experiment,” the GitHub team write.

To help with future open source coding efforts, the experts investigated the following questions:

• If we allow the models access to everything we know now, how well do they fit the data from the past?
• If we only allow the models access to data up until the end of April, how well would their predictions have held up until mid June?
• Are the models broadly in agreement over the future, or are the projections wildly different?

They scrutinised three models: the Imperial CovidSim , the Institute for Disease Modeling Covasim , and the Stanford/Stripe Modeling Covid-19 (MC-19) , running projections on six American states “where the epidemic is fairly far advanced”: California, Illinois, Massachusetts, Michigan, New Jersey, and New York.

Models found to work

Among the review’s finding are that: “Imperial CovidSim does a good job predicting the future, with IDM and MC-19 being less predictive, but still well within the bounds of credibility.”

They added: “Examining the average mean loss over all six states (on the y-axis), we can see that Imperial CovidSim both has the best historical fit, and the most accurate projections.”

While the Imperial model proved most accurate, the team emphasise that all three are of great value: “The importance of these models, however, is that they are not just for predicting the outcome when we stay the course with current policies: they allow us to experiment with scenarios that did not yet happen. Anyone can freely try out new mitigation strategies, and examine their effects far into the future. Society at large should carefully consider such projections under different hypothetical scenarios. The models had it right for the past six weeks, so it’s quite possible they will be right for the next six months.”

The GitHub team have also started to develop the Covid Modeling UI :  “a single UI, where the same scenario of interventions can be run through multiple models. With every model added to the UI, the combined value of the models increases, for experts, for policy makers, and for the public at large. Anyone will be able to run experiments like that reported here, for any number of geographies.”

The full report can be read on GitHub. [LINK]