A new study by CRI Research Associate Ignacio Atal and his colleagues at Manchester Metropolitan University in the UK has been published in the BMJ-Open. The article shows that researchers who develop machine learning tools for medical diagnosis do not properly report the methods they have used in their research. This finding calls for a strengthening of research reporting standards to improve the reproducibility and transparency of machine learning in healthcare.
Over the past decade, access to large quantities of clinical data has led to an increase in the application of machine learning methods to medicine, and in particular to medical diagnosis. Based on large quantities of patient data with a given diagnosis (for example, photos of skin labeled “skin cancer” or “no skin cancer”), researchers train machines to automatically perform these diagnostic tasks. A machine learns to make a diagnosis by mimicking the diagnosis made in these large quantities of data. For example, if you give the trained machine a skin photo of a new patient, the machine will say that the patient has skin cancer if the database contains a skin photo. similar with a diagnosis of skin cancer.
If you want to rely on such a machine to make a medical diagnosis, you need to know the characteristics of the data used to train the machine, such as patient characteristics, how and by whom the actual diagnosis was made, where the data came from, and so on. Without this knowledge, it is impossible 1) to reproduce these studies and 2) to be sure that these results apply to all contexts.
In this systematic review, Ignacio and colleagues analyzed 28 published medical research articles reporting on the development and evaluation of diagnostic systems based on machine learning. For each article, they assessed the extent to which the authors reported the characteristics of patient data used to train their machines. They showed that a large proportion of the articles did not contain sufficient detail on the characteristics of the participants, making it difficult to reproduce, evaluate and interpret the study results.
Diagnostic studies using ML methods have great potential to improve clinical decision-making and ease the burden on healthcare systems. However, studies with insufficient reporting may be more problematic than useful. In the field of biomedical research, there are already frameworks and guidelines that machine learning researchers can use to facilitate their reporting, but most do not follow them. We hope that this work will encourage machine learning researchers in the healthcare field to improve the reporting of their work in order to increase the transparency and reproducibility of research results.
The full text of the study is available here.
