Kotecha et al. shared 4 key steps that healthcare researchers need to develop high-quality AI algorithms.
This first step involves making several decisions that will guide the rest of the development process. What questions are you hoping to answer? What kind of data do you want? Where will the data come from? What are some of the underlying biases that need to be addressed?
The choices made during this step, the authors noted, will impact every other step. If you decide to focus on unstructured data, for example, it may require pre-processing that would not be necessary if your team was only using structured data.
“Underlying biases in the data need to be explored, and then the data transformed into usable formats for machine-learning algorithms,” the authors wrote. “The output of any AI model is only as good as its input; therefore, pre-processing is a critical step to plan a study and understand its findings.”
During this step, researchers should also check for any missing or incomplete data. If some data do appear to be missing, it is important to determine if it is at random or the sign of a significant trend.
Different AI algorithms can perform different tasks. Are you providing the AI with human-derived data and looking for an accurate prediction? Are you teaching the AI how different variables are related to one another? This will help determine which approach is the best fit for your analysis.
Decision trees, random forests, deep neural networks, convolutional neural networks and auto encoders are just some of the algorithm types researchers have at their fingertips. Of course, each one has its own advantages and disadvantages—and there will only be more and more options as time goes on.
It all comes down to this crucial part of the process—grading the effectiveness of your team’s AI algorithm.
“Appropriate validation is needed in order to know how the study findings will apply to the real world,” the authors wrote. “The importance of evaluating machine-learning output is no different from any other prediction task, and there remain challenges in generalizing from one data set to another, and then again to actual clinical practice. External validation should be the default approach for all AI studies, and usually involves estimating performance on completely unseen data. A gold-standard method is to apply a learned model to one or more data sets originating from a different cohort or study, preferably from a different site or time interval.”
This is when researchers measure their algorithm’s accuracy, sensitivity, specificity and other performance metrics. In addition, it is another opportunity to consider any potential biases.
“The explainability of AI approaches is of substantial interest in healthcare,” the authors wrote. “Commonly, AI approaches outperform human experts in specialized tasks, yet they do not give a reason for a particular prediction. Underlying biases in the data might lead to misclassification, and so caution must be used when interpreting AI approaches if they influence the care of patients.”
Read the full analysis, including several examples of high-quality AI in action, in European Heart Journal here.