Transcript
Announcer:
Welcome to Heart Matters on ReachMD. Today, we’ll hear from Dr. M. Melih Dogan, a Postdoctoral Researcher at East Carolina University Heart Institute. He’ll be walking through the GRADE framework for assessing the certainty of evidence from randomized trials. Here’s Dr. Dogan now.
Dr. Dogan:
Randomized trials start at high certainty, because randomization is the strongest design we have. Then you look for specific reasons to downgrade, and there are five: risk of bias, inconsistency, indirectness, imprecision, and publication bias. Each one tells you something different about why to be careful, and I want to talk in detail for each of them.
The first one is risk of bias. This is about whether a study was run in a way that could distort its own results before you even get the numbers. The questions are practical. Was the randomization genuinely random? Did people who received the drug handle it the same way as people who did not? How many patients dropped out, and did they drop out unevenly between groups? Was the outcome measured the same way in both arms? And were all the outcomes originally planned actually reported, or only the ones that turned out well?
Second is inconsistency. This asks whether the trials agree with each other more than you would expect from chance alone. Some disagreement between studies is normal, ordinary statistical noise. But when results scatter far more than that, something systematic is going on. And if you can't explain what, you have to be more careful about the average you are reporting.
The third one is indirectness. This is the one I would most want people to take away, because it's the least intuitive and probably the most consequential. Indirectness asks whether the evidence actually answers the question you care about. It can go wrong in few ways. The patients studied aren't the patients you treat. The dose isn't the dose you would use. The comparison isn't against the alternative you're actually choosing between. But the most important version is about the outcome. A great many trials measure a laboratory value, a cholesterol number, a blood pressure, and a blood sugar. Those are called surrogate endpoints. What we are actually trying to achieve is fewer strokes, fewer heart attacks, and fewer cardiovascular deaths. And medicine has been sometimes burned by assuming the two are the same. But there were so many drugs that lowered the laboratory number meaningfully, but when someone finally ran a trial measuring actual events, it turned out it wasn't helping. Basically, the number moved, but the patient did not. However, using a surrogate endpoint in an early phase trial is not a flaw. Phase two trials exist to answer a specific set of questions. Basically, does a drug do what we think it does biologically, and is there any signal of harm? So the surrogate isn't the trial's mistake. It's the stage of development, and it's what prepares for the ground for phase three.
The fourth one is imprecision. This is about random error—whether you studied enough patients to pin the answer down at all. Every estimate from a trial comes with a range of uncertainty around it, and that range is what a confidence interval shows you. It tells you which values are genuinely compatible with the data you collected. If you are studying something rare like mortality, or, let's say, aortic dissection for thoracic aortic aneurysms, you will need far more patients than you would for something common.
And the last one is publication bias. The fifth one is about what we are not seeing. Studies with striking positive findings get written up, submitted, accepted, and cited. Studies that find nothing often sit in a drawer. So when you gather up everything published on a question, the pile you're holding may already be skewed toward the positive, and the average you are calculating from it will be too optimistic, though there was no fault of anyone's analysis.
So basically, those are the five lenses. Randomized trials start at high certainty, then you look through each lens in turn, and where you find a serious problem, you drop the rating level or two if the problem is severe enough.
Announcer:
You just heard Dr. M. Melih Dogan discussing how to use the GRADE approach to evaluate randomized trial evidence. To access this and other episodes in this series, visit Heart Matters on ReachMD.com, where you can Be Part of the Knowledge. Thanks for listening!