Announcer:
Welcome to CME on ReachMD. This episode is part of our MinuteCE curriculum.

Prior to beginning the activity, please be sure to review the faculty and commercial support disclosure statements as well as the learning objectives. 

Dr. Brufsky: 
Hello, I'm Dr. Adam Brufsky. I'm a Professor of Medicine at the University of Pittsburgh, and Co-Director of the Comprehensive Breast Cancer Center of the UPMC Hillman Cancer Center in Pittsburgh, Pennsylvania. And today we're going to talk about Distinguishing Between High- and Low-Quality Real-World Evidence: A Journal Club for the Real World.

So, I think, you know, the real question that we always ask in Interpreting Studies is, do the observed results represent the true effect? And that means truth in the study. And I think randomized clinical trials help us do that. I think that, you know, are the results of the study true? Or are they an artifact of the way the study was designed? And I think we always have to analyze data in that way. And you know, when we have a big practice-changing clinical trial, that's what happens. We all look at the data and try to figure it out.

And the other option is, are the observed results applicable in other settings? That's truth in real life. In other words, can I extrapolate the results of that large, really important practice-changing clinical trial to real life?

And so again, when we look at those two questions, randomized clinical trials are really good. I think that, you know, they really are high. They're really, because there's really hopefully no bias because of randomization, you know, we can really get to the truth of a study. And I think the problem of trying to get to the truth in a real-world evidence database is that there could be a lot of bias. And if we don't adjust for bias properly, we may draw the wrong conclusions about the study.

I think when we look at truth in real life, randomized clinical trials really don't help us that much. Because they clearly, you know, by their very nature, are very controlled. I mean, they’re patients with poor performance status that can't get on the study. Maybe there are certain minorities or certain age groups are unrepresented. And that's really tough.

But on the other hand, in a real-world evidence study, that's the whole idea is that we're going to see our patients are really treated in the real world by real physicians, in real-life situations. The data is messier, but it does give us a little bit kind of more about truth in real life.

So again, you know, the bias can happen in randomized trials and real-world evidence studies. I think randomized trials clearly are the gold standard, because we have random treatment allocation, and, you know, confounding can be measured. But the problem is that, you know, it depends on what baseline characteristics you have; you can't have an infinite number of baseline characteristics in a randomized clinical trial. So, you know, that really can - you know, that's tough, it's hard to kind of, you know, you hope that the unknown, you know, the unmeasured characteristics are randomly assorted, where real-world evidence is a little bit harder, I think there could be a lot of bias. I mean, again, as we said before, physicians can treat patients that are less sick or more sick with a particular drug, and overestimate or underestimate the value of that drug’s effects. And so that's really kind of what we really have to kind of be careful about, because real-world evidence is not blinded evidence. And I think that you have to be very careful with that.

And so, I think we have things called propensity scores and propensity score matching. And kind of what that does is, you know, it kind of matches the patients based, you know, and tries to kind of make it kind of almost a randomized clinical trial by matching the patients the best they can to kind of really, basically measure the probability of being assigned a specific treatment, you know, that given what’s known about a patient's confounding factors. And so that really tries to kind of even out both groups of the trial; even though they're not randomized groups, they’re two separate groups being studied at the same time, in the same timeframe. So, it tries to do that.

And there are lots of tools that assess the quality of these real-world evidence trials. I mean, I think that there's a variety of checklists and questionnaires that we as academic oncologists use all the time that are really interesting. This ISPOR Questionnaire or an ISPOR Checklist, this GRACE checklist, the ROBINS checklist. I think that what's going to happen as we get more of real-world evidence data generation, as we use it more and more in clinical practice, I think the journals are going to require these checklists or some form of it before we can publish the data. And again, as we as oncologists, you know, as we look at some of these real-world evidence trials, I think one of the things we're going to be asking ourselves is, have any one of these checklists been actually performed? It's kind of almost like, you know, those consort diagrams we see for randomized clinical trials, I suspect we're going to be seeing one of these checklists when we actually evaluate in a real-world evidence study.

So again, I think, you know, talking about real-world evidence, evaluations, I think, real-world evidence is at risk for bias due to lack of randomization and blinding. I think there are multiple confounding variables, the biggest one being physician choice. I think physicians, you know, we choose how patients are going to be treated. The protocols have been telling us that. And there's a lot that goes into that decision based on our own experience with a particular drug or an experience with a particular kind of patient. I think that leads to differences in baseline characteristics.

One thing we really didn't talk about is a lack of data. You know, when we look at medical records, there may be data fields that are important to baseline characteristics that we don't have. And you'll always see in some instances it says other or unknown, and that can be as high as 30 or 40% in some trials. But we do have statistical methods like propensity score matching, to try to minimize bias.

And we have tools and checklists that exist to try to help us evaluate real-world evidence credibility.

And so, with that, I’d just like to tell you again, thank you for listening to me and for this section.

Announcer:
You have been listening to CME on ReachMD.  This activity is jointly provided by Global Learning Collaborative (GLC) and TotalCME, LLC. and is part of our MinuteCE curriculum.

To receive your free CME credit, or to download this activity, go to ReachMD.com/CME. Thank you for listening.