Real-World Data Compared to Clinical Practice: Opposing Perspectives

Announcer:
Welcome to CME on ReachMD. This episode is part of our MinuteCME curriculum.

Prior to beginning the activity, please be sure to review the faculty and commercial support disclosure statements as well as the learning objectives.

Dr. Crowther:
The most important thing to realize about the problem, and I'm going to talk a bit about problems with real-world data is the very simple question, and people ask, when did the bleeding stop? And you can always answer that the bleeding has always stopped. Because people will call you in the middle of night and say, ‘The bleeding is never going to stop.’ And I'd say, ‘oh, yeah, it will stop. There's no question about the fact that will stop.’ So as Paul said, the real problem with real-world data then is defining when endpoints occur, because you need to understand that because there is not a patient who's had bleeding for 5 years, it just does not happen.

So are there advantages and disadvantages to real-world data versus clinical trial data? And there are advantages to both real-world data and clinical trial data. And I've been trying to do studies in warfarin and then Xa inhibitor reversal for almost 25 years. And they - it is an extraordinarily difficult area to do research in. And it's extraordinarily difficult to do research in, because as many of you will know, patients truly only have life-threatening bleeds at 2 AM on Friday. And so you need to have your clinical trial infrastructure ready to go at 24/7/365. And that is extraordinarily complex and extraordinarily expensive. But then more importantly, as Paul just highlighted, the real issue with clinical trial data in this setting, is how do you define when bleeding stops? Because if a person presents with a GI bleed, unless you're going to have an endoscope in them for hours on end to watch when that bleeding stops, you can never truly define when, within a time period that's reasonable, when bleeding stops.

And so I'll go to the right side of this slide first. Clinical trial data is collected in a highly refined group of patients who are subject to very careful inclusion criteria. And as a result of that, the results are going to always be very not generalizable compared to real-world data where it's collected in a much more representative sample. But the dataset is almost always more complete, because in order to do that study, you have a set of data that you're going to collect and then present. And particularly, if the study is being prepared for submission for a licensure or approval process, you know, the company is going to want extraordinarily careful data to be collected. So you'll have great depth of data in a very narrow slice.

And clinical trials are almost always done in experts centers. And so there's always a bias built into clinical trials that the way the study was carried out, even if the protocol is carried out exactly the same in a little hospital somewhere far, far away, and in, you know, one of the biggest hospitals in downtown New York, the big hospital in downtown New York has experience and expertise, and so they're going to do that intervention differently than the average hospital is going to do it. So this is what I call the experts center component.

Moving to the left side of the slide, the advantage of real-world data is it tells you how the product is actually being used in real people out in the wild, which is great. But it's always subject to a different form of bias, many forms of bias, but one of the most important ones is that someone actually had to select that patient to get the intervention. So in the data that Paul just presented, you know, if your hospital doesn't have andexanet, which is possible, you are not going to put any patients in the andexanet arm of that study.

If your hospital has andexanet, then you might put a patient into the andexanet arm of that study. If your hospital is systematically different from another hospital, which we know all hospitals are systematically different, then your data is going to be different than all the other hospitals. Real-world data is always subject to bias because clinicians have actually sat down with the patient and included them. So if the patient's going to die, they oftentimes won't get an intervention because they're sufficiently ill. And it's influenced by many factors, including patient's insurance and payment, the center's knowledge, and really importantly, the center's comfort with using a product. And so when you actually look at real-world data, what are some of the limitations around real-world data? Well, the most important one is that you can only collect the data that is part of the dataset. Whereas, if you're doing a randomized control trial, you can say, ‘I'm really interested in the left leg blood pressure in this person, because I think that's an important outcome.’ But if you're looking at real-world data, no one ever does a left leg blood pressure in clinical practice, so you're not going to get that data, you'll never be able to present it. So really hard datapoints like death, reasonably well discerned in both clinical trials and real-world data. But lots of endpoints just are not collected very effectively, as Paul just said in real-world data. Most real-world studies will have incomplete outcome ascertainment, and the endpoints are generally those that are measured in routine day-to-day clinical practice. This may result in less useful measurements than specific - if specific outcomes of interest are measured.

And there are ways of improving this. So for example, many big healthcare corporations will define a preset set of quality metrics that they want to collect on all people. And so if you can influence those quality metrics at the start of this process, then you can actually collect more valuable data. But that's a relatively limited set of circumstances within that, that can occur.

Oftentimes, when you get real-world data, you get the feeling that the investigators have defined the question after they looked at the data, which is never the way you want to do research because you get biased results. So the other thing you need to do is you still need to stick to the principles of designing a study where you know the question was formulated before they looked at the data. They may need to change the outcomes a little bit to reflect the realities of the data. But you want confidence that these people didn't look at the datasets and said, ‘Well, there's a big difference, let's design a study to highlight that difference.’

If we focus on reversal of anticoagulation, as I mentioned earlier, the big problem with doing reversal studies is it's difficult to determine the effectiveness of hemostasis, since it is difficult or impossible to determine when the bleeding stopped. It's just really hard to do that. But real-world data tends to be really good for looking at hard outcomes. So hard outcomes are things like death, duration of hospitalization, and things like transfusion requirement, because that's objective, it's always going to be captured in the chart. It's very valuable pieces of information, and it's based on real patients and real clinical situations.

Clinical trials data, on the other hand, can be procured or can be designed to collect really high-quality data for the duration and extent of the bleeding for as long as the study goes on. However, that oftentimes imposes severe restrictions on the data later on. So real-world data, you can look to see what happened to those patients who got transfused within the 3 months after hospital discharge. It's really hard to do that in a clinical trial because most of them aren't designed to look at those outcomes.

So if we look at some of the limitations of four-factor PCC studies for DOAC reversal, the most important one, as I mentioned a couple of times, is unclear definitions and a lack of adjudication, variable use of accepted definitions of hemostasis clinical judgments often used, inconsistent follow-up times, as Paul highlighted, inconsistent reporting from the time of the last dose of DOAC to the actual administration of the four-factor PCC, and a very wide range of four-factor PCC doses used from fixed doses to variable doses.

And when you look at the real-world outcomes of interest in examining the data, comparing real-world to clinical trials data, death, duration of hospitalization, and transfusion are highly relevant. They're fairly insensitive markers. Whereas, when a well-done clinical trial can isolate the effects of the intervention and outcomes of interest, but may lose sight of the big picture. For example, not reporting death. And clinical trial data generally will be less useful for infrequent outcomes just because of the size of the study.

So in summary, the happy medium, I think, in implementing something in clinical practice is to use real-world data to help you to understand how and why the information is being used, and what happens when you use it in a broad population, giving good estimates of outcome. But clinical trial data provides you like the microscope that drills down on the specifics of the treatment and allows you to better understand the circumstances within which the intervention works.

Announcer:
You have been listening to CME on ReachMD.This activity is jointly provided by Global Learning Collaborative (GLC) and TotalCME, Inc. and is part of our MinuteCME curriculum.

To receive your free CME credit, or to download this activity, go to ReachMD.com/CME. Thank you for listening.