You're listening to Lifelong Learning on ReachMD. The following program was recorded at the 2018 annual meeting for the Alliance for Continuing Education in the Health Professions. Here is your host Alicia Sutton.
Alicia Sutton: Thank you for joining us. We are at the Alliance for Continuing Education in the Health Professions at their annual meeting in Orlando. My guests are going to talk to me about how to use predictive modeling to maximize educational impact, so I'm looking forward to hearing about this, and you presented it already. Please introduce yourselves.
Jamie Reiter: I am Jamie Reiter, Director of Educational Outcomes at CME Outfitters. Thank you for having us, by the way, this is really exciting. I earned my Ph.D. from the University of California-Irvine, Department of Cognitive Sciences and I developed some mathematical models to analyze data; neuropsychological data, and ever since then it has been my passion to look at different ways of analyzing data, existing data as opposed to gathering new data. So, that is sort of where the predictive modeling came in. I'm super excited and geeked about sharing it with the industry and anybody else.
Alicia Sutton: That's fantastic. Geeks are our favorite interviews. Whitney, please.
Whitney Faler: I'm Whitney Faler. I'm the Director of Accreditation here at CME Outfitters. I earned a Master's in Public Administration from Virginia Tech several years ago and really looked at a lot of organizational change, behavior change within organizations and I really am excited how we can apply that for education when we look at whether it's from an organization standpoint of change, but educational activities because that is an organization when you get down to it.
Alicia Sutton: Sure. So, give us kind of a big picture of what you're learning lab was about here. It was about different modeling, regression modeling in particular, is that correct?
Jamie Reiter: In particular, yes. I am going to let Whitney kind of give just the general overview and then I might add on to what you want to.
Whitney Faler: When we look at outcomes, we are often just checking boxes in the traditional sense, and being able to add a level of did that education actually do something rather than did you like the chicken dinner and was the speaker engaging. It gives us a little bit more as we go through post tests and evaluations that ask for follow up. Did you retain that information? Now, when we can add confidence or behavior as an implication to better inform our needs assessment for the next activity and sometime you're gonna fail, and that's okay because that's going to inform you to be able to be successful in the future.
Jamie Reiter: So, one of the reasons why we value predictive modeling is because we see improvements and outcomes and occasionally we don't, very rarely of course. Whether we see improvements in outcomes or whether we don't, it's really important to understand why we're seeing those improvements because if we can repeat a successful formula of course we want to do that in future activities. If we can identify a reason why something wasn't successful then we can make changes in future activities. So, a lot of times we want to look at demographics and confidence, knowledge and see if that has an impact on clinician behavior. Ultimately, we want to maximize educational impact because hopefully that will lead to improve patient outcomes.
Alicia Sutton: I think in your presentation you referenced a few different models of doing that measurement. Do you have some that you really love?
Jamie Reiter: Of course. So there are several types of predictive models and, kind of going off on what Whitney was saying, usually in outcomes we measure changes from pre- to post and we need to do that. That's important. That kind of tells you the if in terms of outcomes. What predictive modeling does is it tells me the why and there are different ways to do that. There are different forms of predictive modeling. There's regression, naïve bays, neural networks, there's CHAID which stands for chi-square automatic interaction detection, and then there are some others. The workshop covered regression but what we do at CME Outfitters is we use CHAID because that enables us to look at both continuous and categorical variables in a single model. There are some limitations in regression in terms of that and then also I love the output because it's a visual tree format and it shows you kind of the breakdown of the response variables. It's kind of hard to explain but it's a really good way of looking at the data and looking at the results. But, regression definitely has its value and it's more straightforward and I think that's a good starting point for a lot of people which is why I focused on that in this workshop.
Alicia Sutton: How do educational designers get started in thinking about how to construct their education that will be easier to measure? Does it impact it that way? Are you seeing that? I mean, obviously, you're an education company, do you often think about the end first?
Whitney Faler: Yes. I think you hear across the industry to start with the end in mind, add the plan do, study, act, model which is incredibly important from the accreditation standpoint. Everything we do we start with a kickoff after we have our needs assessment, we have our proposal and then we say okay, here's what it's going to look like and that absolutely includes your content and how are we going to measure this. What are our outcomes going to look like? That is our square one for any activity that we start working on.
Jamie Reiter: And part of that is going to be…when you develop a proposal you want to look at your target audience. You want to look at what you're going to measure, how you're going to measure it, and so one of the things that the predictive modeling can help inform is, for example, if you found that academic degree influenced behavior then you might want to target certain audiences based on academic degree. I can give you an example of what we found at CME Outfitters is we found that confidence often times predicts behavior so the more confident you are the more likely you are to perform improved behavior. It makes perfect sense. So, not only is predictive modeling really cool in my mind, you know, to get at the why, but then what do you do with it and one thing we are doing at CME Outfitters is trying to find ways to improve clinician confidence with the hope that that will translate to improved behavior. One of the things we want to experiment with, this is all new, is see if reinforcement activities help build confidence. So, we're starting to build that into our design and to see if, in order to measure whether or not it actually is improving confidence; of course, we have a control group. So these are the kinds of things you can actually take the data from the model and help inform the design of your future activities.
Alicia Sutton: That's good. That's what I was trying to get to and didn't word it very well about how you would go about constructing new education. Knowing what you've learned from other activities that you saw, okay, you know what, you're right. We've got to their confidence and competence in a certain area, so that's good. How do companies go about getting started on thinking through that or measuring? Are there some basic takeaways from the presentation you made to help people get started with modeling?
Jamie Reiter: I'm going to answer your question in two ways. One is, when I started in this industry, I was Director of Biostats and Research at CME, LLC years ago back in 2009 and I noticed a need for statistics in this industry. So, I gave presentations and then I took a hiatus from the industry and went to BioTech Pharma and I came back and I was pleasantly surprised to see more statistics out there. So, I really feel that this industry is forward-thinking and very willing to do what's going to work and try different methods. So, that is more in the broad sense.
Alicia Sutton: That's good to hear that you've seen that change.
Jamie Reiter: Specifically, to providers if they're trying to decide how to get started, I think starting with something like regression just to get your feet wet, there are two questions you want to ask. Which activities do I want to do this on? I would say go for all of them, but that's not always practical. I would suggest maybe starting with activities where you're outcomes aren't as favorable whether it's knowledge or behavior. You pick that activity that is less successful and do the predictive modeling so then you can maybe get a better sense of why it wasn't successful. What a couple of people asked in the workshop today, well how do you decide what variables to use? Unfortunately, because this is so new in the industry, we don't really have guidelines on what variables to choose. So, all I can offer, which I hope is helpful, I had a few points. One is it is subjective, so what is important to your organization. In terms of selecting variables, we have the response variable which is the thing you're predicting like behavior and then you have your predictor variables like demographics, confidence, knowledge, and so selecting those it could be subjective. It could be based on some kind of research precedent where…I knew from prior literature that confidence often times did predict behavior so, of course, I wanted to include that in the model. Different organizations might have read up on different literature, academic degree might impact behavior. Research precedent, maybe what might be important to the supporters. Also, quality of data is super important and then one of the conditions in terms of getting into predictive modeling, you don't want your predictor variables to be correlated. So, if they are then you just drop one of them, and which one you drop is up to you.
Alicia Sutton: That's interesting.
Jamie Reiter: Just a way to kind of get started.
Alicia Sutton: Where do you see this heading? This could be any stakeholder answer. Are the supporters going to start looking for outcomes that are done in a certain way? Are educational designers going to start really thinking through, alright, now we have to get our criteria, our variables in there, established early. Where do you see this going in three to five years?
Whitney Faler: I think you're going to see a lot more sessions at meetings like the Alliance for how do we measure outcomes not just we did it, but how do we actually go through it. What was your process. Sitting in Jamie's session, it's so different from the other sessions because…She pulled up her Excel Workbook and was showing here's how I actually did the math, here are the formulas, and that can be really scary for somebody who does not have a math background. I stopped at algebra at high school and I haven't done it since, but the tools that are available now will allow organizations to get started and then bring on someone like Jamie such as CME Outfitters has done that can really take it to the next level. Just to play with it. Just to get started with it, I think is incredibly valuable for any organization to start to share that yes, this could really make an impact on how we develop education.
Jamie Reiter: And of course, I would love to see it utilized more. With any statistical procedure, you have to be careful how you use it. There are assumptions that need to be met for each different statistical procedure because if you don't meet the assumptions you're results may not be valid. That's something I brought up in this presentation, but I also really wanted to make it accessible and friendly so that people can just kind of get introduced to the idea and start using it. That was my goal, to just share it with the industry. I hope I made it a little bit more friendly in my workshop, maybe not, I don’t know, but that's my goal.
Alicia Sutton: Thank you. Well I'm sure there was great value imparted and people have walked away with some things to try in their own practices. Thank you so much for joining us Jamie and Whitney. We appreciate it.
Jamie Reiter and Whitney Faler: Thanks for having us.
You've been listening to Lifelong Learning on ReachMD, featuring key insights from the Alliance’s 2018 annual meeting. To download this podcast and others in this series, please visit reachmd.com/lifelonglearning.
