Casper, A Situational Judgment Test: All You Need to Know [Episode 513]

Accepted

1 year ago

Categories: Admissions ConsultingAdmissions Straight Talk PodcastGrad School AdmissionsMedical School AdmissionsOther Healthcare Admissions

Casper, A Situational Judgment Test: All You Need to Know [Episode 513]

Podcast: Play in new window | Download | Embed

Shortcuts to Key Insights from this Episode

How to prepare and practice for the Casper.
A detailed explanation on how the subjective test is scored.
The most recent development in making Casper more equitable across all demographics.
What happens if raters identify a red flag.
Will Casper spread to other fields such as social work and law?
Common misconceptions about Casper.
Relevant Links and Related Shows

Do you have to take the CASPer for at least some of the programs you are applying to? Are you concerned about? This interview with the co-founder of Acuity Insights, the folks who provide the Casper, is for you! [Show Summary]

Dr. Dore discusses everything applicants need to know about taking the Casper and what it truly tests.

Interview with Dr. Kelly Dore, Co-Founder & VP of Science and Innovation at Acuity Insights and Adjunct Professor at McMaster University at McMaster University. [Show Notes]

Do you need to take the Casper for at least some of the programs you are applying to, are you concerned about it? Well, this interview with the co-founder of Acuity Insights, the folks that bring us the Casper, is for you.

Welcome to the 513th episode of Admissions Straight Talk. Thanks for joining me today. Are you ready to apply to your Dream Medical Schools? Are you competitive at your target programs? Accepted Med School Admissions Quiz can give you a quick reality check. Just go to accepted.com/med-quiz, complete the quiz, and you’ll not only get an assessment, but tips on how to improve your chances of acceptance. Plus, it’s all free. Again, take the quiz at accepted.com/med-quiz to obtain your free assessment.

Our guest today is Dr. Kelly Dore, co-founder and vice President of Science and Innovation at Acuity Insights, and also adjunct professor at McMaster University. Dr. Dore, welcome to Admission Straight Talk.

Can you give an overview of the Casper exam and its structure? What can applicants expect? [1:31]

Sure. So the exam most people who are familiar with from Acuity is Casper. Casper is an online situational judgment test. A situational judgment test, it’s different than a behavioral interview question you might have. If people are familiar with doing interviews where you say, tell me about a time you managed multiple responsibilities at once. Instead, an SJT presents a scenario in the case of Casper. The majority of those are video scenarios too; you’re watching a 60 to 90-second video. You have a role in it that we tell you at the beginning, and you watch that video and then you get a series of questions after it. So you watch the scenario, see what’s going on, and there’s then three follow up questions talking about what you would do and why you would take that course of action.

The total exam takes around 90 minutes, and during that time, there’s opportunity for breaks. There’s the chance to just reflect, so you’ll watch the video. There’s a chance to have a reflection period before you go into typing your response or recording your response, depending on the moment In Casper. Now Casper broadly is intended to measure social intelligence and professionalism in applicants. It was developed so that we could get a more holistic picture of applicants supporting that holistic selection process and the Casper exam. We’ll go into more detail about it, but it is something that we hope supports the application process.

When can applicants take it? I know it’s online. Can they take it at six in the morning, wherever they are? Can they take it at 10 o’clock at night? [3:01]

Absolutely. So because it is online, and I always say we were online before online was cool. So back in 2010, we started with an online examination because we started in Canada and we wanted to ensure that we weren’t disadvantaging our rural and remote applicants. They weren’t having to travel to test centers to take this examination because the intent of it is it’s used early on in the admissions process. So before you’ve gone for an interview or things like that, this happens. So you don’t want to have to travel or incur costs for accommodation and things.

So the way that Casper works is because it’s a proctored examination, there’s set test times. Now there’s a variety of set test times. It’s not just one time. So when you’re applying, if you’re applying to physiotherapy, you can go in, look for your school and look at all the different test windows that are open to you. Now, the test, as I said, takes about 90 minutes. You have a two-hour window to complete the test. So it’s not that it’s to the moment and then your computer shuts off. You have a little bit of flex time, and of course, folks who require accommodations can apply for that if they need extended time and things like that.

Is there a different Casper rexam if you’re applying to medical school than if you’re applying to DO programs or PA programs or whatever? [4:13]

Yeah. There are different Casper tests that exist. So first of all, the Casper tests that’s even between Canada and the US are different Casper tests. Because they take into consideration some of the cultural nuances and settings that are appropriate for that country. And also the raters who rate that examination, and we’ll talk about that a little bit later, are from that country. So you don’t have people from Canada rating U.S. tests and vice versa.

Now, we also have different levels of tests. So people who are coming out of high school applying to a nursing program, they get a different level of Casper test than somebody may be applying to medical school, and then somebody applying to residency would get an even different level of test. So we’ve got that set up for the level of difficulty. And then in addition, we also have little test groups as well. So MD and DO applicants can write the same examination because we know there’s cross application there, so we didn’t want them to have to write two examinations. And then the others have their own test windows as well. Sounds a little confusing as I’m explaining it, but when you go on and enter your school, you’ll see that the test is set there, it’s set for you at the right level for the school that you’re applying to.

How many different kinds of tests are there? It sounds like there’s a lot. [5:34]

It does sound like there’s a lot. We have an amazing content development team who works with a lot of equity and diversity in mind as they’re developing them. But they also keep in mind they want to get the right level of difficulty. The scenarios that we present, so for example, if you’re applying to a medical school or you’re applying to a nursing program, there’s not clinical content in there that you need to know because we don’t want to give advantage to anybody who’s family is a physician versus those who are maybe first in their family to attend university.

So the content is intended to be relevant to everybody. So that’s why when we have these different levels of tests, we want to make sure that people who are leaving high school have had the opportunity to have the experiences we have in this scenario or can at least relate to them. And then similarly, people applying to medical school, they may have had other examples of things in university they would’ve encountered. So we can have a little bit different scenarios there.

So the goal is obviously to assess judgment, I guess cultural awareness, the non subject matter related aspects of a medical school or healthcare application, correct? [6:35]

Healthcare application, absolutely. So we’re wide across healthcare. So a variety of different types of programs use us to apply to. One of the important things is that while we broadly measure social intelligence rather than professionalism, there’s a lot of sub areas that are covered in that. So communication, collaborations, ethics, empathy, they’re sometimes referred to as the non-cognitive skills or non-academic skills. I actually have heard them recently referred to as the durable skills because the technical skills and the knowledge stuff is constantly changing in the world that we live in. But these are the things that are durable throughout somebody’s lifetime and training, and I really like that term.

I like that too. So that’s what admissions committees are gleaning from it. This professionalism, the socialism intelligence, is there a way for applicants to prepare for it or practice for it? [7:26]

Yes. And because we know, and I can tell you this, when we started this examination in McMaster back in 2010, a single medical school in Canada, there was a test prep company that popped up when we started it and I said, “We can tell you how to write Casper.” And I thought, “I can’t tell folks how to write Casper. I don’t know how they’re doing it.” And we really wanted to ensure that we were creating an assessment that didn’t provide an opportunity for third-party coaching in that same way. We didn’t want you to have to pay in order to do better on the examination. So a couple of things that we did around the structure of that.

So first and foremost, we do have a practice test. There’s a technical requirements check that you can do and a full practice test. And the reason that those are important is that when we did a study actually looking at people’s preparatory methods for taking Casper, whether it was third-party coaching, doing the practice test, sitting in their room thinking about the types of things that were measuring, the only thing that had a little incremental value was really the practice test that they could write. And it’s because we know this, it’s a familiarization with the format.

So you go on, there is a time limit for the answers. And I think that is one of the things that people are always worried about is that you can actually go on to complete some of this. You can do it as many times as you want, if you want to do it 10 times. It’s all about making you feel more comfortable in a stressful situation as much as possible. So I really encourage. This is what I tell all of the pre-health advisors, always get your applicants to do the practice test, make them feel comfortable. They know what they’re getting into, whether that’s typing their response or video recording their response. It just gives them that opportunity to feel a little bit more comfortable with that. And then in addition, we send out a ton of materials.

So on our website, there’s a webinar that you can watch that our applicant support team who’s amazing goes through everything for you, how to sign up, how to do all the things you might want to do to prepare, how to get a quiet space for yourself, et cetera. And then there’s also emails that go out in other literature that we provide to say, “What are our tips and tricks? If we could tell you everything that we would do if we were writing the test, what would we tell you?” And that’s all there free and accessible for folks to go and grab, which is really important because we want to make sure that as much as possible, we’re able to level that playing field. So really encourage folks to go try those out.

I think one of the biggest concerns that we hear from clients and medical and healthcare applicants of all fields is how do you grade and test something as subjective, that’s judgment? [11:04]

That’s a great question. You have to have experience. And I think the other thing about bad judgment is we all do make mistakes. Any of us who said, I don’t make mistakes would be but-

Using bad judgment. [11:31]

Exactly. And the whole point of that though is to learn from it. And if you’ve learned from it, then really at the end of the day you’re doing better because you’re really starting to reflect and think on things. Not everybody does that, unfortunately. Some people still continue on to go make bad judgments, but hopefully most of us are trying to do that. So when we actually started to create this test in the first place, that was part of our consideration. There’s really cool research by Shiphra Ginsburg out of the University of Toronto who’s looked at professionalism. And what she found is that if you get experts, so these folks who are faculty members who are supposed to be able to measure professionalism, and you get them to watch a scenario, what’s interesting is they can’t agree on a course of action.

And that what you realize here is when we talk about these complex professionalism scenarios, there’s no single right answer that exists. There’s lots of shades of gray in all of this, and they’re all impacted by exactly what you say, our own experiences, our own pathway, the things that we’ve encountered, the context around the thinking that we’re doing at the time. I was always the person who when I was writing multiple choice examinations in university, I would write in beside the answer why I picked that one, because I need to contextualize it. Just that gives you a little weird insight into my psyche.

So one of the things that helps you realize is that there’s a lot of thinking that goes into why we pick an answer. So when we created the Casper examination, intentionally we made it open-ended because we don’t think it’s right for applicants to look through and pick from a list of five the most appropriate, because that varies a lot. And Shiphra Ginsburg found the same, that if she had to try and get experts to agree on a right course of action, she had to simplify and simplify and simplify the scenarios again until it was obvious.

So we really want to have these complex professionalism scenarios. What we need to do is actually have open-ended responses where Linda, you and I would see the same scenario, but we’ve had completely different experiences in our lives. So we may take the same course of action, but for very, very different reasons, or we might take different courses of actions, both of which are very appropriate when we contextualize them and we explain our why behind it. And it’s really that why that helps us score Casper. If we just had people write their answers about what they would do, in all honesty, we wouldn’t get the right information. It’s why you take that course of action that really helps us know A, it’s an authentic response. You’re not just trying to pick what I think is right, what you really feel and believe is the right course of action.

And then the other part of it is our rating part. So I always tell applicants this because I think it makes them feel better. Our raters have to do a Casper test so they know what it’s like to write your response or be saying your response and get cut off. So they understand that sometimes you don’t finish your last thought or you don’t, so they understand a lot. The other thing is that when we get them to do the rating, we don’t give them an ideal response to grade against, because again, Linda, when we’re answering a scenario, you might talk about communication and empathy, and I might talk about collaboration and problem solving in my response, both of which are very appropriate or-

Social influences. [14:46]

Or social influences, exactly. So we might take it in slightly different directions because of things that are important to us. So we don’t want our raters going through and just having this pre-prescribed idea of what the right answer is because there is no single right answer. And as they’re doing this, they’re going through and they’re looking at applicants as they go through multiple applicants for the same scenario. So they’re benchmarking as they go through and after they’ve rated 20, they can go back and rescore those responses because I may think something is a seven out of nine, and then after I’m like, “That’s actually an eight or a nine out of nine,” after I see the other responses. So it’s relative to your peers as well. And people, what we’ve seen, and we can give example responses is that some folks say, I would do A, and some people say they would do B and they score the same, and that’s one of the most important things.

The other piece is that there’s still subjectivity in it, that’s right. Most of the world is subjective, unfortunately, whether you like what I’m wearing is subjective or not, whether or not I introduce myself properly or explain the Casper test well is subjective or not. So one of the most important things is this concept of multiple independent responses. So a GPA is valuable because it’s not just a measure of a single course. A multiple choice exam is valuable because it’s not just a single question. Casper becomes valuable because it’s not just a single scenario. This year there’s 14 scenarios that folks are going through to get that score, and it’s your performance on each of those. So you could have a Casper scenario and you say, “I did not answer that one.” Well, you just feel like you didn’t nail it as much as you would like to. That’s okay. The other raters have no idea how you answered that scenario. You just go on and you start again fresh with-

So it’s different raters rating? [16:42]

It’s a different rating-

It’s not by the exam, it’s by the question? [16:48]

Exactly. So we’ve removed this thing called the halo effect. The Halo effect is an unintended first impression bias that we all get. So I come on the call, you’re from Los Angeles and we didn’t have this conversation, but if I told you about this great story I had on my first visit to LA. That might make you think differently than if I didn’t know you were from LA and I said, “I don’t really like that city.” That just creates this unconscious bias in your head, just like a panel interview. So part of the reason panel interviews are difficult is that folks can go in and if you’re really nervous on the first question and you stumble and you don’t feel like you’ve got it, you still have to answer the rest of the interview questions is part of the same group and you’re constantly thinking about how they may have seen your response.

But with Casper, it’s a brand new rate for each scenario that you write. So what that allows you to do is have a 14 different fresh starts as you’re going through and really present your best self as you’re going through. So that score that you get at the end is 14 independent impressions of how you did across 14 different types of scenarios.

And the rater is only rating, let’s say question one? [18:00]

Question one for multiple different-

Another rater is rating question two and another rater question three. I assume you have multiple scenarios, you don’t just have one set of scenarios for each program because otherwise it gets shared and all that stuff. [18:04]

Exactly. Yeah. And the nice thing about that too is that because we know with implicit bias, we all have bias, this unconscious bias that we all have by having multiple raters, it allows us to minimize the potential impact of that bias across the examination.

I viewed the sample videos that you have online, the sample scenarios, and I was struck by the fact that they were all basically ethical dilemmas. [18:36]

And there is ethics in everything, in all honesty, it’s this umbrella over a lot of things. Exactly. And the scenarios we used are probably some of the ones that we’ve gotten feedback over. We actually are going to likely update them. So a lot of them you’ll see, sometimes it feels like communication goes over a bunch of scenarios, and that makes sense because you can’t cover just one dimension in a scenario, and that’s it. These are very, very multi-dimensional. And because we’re talking about professionalism across all of these scenarios, it also is your lens through which you’re seeing them a little bit too. You’re seeing the equity in them, you’re seeing the different pieces in the different scenarios. So one of the things that we’ve heard is that some applicants see one thing through the scenarios. They were like, “These were all about collaboration.” And somebody else said, “These things are all about ethics across the scenarios.” So it’s really interesting because it does allow that different lens for folks to have.

I remember the one about the fellow whose wife wants him to take paternity leave and he’s worried that it’s going to impact his future and his career. It was interesting. And he was talking about it with his colleague, and I was thinking to myself, “Why doesn’t he talk to his wife?” I was thinking, “Discuss it with your wife.” [19:49]

Yeah. There’s a lot there, I think. And I think in what we’re trying to do is no scenario is perfect because they’re all meant to be complex and a little bit.

They were. [20:20]

Yeah. So I think one of the things that we want folks to do is it really stimulates your thinking. Each year we modify the test because we take a lot of applicant feedback. We also use stakeholder feedback and data as well to try and evolve the test each year. So one of the things that we’ve done is after you watch the scenario, we’ve added a 30-second reflection time. So that really allows you to pause and say, “What was going on in that scenario and what was happening before you see the questions?” And very similar to what you’re saying there is because you do get these thoughts of like, well, “Why isn’t he doing that?”

Why is he talking to his colleague? Why is he going to talk to his wife? [20:56]

Exactly.

Casper introduced video recorded responses last year. Are you planning to keep them going forward? It sounds like you are. And did you see any change in scores upwards or downwards with this change and another… Well, I’ll ask you my next question in a minute, but go ahead. [21:03]

Okay, great. So for anybody that wrote the exam last year, they will have seen that we’ve evolved from just having typewritten responses, which is where we started back in 2010 to now also allowing some of the scenarios to be video recorded, just like a Zoom session. And then also other ones typed. And there were two reasons for that. One was applicant feedback that especially in the world that we live in, that they don’t want to just type their responses. They want to be able to communicate. And that ability to capture verbal communication is also so important because not every program can have an interview and not every program wants to wait to really value that communication piece until the interview time. That was, in fact, the first reason why we created Casper is that in our medical school, we were having to weight GPA so heavily, we used the MCAT verbal reasoning component back in the day, but we weren’t able to really measure these personal professional attributes early enough on.

So that was the original inception for why Casper was created. But when we started doing it, when we were at the edge of technology back in 2010 and scary as that is to sound, we were really just doing type responses because the internet wasn’t where it could be for people to actually capture videos. Now, the data has also said about videos that you can actually further minimize demographic differences with the use of videos, which is very counterintuitive, which we talked about implicit bias and readers actually seeing the videos. So before last year, we actually piloted it for two years before that, so where people could opt in and voluntarily complete the video at Casper. That allowed us to start collecting data and understanding because I’ll be honest, I wasn’t convinced when I read that first study that we would see demographic differences decrease, but amazingly enough, that’s what we saw.

So SJT, situational judgment tests and Casper in particular tend to have significantly smaller demographic differences than things like GPA, GRE, MCAT, those standardized academic tests. So because we’re measuring something different, we tend to have smaller demographic differences. So that means fascinating differences between the two groups. However, there were still demographic differences, they still existed, those group differences.

So with the addition of AV, we’ve found that we’ve been able to even further decrease those demographic differences. The strongest area where we’ve seen that is actually for Black or African American applicants who write the examination. So last year everybody wrote it who completed a Casper test, but we didn’t use the scores because we really wanted to test and make sure we were introducing something that made sense, was defensible in what we were doing, but also really did provide that demographic differences decrease that we wanted to see.

And we saw the results repeated for every different group. So whether that was nursing or OT/PT or MD, DO or GME for applicants to residency, we saw it time and time again. So for that reason, we made the decision to include it this year in the examination, which is in all honesty, so exciting to me because while we’ve made changes to the exam each year to try and make it more equitable, this is really, I think one of the first times a standardized test has really made a monumental change that’s demonstrated value in terms of demographic impact.

I guess that’s the difference in literally looking at people as individuals as opposed to as members of a group. [24:44]

I think so. And despite the fact that they’re videos, and again, I keep going back to the fact that this was counterintuitive to me at first. We’ve been able to see through a combination of the way that we train raters, but also because of the snapshots that we’re getting and we’re having, again, those multiple independent impressions, we’re able to capture data that not only predicts for future performance is defensible, but also now has increasingly smaller differences between groups that are applying, which just aligns with our mission of creating more equitable assessments in creating a world served by exceptional professionals.

Do medical schools or healthcare programs, whatever program they’re applying to have access to the actual exam, the videos, or do they just get the raw score? [25:32]

Great question. So applicants who write get their quartile when they write. They understand where they fall.Part of the reason that we did that, and I just want to explain that, is that Casper is not like the GRE or the MCAT. It’s not a cut score that schools have-

Like you get a 512 or a 515- [25:59]

And you’re in or you’re out. It’s not that way. It’s used in a way to support the holistic assessment of applicants. So it’s used in a rubric or in a formula along with people. So it’s not a cut score in that same way, I want to make sure people understand if they find out they’re in the bottom quartile, it does not mean that they’re ineligible for getting in.

So all they find out is what quartile they’re in? [26:22]

They find out what quartile… The schools actually get the scores and they get the average score. So they don’t see the score for the individual scenarios. The only time that they would do that is if our raters do rate for red flags. So if there’s egregious racism, anything like that, that is in the response, the rater flags that. Now we at Acuity don’t make a decision about what to do with that that’s passed on to the school, whether or not they want to consider it a flag or not. So in those cases, that would be the only time. But in all honesty, I’m grateful to say that those are few and far between, which is very important.

So they get that they don’t see the videos or they don’t get the individual responses because we want them to be able to use the data. And what we know, and sadly enough we’ve heard, is that we know personal statements have limitations. They have bias in them, they tend to be not personal statements. They tend to be the collaboration of an entire community to write sometimes. Because if you have three months to write a high stakes personal statement, I’m going to get everybody I know to help give me feedback as you should.

That’s by the way, in my opinion, a recipe for disaster if you get everybody to comment. [27:32]

I could not agree with you more.

So we know, unfortunately, that a lot of the personal statements that are written don’t get read because schools don’t have the resources to sort through them. That is unfortunately in some cases, the sad fact. Schools are getting better at that, which is the most exciting part because they understand it’s important to have more data points on applicants. But what we know is that if we gave schools every single scenario, they would just get overwhelmed with data. And each individual score itself is not a defensible score. It’s the collective that is a defensible score, and that’s what we want them to focus on.

Is the Casper predictive of success in healthcare programs, and as a clinician, has there been any studies to correlate Casperto passage on the Step exams or other licensing exams, especially the clinical ones or to the objective structured clinical examinations in medical school or other exams? [28:19]

Those are the key questions. It’s like you might be measuring something, but are you measuring something that matters? And the exciting thing for us is that we’re seeing that data. What’s exciting in my mind is that that data is repeated school after school, after school. We’re seeing it repeated. What we tend to see, so in medical school for folks who are applying elsewhere, tends to be the first couple of years. It’s changing a little bit, but they tend to be more lecture-based where you’re focusing on anatomy and physiology and all of those really important things. But Casper doesn’t correlate with those things. We know Casper doesn’t correlate with GPA. It doesn’t correlate with those. So it doesn’t correlate with your measures of your anatomical knowledge, that shouldn’t. If it was, we would be measuring the wrong thing.

Where we see Casper starting to correlate is in those third and fourth years on OSCE performance. So what we tend to see is that it adds strength to the prediction, so additive to GPA or additive to the MCAT. So it makes it stronger. So it’s a combination of knowledge and personal professional attributes. Not surprisingly that they’re measuring in OSCE. We actually had the good fortune of doing a study with the National Board of Medical Examiners who do the Step exams at the end of medical school. And what we saw, not surprisingly, is that your Step one exam does not correlate with Casper. That is not surprising. It shouldn’t.

That’s a knowledge exam? [29:56]

Exactly.

That’s step two and step three? [29:59]

So Step 3, we didn’t have the data for unfortunately. Step 2 CS, we saw some increased correlations. What we saw, if we look at the Canadian National Licensing exam, and we published a study on this as well with the Medical Council of Canada, is that what we see is that Casper correlated with a component on the national licensing exam in Canada, which is roughly equivalent to the Step 2 CS, which had a component called the CCLEO and PHLO, which is the cultural, communication, legal, ethical, organizational and population health components. So big words, yes. What it predicted for if you did well on CCLEO and PHLO, it predicted good peer review out into practice and lack of professionalism complaints. So we saw Casper correlating that, which is a surrogate for later performance of course, and we’re ongoing doing these studies, trying to collect more data.

When we look at higher education or other things, such as occupational therapy, we’ve seen correlations with OSCE exams as they’ve gone through over and above GPA. So that’s like, is it telling the school something useful? We’ve seen that data repeated in physician assistant programs, and most recently, even in teachers education and business schools, we started to see as people go into co-ops and get their work placement ratings very similarly, we started to see that. We also saw, when we think about beyond residency programs that are starting to use Casper, when they start to use it, they saw a decrease overall in both professionalism incidents as well as formalized remediations they had to do in professionalism. And why that’s important is that we know we’re actually relatively good at remediating knowledge.

So if somebody has a knowledge gap, we tend to have good solutions and a good pathway for them to follow to get them better. When it comes to professionalism, we don’t have those same tools. We don’t have those same frameworks in place to remediate and support. We tend to rely a lot on external resources, and they tend to fail a little bit more. So for that reason, we know it’s adding something to the process of helping to find people who are incredibly intelligent but also strong in their personal and professional attributes.

How does Casper differ from the AAMC situational judgment test, which I think is called PREview? [32:21]

Great. So the AAMC for medicine has come out with a situational judgment test called PREview. What’s interesting is that they’re completely different formats. So even though they’re both situational judgment tests, they actually are collecting different information on the same thing. So very much, I use the analogy of when we’re talking about admissions to medical school, you have GPA and MCAT. They’re both measuring academics but different pieces of it. And I think the same is true with PREview and Casper. So PREview is a fixed response SJT. So where you have a series of response options after you’ve seen a scenario. So very similar, you get a text-based scenario with PREview, and then you rate the effectiveness of different options. The limitation to that is that, of course, those are pre-selected options from a group of subject matter experts.

So we’ve got a fixed option. You mean multiple choice options? [33:18]

Yes, multiple choice options. And you rank the effectiveness of each. Yes, absolutely. PREview is really measuring your knowledge of what is the right thing to do in that scenario. Do you know from when you look at the list, which is the right thing to do, which is an important data point. Casper on the other hand, is measuring what you would do in that scenario. So you would take this course of action and why would you take that course of action? And it allows a little bit more flexibility in which one you would personally pick. Because what I know about generating response options is that even if I were in a room and I have been with incredibly brilliant folks at a conference workshop or something where we’re all brainstorming, we will not come up with every answer.

Students and applicants to health science programs and higher education will blow your mind with the way that they think through these complex problems and the ideas that they will come out with. And they’re things that we would’ve never ever thought of. And to me, those are the superstars who can really be creative and agile in their thinking and come up with something really from the heart based on their own experiences and things like that. And that’s the part that I think we miss otherwise. So fundamentally, they’re both situational judgment tests, which is a very, very defensible way of measuring things, but the way that we’ve set up the questions and scenarios actually measure different and I think complimentary things.

I wonder if any medical schools are going to start requiring both of them because they are complimentary. [34:46]

Yeah, I think they are. It’s interesting as schools start to adjust and shift moving away from just really MCAT or GPA or GRE or whatever the right assessment is that they might require or really a GPA heavy application to considering other things. We do know that some schools do require both of them.

You also offer an exam called DUET. And what is DUET and what does it test? How does it test it? I mean, we focused very much on Casper, so maybe how does it differ? [35:15]

So what we did is wanted to really look at what are some of the things that applicants think are really important and programs think are really important, but maybe we don’t measure really well. And one of them that came to mind once we developed Casper was this concept of “fit.” Is this program the right fit for me? Is this applicant the right fit for our program? And there’s a lot of problems with the term fit. It is a little bit assuming that there’s a single fit that people have, and if you don’t look like this cookie cutter shape, then you are not the right fit for a program, which I don’t think is right. We know that the more diverse a class is, the better for both the class and the program outcomes and the profession as a whole.

So we wanted to really say, “Okay. How do we take this concept that we know is really important to people, but do it in a more defensible way?” Because right now, programs are measuring this concept of fit by really very arbitrary ways. When I interviewed folks saying, “How do you measure fit? What do you look for?” Their answer was, “Well, I know it when I see it.” And oftentimes that means they looked like me or sounded like me or did something. So what that does is perpetuates some of the biases that we have in the process or doesn’t allow everyone a fair shot at the application. So we really wanted to say, what else could we do? And so DUET has that in mind. It’s how do we take the values and priorities of an applicant and the values and priorities of a program and actually align them in a way that makes sense.

So the way that we do this is a forced choice or paired comparison selection process. So it was a very long process to create this. We have one version for residency and one version now for MD and DO with other versions coming out soon. So what we did is get a lot of stakeholders including applicants to these programs, current students, faculty, all sorts of different stakeholders to say, “What are the things that really matter when you’re talking about fit?” Now, this isn’t taking into account geography or all those other things that also matter. No silver bullets in this conversation. We don’t have the answer to everything, but incrementally, how do we add data? The analogy I use is to add more pixels to the picture to get a clear lens of who it is that are actually applying to our programs.

So what we know is that with DUET, we were able to get three categories with multiple characteristics. And I’ll just use the example of if you could have a program that promotes diversity or has a reputation for excellence, which would you want? And the answer is, well, I’d want both. So there’s no way to give applicants or programs just a list of these characteristics. So we have them go through this comparison process. I use my kids as the analogy where during COVID, like many people, they wanted a pet. If I just said to them, “Do you want a dog, a cat, a fish or a turtle?” They would say yes to all of them, and I would live in a zoo. So what I could have said, “Okay. Do you want a dog or a cat, a dog or a turtle, a cat or a turtle?” And have them pick between each of those pairings. And what I would end up with is a numerically prioritized list of these animals. We did end up getting two kittens, by the way.

So what we do is we do that similar process with values like promoting a culture of diversity and a reputation for excellence. But instead of just picking A over B, it’s how important is one over the other, because it’s not just a simple, ”I care about one and don’t care about the other.” It’s, you’re allowed to have that sliding scale of importance for those. And by doing this, we’re able to get a sense of who the applicant really is or what the applicant’s priorities are and how they align to the program. Now, it doesn’t mean if somebody aligns with you not perfectly that they’re not the right fit, that means A, you have an opportunity to support them when they get into your program, which is a really great piece of data. But also you have an ability, we’ve given programs a chance to actually look at what they say are their values and priorities and say, “Is that right? Is that what you want to be doing?” And that’s been a really great opportunity.

So the DUET assessment as a quick summary takes about 10 minutes to do. It’s an online examination. You do it on demand. So anytime you register for your Casper test, you can go and sign up and complete DUET. We just say do it within the two week window of when you’ve completed your Casper test so that the school has your data. Then once you’ve completed it once, that’s all you have to do. So an applicant completes it once and it’s sent to every program that they’re applying to that requires DUET, and that’s compared to the program profile. Now, the programs go through the DUET process too.

I was just going to ask you. [40:40]

They take the 10 minutes and they go through, and what’s great about it is it’s not just the admissions director who’s doing it or the program director. We get a combination of faculty and current students to do the DUET, to actually create the profile. So it’s not just what do they aspire to be, but the students in the program are actually saying, “What is it actually?” And that’s where it’s great data for the program to have to say, “Is that actually what you want to be doing?” But also it really gives a much more authentic sense of this concept that we call fit. So we’ve started to use the term instead of fit, we call it applicant program alignment, which really-

Just heard that term for the first time, I think yesterday. [41:18]

And it’s this term alignment that really to me is a sense of how aligned are you? Because you could choose and not be very aligned to a program, but go to them for a variety of different other reasons. They could be geographically the right school for you. There could be a lot of other things, but it’s knowing that going in so you can respond and reflect on that.

Now with alignment as the goal, will there be a lessening of diversity of opinion and thought? [42:36]

Great question. So one, I would hope not, and the reason for that is this is alignment in a few very specific areas. So it’s-

So alignment of values to a certain extent. [42:57]

Values and priorities, absolutely. One of the things, and part of the reason I love Caspe ris that it allows that diversity of thought and very different perceptions of what a right answer is. With DUET, you’re getting a sense of, is this person really looking for a research heavy institution and that’s not what you are, then maybe you can’t support them. I always say that most of us are not weeds. There are some people who can grow in any soil you put them in. I am not one of those people; I need the right soil in order to be my best self. So it’s about making sure, are we putting them in the right soil there or are you prepared for the fact that you’re bringing in somebody who requires a very different soil? And so you need to be prepared to support that person because that’s an intentional choice, which is great.

I also really want to emphasize though that these are two data points in what should be many data points about the selection process. So it shouldn’t be just that we get people with a 3.8 or higher GPA in, because to me that brings in only a very small proportion of who could be a great nurse or healthcare professional. When we think about adding these other data points, that’s what helps us broaden and get that more holistic impression because we want folks who have had different opportunities in life, maybe folks who have never volunteered or never been able to shadow clinically because they’ve been working or supporting families or doing other things that are incredibly important in our society.

So how do we make sure that across the whole spectrum of the admissions process, we’re not putting artificial barriers in place. So for example, in the case of shadowing, there’s now these virtual shadowing opportunities that folks can do. And to me, that is just one of the best things that I’ve seen come out of the COVID pandemic because what that means is that you don’t have to know a clinician already or be able to get connected. You can just sign up for one. And it doesn’t require you to necessarily go and do that when that might not be feasible for what else you have going on in your life.

So it’s one of the silver linings I’ve tried to find in the way that admissions was completely disrupted during the COVID pandemic. Absolutely. I think we’ve actually found some really strong silver linings in this process. I also think virtual interviews are one of them too, from an equity standpoint of having to pay to travel and things like that. So I think going back to your original question, there is this tension because it’s really hard when folks don’t align with our way of thinking, for example, because that means they’re not going to be easily agreeable with what we’re doing.

But I think actually that breeds a better health professional because all of your patients aren’t going to be aligned to the way you’re thinking. All of your other caregivers that you need to work with in order to create that amazing healthcare team are not going to align with your way of thinking. So how do you learn to have that flexibility, that ability here, and understand others that provide a way to start articulating the reasoning behind what you think, not just what you think as a way to start building that pattern of communication. It’s going to help not only in the school but beyond. So I mean, that’s my opinion on why I think it’s going to happen, but I hope that by adding these data points, we’re not going to be limiting our applicant pool that way.

Now, Casper, correct me if I’m wrong, started at one school about 12 years ago? [46:33]

Yeah. The research on it started 20 years ago to date myself.

20 years ago. All right. But it actually was implemented initially at McMaster in 2010. [46:44]

Yeah, 2010.

Do you see it spreading to other fields? I mean, professionalism is important, social skills are important in tons of areas, whether it’s law, business, engineering, whatever? [46:50]

Absolutely. So we already have teacher education programs using our tool.

Social work? [47:15]

Yeah. And we’ve had other schools in business and engineering express an interest because in many of those cases, to become an effective professional, to become an effective engineer, you need to communicate and collaborate. You’re not a single person. You’re on a team. So all of those things are becoming increasingly important. I have this vision that one day by having these assessments, unfortunately unless something is measured, we don’t put a lot of value on it when we think about admissions, as you probably know more than anybody else. So I hope that by assessing it, a defensible way that tries to be as equitable as possible and accessible as possible, that we’re actually building and changing the culture of the way that we think broadly is a society that grades and academics are incredibly important, but the way that we treat each other as human beings is even more important.

It’s something I always feel like I try to instill in my kids as I am teaching them about what’s important in life, it’s not always getting the perfect test score. It’s about are you being kind to your classmates? Are you having these, are you being inclusive? Are you doing those sorts of things? And I hope as a society, that’s actually the way that we’re moving. So we’re starting to see that momentum across higher education that people are saying. There’s actually difficulty with some of the existing standardized tests and measures that they use. So what else is there? What else is important in how we’re selecting our teachers or our engineers, our lawyers of tomorrow?

Do you see the Casper as a complement to the existing aptitude tests or as a potential substitute? [48:53]

Well, I don’t think it’s a substitute-

Situational judgment test, I should say. [49:06]

Yeah. Situational judgment test. I don’t know if it’s fully a substitute in that it’s not measuring the same thing. So if you are measuring-

I want to know that somebody can do the work required from a specific program. [49:17]

Yeah. That’s a very, very different measure. Although I think that measure, the pendulum has swung so far that we’ve almost set our standards for that way too high. But I think when we talk about Casper, it’s that there’s a need to balance that we don’t just want academics. We also want to be looking at the whole person who’s applying. So it’s about having other measures that allow us to really capture not just how absolutely intelligent you are, but who you are as a human being, because those two things combined, that’s going to make you successful. And if you need support in one of those areas, then how can we support you?

This has been absolutely amazing. My last question is, what would you have liked me to ask you? [50:04]

Maybe some of the misconceptions about Casper?

I just set myself up to answer a tricky question. So I think people believe that it’s just a barrier or a hoop to jump through. I want to convey that the reason we originally developed this is to make sure that as much of the whole applicant is actually included in the admissions process as possible. That you aren’t just your grades, you aren’t just a data point on an academic test that you’re much more than that. And we really, really wanted to make sure that that was included. So that’s probably one of the misconceptions that I really want to get away from. Also, to highlight that we have accommodations in MD/DO programs. We have a fee assistance process for the application process. We’re actually encouraging other groups to develop these FAP programs so that we can honor them in the application process. So we really want to make sure that that’s right.

I will actually add one more thing because it’s such a hot topic right now, is ChatGPT. Because the question is, can you gain Casper with ChatGPT? And we’ve done a lot of research on this trying to understand that, and we’re excited because two things, one, the majority of our scenarios are so complex in their videos, so it’s harder to input in the time allowed to ChatGPT. The other thing, and I want to highlight this to people who may be using them for personal statements or other things, is that ChatGPT really just generates a very generic answer. And unfortunately has no passion in it. It’s not necessarily up to date in terms of the societal or cultural things that are going on that may impact your beliefs or perspectives on things.

So there’s a word of caution and the fact that what we tend to know is that because we’ve tried to answer Casper with ChatGPT internally, and that you don’t actually always score that well because it gives a very generic answer; it can have lots of mistakes in it and things like that. And the problem is when those mistakes are there, and you only have a time limited thing, you may get your response submitted in a way that actually isn’t reflective of you. So the best advice I can give folks is find a quiet space, make sure nobody’s streaming videos, next door to you, but just really try and be yourself. Because it’s when you come across as who you are as an individual, who you are as a human, your own experiences, and you really express those through your responses, your why, you would take course of actions, those are the people that excel.

Where can listeners learn more about Casper and DUET for that matter? [53:25]

So if you want to just go to our website, which is acuityinsights.app, you can learn more about the different assessments. You can see some example Casper scenarios as well, so that you can feel familiar with it. When you sign up to do the Casper test, you can take the full practice test. So go on, learn more and don’t be afraid to reach out with questions.

Relevant Links: