What is the Difference Between Testing and Ranking Students?

This podcast was originally published on In Their Own Words for The W. Edwards Deming Institute

Are tests like the SAT – and a potential National Merit Scholarship that goes with a good score – the same as grading or ranking students? David and Andrew discuss the differences.

TRANSCRIPT

0:00:02.4 Andrew Stotz: My name is Andrew Stotz, and I’ll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today, I’m continuing my discussion with David P. Langford, who has devoted his life to applying Dr. Deming’s philosophy to education, and he offers us his practical advice for implementation. Today, his topic is, The Difference Between Testing and Ranking Students. David, take it away.

0:00:29.4 David Langford: Okay, well, if you’re an educator, that should be a trigger enough for you to pay attention and listen to this. [chuckle] So I wanna tie this in with Deming thinking and the difference and what people are trying to do with equity and all kinds of things that are going on today. And it’s pretty relevant too, because I just watched a newscast that a school district was delaying or even not announcing their National Merit Scholars because of the fear that it would make other children feel bad because they weren’t recognized like that. So it’s kind of like what Dagwood set in the comics one time sounds like a good idea till you think about it. I wanted to discuss that today because over the last 40 years, I’ve run seminars and talked about Dr. Deming’s focus of rating and ranking and grading, and he’s just really against grading, and I pulled up a quote from The New Economics, he says, our educational system would be improved immeasurably by abolishment of grading. Okay. So there’s a difference between grading and actually just testing. So let’s just talk for a minute about the National Merit Scholarship Program. So I just read some stats on that, about one… Over one million students actually apply for a National Merit Scholarship each year, and only about 50,000 are selected, and how are those kids selected?

0:02:09.4 DL: Well, when you’re a junior in high school, you take the SAT test, and if you have one of the 50,000 scores of the highest in the nation, then you could be named as a national merit scholar, and that could mean a lot of things. I mean, it could help you get scholarships to universities, it could really look good on your resume for the rest of your life, it could mean a lot of things. So is that the same thing that Deming talks about by grading and ranking people, and I would say no, because what really should be happening instead of thinking that by honoring or naming people that, or recognizing people that took this test and got one of the top 50,000 scores that that’s gonna make other people feel bad, therefore we’re not gonna announce that or we’re not going to recognize that is not the same thing. When you’re grading and ranking people, you actually have to grade them, grade their performance in order to rank them, and talking about Deming’s concept of profound knowledge, the variation in that is huge, and the psychology in that is huge. I’m sure that almost every single person can relate to a classroom where they probably told somebody, I just, I don’t think this teacher likes me. I don’t know why but I just don’t think they like me for some reason, and no matter how hard…

0:03:42.4 AS: And it’s confirmed at the end of the semester when I get my grade.

0:03:46.9 DL: Yeah, no matter how hard I try or whatever, I just don’t think they like me, and I know it’s happened to me at times, and I just… Well, I just, I got choices that I could drop the class or I could just put up with it and go through with that. So psychological things like that could enter in and then all the social-economic stuff that we’ve got going on now could enter in and ethics could enter in, and all kinds of things could enter into someone giving someone a grade, like in a classroom and then ranking them against other students, right? That’s a totally different thing. If I could give this, the school district and I’m not gonna name them ’cause I don’t wanna get in trouble or anything. But if I could give this school district advice, what you should be trying to do is get as many students as possible into that level of National Merit ranking, because it’s not limited, as far as I know, it’s not limited that you can only have one per school or something. You could have as many as qualify, and that would show what an elite school you are actually, that you have more people qualifying for a national merit scholarships than any place else. And drive other people to think, Okay, if they can do that, I can do that, right.

0:05:11.1 AS: And can we… Can we go into that more detailed, just so we really break it down. To understand when someone, I guess, voluntarily as a student does with this National Merit Scholarship, goes into some sort of competition or measurement or something like that, that that’s different from a school teacher and a school administrator observing the behaviors and actions of the students and then coming up with a ranking of that environment that they’re living in every day. Explain how that’s different.

0:05:43.4 DL: Yeah, just a test that they take and they all go in to the room, 300 kids go in a room and they take this test and whoever gets the best score qualifies. That’s all there is to it. So you have no idea if they’re a male, female, tall, short, skinny or not. None of that enters in, so there’s no real psychology to it, you just go in and take the test and if you gotta… You get the score, you get that, you get to ranking or it’s not really… It’s not a ranking it’s, you just achieve that level of being able to pass that test.

0:06:17.6 AS: And as a… Okay, so from a school perspective, I can see that, then the next question is, from a bigger picture society perspective, is that person now ranking themselves or is there some problem with that from a country perspective that people are entering a competition like that?

0:06:37.4 DL: Well, would I want to put in the hard work it takes? Because when I look at kids that achieve that level of performance, I see years and years, 10, 12 years, some cases of hard work of always working hard to be a top student, and they may or may not be ranked as the 4.0 students in their schools. That has nothing to do with that, but they may be really good at taking tests or they’d be really good at studying for this, or they may have family members that are super supportive, maybe you have two parents that are both college professors, right. Well, I would think that they would have more emphasis on a National Merit Scholarship and the importance of that and be communicating that throughout this child’s entire life than a sharecroppers child in Georgia, that has nothing to do with the school system, except taking his kid to school every day, right? Those are totally different situations.

0:07:46.3 AS: And in that case, if that person, let’s say that person’s… Let’s say a family has… This is the first kid to have a chance to go to university as an example, and if that family found out about this National Merit Scholarship and they told their son or daughter, Hey, why don’t you set that as a goal to try to take that exam when you are 14, or 16, or 18? Is there a problem with that?

0:08:14.8 DL: No, I don’t see any problem with that at all. You have a bar that you’re setting, and if you get over this bar, then basically you win, but it has nothing to do with rating and ranking the individuals.

0:08:28.4 AS: And it’s part of it that it’s like a third party, a separate entity that you’re going to. It’s kind of a voluntary thing as opposed to a system that’s imposed on the teachers and the students, and everybody in the school.

0:08:40.8 DL: Has nothing to do with your school, basically, you could be the best or worst schools in the world and either pass this test or not pass. And it’s not about passing his test, it’s who actually gets the best scores. One of the top 50,000 scores for you to be named this.

0:09:02.3 AS: And if we look at these teachers in that school that have decided and the administrators who have decided to do this action, let’s just say that their intentions are good, in the one sense that, like we’ve talked about here, when a student does really well in assignment, the idea that you’ve talked about is, Hey, how did you do that? Why don’t you explain that to the other students and share what you’re doing and stuff. I suspect what they’re afraid of is that it’s glorifying these really elite students within the school, and that the other students don’t, either don’t get the opportunity or they feel less of themselves. The teachers are trying… Let’s just assume that the teachers are trying to do something good, but they’re maybe misdirected. What would be a better idea within the school?

0:09:56.6 DL: Well, I wouldn’t refrain at all from recognizing those students and saying, Hey, these are the ones that took the test and are now National Merit Scholars and, I suppose there could be an over-glorification of that, that you could go overboard with that, but to those students that are actually taking that test, they obviously know what it means, right. And the recognition that could come with it, and that could be at their college scholarships that you’re a National Merit Scholar and that, I could look really good going to a major university or something, and it could actually end up in dollar values. And I think that’s what are the things that the parents were complaining about is by not naming these kids in a timely fashion, apparently they withheld the names of these kids, because they withheld the names of these kids, some of them would miss out on being able to put that on their scholarship applications to universities, or even if you’re just going to go get a job, that would look good on a resume and things like that, but the difference to me is that’s not a rating and ranking, it’s simply a count data, right.

0:11:15.7 DL: Everybody takes the test and whoever got the top scores, then they get the recognition. So, I think is all there is to it.

0:11:23.8 AS: If we were to look at another parallel and just trying to understand how Deming thinks and this concept, let’s take a wrestling team as an example, where there is a team score kind of thing and an individual performance as opposed to, let’s say a football team where really it’s a team performance. And let’s say that the wrestling coach has worked hard with their team and they’re doing really well, and they’ve got a couple of really strong wrestlers and they compete and they win the state championship, and two of their wrestlers win the best in their weight class or whatever that is. Should that be celebrated by the school as an accomplishment, or is that rating and ranking, how do we view something like that?

0:12:15.2 DL: No, of course, it should be celebrated and kids recognized, etcetera, because those things take a tremendous amount of hard work, I don’t know if you’ve ever were a wrestler, but I did that once upon a time.

0:12:28.4 AS: I looked at it once and I thought, Yeah, I can’t work. I’m not gonna work that hard on that ’cause that looks brutal.

0:12:35.1 DL: Yeah. So not only do you get your own personal score, but those scores are all added up as a team, and that team score is what determines if your team beat somebody else’s team, or you become the state championship team, etcetera. But the schools that are really good at developing wrestlers, right, they don’t think about just having one person who’s state championship level, right. They’re developing a whole deep program that year after year after year, they have a plethora of top wrestlers that are moving into that upper echelon and can work that through. And there’s also a good example, when I was the first year band teacher, the school I was at the high school was really into wrestling, and so I asked the wresting teacher, I said, Would you like to have a pep band at the wrestling meet and first he thought it was kind of nuts and he says, why, you know, I don’t know, he said, Let’s try that. Let’s see what that was like. So I got a bunch of volunteer kids and we get a whole drum core and everything, we choreographed this whole thing. So there was still like music going, announcing the wrestlers, there was music in between and then there were drum beats going on, everything.

0:13:55.6 DL: Well, we ended up wrestling against this school that they had hardly ever beat and we just clobbered them, because the psychology of what we created was this momentum of…

0:14:07.9 AS: Energy.

0:14:09.5 DL: Wow, we’re invincible, and we’re one of the top programs in the state, and so on, amd so forth. So I thought the wrestling coach was gonna kiss me afterwards, and so he really liked that, but I mean, that’s really kind of a good example I think that you can manipulate these things to a large degree, psychologically, if you think about profound knowledge and the psychology behind things, you can manipulate things to get the data to show different things. Were these kids all the best wrestlers? Now, I’d say probably we intimidated the other wrestlers and in an equal environment, some place, our kids may have not been able to beat these other kids because of what went on.

0:14:57.2 DL: But the point is that you’re trying to develop the depth of a system in a program, so that you continually have great wrestlers, not relying on the fact that once every 12 years, we just have some naturally gifted kid that comes in. I saw this when I was a teacher in Alaska, and we had this student as a junior in high school, and he could pick up a 50-gallon barrel of oil and pick it up and put it in the back of a pick-up, and that was his job, and he came to the school and the wrestling coach said, Well, how would you like to come out for the wrestling team? And he said, Well, I’ve never done that before Junior in high school. Well, he ended up being state champion two years in a row, and basically he didn’t have near as much training or talent as anybody else, but if he ever got a hold of somebody, they were done because he would just like…

0:16:00.2 AS: He’d put ’em in the truck.

0:16:00.3 DL: And just force them to the ground. So to me that’s… And that had nothing to do with our wrestling program or the development, or anything. It’s just a kid that had grown up super talented, or super strong. To me, it’s also sort of the basic same kind of thing we’re talking about with this PSAT test and the National Merit Scholarship, etcetera, etcetera. Are you really recognizing who’s the most brilliant or who’s just really worked the hardest. There’s probably an element of both from the neuro-science standpoint, there’s development of all of those neural structures and everything else that enabled these kids, but I would also submit that probably some of those kids were just much better at photographic memories of remembering stuff and excelling. They’re just born with that, and it just was much easier for her, them to get there, but that doesn’t preclude other kids that really wanna work really hard at preparing for that test and really working for them, and that’s a goal or an aim that they might have that they really wanna try to do that.

0:17:09.1 AS: I would love to wrap this up by just kind of circling back to what’s the objective of school, what’s the objective of a business, what is the function of an individual within that system, what is the function or the objective of the management of that system and of the individual? What are we trying to do so that we just go back to first principles to make sure that the listeners, the viewers are going back to those first principles to say, Let’s make sure we’re doing the right thing. So can you describe for, as simply as possible what you think.

0:17:45.7 DL: To wrap this up, I will give you two words that I learned from Deming that just became imprinted on me over the years, and that’s artificial scarcity. So when you’re creating an artificial scarcity of top marks or top performance or anything like that, then that’s bad, that’s gonna have a detrimental effect on people. And we’ve talked about valedictorians and all those kinds of things, those are… That’s really an artificial scarcity, you’re actually… That’s why some of the school districts are grading kids to 1/1000 of a point, et cetera, because they got too many valedictorians. Well, that’s just the opposite of what you should be thinking about. You should be thinking about, can we get more and more and more people to this level, the same thing that we’re talking about with the wrestling program, can we have a program that’s producing more and more and more better athletes and that’s a true system and a program. And that’s the same thing. So you always wanna watch out, it might create an artificial scarcity. I have five children, and the example is, would I ever rank my five children, and say, Who’s the best or who’s not?

0:19:06.4 DL: Well, anybody who knows anything about parenting would say, No, that’d be a very stupid thing to do. Right. It’d be very foolish to do that, and they all have different gifts, they all have different skills and gifts and the backgrounds, et cetera.

0:19:21.8 AS: I’m thinking about also natural scarcity, where let’s say a family does not have the means to put all five of their kids through school, and they have to choose one and say, Look, we’re gonna put everything behind, and everybody knows that Bobby is the one that we think can be successful with the money that we have for University as an example, which I would say it’s more natural scarcity than artificial scarcity.

0:19:47.0 DL: Yeah, even that, to me, that’s a concept that may have been true 60 years ago, it’s not true today, every single kid that wants to go to, even if you don’t have the scores to get into a certain school you wanted to get into, okay, go to a community college for two years. And in many States, the State pay for it and it’s for free, so that’s a level of trying to level the playing field that… So it’s not just reliant on the rich that can get to that level.

0:20:18.9 AS: So let’s go back and try to… I just wanna try to wrap up what you’re saying about the goal is to try to… How do we get more people to this level? And what I’m thinking about is PDSA, what I’m thinking about is training, figuring out what’s working, and then bringing that…

0:20:36.7 DL: Systems thinking, Psychology, understanding variation, it’s Deming System of Profound knowledge is what you need to be applying, that your system gets better and better and better and better, so that virtually anybody that comes to your school, maybe they won’t rise to that level of one of the top 50,000 in the country, but everybody is getting better and better and better, and what are we doing in the system that’s preventing more and more kids to get to that level of performance, just’s the way you wanna think about it. So you’re not creating an artificial scarcity of people.

0:21:13.8 AS: I remember Elon Musk being quoted as saying something like, We need to launch more Rockets, when he was talking about how to get better at what they were doing with landing and reusing the rockets and all that, and I just think about in my case with my valuation masterclass Boot Camp, which is a purely online system focused on a very specific thing, it’s voluntary where people are signing up, and so it’s very different than, let’s say a public school. But the point is, is that every time we launch, we have new things that we apply from what we learn in the prior one, and as I tell my students in the current valuation masterclass boot camp, number seven. If they’d studied at number five, it’s a completely different course, and I’m just thinking about all the different iterations and we stick with the things that work, and then we build and add on the next thing, and that’s ultimately, I guess the job of us inside of business, inside of school, inside any process is, how do we find what works.

0:22:14.2 DL: What you’re trying to do is to create a system where people are gaining knowledge that’s useful and applicable in the future. One of the quotes that Deming had was, Why would I rate and rank my students, how can I determine who amongst them is gonna be great in the future, so why would I wanna limit them now with a grade. It took me years to understand what that meant, but until you’ve actually seen hundreds of students move through and students in high school and stuff where you think, Oh, that kid’s not gonna, they’re never gonna amount to anything, and all of a sudden they’re state senator or they’re doing something 20 years from now that you have no idea. But maybe they had to overcome your rating and ranking in order to think that that was possible, or that they were capable of doing that kind of thing.

0:23:11.1 AS: Yeah. And I went back to my high school records and I found that my GPA in high school was 2.6, I was firmly in the middle of my ranking in my high school. I was getting high basically most of the time and doing other stuff, and I wasn’t really paying attention, my parents weren’t pushing me that hard, they were just like, Try your best and whatever, and they didn’t wanna see me drop out, but I was definitely on that path, and I think most people thought I wouldn’t succeed. But then my last semester of university, I had seven classes and six of them I got As, and the seventh one I took at another university and I got a B. And something switched in me and I overcame that rating and ranking, and the fire of learning was lit under me, and I think maybe we’ll wrap it up by saying that part… The whole objective of what we’re trying to do is develop systems and processes that really work to set children on fire with the excitement of learning and figuring things out and finding things out with the objective that they’re gonna live a better life, they’re gonna have more joy and more, they’re gonna understand things around them, they’re gonna be able to make an impact around them, and if we can do that, I would say we’re doing a pretty darn good thing. Anything you would add to that?

0:24:32.9 DL: Yeah, well, it leads into… And maybe we can discuss this in a later podcast too about, I’ve worked with a lot of universities and stuff, and I’ll meet with them and I’ll hear phrases like, Oh, we’re one of the most selective universities in the state or the nation, or whatever, and we turn out the best graduates. Well, just go to our random selection of students, have a bar that you want everybody to get to a certain level, and when they do, everybody’s name goes into the hat, and you draw out however many slots you have open. Now, everybody would know how they’re chosen and if you can take those randomly selected students and turn out the best graduates in the country, I would acknowledge that, yeah, you’ve got a tremendous school, but if all you’re doing is selecting, going through a rigorous process to select the people that are gonna fit your program, you’re probably not doing much of anything, and you’re not really developing a system of greatness where virtually anybody that comes here is gonna become great. And I want to submit kind of to wrap this up that every teacher is going through that very same thing, because students are thrown into their classrooms, usually and just randomly…

0:25:55.9 DL: Random selection, right? So if you can develop a system by which, no matter who is thrown into my class, even kids with special needs, I’m able to move them to a level of performance that nobody else is able to get these kids to, the very same kids. And next year they go into another class and they’re not able to achieve that. Right. I would say you probably have created a fantastic teaching system, that no matter who I get, I’m over time, I’m able to get them to a very high level of performance. And I think that’s the same thing that this whole podcast is about, you should be thinking about getting everybody to that level, and what are we doing as a system that’s standing in the way that’s preventing people from getting to that level of performance. So you mentioned a company, right, you don’t just want one great worker, right. You want everybody to be great, otherwise you don’t have a system that’s continually producing great products.

0:26:58.3 AS: Yeah. Well, David, on behalf of everyone at the Deming Institute, I wanna thank you again for this discussion, and for listeners, remember to go to Deming.org to continue your journey and listeners can learn more about David at langfordlearning.com. This is your host, Andrew Stotz, and I’ll leave you with one of my favorite quotes from Dr. Deming, and it just never gets old. People are entitled to joy in work.