Neal McCluskey: Welcome everybody to what was going to be a Facebook Live event, now it’s I guess you might say a Facebook dead event. But, we are still going to have, I think, a very interesting discussion with Dr. Patrick Wolf about Achievement, Attainment, Something Else? What really matters in education? We were going to be taking questions and comments. We actually got a few before the event started or was supposed to start. But if you have questions after this and you want to send them to me, my email address is nmccluskey@​cato.​org. That’s NMcCluskey@​cato.​org, so if you see this video and you have questions for me or you have questions for Dr. Wolf, you can send them to me. We are also going to go over some of the questions that we got beforehand. I am, if you don’t know, the director of the Center for Educational Freedom, Neal McCluskey, as you just got from my email address. At the Center for Educational Freedom we do, obviously education here at the Cato Institute. We cover everything from pre‑K to higher ed. Depending what is in the news, we do a little more of each of those things, but the nucleus of what we work on is school choice and K through 12 education, but in particular school choice at that level. I’d like to introduce Dr. Patrick Wolf. He is the Distinguished Professor of Education Policy and 21st Century Endowed Chair in School Choice, in the Department of Education Reform at the University of Arkansas College of Education and Health Professions which is nearly impossible to say without taking a breath. And I think I did it. I have a few, Dr. Patrick Wolf writes all sorts of stuff all the time, but a few books of his: The School Choice Journey, School Vouchers and Empowerment of Urban Families, an excellent book about school choice. He co-wrote that with Thomas Stewart. Here is one of my favorites that he co-edited, Educating Citizens: International Perspectives on Civic Values and School Choice. And then, I also made a book that was going to be a joke, but it could be a real book, I Screamed from Eagles, my Descent into Despair, by Patrick J. Wolf, a rabid Vikings fan as you can see from this picture, he is wearing a Vikings helmet and uniform so you know he is a fan. And it was just terrible for him when he lost to the Eagles and you can send me emails about that if you want. I will send them to him and he can answer any questions you have about what went wrong with the Vikings in the playoffs. But, Pat, if you would like to tell everybody a little bit more about yourself. And then if you want to go right into Achievement, Attainment, what are we talking about here? Which of these things matter? That would be great.

Dr. Patrick Wolf: Sure. I was actually born in Washington D.C. My mother went into labor when she was strolling through the National Zoo with my older sisters. And I was born in Georgetown Hospital when they had a maternity ward. But I got out of the swamp pretty quickly and moved to sunny Minnesota for my upbringing and my inculcation as a rabid Minnesota Vikings fan. So, for the rest of the time I think we’ll just talk about that New Orleans Saints game against the Vikings.

Neal McCluskey: Well, we know you returned to D.C. for a while because you were at Georgetown University.

Dr. Patrick Wolf: I was.

Neal McCluskey: I didn’t give your whole bio, but…

Dr. Patrick Wolf: I was. And now I’m in Fayetteville, Arkansas. So, I call the Hogs and cheer them on and basically study what happens to children, to families, to communities when they are given more educational options, including private schooling options.

Neal McCluskey: So, let’s talk about that. And in particular, so you have a new report from the American Enterprise Institute, but that’s okay because we like them. Not every report has to come from Cato. But it is about a topic that actually, it seems to me, has been percolating up for the last four or five years, maybe even longer than that. And the question is, what is the right way to measure? What is the right metric for success in education? And we see it a lot in school choice but No Child Left Behind really applied it to all schools. But for a long time, the end-all and be-all of our assessing education seemed to be what are the test scores? What are the standardized test scores? Is your public school good? Does school choice work? Is your child learning? It all was reduced to standardized test scores. And then we had a backlash against that, I think just publicly, or the public broadly was saying we are getting tired of this. And then you and colleagues started to look at, well, and does this even correlate with attainment? So, can you tell us more about that?

Dr. Patrick Wolf: Sure. It started with my evaluation of the D.C. Opportunity Scholarship Program about a decade ago for the U.S. Department of Education. And we found sort of mediocre test score impacts, maybe some gains in reading. No gains at all in math. And these big positive effects on the likelihood of high school graduation. So, there was that disconnect in that one study. My team went on to evaluate the Milwaukee Parental Choice Program and we found something very similar. Suggestions of maybe slight positive impacts in reading, but large, significant and consistent effects on the likelihood of graduating from high school, attending college, persisting in college. So, my junior colleagues and I, Collin Hitt of Southern Illinois University and Mike McShane of EdChoice, wanted to do a more broadly scoped examination of, do we see this pattern consistently in school choice evaluations where test score effects don’t match up with attainment effects? And what we found in various studies of school choice, charter schools, open enrollment, vouchers, etc., is a strange pattern. The most common pattern is that the attainment effects would be much better than the achievement effects. That was the most common. But we’ve also seen cases of public charter schools that absolutely kill it on the test score side and have no benefits in terms of how far students go in school. So, you know, when we put it all together we just find this lack of a close connection between test score effects in a school choice evaluation and attainment effects in a school choice evaluation. And we think that is really important for the reason you gave, that we have been so fixated on test scores for so long, maybe this is a wakeup call that we should consider other measures of success for choice programs.

Neal McCluskey: But what’s the theory, why do you think this happens? So, I think the average person would think, look if your test scores are going up, if you are learning more, or you are doing better in school, however you want to characterize that, why don’t we see sort of a one-to-one correlation where those test scores go up then we see attainment go up because the learning, the education, however we want to refer to it, because the test scores are going up we are getting more of it, we are getting better education, wouldn’t we expect to see that translates into a whole lot more attainment? Why might there be this disconnect?

Dr. Patrick Wolf: Sure, I think part of it is that test scores are a narrowly targeted measure of educational success. You know, it’s just in the particular educational domain, so English, language arts, or math or science, really just focused on those areas and then just a certain amount of content in those areas. So, it’s getting a very small sliver of what we would broadly consider to be learning. Attainment is more of a general measure of success in education where a student has stuck with the educational project longer. And it could very well be that that depends, sure, the more they learn the more likely they are to stay in school. There is definitely that connection at the individual student level. But it could be that schools of choice specialize in focusing maybe more on the test score side. This is what many researchers called academic press. They are really pushing the ability to score well in the tested domains of ELA and math, academic press. And maybe students kind of get burned out by that. They do great on the test scores and then they just decide, okay I’ve done this, but I’m not continuing, whereas other schools and school choice programs might specialize in promoting non-cognitive skills and abilities of students. This would include grit, conscientiousness, persistence. And it could be these non-cognitive character traits that matter more in terms of how far students go in school than the knowledge they are able to demonstrate on a standardized test. So those are two explanations for why this disconnect might take place.

Neal McCluskey: Now, one of the concerns, I think, for any of these measures we have been talking about is you know, they can sort of be gamed, people can push to get the particular outcome they want. And we’ve seen some examples of what looked like schools kind of artificially goosing graduation rates or sometimes tests. And so one of the questions that we got on Twitter earlier was from Lou. He says, I’m conflicted on voluntary teachers’ unions. They are there to protect members like when administrators go haywire on school discipline — here’s the important part of what we are talking about — graduation rate pushes, etc. So, how big is this problem? If we make any of these, any major metric for whether or not a school is good or teachers. How big is the danger of those things end up being sort of gamed and then don’t really tell us what we think they are telling us?

Dr. Patrick Wolf: Sure. You are talking to what we refer to as Campbell’s law in social science. And it is basically that the less consequential a measure is for human beings, the more accurate it is going to be. And the more consequential it is the more important it is for accountability purposes for rewards and punishments, the greater the incentive to game it and to figure out a way to make the numbers go up no matter what. And I think it is a bit of a problem in the area of education policy. In terms of standardized test scores there has been a number of studies showing that ways that schools and school personnel can game standardized test scores result at the margin. The most egregious case was widespread cheating in Atlanta, in Atlanta public schools. And people actually went to jail for that. So, they were caught and there were consequences. And you referred to the recent graduation rate scandal here in Washington D.C. at Ballou High School where students who hadn’t even attended school for large chunks of the year were awarded diplomas that they clearly didn’t deserve. That was such an extreme case because they touted 100% graduation rates for a highly disadvantaged population of students. That was put under close scrutiny and we saw that there was widespread cheating involved. So, there is an opportunity to cheat, to game a little bit. I would call on teachers, as professionals, as education professionals, to hold to their standards that, you know, a student does not get credit for a course unless they actually master a sufficient portion of the content. Students don’t graduate from schools unless they actually show up and engage in the educational project. So, I’m hoping that the reputational stain of the Ballou grad rate scandal will be a message and a lesson for educators across the country, in public schools, charter schools and private schools, that yes, we want them to graduate a higher percentage of their students but they have to take the high road and do it the right way.

Neal McCluskey: We talk about these measures, attainment through test, I mean achievement through test scores, attainment graduation, but one of the things people have talked about for a long time, maybe longer than I have been paying attention, but we always talk about well we want people to learn, kids to learn critical thinking. There was a time when people would say we want 21st century skills which was very rarely defined in a meaningful sense, but Kevin wrote to us and said, what matters is that you can read, write and think for yourself. How can schools, and people attending or sending their kids to schools, how do you measure something like whether or not a child is being sort of trained, or however you might put it, to think for themselves? How can we ever know whether a school is doing a good job or a bad job of getting kids to do something like that?

Dr. Patrick Wolf: That’s an excellent question, Kevin, and I think one way, some indicators of whether students are prepared to think for themselves upon graduation come from measures of civic values and civic participation. I know this is an issue that you are very interested in, Neal, stemming from your dissertation research on the topic, and it is something I was very interested in editing the book Education Citizens, Educating Citizensis that basically we have some indicators. If…

Neal McCluskey: Educating Citizens, by the way, right here.

Dr. Patrick Wolf: There we go.

Neal McCluskey: Yeah. Find it on Amazon.

Dr. Patrick Wolf: That basically citizenship behaviors, generally people engage in them when they have a strong level of self-confidence. That they know what they are doing, they are motivated to do it. They are going to engage in civic activity in terms of voting, in terms of participating in rallies, writing their member of Congress, all these sorts of citizen behaviors upon which our democratic republic rests. And there actually are a number of studies, more than fifty studies, of the effects of schools of choice on these civic activities. Because it is a great concern. If government isn’t running schools how can we be confident that the public purposes of education, the citizen purposes of education, are being fulfilled? The numbers are actually very encouraging. The overwhelming set of findings suggest that schools of choice do as well or perhaps even better than assigned public schools in promoting these self-confident behaviors of citizens that are associated, connected with the idea of them being able to think for themselves. Definitely if you are stating an opinion to a member of Congress, if you are testifying before a legislative body, you feel pretty confident that you can think for yourself.

Neal McCluskey: So, the big thing that is going to happen next week, April 10th, is that we are going to see a release of the National Assessment of Educational Progress, the state NAEP, or the main NAEP, I think the official term is main NAEP, but it is fourth and eighth graders, reading and math, and already people seem to be sort of seeding the ground for how they are going to deal with whatever the results are. And we saw former Secretary of Education, Arne Duncan, start talking about we know that school education reform over the last fifty years or so has worked. I guess it’s sort of a multi-part question. What do you expect to see, maybe in the upcoming NAEP? What can we really take from the NAEP, because this is again one of these things where we get one test score and we say this test score tells us everything we need to know about our education system. So, what do you think are the limits to these NAEP scores and what do you think of people maybe kind of getting ready, all ready to spin whatever the results are that we are going to see?

Dr. Patrick Wolf: Sure. Well, I have not seen the NAEP results. I am not one of the privileged folks who are briefed on them. But, what we are hearing, the buzz, certainly suggests they are probably going down, that it’s a negative change, a decline from previously. And the previous set in 2015 was a decline from 2013. Now, a few things about the NAEP. It does not move very far very fast. So, there is a real static kind of nature to the NAEP. But, once it trends in a direction, you know, it can continue for a little while. The hope certainly, after some reasonably intensive period of reform in the last decade or so, was that there would be a strong upward trend. But, you know, we are seeing the suggestion of maybe a bit of a dip instead. I really hesitate, and I encourage our readers and listeners and watchers to hesitate to draw strong conclusions from the movement of this. This is a consistent task. It is supposed to evaluate the general level of proficiency for students in the United States. But, the reform efforts have been a real smorgasbord and in some parts of the country they have been more focused on parental choice. In other parts of the country they have been more focused on Common Core Standards and test-based accountability. And so, there isn’t really a single thing called education reform that has been administered as an intervention and for which NAEP can measure the results. That being said, I just think it is a bit discouraging if we see another downtick in NAEP results because there has been, there certainly has been a lot of money, a lot of resources, put into a myriad number of reforms across the country. And it will take some time to unpack. Maybe focus on particular states that have done certain reforms intensively and see if they have been able to buck the downward trend of NAEP or not.

Neal McCluskey: Yeah. Well that’s one of the good things about the state NAEP is at least you can get down sort of to the state level.

Dr. Patrick Wolf: Right.

Neal McCluskey: But do you think it’s sort of symptomatic of how we often, or maybe, maybe it seems to me that it is symptomatic of politics. That one of the reasons test scores are always so important, and when we get NAEP scores because they are national scores, they always get a lot of press. Is part of the problem that politics is sort of driven by sound bites, and an easy way for someone to say what they think about the school system is oh, the tests are up, the tests are down, and let that just be all you have to say for your assessment of the schools? Do you think part of the problem is that that’s just politically the easy thing to do?

Dr. Patrick Wolf: It is, Neal, and it is just part of this over-reliance on test scores that I think is unhealthy for those of us who want to see an authentic improvement in the educational experiences of America’s children. When you actually visit schools, public schools, private schools, charter schools, you get a sense of how complicated the educational lives of children are, especially disadvantaged children in the inner city, and how the school personnel really have to respond to a lot of challenges in their attempt to make the educational experience helpful and meaningful for students. And, you know, a lot of that is missed by standardized tests at the individual student level and certainly aggregated to the state and national level. When we held a focus group as part of our school-choice journey research — we held a focus group with 39 parents in Washington D.C. who were participating in the D.C. Opportunity Scholarship Program, we asked them what metric, what measure, they use to assess the success of their child in school, how well their child is going at school. And we gave them options — test scores, grades, homework, student’s attitude towards school, attendance, teacher feedback, and of the 39, a grand total of zero chose test scores as their metric of how well their child is succeeding in school. They focused on measures of student attitude and behavior. For two reasons I think. One, they know that that really matters for the ultimate success of their child, how committed the child is in engaging in the challenge of education, and secondly because they can, because they are parents. They know the child better than anyone else in the world. So, if test scores are not terribly important to parents, I think we as policy analysts and as policy makers, need to think long and hard about the reverence that we have accorded to test scores and whether or not you know, we have kind of got ahead of our skis on that.

Neal McCluskey: Yeah, well, I mean, I think we are going to probably to debate this attainment, achievement, whether any of those things are really what matters. I think that ultimately it is the case that every child is different. But, Washington, state capitals are not going to let us just — anytime soon, I hope — as soon as possible, but just let parents make decisions for themselves. So, this debate is going to continue. Thank you for joining us today. I want to apologize to everybody who is trying to desperately get their questions through on Facebook. I know we were on and off. I hope that the parts that people saw when we are live were really outstanding, and I am sure they were because everything we did during that time was outstanding. But if you have questions, even if you weren’t able to send them through Twitter, you can again send them to me. My email address is nmccluskey@​cato.​org. That’s NMcCluskey@​cato.​org. If you have really good questions I will forward them to Dr. Wolf. I also want to note that on April 19th we will be having a live event at the Cato Institute, but we will also be online. We will also be taking questions using #CatoCEF. And that is what should school choice look like? So, we are going from how do we measure stuff to what’s actually a good choice? What might be a bad choice? And that will be with Professor John Merrifield, Halley Potter, Mike Petrilli and our own Corey DeAngelis, who was one of your students and still technically I think is one of your students, correct?

Dr. Patrick Wolf: For a few more days.

Neal McCluskey: Yeah. So, soon the morning period will start, but not the same mourning period as he had when the Vikings lost to the Eagles. And I will actually be moderating that April 19th event at noon Eastern time: What Should School Choice Look Like? And again, Dr. Wolf, thank you very much for joining us and thank you all.