Paul Matzko: Welcome to Unintended Consequences, a podcast about what can go wrong with government regulation. I’m your host, Paul Matzko, and with me as always is Peter van Doren, the editor of Regulation magazine.

We have a set of three articles today, all broadly having something to do with the medical profession, medical practice, and pharmaceuticals—so health and medicine in a broad sense, but covering very different topics. We have one about Ivermectin (which is going to be controversial), one about whether psychologists should be allowed to prescribe, and one about warnings for pediatric antidepressants. They are very different fields, but there is a unifying theme here, right Peter?

Peter van Doren: It’s the role of science and statistical inference in decisions. A common phrase one hears from both the left and the right is “just follow the science,” as if you can draw an arrow directly from something called scientific inference to whether you should or shouldn’t do something. That’s the meta-glue holding these three things together.

Paul Matzko: It’s become a culture war topic and a fight between two very different communities, though they are united in a funny way. On one side, you have the “do your own research” crowd, who often end up doing very bad research. To them, research means going online, Googling it, and getting an answer from someone else who claims to have done research. On the other side are the people who wear t‑shirts that say, “I effing love science.” It’s not so much that they love science because they are doing real scientific research themselves; rather, they are identifying with a community and a vibe that trusts various scientific institutions and apparatuses. In both cases, you have people making judgment calls without actually using the scientific process themselves.

Peter van Doren: Correct. They are trusting very different sets of experts.

Paul Matzko: While both cases seem extreme, we all kind of do this. Peter and I are not scientists. I’m a historian, though I grew up with a father who was a Clemson-trained chemist, so we talked about double-blind studies at the dinner table—a typical American dinner conversation. But I’ve never run a scientific trial, and I do not regularly read scientific journals. You’re an economist, Peter, which is not exactly a science in quite the same way.

Peter van Doren: Correct, though it’s closer than history. We are making judgment calls all the time about which investigators and experts to trust. We’re in this mess too, even if we’re a little more respectable than the two communities I’ve just described. We’re really grappling with the same kinds of things.

Paul Matzko: Let’s start with the really charged article first. This is a piece by Charles Hooper and David R. Henderson called “Reconsidering Ivermectin.” Neither of these guys are anti-vaxxers, but they are pro-Ivermectin. Most of us will recall that this was a medicine originally prescribed by veterinarians to treat parasites in horses—to deworm equine creatures—and also used in humans as an anti-parasitic. It got popularized as a treatment for COVID-19 early on in the pandemic because it’s cheap and widely available. On a personal note, this was actually the second kind of alternative treatment to go viral. The first was hydroxychloroquine, which is commonly used as a lupus medication. Originally, hydroxychloroquine was a treatment for yellow fever and was given to people suffering from malaria. It didn’t do much good against malaria, but they noticed that patients with joint issues reported a lot of improvement, so it became a lupus medication. I have people in my family who suffer from lupus, so it was obnoxious during the pandemic when it became hard for them to fill their prescriptions because of the sudden demand.

I should put my cards on the table: I was reflexively skeptical when I started reading this article. Privately, I thought Peter had gone off his rocker putting this in the magazine. But Hooper and Henderson are serious folks operating in good faith. They aren’t scientists, they are like us, but let’s steel-man their argument. The role of dissidents in intellectual activity is important. My prime example in science is Galileo. Talk about people attacking somebody—Galileo had a rough patch for his theories about the sun and the earth, and guess what? He was right. In any given era, trying to figure out which dissidents have useful things to say versus who the quacks are is a major judgment call. It’s only looking backward, once things are resolved, that it looks easy. Do the people who say Ivermectin ought to be on the table have a point, given that it’s cheap and could play a big role in antiviral control in the developing world?

Peter van Doren: That’s why I published it. I think it ought to be part of the conversation, but it’s up to the reader to decide. When Paul and I discussed this at lunch, we talked about how ordinary people—and even scientists—discuss disputes about knowledge. Are those disputes translatable for the public, and are they conducted fairly? What does it mean to have a neutral discussion of the evidence? Some would argue that’s not possible because science is a human endeavor, and humans have views. It’s hard to transcend political and moral biases to be a neutral observer from beyond. I like to think of myself that way, but as Paul and I were discussing off-microphone, that image of ourselves is probably not correct.

To get to the nutshell, science is about trying to differentiate a result from no effect. That’s the central issue, and that’s why we run experiments where you control everything. In drug trials, you give a treatment to some people and not to others, hoping that randomization controls for everything else so that the only varying factor is the experimental design. Then you look at the results, which brings up a big question: how confident do you want to be that the difference in results is not just the result of random error? How big does the difference in the experimental group need to be relative to the control group for you to declare with confidence that the experiment caused the difference—for instance, that taking Ivermectin reduced mortality in those with COVID? Simple statistics dictates that the smaller the number of people you try it on, the less confident you are that you’re picking up a real effect versus an artifact of some other variable. If the sample is just two people—say, my wife and I both get COVID, I take Ivermectin, she doesn’t, and I recover faster—maybe that’s just because she is immunocompromised, or struggling with something else, or because the illness affects women differently. There are a gazillion variables. A bigger sample—20 is better than two, 200 is better, 2,000 is even better—increases your confidence that you’re finding something statistically significant rather than just random chance.

Paul Matzko: Well said. But how confident is “confident”? This is where they get into the debate over a 95% confidence interval versus more or less than that. What are we talking about when we say 95% confidence?

Peter van Doren: In words, it means: if there is really nothing going on—assume there is zero effect from Ivermectin—what probability of random variation are you willing to accept? Just through random variation, some people will get better anyway. What percentage of people do you want to observe getting better before you are confident that it’s the Ivermectin doing the work rather than random chance?

The answer to that question is not scientific. That’s the big bottom line to this part of the podcast: science itself resides on an unscientific foundation. It’s not because anyone is corrupt; it’s just a choice. Do you want to be 80% confident, 95% confident, or 99.9% confident? In other papers I’ve written, I bring up the example of experimental physics at the Large Hadron Collider in Europe. They send electrons around, blow them up, and look at the bubble chamber. Do you know what confidence interval is required in experimental physics to get a paper published? It’s six nines: 99.9999%. Social science couldn’t even exist under that standard. A 95% confidence interval in physics is considered garbage. Nothing the social sciences do fits the criterion of publication in experimental physics. Why are physicists that way? Because they have decided they want to be that confident before they publish, probably because of the consequences. There’s a line in the Oppenheimer movie where it’s a big deal that there’s a near-zero—but non-zero—chance that the experiment sets the atmosphere on fire. In that scenario, you definitely want a 99.9999% confidence interval, as opposed to a historical query where it’s fine to be a little less confident because a newly discovered document won’t end the world.

That brings us to the second issue. That standard applies to individual studies, which have different confidence intervals based on sample size and other factors. But the article we are discussing deals with a meta-analysis—a collection of different papers, all with different confidence intervals. You have to make a decision about which papers count in your meta-analysis and which don’t.

Paul Matzko: That feels like the crux of the argument in this paper: a disputation about which studies should be included and which shouldn’t. My impression is that the authors are unhappy that the meta-analysis they are critiquing excluded papers that made Ivermectin look more effective because their individual confidence intervals were lower.

Peter van Doren: Correct. The nutshell is that we are now having a meta-dispute about a dispute. Again, the lay listener may just want science to tell them what to do, but the answer is that it can’t. In the end, decisions about taking medications require a cost-benefit analysis for that specific individual, and only the individual can do that. When I get advice from a doctor, whether I want the study to be 93%, 95%, or 99% confident depends on how big a deal my disease is. People who are dying often don’t want FDA control; they want access to everything because they are out of options. I get that, although my viewpoint is that giving everybody everything they want means we don’t learn very much. In the end, you need trials that help you understand what does and doesn’t work. The individual may still decide to try something even if science says it doesn’t work, but that’s a different question.

Paul Matzko: Wading through their blow-by-blow critique was tough because they list what they call the “knackered nine”—nine studies they think shouldn’t have been included—and argue for including others instead. Props for the branding, I like “knackered nine.”

Peter van Doren: Notice the non-neutral language. This was not neutral, scientific-journal-sounding language.

Paul Matzko: Exactly. It was impossible for me as a layperson to know which of their complaints were legitimate. As a specific example, there was one study with a high confidence interval concluding that Ivermectin doesn’t work. The authors’ issue with it was that the researchers didn’t guarantee the subjects took the medication with food. People were taking Ivermectin on an empty stomach. As someone who takes medications every now and again, I know some tell you to take it with food and others on an empty stomach. It’s a plausible complaint, but it’s also plausible that it’s totally meaningless in this case. How am I supposed to know?

Peter van Doren: We then enter a field of study called pharmacokinetics, which is all about how we swallow pills, how they pass through the stomach, and their absorption rates. Why are some drugs injected and others aren’t? Because the acid in the stomach degrades certain drugs to the point where taking them by mouth won’t do any good, whereas injecting them bypasses the stomach and puts them right into the bloodstream. There is a lot of underlying knowledge that trained people have. Are we in a position to adjudicate these disputes? No.

Paul Matzko: I was struck by the lack of clarity in these debates. We talk about doing your own research, but at the end of the day, I don’t think we actually can. I consider myself smart, yet this article is a bit of a slog because of the dense detail about sample sizes. But one important concept they use a lot is the word “underpowered.” This just means what you said earlier: the number of people in the trial needs to be large enough so that if the effect size or death rate difference is small, you can actually statistically distinguish an effect from no effect. In some of my other work, I’ve noted that because modern Americans are basically healthy, trying to find underlying mortality differences between different treatments requires sample sizes in the millions. We can’t afford that, so those trials will never be run. There are many medical questions the public has that actually cannot be scientifically answered because the events are rare enough that the required sample sizes are too expensive to gather. This becomes a barrier to medical progress because potentially good treatments are simply never tested.

Peter van Doren: All true. On a similar theme, our second article puts the focus back on facts: “Correcting Mischaracterizations of Prescriptive Authority for Psychologists” by Jacqueline Marie Galusha. Here we are back to a question of “who watches the watchmen?” Who gets to decide who is allowed to prescribe? Should only psychiatrists (MDs) be allowed to prescribe, or should psychologists (PhDs), the people you actually go to for therapy, be allowed to prescribe as well? It’s an MD versus PhD food fight.

To put it in a nutshell, everyone agrees on the problem: the number of psychiatrists is falling, the ones who remain are aging, and it can be incredibly hard to find someone in rural areas to manage prescriptions for antidepressants or anxiety. We have a mental health workforce collapse. But not everyone agrees on how to solve it. The American Medical Association (AMA), representing psychiatrists, says absolutely not—only psychiatrists should prescribe these medications. This article, aligning with the American Psychological Association (APA), argues that psychologists should be allowed to prescribe too, presenting data suggesting it is completely safe and even beneficial. Medical regulation happens at the state level, so there is state-by-state variance on this. This dispute is very similar to others we’ve published in Regulation regarding dentists versus hygienists, or nurse practitioners versus MDs. With healthcare costs rising, economists ask why supply is restricted, and one reason is strict credentialing by the state. Should we open up the mental health world to prescribing psychologists? This author stridently argues yes, pointing out that these psychologists have advanced clinical training beyond their PhDs, and she provides data showing they actually have fewer adverse events than psychiatrists. Psychiatrists are MDs who spend a lot of time learning about general medicine rather than focusing exclusively on mental health drugs.

When scientific communities have these disputes, how should laypeople adjudicate? I don’t know. But because we are at Cato, I say let a thousand flowers bloom. Let people practice, and we will gather evidence on adverse events. The data presented in this article suggests that while psychiatrists may have an economic worry about competition, the medical worries are overrated.

Paul Matzko: It definitely feels like guild politics and economic worry, which isn’t surprising. I was reminded of this because my partner used to be a health professions advisor, and she told me the history of Physician Assistants (PAs). The entire profession came out of the Vietnam War. You had all these guys coming home who were highly trained as combat medics, but they weren’t allowed to do anything in civilian medicine unless they went to medical school. The system realized these people had useful skills and created the PA pipeline. In most states, PAs are strictly subordinated to doctors, but it varies—some states allow them prescribing authority, similar to the ongoing debates surrounding nurse practitioners. When you see this same turf war popping up in every single medical sub-profession, you realize we are talking about guild politics.

As a parent of an 11-year-old, the discourse around teen mental health—depression, anxiety, and suicide risk—is very much in the air right now. Usually, that conversation revolves around social media, like Jonathan Haidt’s The Anxious Generation, arguing that smartphones and Instagram cause poor self-image and mental suffering. A knock-on part of that conversation is the worry over over-prescription—the idea that we are creating a mental health crisis and then over-prescribing medications for ADHD, anxiety, and depression in young people. What was surprising to me about this article is that it feels entirely counter to that trend. The argument here is that we have a shortage of people allowed to prescribe pharmaceuticals, at the exact same time that we are seeing record amounts of pharmaceutical prescriptions. That felt odd to me, so I did a Google Ngram search. References to “psychiatrists” in literature have fallen by almost half since the 1970s, while references to “psychology” have risen by 60%. In other words, popular culture and practice have already shifted toward psychologists.

Peter van Doren: In the 1960s, if you watch old Woody Allen movies or sitcoms, psychiatry was always linked to Freud, sexual repression, and your mother. That’s largely gone now. Freud’s intellectual standing has been tarnished, and his theories are no longer prominent in medical training for psychiatrists. But you can see why psychiatry was so embedded in popular culture back then—it fit cultural stereotypes about why people are the way they are. This article talks about a repressed minority, but according to the data, there are only about 236 prescribing psychologists in the entire country. It’s a massive educational gauntlet to go through, so not many want to do it, but the author thinks it should be more widely accepted. I agreed, which is why we published it. I am inclined toward the laissez-faire approach.

Paul Matzko: If someone were to push back on her stance, they could argue that because the current number of prescribing psychologists is so small, it’s a non-representative sample. These 236 people are likely extra-educated and highly motivated. If the floodgates open and we go down the quality food chain to the bottom quintile of psychologists, the results might not be as good. As Seinfeld joked, someone always finishes at the bottom of every medical school class, and Elaine manages to get treated by that person. What’s interesting is that while we are sympathetic to her stance of not blindly trusting the official proclamation of the AMA guild, we are essentially choosing to trust a smaller community that says, “Trust us instead, we have studies showing it’s fine.” As laypeople, it’s like trying to decide whether to trust the First Assembly of God or the Second Assembly of God. We were talking off the air about divisions among elites regarding science versus religion, and laypeople are caught right in the middle. It’s tough.

Our third paper is titled “Acts of Commission Leading to Acts of Omission: Miscarriage of Justice for American Youth in the Black Box Warning for Antidepressants” by Arif Khan et al. This is close in spirit to the psychology and psychiatry debate, but it focuses on what happens when these medications are actually prescribed to young people struggling with mental health. Peter, what were the acts of commission and their consequences?

Peter van Doren: It involves Selective Serotonin Reuptake Inhibitors (SSRIs). They were originally approved for adults, and then incentives were introduced to run clinical trials for children, because getting medications approved for pediatric use is difficult. In the early 2000s, these medications started being prescribed to children for depression. Then, the BBC ran a documentary claiming these drugs were harmful and caused suicidal thoughts and actions in children. The FDA convened a committee to examine the trial evidence used for US approval and found a 4% rate of suicidal ideation in the experimental group taking the drugs, compared to a 2% rate in the placebo group.

Paul Matzko: So that margin between 4% and 2% was considered too small to provide a high confidence interval?

Peter van Doren: Given the size of the trial, that difference was not statistically significant under the traditional 95% confidence rule. Even so, the FDA—which is inherently risk-averse—imposed a strict “black box warning” on these drugs for pediatric use. The authors of this article argue that because of malpractice concerns in the US legal setting, the medical community reacted to this warning by drastically cutting back on diagnosing depression and prescribing these medications. A black box warning doesn’t mean you can’t prescribe a drug, it’s just a massive red flag on the packaging. But in a real-world malpractice lawsuit following a tragedy, a lawyer will look at the doctor and say, “There was a black box warning. How could you prescribe this anyway, you heartless person? Give us millions of dollars.” So doctors stopped prescribing them.

The authors performed a time-series intervention analysis looking at suicide rates by age in the United States. Lo and behold, after the black box warning was implemented and prescribing plummeted, the suicide rate among young children rose significantly. They argue that subsequent trials have proven this initial risk aversion was unwarranted, but the black box warning remains, which they believe is a mistake that should be revoked.

Paul Matzko: So we are right back into a conversation about confidence intervals, and whether you should act on a null result that isn’t statistically significant. You can make a plausible case for the FDA’s side: even without statistical significance at the standard rate, the consequences of getting it wrong with SSRIs—which carry heavy risks—are incredibly severe, so early risk aversion makes sense. But with Ivermectin, you can make the opposite argument: even without high confidence intervals, the risk of taking Ivermectin is generally very low and cheap, so people should be allowed to experiment. It’s fascinating because we are dealing with the exact same statistical question, but landing on two completely different answers depending on the severity of the treatment’s downside. That feels completely rational, but it cracks open the black box of science to show it’s a lot fuzzier than it looks from the outside. You can’t just say “the studies show X.” Which studies, at what confidence interval?

Peter van Doren: The studies always show exactly what they show. The real issue is: what decisions should we make given what the studies show? When I teach younger people, I tell them there is nothing wrong with science; science is just a controlled conversation among smart people disputing data within set boundaries. Where the fighting starts is over what one should do with their life based on that science. Someone might say, “I don’t require the strict confidence interval you require because I am more risk-seeking and less risk-averse.” That varies across individuals. As you said, the error of taking Ivermectin is simply that it might not work. The error of taking a drug that might induce suicidal thoughts is a much bigger deal, so you can see why an FDA analyst would lean toward a black box warning even without statistical significance. I get it. But the takeaway from this article is that the real-world consequence of that risk aversion was an increased suicide rate for young people who went untreated.

Paul Matzko: I want to shout out one specific statistical wrinkle in this paper. At first, I was skeptical of their mechanism. How do they know this is causative rather than correlative? Just because teen suicide went up after the black box label was imposed doesn’t inherently prove causation. But they made a very strong counter-argument: you do not see a corresponding rise in suicide rates among adults during this same period, and adults were not subject to the black box warning. That’s a decent argument. If the paper had only shown the correlation, I would have dismissed it, but pointing out the divergence between the youth and adult trends makes it worth listening to. Of course, science is a continuing conversation, and they could still be wrong. If Jonathan Haidt is right, the gap between kids and adults could be explained by the fact that kids adopted smartphones and social media at a much faster rate than adults. If you controlled for per capita phone usage among young people, the correlation with the black box warning might totally disappear, explaining the spike through screen use rather than the lack of medication.

Peter van Doren: That is exactly what a referee report on this paper might say. But we are back to our core point: the utter lack of clarity in these public debates. You and I, as non-scientists, spent hours digging into these papers and we are still left saying, “I don’t know.” We would never be interviewed on the Today Show.

Paul Matzko: The Today Show is aimed at parents who want a definitive yes or no answer. What we are saying is, “It depends,” which is a tough sell for television. But I think we are right and the soundbite version is wrong. There is rarely a simple “safe” or “unsafe.” There are just choices with probabilities. Sometimes bad things happen with very safe things, and sometimes good things happen with unsafe things. That’s just the way it is.

That’s our show for today. Thank you for listening, and please leave a review on your podcast platform of choice—it really helps more people find us. Thank you to Donald Lamar for producing. Until next time, be well.