Facts, Opinions, and the Pew Research Center’s Pseudoscience

This article appeared on EconLog on October 29, 2019.
Share

Do you remember those grade school exercises where you had to divide a bunch of statements into facts and opinions? The trick to getting an ‘A' was easy: if a statement could be looked up in a reference book or checked by simple observation—e.g., "Topeka is in Kansas," "An isosceles triangle has two sides of equal length," "My sneakers are white,"—you labeled it a fact (even if it was false!); otherwise it was an opinion.

At the time, I gave the exercises little thought, but they should have bothered me. I certainly considered "Hitler was evil," "Catherine Bach is beautiful," "Johnny Carson is funny," and "The food in the school cafeteria is lousy" to all be facts, though none of them could be checked in reference books. On the other hand, "Elvis is alive and working at a Denny's in Tucson" could be checked, but that didn't seem like a fact to me.

It wasn't until my first formal logic class that I thought about those exercises critically. In logic, all statements are claims about the world. Facts are accurate statements (though many facts will never be known because no one will ever prove their accuracy—or even conceive of them), while opinions are expressed statements that are believed to be true but may or may not be facts. "Topeka is in Kansas" is a fact and is an opinion held by many people, while "Topeka is in Virginia" is neither a fact nor—as far as I know—an opinion (because I don't know anyone who believes that).

Normative and subjective statements can also be facts and opinions, though it can be argued that many of them are vague. For instance, "The food in the school cafeteria is lousy" may be better understood as "Most people dislike the food in the cafeteria."

One of my logic teachers, Virginia Tech emeritus professor Harlan B. Miller, felt so strongly about this that he wrote in his logic text:

You may have been unfortunate enough, in grade school or middle school, to have been taught to contrast "fact" with "opinion" and to sort statements into these two categories as exclusive and exhaustive. If so, forget it. Try to clear that distinction from your memory banks. It's nonsense or worse.

Sorting statements as fact or opinion is as sensible as sorting physical objects as green or plastic. Some things are one, some the other, but lots are both and lots are neither. There are lots of facts that are no one's opinions (the unsaid truths), lots of opinions that certainly don't correspond to facts (The Sun at your supermarket checkout is full of them), lots of unbelieved falsehoods that are neither facts nor opinions, and lots of things, including all known propositions, that are both fact and opinion.

It's a fact, and my opinion, that 7 + 7 = 14. It's my opinion, and a fact, that Hitler was morally depraved. Many (not all) facts are the opinions of somebody. Many (not all) opinions are (correspond to) facts.

So flush the fact/opinion contrast from your system. It should never have been there.

For good measure, he cites this article by Cal State–Sacramento philosophy professor Perry Weddle and admits to the paranoid suspicion that the fact/opinion exercises are part of "a plot to sell ethical emotivism by warping the minds of ten-year-olds."

I've occasionally thought about those exercises over the years, wondering if grade schoolers still do them. I hoped that, as more and more people studied logic, they—especially K–12 educators—had relegated the fact/opinion dichotomy to the dustbin of pseudoscience along with alchemy and phrenology.

Then, this week, I came upon this headline and deck on The Atlantic's website:

Older People Are Worse Than Young People at Telling Fact from Opinion

Given five facts, only 17 percent of people over 65 were able to identify them all as factual statements.

The article explains that the Pew Research Center's Journalism & Media unit has been administering its own version of the fact/opinion exercises to adults, then issuing hand-wringing reports on the inability of many test-takers to "correctly" separate the statements. According to Pew,

Factual statements are unambiguous statements that one can prove to be accurate or inaccurate based on objective evidence. Opinion statements are based in people's beliefs and values, whether political, religious, moral, cultural, or some other belief system, and therefore lack traditional standards of objectivity. And borderline statements have elements of both factual and opinion statements — they can be based in objective evidence, but claims are vague enough that they can neither be unambiguously proved nor disproved by factual evidence, in part because such evidence is often conflicting, incomplete, contested or involves making predictions.

That certainly sounds like the distinction from grade school.

The Atlantic article's author, Alexis Madrigal, accepts the dichotomy uncritically and argues (torturously) that people's inability to separate fact from opinion should be blamed on the repeal of the Federal Communications Commission's Fairness Doctrine, the rise of social media, and the existence of Breitbart, Fox News and MSNBC. A quick Google search finds that Madrigal is not alone in accepting Pew's fact/opinion dichotomy uncritically; so does Justin Doom at ABC NewsJoe Concha at The HillMairead McCardle at National Review OnlineJohn Walsh at Business InsiderAngela Moon at Reuters, and even Rush Limbaugh.

It's tempting to dismiss this as a mere semantic disagreement between philosophers and logicians on one hand and the Pew folks, media, and K–12 educators on the other. But if you look at Pew's "test answers" for 10 statements it deems unambiguous facts or opinions, you find that Pew has trouble following its own definitions.

For instance, it classifies the following three statements as "opinions":

  • Immigrants who are in the U.S. illegally are a very big problem for the country today.
  • Government is always wasteful and inefficient.
  • Increasing the minimum wage to $15 an hour is essential for the health of the U.S. economy.

In public policy, analyses are regularly published arguing that these statements are true or false, and thus facts or falsehoods. These analyses make extensive use of "objective evidence" and "traditional standards of objectivity" such as empirical data and statistical analysis—the stuff Pew claims are the hallmarks of "facts."

For instance, the notion that illegal immigrants are "a very big problem" rests on ideas about crime and public safety, political and social dynamics, labor economics, and government fiscal effects. Those topics can be evaluated with empirical data, historical examples, economic modeling, and basic mathematics—that is, "traditional standards of objectivity." My Cato colleagues Alex Nowrasteh and David Bier muster mountains of "objective evidence" on the effects of immigration and provide a careful, reasoned framework for evaluating them, concluding that illegal immigrants are in fact not a very big problem (or any problem at all, on net) for the country today.

Ah-ha, the Pew folks would likely respond, but analysts at the immigrant-wary Center for Immigration Studies likewise muster evidence and assemble a rational framework, and they conclude that illegal immigrants are in fact a very big problem for the country today. It can't be the case that both Cato and CIS are right—so, Pew reasons, there must be no objective fact of this matter, just opinion. But just because there is disagreement doesn't mean there's no fact, just as disagreements between physicists don't mean there's no facts in physics. Rather, these disagreements mean the facts are in dispute.

Similar arguments can be made about government wastefulness and inefficiency (two concepts that are well-defined in economics) and whether a minimum-wage increase is "essential" (or even beneficial) to the nation. In fact, all five statements that Pew identifies as opinions and not facts clearly are claims to fact. They are all subject to objective analysis (though, again, that analysis may not settle the issue—at least not today) and are either facts or falsehoods as well as opinions.

What about the statements Pew labels as "facts"? All five do accurately describe the world, but at least one of them is not a "fact" according to Pew's definition of the term. Consider this statement:

  • ISIS lost a significant portion of its territory in Iraq and Syria in 2017.

ISIS certainly lost most of its territory over the last few years. But does the amount lost in 2017 count as "significant"? That depends on what is meant by "significant." In social science, statistical significance usually (but not always) means there is less than a 5 percent chance the correlation of two variables is the result of random chance. By extension, does that mean ISIS must lose at least 95 percent of its land over the course of 2017 in order for the loss to be "significant"? Or does "significance" in this context depend on "people's beliefs and values … and therefore lack traditional standards of objectivity"? If we follow Pew's definition, then this statement is an "opinion," not a "fact."

Pew's dichotomy of fact and opinion is problematic—so problematic that Pew violates its own definitions of those two terms even when it tries to use them carefully. Rather than casting aspersions upon the people who ostensibly misidentify "opinions" as "facts" and vice-versa, Pew should appreciate that those people take value judgments seriously enough to believe they accurately reflect the world we live in—that is, that they are facts.

That leads to the deeper problem underlying the Pew exercise. If value judgments are purely matters of opinion and wholly removed from fact, then most if not all policy disagreements—and most if not all human disagreements—cannot be settled rationally. "Slavery should not be legal in most cases" is not a fact according to Pew, but an opinion that cannot be true or false. Likewise "The Holocaust was immoral," "Sexual harassment is a big problem in the United States," and "The immigrant caravan heading toward the United States is a big problem for the country."

In fairness to Pew, I'm sure this exercise is motivated by a noble idea: to get people to think critically about their value-laden beliefs about the world, which in turn should encourage civility to others with different points of view. But by making fact and opinion mutually exclusive, Pew goes too far, removing the very reason—trying to understand the world—for critical thinking and civility. If opinions can't be facts, then societal disagreements on these matters can only be settled through power-battles between opposing tribes, with the strongest tribe dictating to everyone else. The Pew Research Center's nobly intentioned project becomes the perfect clarion for the Age of Trump.

This is not to say that policy disagreements don't entail differing personal tastes, preferences, and risk tolerances. De Gustibus Non Est Disputandum, after all. That's why I'm a libertarian who believes in robust individual liberties and limited government—institutions that help people with different preferences maximize their freedom to follow those preferences while living peaceably with others in a well-functioning society. The same is true when the facts are unknown or in dispute. But differing tastes don't mean that we should deem matters of judgment, values, and beliefs—and all matters subject to disagreement—to be fact-free zones. Doing that would undermine the very importance of these disagreements.