You Ought to Have a Look is a regular feature from the Center for the Study of Science.  While this section will feature all of the areas of interest that we are emphasizing, the prominence of the climate issue is driving a tremendous amount of web traffic.  Here we post a few of the best in recent days, along with our color commentary.

—

There was an interesting stream of articles this week that, when strung together, provides a pretty good idea as to how the scientific literature on climate change can (and have) become biased in a hurry.

First up, consider this provocative article by Vladimir Jankovic and David Schultz of University of Manchester titled “Atmosfear: Communicating the effects of climate change on extreme weather.” They formalize the idea that climate change communication has become dominated by trying to scare folks into acceptance (and thus compliance with action). The abstract is compelling:

The potential and serious effects of anthropogenic climate change are often communicated through the soundbite that anthropogenic climate change will produce more extreme weather. This soundbite has become popular with scientists and the media to get the public and governments to act against further increases in global temperature and their associated effects through the communication of scary scenarios, what we term “atmosfear.” Underlying atmosfear’s appeal, however, are four premises. First, atmosfear reduces the complexity of climate change to an identifiable target in the form of anthropogenically forced weather extremes. Second, anthropogenically driven weather extremes mandate a responsibility to act to protect the planet and society from harmful and increased risk. Third, achieving these ethical goals is predicated on emissions policies. Fourth, the end-result of these policies—a non-anthropogenic climate—is assumed to be more benign than an anthropogenically influenced one. Atmosfear oversimplifies and misstates the true state of the science and policy concerns in three ways. First, weather extremes are only one of the predicted effects of climate change and are best addressed by measures other than emission policies. Second, a pre-industrial climate may remain a policy goal, but is unachievable in reality. Third, the damages caused by any anthropogenically driven extremes may be overshadowed by the damages caused by increased exposure and vulnerability to the future risk. In reality, recent increases in damages and losses due to extreme weather events are due to societal factors. Thus, invoking atmosfear through such approaches as attribution science is not an effective means of either stimulating or legitimizing climate policies.

With a dominant atmosphere of atmosfear running through climate science, its pretty easy to see how the scientific literature (which is contributed to and gatekept by the scientific establishment) rapidly becomes overrun with pro-establishment articles. Silas Nissen of the Danish Niels Bohr Institute leads a team that investigated how “publication bias” leads to an unbalanced (and perhaps misleading) scientific knowledgebase. In their paper “Publication bias and the canonization of false facts” they describe how the preferential publication of “positive” results (i.e., results which find something seemingly “ interesting”—to the researcher, the publisher or perhaps funding agency), leads to a biased literature and, as a result, a misled public. From their abstract:

In our model, publication bias—in which positive results are published preferentially over negative ones—influences the distribution of published results. We find that when readers do not know the degree of publication bias and thus cannot condition on it, false claims often can be canonized as facts. Unless a sufficient fraction of negative results are published, the scientific process will do a poor job at discriminating false from true claims. This problem is exacerbated when scientists engage in p-hacking, data dredging, and other behaviors that increase the rate at which false positives are published…To the degree that the model accurately represents current scholarly practice, there will be serious concern about the validity of purported facts in some areas of scientific research.

Nissen and colleagues go on to conclude:

In the model of scientific inquiry that we have developed here, publication bias creates serious problems. While true claims will seldom be rejected, publication bias has the potential to cause many false claims to be mistakenly canonized as facts. This can be avoided only if a substantial fraction of negative results are published. But at present, publication bias appears to be strong, given that only a small fraction of the published scientific literature presents negative results. Presumably many negative results are going unreported. While this problem has been noted before, we do not know of any previous formal analysis of its consequences regarding the establishment of scientific facts.

And once the “facts” are ingrained, they set up a positive feedback loop as they get repeatedly “reviewed” in an increasingly popular pastime (enjoyed by national and international institutions alike) of producing assessment reports of the scientific literature, oftentimes as the foundation and justification for policymaking, such as the demonstrably atrocious “National Assessments” of climate change in the U.S. published by—who else—the federal government, to support—what else—it’s climate change policies.  In his new paper, “The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses,” Stanford’s John Ioannidis describes the proliferation of systematic review papers and why this is a terrible development for science and science-based policy. In an interview with Retraction Watch, Ioannidis discusses his findings. Here’s a taste:

Retraction Watch: You say that the numbers of systematic reviews and meta-analyses have reached “epidemic proportions,” and that there is currently a “massive production of unnecessary, misleading, and conflicted systematic reviews and meta-analyses.” Indeed, you note the number of each has risen more than 2500% since 1991, often with more than 20 meta-analyses on the same topic. Why the massive increase, and why is it a problem?

John Ioannidis: The increase is a consequence of the higher prestige that systematic reviews and meta-analyses have acquired over the years, since they are (justifiably) considered to represent the highest level of evidence. Many scientists now want to do them, leading journals want to publish them, and sponsors and other conflicted stakeholders want to exploit them to promote their products, beliefs, and agendas. Systematic reviews and meta-analyses that are carefully done and that are done by players who do not have conflicts and pre-determined agendas are not a problem, quite the opposite. The problem is that most of them are not carefully done and/or are done with pre-determined agendas on what to find and report.

Ioannidis concludes “Few systematic reviews and meta-analyses are both non-misleading and useful.”

Together, the chain of events described above leads to what’s been called an “availability cascade”—a self-promulgating process of collective belief. As we’ve highlighted on previous occasions, availability cascades lead to nowhere good in short order. The first step in combatting them is to recognize that we are caught up in one. The above papers help to illuminate this, and you really ought to have a look!

References:

Ioannidis, J., 2016. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses, Milbank Quarterly, 94, 485–514, DOI: 10.1111/1468-0009.12210.

Jankovic, V. and D. Schultz, 2016.  Atmosfear: Communicating the effects of climate change on extreme weather. Weather, Climate and Society, DOI: http://dx.doi.org/10.1175/WCAS-D-16-0030.1.

Nissen, S, et al., 2016. Publication bias and the canonization of false facts. Archived at arXiv.org, https://arxiv.org/abs/1609.00494