Fact-Checking Common Policy Misperceptions

Evidence-based recommendations for journalists

In this summary of her research, Emily Thorson of George Washington University offers recommendations to journalists who write about public policy issues.

Many Americans are substantially misinformed about key aspects of American politics and policies. However, misperceptions about politicians (for example, President Obama’s birthplace) have often received outsized attention from both the public and journalists as compared to misperceptions about policies. This paper uses a survey and experiment to show that Americans hold a number of misperceptions about the government policies that shape their lives.

But contrary to the view of Americans as intractably bound to their false beliefs, experimental evidence shows that fact-checking works. Many people are quite willing to accept corrections, especially on matters of policy rather than electoral politics.

The results suggest that by fact-checking these misperceptions in news coverage of policy issues, journalists may be able to substantially reduce misperceptions.

Methodology

An online survey was conducted over two waves in November of 2014 using a representative sample recruited through YouGov (N=1000). The survey included 13 factual questions about a wide range of policies, as well as an over-time experiment designed to assess whether misperceptions could be effectively corrected. Question wording and topics were informed by a series of open-ended interviews about policies conducted with forty Americans in the summer of 2014.

Each participant was asked to choose which of two statements (presented in random order) was more correct. They were then asked to indicate whether they were “very confident,” “somewhat confident,” or “not at all confident” in their answer. The 13 pairs of statements are listed below.

Results: Breadth of Misperceptions

On average, people answered slightly more than half of the 13 questions (7.5) correctly. More educated people were more likely to answer more questions correctly, as were people who regularly read newspapers and were interested in politics. Neither party identification nor gender was related to the number of correct answers.

The following figure shows the total percentage of respondents who answered each question incorrectly as well as the distribution of confidence around their answer. The four questions that more than 50  percent of people answered incorrectly concerned China’s ownership of U.S. debt, interest paid on that debt, the relative cost of the health care versus military, and undocumented immigrants’ eligibility for food stamps.

Percentage Incorrect, by Confidence Level

(N=945; sampling error of plus or minus three percentage points)

Percentage Incorrect, by Confidence Level

(N=945; sampling error of plus or minus three percentage points)

The darkest bar represents those who are “very confident” in their incorrect answer. This estimate of the “confidently incorrect” represents the most conservative estimate of the breadth of misperceptions. Looking only at this number, the three most common misperceptions (those that over 20  percent of the population hold strongly) concern China’s ownership of U.S. debt, the funding of Social Security, and the size of government.

The middle section of each bar shows the percentage of people who were “somewhat confident” in their misperception. For eight out of the thirteen questions, more than 30 percent of respondents expressed that they were “somewhat” or “very” confident in their incorrect answer.

Results: Effectiveness of Corrections

After giving their answers to the thirteen questions, participants were shown the correct answer for each one. A month later, they were surveyed again and asked the same set questions. In this second survey, the average number of questions answered correctly rose significantly (from 7.4 to 8.9). A majority (60 percent) of respondents did better at Time 2 than at Time 1, while 20 percent received the same score and 20 percent did worse.

For every question but one (Medicare), the percentage of people answering incorrectly decreased between Time 1 and Time 2.  The largest drops were for TANF time limits (from 54 percent wrong to 31 percent), China’s debt ownership (from 67 percent wrong to 46 percent) and immigrants’ eligibility for food stamps (from 57 percent wrong to 40 percent).

People who reported being interested in politics were most likely to learn the correct answers, and learning was not more likely among Republicans or Democrats.

Conclusion & Recommendations

The results of this study demonstrate the importance of looking beyond electoral politics when identifying and correcting misperceptions. While many Americans are misinformed about key aspects of public policy, fact-checking can effectively reduce these misperceptions, and the effects are surprisingly long-lasting.

Correcting misperceptions. This study employed a straightforward intervention: participants were shown a single sentence with the correct information. Its effectiveness suggests that adding simple explanatory information to news coverage may go a long way towards correcting misperceptions. For example, including a sentence describing how Social Security is funded in a piece discussing a candidate’s stance on the issue may help to correct misperceptions that people may have about the program.

Identifying misperceptions. The misperceptions assessed in this survey were generated through an interview process in which a range of Americans were asked to explain different policies. This open-ended process can help to identify areas of confusion for citizens that may not be immediately obvious to journalists or academics. Informally, conversations with non-experts can help to identify issues that need more explanation. More formally, media outlets may want to consider including open-ended questions on their reader surveys to gauge their audience’s need for clarification around different issue areas.