How will we cover public opinion polls?

What is a Critical Conversation? | Printable discussion guide

Public opinion polls — both those conducted for your own news organization and those done for others — can be challenging to cover. There are the tricky mechanics of the polls, which need to be accurately explained, and there is the larger question of where polls fit in your overall coverage. Finally, it’s important that journalists writing about polls understand them — the different types of polls, their varying quality, and their methodologies. This discussion is aimed at helping you navigate some key challenges.

Who’s in this conversation?

This topic will necessarily include politics reporters and editors. Data and graphics team members should also be included, as should standards editors. The social media and audience teams should have insights into these issues, especially for the discussion on writing accurately about polls.

The agenda

1. Should we even cover polls?

In a data-driven world, this may seem like a radical question. But it’s a discussion worth having, because polls can be problematic in several ways, some of which are described below — so you may want to answer this question at the end. In the meantime, here’s a note from the editor of the Cleveland Plain-Dealer on why he doesn’t cover them.

2. If we cover polls, how can we ensure that our coverage includes depth and context?

Media critics have called for demoting “horse race” coverage and instead favor paying more attention to what New York University’s Jay Rosen calls the “not the odds, but the stakes.” Polls are inherently horse race-oriented. But they also provide a snapshot of public opinion — important for journalists in capturing the political landscape. So you may want to look for opportunities to frame polls in ways that add meaning and value — in other words, make the story about more than just the poll.

Consider these actions:

• Examine the cross tabs of major polls for story ideas that might be especially relevant to your readership. (The New York Times, for example, publishes the cross tabs of its major polls.)
• Examples of drill-down stories: The Times wrote about support for Donald Trump among Latino voters. This Wall Street Journal story focused on the views of suburban women on abortion.

3. How can we convey the limits of polls without undermining the insights they offer?

Polling experts say one mistake they see in polling coverage is when journalists suggest that a poll says more about the electorate than it actually does. They are a snapshot in time, not a projection of who will win an election held weeks or months later. Washington Post Columnist Jennifer Rubin is among those concerned that journalists overstate the meaning of polls.

This is a discussion that can cause tension in story framing. Polls have limitations, and it’s important to be transparent about those limitations. But editors can be hesitant to emphasize these limitations for fear of weakening the story. We are trained, after all, to look for change and explain it. But sometimes the changes just aren’t that meaningful from one poll to the next — especially if the change falls within the poll’s margin of error.

Consider these actions:

• Recognize and openly discuss this tension. There should be no penalty if someone plays the role of “resident skeptic” by saying a poll isn’t just that meaningful compared with the last one.
• Consider your transparency wording to show the limitations of polls. Look at how other news organizations have explained their polls and their methodology, and consider whether their approach would work — or could be adapted — for you. Here is an example from The Atlanta Journal-Constitution. Here’s ABC News. API’s partners at Trusting News have also written on this topic.
• Ask someone on staff to look at your past coverage of polls for insights into how to frame current polling coverage. What lessons did they take away?

4. How can we ensure we’re writing accurately about polls?

Once you’ve decided to write about a poll, it’s important to get the terminology right. Journalists can get tripped up when describing the margin-of-error, for example.

Consider these actions:

• Make sure a person or a team fully understands margin-of-error, demographic weighting, the difference between a poll and a focus group and other polling mechanics so they can answer staff (or audience) questions about it. There are some resources they can use:
  • The Pew Research Center offers a short course on how polling works.
  • A Roper Center (Cornell University) glossary.
  • A Journalist’s Resource piece on the margin of error.
• Create boilerplate language or a standing sidebar to run when you publish poll stories that explain these issues.
• Set up a brown bag information session with a trusted polling expert from a local university or think tank who can walk the team through some of the technicalities of polling. Tell staff to have questions ready in advance.

5. Which polls should we cover and which should we avoid?

There are cases in which polls come to you that seem newsworthy, but may require discussion. For example, how should we handle it when someone leaks us an internal poll? What about an internet survey by an interest group? Are our journalists able to discern “junk” polls from quality ones? How should we handle it when a politician talks about a poll in a newsy way, but it’s a poll you’re hesitant to amplify?

Consider these actions:

• Make staffers aware of the perils of covering internal polls. In a National Press Club Institute webinar with experts, the Cook Political Report’s Erin Covey talked about how campaigns use internal polls to advance their own agenda — and why journalists should be wary of them.
• Familiarize yourself with polling standards. 538 has pollster ratings that includes an explanation of their standards.
• Ask trusted political scientists (see brown bag suggestion above) how they gauge poll quality.

6. How can we use polls to engage our community?

People often love to know where they fit with the rest of their community. If you’re going to do a poll, or write about one, think about whether there are ways to use the data to engage your community. This helps build audience engagement and gives people a sense that you are listening to them as well as informing them.

Consider these actions:

• Ask appropriate people — probably the politics, audience and graphics teams working together — to think about using polling data to engage your audience. Consider a feature that asks people some of the same questions the poll respondents were asked. Is there an interactive feature that people could use to show how their views align with or depart from the poll results?