How to build a metrics-savvy newsroom

Journalists have a reputation for dismissing data about their stories. But it’s undeserved.

After all, the axiom “if it bleeds, it leads” describes a newsroom practice driven by the metrics of newsstand sales and broadcast ratings. Pulitzers and Emmys are qualitative data that indicates a reporter’s work is respected by her peers.

Journalists don’t hate data. They just prefer data that’s easy to understand (because they’re busy), gives them immediately useful information about how to serve their audiences (because they’re still busy), and confirms that their work makes a difference to their newsrooms and communities (because that’s the mission).

But, while journalists might welcome feedback, the digital audience metrics served to newsrooms are often frustrating and more confusing than helpful.

Many of the reporters, producers and editors we spoke to for this study either say they don’t understand what the numbers say, or they’re worried that the metrics they see can’t really tell them how to serve their audiences.

Pageviews. Time on site. Subscribers. Open rate. Engagement. Bounce rate. Conversion. All of these describe audience behavior. But, unlike surveys, comments or face-to-face conversations, they don’t explicitly reveal what readers actually think and feel about the news.

Most digital metrics are an approximation for what news organizations really want to know. And that is …

“Love. How much are we loved by a certain reader, know what I mean?” said Sari Zeidler, Quartz’s director of growth. “We can see how often people are returning and what journey they’re taking down the funnel, but what we really want to know is do they love us, and how do we impact them? That’s what journalism is about: Did we have an impact on their life? We can know they open an email every day, but I want to better understand who is this person and why do they love us and how do they love us. What makes them so hardcore?”

And, until we can measure that, journalists will be skeptical. The abiding challenge is that metrics alone can’t tell a newsroom where to focus its attention. Metrics must be matched to strategy, to culture and to the specific job that each journalist was hired to do.

This report examines the best ways for newsrooms to think and communicate about metrics. It is part of the American Press Institute’s collection of Strategy Studies — in-depth reports that provide practical guidance for addressing important journalism challenges.

As we set out to write this study, we began with a few goals.

We wanted to speak to lots of newsrooms with lots of different resources and audiences. We knew that a strategy that works for Financial Times or NPR might not work at the Anchorage Daily News or the Virginian-Pilot, purely based on resources. We also wanted to talk to journalists who have seen successes and failures as they try to get their newsrooms more involved with audience data. And we spoke with non-journalism media, including Sesame Workshops, that have found long-term success in blending audience data with editorial judgment.

We also wanted to talk to the companies and nonprofits that provide metrics dashboards, consulting and other services and tools for newsrooms, including API’s Metrics for News, Chartbeat and Parse.ly. As bridges across many news organizations, metrics services providers and analysts have a broad and valuable view.

In the course of our reporting, we interviewed two dozen journalists and data analysts across 20 organizations, hunting for practices that are broad enough to be useful to most newsrooms, but specific enough that they provide at least a basic blueprint.

As you read further, we encourage you to look for examples from organizations that have a business model similar to your own.

Some of the organizations we studied pay the bills with advertising dollars; those organizations, like New Tropic and Evergrey parent company Whereby.us, are increasingly focused on building narrow, high-devotion audiences rather than chasing a million people who will click (or sign up) but never return to the site or open a newsletter.

Other organizations are shifting toward a subscriber model, and they are tightly focused on understanding how to move people from casual readers to frequent readers and, ultimately, subscribers. Those organizations, like the Dallas Morning News, are spending a lot of time thinking about that conversion process and zeroing in on simple things they can do to speed up the time it takes for someone to subscribe.

All of the organizations we surveyed had one common priority: They agree that building and sustaining high-quality journalism has to be at the center of any revenue strategy.

Metrics for decisions vs. metrics for learning

Educators would identify a lot of the “how you did” metrics in newsrooms these days as “summative assessments.” In classrooms, this is a grade — the 99 you got in news writing, the 62 in statistics.

At media companies, summative assessment is valuable for managers and executives who need to decide whether to keep or kill a product; whether to keep producing social video or shift that time into newsletters; or whether to submit a story for an award.

This is how most journalists have encountered metrics — as a final figure that they can’t change. Your story got this many pageviews. Your post got this many comments.

While summative assessments are useful for executives, those kinds of metrics rarely create fresh learning for staff. Why? Because they’re not analyzed and explained at a point when reporters and editors can change anything.

The other way to use metrics is what educators call a “formative assessment.” It focuses on how to achieve a goal, rather than judging whether a goal was met.

At Financial Times, an audience engagement team goes from desk to desk, helping every team understand the numbers. “They’ll do a health check to help you interpret your audience compared to a typical FT audience, and identify things that are interesting, and together come up with an experiment to test,” said Chief Data Officer Tom Betts.

With that help, reporters have figured out the best publication times — for the FT, the 8 a.m. to 9 a.m. start of the business day is the live-or-die window. “If you are filing copy late, you’re missing most of the audience,” Betts said.

Once a reporting team has tested that kind of idea, they generally don’t need further persuasion — just the chance to change their workflow to take advantage of new knowledge.