Build a positive feedback loop

If a newsroom wants reporters to learn from experimentation, then the end point of every project is feedback and inspiration.

“The most important part of being an analyst is to take all this data and say, ‘This is what we value as good,’” said Parse.ly’s Kelsey Arendt. “If you’re only reporting on numbers, you’re purely reactionary and you’re not helping people do their jobs really well. Take a step back. Say, ‘Hey, team, this is what we value, this is why it’s important, and shout out to you, shout out to you, too.’”

Arendt was getting at another important point: Celebration is a powerful signal.

Our simplest and favorite question to ask as we were surveying newsrooms for this study was “When do you pop the champagne corks?” For most, the answer was still “when we win awards.” But there is a slow shift toward rewarding metrics victories.

“On a weekly basis, we try to reward good work — something with an engagement score, or someone noticed that our obit numbers seemed to have been undercounted by about a million,” said Erica Smith at The Virginian-Pilot. “The person who caught that, that was a thing we celebrated, and a lot of times that comes with a check for $100 or something.

“We’ve tried to get away from rewarding people with metrics and more about the journalism that they’re doing, but it’s often a combination of that. If you have a crappy story that everybody read, we’re not going to celebrate that the same way.”

Celebrations are, in some ways, a contradiction of how journalists have experienced metrics over the past decade. Many people told us that in any discussion about metrics their newsrooms suffer from data fatigue, choice paralysis and something like fear.

More often than not, we’ve observed, journalists worry that story-performance data is a scorecard used to make decisions about layoffs, not a tool to help newsrooms.

“Are we excited about data as a means of discovery or are we afraid of data as a means of punishment?” asked longtime journalist Jill Nicholson, now director of customer education at Chartbeat. “People are afraid of any concrete thing that might put them on the chopping block. It’s all about the culture. There are some leadership teams that use metrics as a stick to interrogate people. … If your leadership is hammering on about pageviews, seeing these numbers can stress you out.”

Several interview subjects also expressed the concern that publishers and managers are more interested in big traffic than in good storytelling. None of these journalists was averse to big audiences. But they were afraid that a relentless focus on high-traffic content would supplant crucial but less enticing beats, like local government.

Publishers and newsroom leaders who want their journalists to fall in love with metrics can get their teams on board if they respect those fears, and provide a meaningful response to them.

At NPR and in other newsrooms, an internal newsletter celebrates successes and milestones and analyzes failure in a way that makes it a natural part of experimentation, rather than a statement about the experimenter’s quality as a journalist.

“The format is what we call a dashboard report which looks at all of the different platforms and audience/reach consumption growth over last week and previous eight weeks,” said Dan Frohlich, a digital analyst. “When we have something more in depth, we regularly type that up and deliver that to the entire newsroom. I also have been sitting in the newsroom once a week to give a face to where people could ask questions.”

Notice who’s noticing

NPR’s analytics team doesn’t just measure how external audiences use the news.

Because their feedback to the newsroom and their reporting to leadership takes the form of a newsletter, the analytics team also measures how the newsroom is opening, studying and using their newsletter content — a way for them to do a formative assessment for their own (internal) audience and follow how recipients are engaging not just with analytics, but education about analytics. It also lets them identify the most willing and enthusiastic collaborators in thinking about the marriage of audience data and journalism.

“The weekly emails go out to executives as well,” Frohlich said. “We can see it through MailChimp — who is opening the emails we send, and who is forwarding the email to other people. We also have a monthly meeting on research where we invite most leaders from the organization and leaders from digital media and others so we can track attendance there.”

Jill Nicholson at Chartbeat suggests that organizations pay particular attention to and talk widely about experiments that cross departmental borders.

“I think that alignment across all of those groups is really important,” Nicholson said. “Is your editorial team trying to prioritize the same reader behaviors that your product team is and your marketing team is? Are we rowing in the same direction?”