As part of our efforts to expand and improve fact checking, the American Press Institute regularly presents tips on how media organizations around the country chase down facts. Today, Meghan Hoyer, data journalist at USA TODAY, explains how she got to the bottom of some mystifying data that just couldn’t be true.
Background: After a Georgia father was charged with murder in a case where his young son was intentionally left in a hot car, my colleague Marisol Bello came to me with a question: Could we find out how often parents kill their children, and learn of the circumstances in those deaths? I’ve worked with FBI homicide data for nearly two years on a project about mass killing, and knew that information could be teased out of supplemental homicide reports (SHR), but that the SHR was notoriously spotty because many cities fail to provide details on murders. So I started looking for other research on the topic.
A little digging found what seemed to be the gold standard of analysis and stats: a piece authored by an Ivy League researcher. It was published in a peer-reviewed journal earlier this year and used FBI homicide data to deliver annual totals and details on the issue of filicide. The research had been lauded by Brown University and picked up in a number of media sources at the time, because according to the researchers, 3,000 children on average were killed by their parents each year between 1976 and 2008. But when I went to the raw FBI data to get comparable stats for more recent years, I couldn’t get my numbers to come even close to those findings – the researchers’ numbers were roughly six times my tallies. Something was wrong.
Fact to check: Did 3,000 children die by their parents’ hand each year? And if so, can we find more recent figures on the same types of incidents? If not, can we get accurate figures on the number of filicides in the United States?
How I got that fact: First, the numbers published in the research paper needed a quick common-sense check: On average, about 16,000 people die in homicides each year in the U.S. The researchers’ figures would mean that nearly 20 percent of those killings were children killed by their parents. That obviously was not right.
So I emailed the researchers about their methodology. Instead of using the raw FBI data, they said, they had used cleaned-up figures from James Alan Fox and Marc Swatt of Northeastern University, and that’s why their figures differed from mine. So I downloaded the same data set and immediately realized the problem.
Fox and Swatt’s data is built for advanced statistical analysis, and as such, has multiple imputations and weights – this allows academic researchers to fill in holes that we know exist in FBI data, either where cases are missing entirely, or where certain details (the relationship between victim and killer, for instance) are missing. Each killing was broken into six lines: the original record, and five different imputations with different weights applied and missing values filled in. But in this report, all the cases had simply been added up, which basically multiplied the raw case number by six.
No weights had been applied to cases – which meant cases where a relationship wasn’t known, or where cities didn’t report all the data, were omitted. To get a more accurate count of filicides, I first used the imputed values to fill in the blanks where relationships weren’t provided to the FBI by the reporting agencies. I then used Fox’s national pweight, or probability weight, to account for missing records. To get more recent years’ figures, I called Fox, who provided me their updated database which spanned from 1976 through 2012.
The upshot? We found that on average, about 450 children are killed by a parent or stepparent each year.
Brown University has since issued a correction to their press release on the researchers’ findings. Marisol and I used the data and our findings in a were published in USA TODAY, along with another follow-up story published this week.
— Meghan Hoyer, USA TODAY
Have a tip you’d like to share? E-mail us.
Interested in our weekly newsletter on fact-checking news, research and efforts around the country? Sign up here.
You might also be interested in:
API's summit on Elections, Trust and Democracy led to a 5-step process for how news organizations can do this work now — and year-round.
News leaders have told us that they want to better support reporters and editors who will be covering an election for the first time. That’s why we’re sharing portions of this media guide to covering elections and voting from The Elections Group.
These guides are here to help you save time while getting some core thinking shared among your staff, on the record. So when you face a specific development you’re not saying, “I wish we’d talked about this in advance.”
