Because my background is in economics and not religion or media, the media malpractice that gets me worked up the most usually involves numbers. No matter which newsroom I've worked in, the presence of numbers or numerical analysis seems to make journalists lose any brainpower. You should see what it's like when reporters try to analyze polling data or governmental budgets. It's just not pretty.
A reader sent along a story about a new nationwide study about porn usage. The story appears in the New Scientist. I have to be honest, if this story is any indication, that name must be satirical. There is literally nothing scientific about the report, headlined "Porn in the USA: Conservatives are biggest consumers."
There is literally not one scintilla of evidence to back up the claim, written by reporter Ewen Callaway. I'm not saying conservatives aren't the biggest consumers of porn. I'm just saying that there is nothing in the story to substantiate that they are. Maybe new science is about making stuff up?
The story begins by quoting the study's author Benjamin Edelman at Harvard Business School noting that there is little variation across the nation when it comes to consumption of on-line porn. And yet . . .:
However, there are some trends to be seen in the data. Those states that do consume the most porn tend to be more conservative and religious than states with lower levels of consumption, the study finds.
"Some of the people who are most outraged turn out to be consumers of the very things they claimed to be outraged by," Edelman says.
Um, this is your first clue that something is amiss. The data groups consumers by "states" and yet we then extrapolate based on anonymous credit card receipts that people who are outraged by porn are consuming it?
How do I explain to Edelman and Callaway that states are places where many people live and that in each state there are variations in behavior in attitudes? You can, say, live in a state that has legalized marijuana usage for medicinal purposes because a majority of voters supported that. And yet you can also oppose marijuana for any use. You can, say, live in a state that voted for President Barack Obama and yet not have voted or not have voted for him. See how that works? This seems like such an unbelievably obvious point to make and yet look at how the story lumps millions of people together as if they are all lock-step voters:
The biggest consumer, Utah, averaged 5.47 adult content subscriptions per 1000 home broadband users; Montana bought the least with 1.92 per 1000. "The differences here are not so stark," Edelman says.
Number 10 on the list was West Virginia at 2.94 subscriptions per 1000, while number 41, Michigan, averaged 2.32.
Eight of the top 10 pornography consuming states gave their electoral votes to John McCain in last year's presidential election – Florida and Hawaii were the exceptions. While six out of the lowest 10 favoured Barack Obama.
I don't if Callaway or Edelman -- or both -- are to blame for the idiocy of the above three paragraphs. But someone needs to explain to people that these numbers don't mean a thing.
I mean, is the Utah number so high because some high number of repressed Mormons are logging on to some cheesy porn site? Or is it so high because all of the non-Mormons use porn as an outlet in their private time because the state is so pious otherwise? Or is it some altogether different explanation related to, say, gender ratios? We have literally no idea because Edelman didn't have data that would even come close to answering that question.
Are John McCain voters obsessed with porn? Or are Barack Obama voters who are marginalized in Red States seeking comfort from porn? We have, again, literally no idea. To infer otherwise is nothing less than lying. And it's not just inference, if you recall that headline.
And then check out this:
Church-goers bought less online porn on Sundays -- a 1% increase in a postal code's religious attendance was associated with a 0.1% drop in subscriptions that day. However, expenditures on other days of the week brought them in line with the rest of the country, Edelman finds.
Residents of 27 states that passed laws banning gay marriages boasted 11% more porn subscribers than states that don't explicitly restrict gay marriage.
That first line can not be known from the data. Church-goers may buy more porn on Sundays, they may buy less. We don't know because the data set that Edelman was using didn't identify individuals by their church attendance. Are 100 percent of the people in postal codes with increased religious attendance going to church? Obviously not. So how do we know which percentage of the churchgoers are in his credit card data and which percentage of non-churchgoers are in his credit card data? We don't know. We just don't know. To say otherwise is hackery.
Now there have been studies that ask INDIVIDUALS (as opposed to, um, STATES) about their porn usage and various other sex-related questions. And when it comes to INDIVIDUALS (as opposed to, again, STATES), 60 percent more Democrats than Republicans report having watched at least one porno in the last year. That was from the 2006 General Social Survey. While there's no breakdown for conservative versus liberal, it does seem to contradict the claims by the New Scientist and Edelman. And it has the bonus of being data-driven as opposed to existing solely in the fevered imaginations and poorly-done statistical analysis of two individuals!
I read the study and it really is shocking that an assistant professor of business administration at Harvard University could be so clueless about what his data showed. I fear for the state of higher education. But whether or not a professor tries to make data say something it doesn't even come close to saying, reporters should remember that their job is not to be so gullible. As news coverage spreads, hopefully reporters won't fear or reject statistical analysis as much as Callaway. It's not looking good thus far.