Tuesday, May 22, 2012

Public opinion surveys, too?

Last week, I ran a post arguing that just about all scientific studies (or at least, their "conclusions") were warmed-over B.S. designed to confuse a public with limited knowledge or understanding of statistics. This week, I came across an article from the Pew Research Center that indicated that public surveys are in danger of becoming similarly unreliable.
For decades survey research has provided trusted data about political attitudes and voting behavior, the economy, health, education, demography and many other topics. But political and media surveys are facing significant challenges as a consequence of societal and technological changes. 
It has become increasingly difficult to contact potential respondents and to persuade them to participate. The percentage of households in a sample that are successfully interviewed – the response rate – has fallen dramatically. At Pew Research, the response rate of a typical telephone survey was 36% in 1997 and is just 9% today. 
The general decline in response rates is evident across nearly all types of surveys, in the United States and abroad. At the same time, greater effort and expense are required to achieve even the diminished response rates of today. These challenges have led many to question whether surveys are still providing accurate and unbiased information. Although response rates have decreased in landline surveys, the inclusion of cell phones – necessitated by the rapid rise of households with cell phones but no landline – has further contributed to the overall decline in response rates for telephone surveys.
The folks at the Pew Center also included this handy chart, which shows just how much deterioration has occurred in just the last 15 years:


There's a lot more to the article than just that blurb, and I'll admit that I'm not fully doing it justice by focusing only on this one dynamic. But I think it's an incredibly important trend, and one that most people are completely unaware of--nobody really knows these days how their statistical sausage is made (let alone their actual sausage, but let's just leave that for another day).

The problem with unreliable public opinion surveys, much like unreliable scientific studies, is that their results often impact future behavior. For example, if I'm considering voting for Ron Paul in an upcoming election, but opinion surveys show that he is polling only about 10%, I'll likely re-consider my vote because he looks "unelectable", regardless of whether or not this is actually the case. On the flip side, a poll that overstates a candidate's popularity could perversely make him more popular among other voters (in fact, I'm pretty sure this is what happened with Rick Santorum). Therefore, public opinion surveys can at times become self-fulfilling prophecies, which is always a bit of a problem.


But response rate isn't even the only problem that opinion polls are currently facing. I've recently participated in multiple phone surveys (one political, one economic), and both times I was literally laughing at the questions as I heard them because they were structured in ways that couldn't possibly reflect my actual thoughts or beliefs--they required overly simplistic answers, or else falsely limited my universe of possible answers. (An example: one caller asked if I aligned myself more closely with the Republican or Democratic party platforms. I responded "neither", and she told me that wasn't an option. I asked her if she was serious. She said she was. I laughed and said fine, flip a coin for me. She refused. So I flipped a coin. It came up tails. I went with the Democrats.)

Our problem is that we've all become increasingly obsessed with data, even as the quality of that data has steadily declined. But if you demand that your data tell a story, they'll tell one for you, even if it's an imaginary story. With response rates this low, that's basically what public opinion surveys have become--imaginary stories to be reported in national news outlets as if they were real news. It's sort of sad, really.

[Pew Research Center]

2 comments:

  1. what's 'sort of sad, really' is that you would re-consider voting for ron paul simply because he looks 'unelectable'. nice self-fulfilling prophecy. i wasn't aware that the right to vote was a multiple-choice option. i also wasn't aware that our forefathers fought for the right to call heads or tails every four years. seems like they could have just flipped a coin with king george and saved themselves a lot of time and bloodshed. 'heads i don't pay tribute to you and your system of excessive taxation to fund your bullshit endeavors back in europe; tails i do'. why don't you man up and vote for whoever you want to vote for on principle. only an idiot would think that his vote mattered one way or the other except on principle anyway. i mean, you have to be either a moron or a narcissist, or maybe a moronic narcissist, to think that if you vote for ron paul and then mitt romney wins, it was because you didn't vote for obama. what's 'sad, really', is that we don't seem to have any men of principle anymore. everyone is too busy figuring out the 'easiest' way to meander through life without having to do anything for themselves, or without having to take a stand on anything, or ruffle any feathers, or make anything remotely difficult on themselves to realize that that isn't freedom at all. you end up with a nation full of smug motherfuckers like the guy in the apple commercials

    http://www.youtube.com/watch?v=VCL5UgxtoLs

    this mass is ended. let us go in peace to vote for whichever pre-approved candidate offends us less. and then pretend it would have been any different if the other pre-approved candidate had won.

    it's literally like arguing over whether pepsi or coke is better. they are both shitty cola products that ruin the ph of your bloodstream, causing your system to steal calcium from your bones and teeth to correct the acid-alkaline balance in your body to prevent you from immediate death. but then i guess that probably doesn't apply to you. seeing as how you only drink coke, and would never drink pepsi. also, you would probably never drink rc cola, either. it's just unelectable.

    ReplyDelete
  2. Alright, jerkoff, let me rephrase... "one might reconsider one's vote because he looks 'unelectable'". My mistake was in using the active first-person voice in what was actually a hypothetical sentence. I apologize for my grammatical laziness, similar to your choice to eschew capitalization. Ohhh I'm a fucking rebel, I don't use capital letters, I'm like a motherfucking modern-day e.e. cummings. Bitch, please. What would really be rebellious is to type in German, now that I could stand up for. I want some umlauts and eszetts up in this bitch, pronto. Ron Paul would.

    As a point of fact, I did vote for Paul in the VA primary. He got 40% of the vote here because Gingrich, Perry, and Bachmann were too stupid to get on the ballot. He didn't win many delegates, but fuck it. I'll vote for him in November, too, even if he's not on the ballot. Because fuck it and fuck Romney. Unless Romney tabs Paul as his running mate, in which case my head will explode.

    Also, I hate that Apple douchebag. And Apple, as I've made perfectly clear here on several occasions. And I don't like RC Cola either, because of its bullshit kowtowing to the Crown (it's in fact ironic that you would give RC Cola your tacit support, given your rant about King George--I'm sorry "king george"). Instead, give me Jolt Cola, the only soft drink brand to ever stand up for what it is and not pretend to be something else. I WILL ROT YOUR TEETH OUT OF YOUR SKULL BUT YOU'LL BE TOO AMPED ON CAFFEINE TO GIVE A SHIT. JOLT COLA, MOTHERFUCKERS. Hell yeah. Now that's change I can believe in.

    ReplyDelete