Wednesday, May 28, 2014

How not to report a poll

The Toronto Star has a significant-looking headline this morning: "Justin Trudeau, Liberal support, dip over pro-choice stance".

The first five words were mostly correct, the rest were a bit of a leap. This concerned a Forum Research poll showing the Liberals down three points since the end of April. But that is all the poll showed, as no questions relating to abortion were included with the survey and, thus, there is no evidence to link the (statistically insignificant) drop of support to his abortion stance. It could have been related, or the drop could have been because Trudeau looks a little bit like Brian Boyle of the New York Rangers (the Liberals did drop in Quebec, after all).

This wasn't entirely the fault of the Star, however, as the headline and the lede were merely echoing what the analysis included with the poll was saying.

Unless they are expressing their opinions as close observers of politics separate from the analysis of a poll, pollsters must limit themselves to what their own numbers show. Otherwise, they are giving their opinion the semblance of authority, of evidence-based observation, when that is not the case. There was nothing in the Forum poll that suggested the drop was related to Trudeau's abortion stance, unless you consider an increase of support for the Liberals among evangelical Christians to be sarcastic.

I wrote about this poll and this issue in more detail for The Huffington Post Canada today. Please go check it out.

19 comments:

  1. Thank goodness for blogs like yours that cut through the rhetoric with rational thought!

    ReplyDelete
    Replies
    1. Rhetoric? That's giving it too much credit. I'd call it B.S., and that's just because Eric would block anything stronger.

      Delete
  2. Good article. Headlines like this show why your work is so necessary.

    You could also mention that this 3% dip corresponds to a decrease from the impossibly high value of 39% (which I doubt anyone believes) in Forum polls to a value closer to the consensus value of 36%, a more realistic estimate that has held steady for almost a year.

    ReplyDelete
  3. No ... new ... Ontario ... poll ... Need ... fix ... Aghhh!!!

    ReplyDelete
  4. Which sort of ties in with what I said earlier about pollsters having political biases ??

    ReplyDelete
  5. Meh.

    Last Forum poll, Trudeau was up among Catholics 41 to 25. Not it's 36 to 27.

    Pretty substantial, and his spat with the Cardinal seems like a plausible cause.

    ReplyDelete
    Replies
    1. It's not implausible as a cause of that drop, but 5% of self-identified catholics who actually care what a cardinal thinks is not 3% of the population.

      Delete
  6. I'm glad you wrote this, Eric. This is exactly what I thought when I looked at the Forum result & then scrolled through your site looking for a link to previous data. Pollster (and lazy Newspaper) trying to grab a headline, with no substance behind it. This is what can undermine the polling industry, in my opinion.

    ReplyDelete
  7. People posting here do this type of thing all the time, subscribing singular events to differences between different polls, even different polling firms. Events like Trudeau dropped the F bomb, Jim Flahertys death...but without any evidence...and even saying its so obvious that x event is causing a change, as justification.

    ReplyDelete
    Replies
    1. I think if a some guy on a comment forum speculating is different from a newspaper headline.

      As for the Toronto Star article/Forum analysis. I found it pretty weird since the Trudeau Liberals went down by a mere 3 points, which is not significant looking at the margin of error.

      Delete
    2. In and of itself it is not significant, but if several more polls show a similar pattern, it would at least be somewhat significant. The bottom line is we need to wait for more polls to confirm the trend.

      Delete
  8. This is not unlike seeing 'The Liberals are up two at the expense of the NDP" if from poll to poll the Liberals are +2 and the NDP -2 say. I mean unless it was asked we don't know. It could be the Liberals got 1 from the PCs, and one from the NDP. The NDP gave 1 to the PC and 1 to the Liberals.

    ReplyDelete
    Replies
    1. I agree entirely with your point. The reflexive shorthand analysis of mainstream reporting on polls is irritating at least, and misleading and damaging at worst.

      Delete
    2. I wish I could thumbs up a post because I certainly would this one. And most likely the +2% and -2% occurred with a margin of error of 3%.....

      Delete
    3. Which is why everyone should take my statistics classes. Or, well, that is how I try to sell it to my students.....

      Delete
  9. Meanwhile, I notice that the BEST PM index by Nanos for May show Justin Trudeau going up by 3 points.

    ReplyDelete
  10. I doubt the same blog post would have been made if it was Harper who was dipping in the polls and Forum was linking his drop to the Senate Scandal.

    ReplyDelete
    Replies
    1. http://www.threehundredeight.com/2013/05/sampling-weighting-and-transparency-in.html

      In the poll analyzed in that post, the Liberals were at 44%, Conservatives dropped three points to 27%. Rather than crying hooray and setting off fireworks, I analyzed the sample because it was problematic and, you know, what this site is all about.

      Delete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.