Thursday, January 30, 2014

Mulcair favoured as PM due to likely coding error (updated)

When a poll result is counter-intuitive, and not marginally so, you should probably take a look to make sure you did everything correctly. Last week, The Hill Times reported on a new poll by Forum Research with a headline that read: "Libs gain at expense of Conservatives, but Mulcair favoured as leader who would make best Prime Minister".

That didn't make much sense. The Liberals were ahead of the New Democrats in the poll by 12 points, yet Justin Trudeau trailed Thomas Mulcair on the PM question by 10 points? The reason it didn't make much sense is because it was probably wrong, and a coding or transcription error on the part of Forum Research is to blame.

UPDATE: As appeared to be the case, Forum Research has confirmed there was an error in its two reports that listed the numbers that should have been Trudeau's as Mulcair's and the numbers that should have been Mulcair's as Trudeau's. So, the error crept in when Forum took the data from their polling and transcribed it into their reports, both of which contained analysis as to why Mulcair was polling ahead of Trudeau on this question. 

This means that for these two polls, Trudeau was ahead on who would make the best Prime Minister with 29%, with Mulcair at 19% on Jan. 16-17 and 18% on Jan. 23-24. Forum says it has corrected the issue.

I've obtained the detailed reports of Forum's most recent polls, and the firm has some numbers out for a Jan. 23-24 poll that shows generally the same numbers that they recorded in the Jan. 16-17 poll reported by The Hill Times. Here is the table showing how the numbers have evolved on this question:

Forum report for Jan. 23-24 poll

What happened between Dec. 12-13 and Jan. 17 to change things so dramatically? Voting intentions did not shift to any similar degree between that time. Polling by Nanos Research concerning who would make the best prime minister has consistently tracked very closely to what Forum has been recording - until these last two polls, that is.

Forum releases a lot of data tables in their reports, so it makes it possible to see how the various demographic and regional breakdowns look. For the Best PM question, the numbers don't make much sense. The Liberals leading with 58% support in Atlantic Canada to 23% for the NDP, yet Mulcair gets 47% to 12% for Trudeau on the Best PM question in the region?

But then you get to the table of Best PM by voting preference. You can see why this is a problem pretty quickly:

Forum report for Jan. 16-17 poll

Apparently, Mulcair is the overwhelming choice of Liberal voters and Trudeau is the overwhelming choice of NDP voters. Time for a leader swap!

If you run the numbers using this table exactly as presented, you get pretty much the exact same national results as reported by Forum. But let's assume that this is the table that is wrong, rather than the overall numbers. That would mean that 53% of 609 Liberal voters prefer Trudeau, and 65% of 381 NDP voters prefer Mulcair. Perhaps the columns were just misplaced on this chart. If you do that, however, you come to a national tally of 24% on this question for Trudeau and 23% for Mulcair.

So, it seems that somewhere in Forum's coding or transcription process, the numbers were swapped between Trudeau and Mulcair, or Trudeau and Mulcair's names were put in the wrong spot on the chart (meaning Trudeau would be the one at 29%, and Mulcair at 19%). Either that, or their detailed report is rife with typos. Both the Jan. 16-17 and Jan. 23-24 polls show the same error, while previous polls have everything in the right place.

It isn't entirely The Hill Times's fault for the misinterpretation, the article was written by a great, but very busy, Parliament Hill reporter who can be forgiven for the omission as the analysis by Forum in the report itself talks about how Mulcair is ahead of Trudeau on the Best Prime Minister question. But there is a lesson in here, though. If a number doesn't look right, it probably isn't right, and both pollsters and journalists should be damn sure to double check to make sure everything is as it should be.

56 comments:

  1. I think it is more likely that the rows were switched (not the columns) and 29% prefer Trudeau and 19 prefer Mulcair. That seems to make more sense and follows the pattern from September to Decmber.

    ReplyDelete
    Replies
    1. Yes, that is probably what happened. But their 2013 polls had the names in the same order.

      Delete
    2. That also makes sense for the choice of Bloc respondents; it seems odd that 27% of the Bloc would back Trudeau and 14% Mulcair when vice versa more aligns with party and leader policies.

      Delete
    3. Yes, that is what I thought had happened but in reading my post I see I didn't make that clear (I've updated).

      I was just trying to give Forum the benefit of the doubt that the error was in the layout of the chart, not the data itself (i.e., "Oh no, just in that chart the LPC and NDP numbers were in the wrong spot..."

      Delete
  2. Éric, are the Forum reports you allude to publicly available anywhere at this time?

    Dom

    ReplyDelete
  3. This one does look like a transcription error but perhaps you are too young to remember when Ed Broadbent lead every poll for best Prime Minister, but the regularly came in third place during elections. It can happen.

    ReplyDelete
    Replies
    1. Ed Broadbent was never the 2nd choice in his own party to the Liberal leader as this forum poll suggests.

      Delete
    2. If my memory serves me correctly when Broadbent was deemed best PM the NDP also lead in the polls.

      Delete
  4. Pretty bad on Forum's part. A flip in the numbers like that should at least raise a few eyebrows and lead to further checking. As if the polling industry in Canada wasn't enough of a joke already.

    ReplyDelete
  5. Would the polling company be legal liable if you had them do market research and you made a costly business decision based on their bad polling information?

    ReplyDelete
    Replies
    1. Depends on the contract but, generally speaking one can sue for incompetence or failing to complete a task that is paid for or agreed to.

      In market research of course there exists an area "margin of error" for example where it would be difficult to prove an error occurred or if an error occurred it may be reasonable. However, if you pay a company to do market research and they submit a report that does not include the data you paid for that would be negligence.

      Delete
  6. Echoing Ryan, how is it - particularly when a glaring spotlight has been on polling firms for over a year now - that Forum can realease not just one, but two polls with such a seemingly obvious error?

    ReplyDelete
  7. I may be wrong, but I don`t think it`s a mistake. I think it is very possible that Liberal voters prefer Mulcair`s competence and credibility. I can`t wait to see what Forum says about this.

    ReplyDelete
    Replies
    1. Nah, it simply doesn't make sense that 65% of Liberal supporters prefer Mulcair vs. only 8% who prefer Trudeau, while 53% of NDP supporters prefer Trudeau vs. only 13% who prefer Mulcair.

      Dom

      Delete
    2. Yah, not only is this not true, the real poll data actually shows that Trudeau is more popular with the NDP than Mulcair is with Liberals.

      Delete
    3. Mathieu,

      If that was the case we would see other examples and we would also see the NDP higher in the national polls.

      If Liberals were going to convert en masse to the Dippers they would have done so a long time ago.

      Delete
    4. Mathieu had done one of those double posts, and they were slightly different. Based on the one I deleted, I think he was being sarcastic.

      Delete
    5. Well played then. Well played indeed.

      Delete
    6. No, I wasn`t being sarcastic, but now that I see that it was a mistake that`s fine too. I just thought that it wasn`t necessarily contradictory to support one party but think a different party`s leader would make the best PM. Ex: A Bloc voter may know (rightly) that their leader could never be PM, and so answer the best PM question with another leader. I imagined that Liberal voters may like the party, but not believe it possible for them to win, and so say that Mulcair would be a better PM. Either way, we now know that wasn`t the case :)

      Delete
    7. I don't think you'd ever see a party's supporters have such an overwhelming preference for a different party's leader.

      Technically speaking a Bloc leader could become PM though. If the Greens by some miracle managed 50+ seats, and the Bloc got a near sweep in Quebec, and the other parties were divided fairly evenly, there's also enough seats there to just barely come out a head. Lol.

      Delete
    8. Very unlikely even with a plurality of seats the Crown would appoint a separatist as PM.

      Delete
    9. Ryan,

      I don't think Blocquistes would be able to swear the Privy Council oath.

      Delete
    10. The much more plausible way for a BQ leader to become PM is in a coalition government.

      Delete
    11. It's fun to pretend though, eh?

      I doubt the other parties would support a BQ PM though. The cost of breaking the "cordon sanitaire" is too great (as Dion found out).

      Delete
    12. Agreed. Though it's looking more and more like this sort of speculation about the BQ is like speculating about alternate outcomes of the Punic Wars, etc.

      Delete
    13. I don't think they'll go extinct even if they end up existing in a greatly diminished role. They made need to retool and rethink their approach to Ottawa though. Plaid Cymru seems to survive just fine in Westminster and they have far less favourable conditions than the Bloc.

      Delete
    14. I would also expect the BQ to stay around if only for publicity's sake. Plaid and the SNP are good examples each only wins a handful of seats in Westminster but, like the PQ play much more significant roles in the Scottish Parliament and National Assembly for Wales.

      Delete
  8. Glad to see the confusion has been settled. Now, if only Forum would start diligently posting their results again like they used to. It looks like they conducted one federal poll in December and two more in January, none of which they have shared yet, and on top of that the recent Hill Times article now appears to be locked.

    Dom

    ReplyDelete
    Replies
    1. Other firms conduct polls they don't release - think of the wobbly line in Harris-Decima's charts. They are in the field every week, but we only hear from them every few months.

      Delete
    2. Yeah, but I appreciate Forum's usual diligent reporting, as I imagine you must too since it gives you more data to work with. I just think it would be a shame if they started slacking on it.

      Dom

      Delete
    3. Update: Glad to see they've now come through. The results of their December poll are also shown in the new reports.

      Dom

      Delete
    4. undermedia,

      Just wanted to say, great work on the Wikipedia article re: polling post 2011. Nicely done.

      Delete
    5. Thanks chimurenga. It's not terribly sophisticated, but I think having the graph and table of all the polls since last election is useful and informative. It's hard to contend with fancy-pants Éric here though, what with his exclusive inside access to unpublished polls, elaborate poll averaging algorithms and all. ;)

      Dom

      Delete
  9. Gee Eric you are killing me. I'm desperately waiting for updates on the Forum and Nanos ON polls. Please!

    ReplyDelete
    Replies
    1. There's a CROP poll that's been out for a week or so too I think.

      Delete
    2. These polls have been added to the averages, BTW.

      I am but one man (with a book to write!).

      Delete
  10. Éric,

    After finishing that book, you aren't planning on running for something, are you?

    Wow, what a slogan that would be: I am but one man.
    LOL. Keep up the good work which we greatly appreciate!

    ReplyDelete
  11. This whole thing is going to become a great example in one of my classes about why people should do exploratory data analysis before anything else.

    ReplyDelete
  12. And whichever way the poll should be it's still bad for Harper.

    ReplyDelete
    Replies
    1. Polls are surprisingly good for Harper considering the bad press over the last few months. Conservative support has stabalised in the high 20's which and it appears Liberal support has peaked in the low 30's. In a competitive race Harper with the power of incumbency has the edge.

      As for the reversed poll numbers much better for the Tories if Mulcair is leading Trudeau. M<=ulcair and the NDP are much less of a threat toward Conservative seats in Ontario.

      Delete
    2. And as time goes on Dan that incumbency edge just gets worse and worse. Harper is on a losing slide.

      Delete
    3. Tell that to the BC NDP Peter. Election's not won yet.

      Delete
    4. No Ryan two more years should cement his removal if he doesn't go first himself !

      Delete
    5. Pete,

      Hahaha.

      Only a fool would underestimate Harper.

      Delete
    6. We are a long way from the election and in the latest polls the Conservatives appear to have arrested their slide and are enjoying a small rebound.

      Also the power of incumbency I think is very strong. In BC, Alberta and Ontario the incumbents (McGuinty, Redford and Clark) won re-election even though they had been in office a long time and were not particularly popular amongst voters.

      Delete
  13. Here is another inconsistency in a poll:

    http://www.nationalnewswatch.com/2014/02/02/poll-third-of-canadians-say-pms-trip-to-israel-a-success-most-have-no-opinion/#.Uu7FyvlSYlR

    In this poll it says that 50% of Canadians paid attention (or knew about) Harper's trip to Israel.

    That is believable.

    Out of these "Fifty-nine per cent of those polled believe Harper’s visit will have a positive impact on Canada’s ties with Israel."

    What were the other 31% Thinking??? What was their educationally background? Where they stoned or just drunk?

    It is arguable whether a closer best friend relationship with Israel is good for Canada but saying that this trip is not good for Canada Israel relationship is like saying that recording "Satisfaction" was not good for the Rolling Stones career.

    Polling in Canada. You have to evaluate the overall information of any poll that gets on such an obvious question wrong.

    They need to add a trick question on each poll to weight it... suggestions:

    Is President Obama an African-American?

    ReplyDelete
    Replies
    1. Maybe the other 31% thinks it won't have any effect. Israel and Likud are different entities.

      Delete
    2. BCVoR,
      It depends on what you include in the concept of "Canada’s ties with Israel". It may seem a neutral term, but it's actually value-laden. If one is referring to business and political ties between the two countries, then by that narrow definition a positive impact is likely. But considering the concept more broadly, then it's likely to be negative. The issues of Israel's treatment of Palestinians and its aggressive policies in the region have an undermining effect on relations between Canada and Israel. However much Harper favours those policies, most Canadians do not. But more significantly, such policies have the demonstrated effect of increasing insecurity in the Middle East, leading to escalations of violence, with repercussions that are likely to come back to haunt us in the future, as long as the Canadian government supports Israel in this way.

      Delete
    3. Is President Obama an African-American?

      Yes and no. He is African by race and American by citizenship but, he is not an African-American ethnically at least so far as these things can be attributed or in so far as African-American is a separate ethnographic group. Therefore the question: Is President Obama an African-American can only be answered by Mr. Obama himself. If he self-identifies with African-American culture he would probably answer yes. If he does not identify with that culture he would probably answer no.

      Delete
    4. Ha Ha Chimurenga..

      So how do you answer the question do you love your Spouse and Family? Do you evaluate the impact of this choice on the world ? Is the world better off because of your choice.


      You can argue, as I have previously stated, as the the value of relationship but there can be no argument that the current visit strengthen the mutual support and relationship between the countries.

      Bede

      It is pretty clear that Mr. Obama clearly identifies as African-American. It has been widely broadcast that he is the first African-American president. He has had ample opportunity to reject this classification and has chosen not to.

      Anyone answering that Obama is not African-American.... whatever their rationalization or thinking... will not provide value to a survey representative of the universe being studied.

      Maybe The question should be list the last 3 PMs of Canada. If you can't do that you are not a representative sample of the universe being sampled.

      Delete
    5. "Maybe The question should be list the last 3 PMs of Canada. If you can't do that you are not a representative sample of the universe being sampled."

      A lot of people couldn't answer that question though, and they still vote.

      You can poll people on whether or not gravity exists, but the answers don't change if that's true or not. We do polling because we want to know people's opinions and views. If you're looking for something else from polls you're not going to get it.

      Delete
    6. @Ryan

      The idea of polling is trying to get the sample that reflect the universe in question.

      The universe in political polls is all Canadian's of voting age that will vote in the next election.

      Chances of people who do not know the last 3 PMs in Canada are extremely unlikely to actually get out and vote. They have no interest in the political process.

      There are going to be some confused people who would not understand the question on Canada-Israel relationship being stronger by Harper's support and visit to Israel. But people who do not understand basic concepts are not going to actually make the effort to vote.

      From your posts I clearly understand that you will not be voting for Harper or any Conservative in the next few elections... But I would be shocked if you did not see Harper making a Stronger relationship with Israel.

      You might not agree with it and think it is bad for Canada and/or the middle east but surely you would see Harper strengthening the relationship. Wouldn't you?

      Delete
    7. @Ryan Here is another conntrary strawman

      "I never voted before and likely won't vote ever but next time I will vote for Harper because he sends me my a GST refund check.

      A few people like thus in the sample after they get normalized to represent the under 25 Canadian demographics would produce poll showing Harper sweeping into another majority based on youth support.

      You can see that these non-informed non-voters have to be filtered out of the survey.

      Delete
    8. No they don't. You can screen by likely hood to vote, but not solely on how informed they are or their motives or intelligence. It's not a scientific survey otherwise.

      What if we ask "how old is the Earth" in a survey in the United States? A huge number of voters would get that question wrong, but they still vote in droves.

      Delete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. Please keep discussion on topic.