That didn't make much sense. The Liberals were ahead of the New Democrats in the poll by 12 points, yet Justin Trudeau trailed Thomas Mulcair on the PM question by 10 points? The reason it didn't make much sense is because it was probably wrong, and a coding or transcription error on the part of Forum Research is to blame.
UPDATE: As appeared to be the case, Forum Research has confirmed there was an error in its two reports that listed the numbers that should have been Trudeau's as Mulcair's and the numbers that should have been Mulcair's as Trudeau's. So, the error crept in when Forum took the data from their polling and transcribed it into their reports, both of which contained analysis as to why Mulcair was polling ahead of Trudeau on this question.
This means that for these two polls, Trudeau was ahead on who would make the best Prime Minister with 29%, with Mulcair at 19% on Jan. 16-17 and 18% on Jan. 23-24. Forum says it has corrected the issue.
I've obtained the detailed reports of Forum's most recent polls, and the firm has some numbers out for a Jan. 23-24 poll that shows generally the same numbers that they recorded in the Jan. 16-17 poll reported by The Hill Times. Here is the table showing how the numbers have evolved on this question:
 |
| Forum report for Jan. 23-24 poll |
What happened between Dec. 12-13 and Jan. 17 to change things so dramatically? Voting intentions did not shift to any similar degree between that time. Polling by Nanos Research concerning who would make the best prime minister has consistently tracked very closely to what Forum has been recording - until these last two polls, that is.
Forum releases a lot of data tables in their reports, so it makes it possible to see how the various demographic and regional breakdowns look. For the Best PM question, the numbers don't make much sense. The Liberals leading with 58% support in Atlantic Canada to 23% for the NDP, yet Mulcair gets 47% to 12% for Trudeau on the Best PM question in the region?
But then you get to the table of Best PM by voting preference. You can see why this is a problem pretty quickly:
 |
| Forum report for Jan. 16-17 poll |
Apparently, Mulcair is the overwhelming choice of Liberal voters and Trudeau is the overwhelming choice of NDP voters. Time for a leader swap!
If you run the numbers using this table exactly as presented, you get pretty much the exact same national results as reported by Forum. But let's assume that this is the table that is wrong, rather than the overall numbers. That would mean that 53% of 609 Liberal voters prefer Trudeau, and 65% of 381 NDP voters prefer Mulcair. Perhaps the columns were just misplaced on this chart. If you do that, however, you come to a national tally of 24% on this question for Trudeau and 23% for Mulcair.
So, it seems that somewhere in Forum's coding or transcription process, the numbers were swapped between Trudeau and Mulcair, or Trudeau and Mulcair's names were put in the wrong spot on the chart (meaning Trudeau would be the one at 29%, and Mulcair at 19%). Either that, or their detailed report is rife with typos. Both the Jan. 16-17 and Jan. 23-24 polls show the same error, while previous polls have everything in the right place.
It isn't entirely The Hill Times's fault for the misinterpretation, the article was written by a great, but very busy, Parliament Hill reporter who can be forgiven for the omission as the analysis by Forum in the report itself talks about how Mulcair is ahead of Trudeau on the Best Prime Minister question. But there is a lesson in here, though. If a number doesn't look right, it probably isn't right, and both pollsters and journalists should be damn sure to double check to make sure everything is as it should be.
I think it is more likely that the rows were switched (not the columns) and 29% prefer Trudeau and 19 prefer Mulcair. That seems to make more sense and follows the pattern from September to Decmber.
ReplyDeleteYes, that is probably what happened. But their 2013 polls had the names in the same order.
DeleteThat also makes sense for the choice of Bloc respondents; it seems odd that 27% of the Bloc would back Trudeau and 14% Mulcair when vice versa more aligns with party and leader policies.
DeleteYes, that is what I thought had happened but in reading my post I see I didn't make that clear (I've updated).
DeleteI was just trying to give Forum the benefit of the doubt that the error was in the layout of the chart, not the data itself (i.e., "Oh no, just in that chart the LPC and NDP numbers were in the wrong spot..."
Éric, are the Forum reports you allude to publicly available anywhere at this time?
ReplyDeleteDom
I don't believe so, no.
DeleteThis one does look like a transcription error but perhaps you are too young to remember when Ed Broadbent lead every poll for best Prime Minister, but the regularly came in third place during elections. It can happen.
ReplyDeleteMost definitely.
DeleteEd Broadbent was never the 2nd choice in his own party to the Liberal leader as this forum poll suggests.
DeleteIf my memory serves me correctly when Broadbent was deemed best PM the NDP also lead in the polls.
DeletePretty bad on Forum's part. A flip in the numbers like that should at least raise a few eyebrows and lead to further checking. As if the polling industry in Canada wasn't enough of a joke already.
ReplyDeleteWould the polling company be legal liable if you had them do market research and you made a costly business decision based on their bad polling information?
ReplyDeleteDepends on the contract but, generally speaking one can sue for incompetence or failing to complete a task that is paid for or agreed to.
DeleteIn market research of course there exists an area "margin of error" for example where it would be difficult to prove an error occurred or if an error occurred it may be reasonable. However, if you pay a company to do market research and they submit a report that does not include the data you paid for that would be negligence.
Echoing Ryan, how is it - particularly when a glaring spotlight has been on polling firms for over a year now - that Forum can realease not just one, but two polls with such a seemingly obvious error?
ReplyDeleteI may be wrong, but I don`t think it`s a mistake. I think it is very possible that Liberal voters prefer Mulcair`s competence and credibility. I can`t wait to see what Forum says about this.
ReplyDeleteNah, it simply doesn't make sense that 65% of Liberal supporters prefer Mulcair vs. only 8% who prefer Trudeau, while 53% of NDP supporters prefer Trudeau vs. only 13% who prefer Mulcair.
DeleteDom
Yah, not only is this not true, the real poll data actually shows that Trudeau is more popular with the NDP than Mulcair is with Liberals.
DeleteMathieu,
DeleteIf that was the case we would see other examples and we would also see the NDP higher in the national polls.
If Liberals were going to convert en masse to the Dippers they would have done so a long time ago.
Mathieu had done one of those double posts, and they were slightly different. Based on the one I deleted, I think he was being sarcastic.
DeleteWell played then. Well played indeed.
DeleteNo, I wasn`t being sarcastic, but now that I see that it was a mistake that`s fine too. I just thought that it wasn`t necessarily contradictory to support one party but think a different party`s leader would make the best PM. Ex: A Bloc voter may know (rightly) that their leader could never be PM, and so answer the best PM question with another leader. I imagined that Liberal voters may like the party, but not believe it possible for them to win, and so say that Mulcair would be a better PM. Either way, we now know that wasn`t the case :)
DeleteI don't think you'd ever see a party's supporters have such an overwhelming preference for a different party's leader.
DeleteTechnically speaking a Bloc leader could become PM though. If the Greens by some miracle managed 50+ seats, and the Bloc got a near sweep in Quebec, and the other parties were divided fairly evenly, there's also enough seats there to just barely come out a head. Lol.
Very unlikely even with a plurality of seats the Crown would appoint a separatist as PM.
DeleteRyan,
DeleteI don't think Blocquistes would be able to swear the Privy Council oath.
The much more plausible way for a BQ leader to become PM is in a coalition government.
DeleteIt's fun to pretend though, eh?
DeleteI doubt the other parties would support a BQ PM though. The cost of breaking the "cordon sanitaire" is too great (as Dion found out).
Agreed. Though it's looking more and more like this sort of speculation about the BQ is like speculating about alternate outcomes of the Punic Wars, etc.
DeleteI don't think they'll go extinct even if they end up existing in a greatly diminished role. They made need to retool and rethink their approach to Ottawa though. Plaid Cymru seems to survive just fine in Westminster and they have far less favourable conditions than the Bloc.
DeleteI would also expect the BQ to stay around if only for publicity's sake. Plaid and the SNP are good examples each only wins a handful of seats in Westminster but, like the PQ play much more significant roles in the Scottish Parliament and National Assembly for Wales.
DeleteGlad to see the confusion has been settled. Now, if only Forum would start diligently posting their results again like they used to. It looks like they conducted one federal poll in December and two more in January, none of which they have shared yet, and on top of that the recent Hill Times article now appears to be locked.
ReplyDeleteDom
Other firms conduct polls they don't release - think of the wobbly line in Harris-Decima's charts. They are in the field every week, but we only hear from them every few months.
DeleteYeah, but I appreciate Forum's usual diligent reporting, as I imagine you must too since it gives you more data to work with. I just think it would be a shame if they started slacking on it.
DeleteDom
Update: Glad to see they've now come through. The results of their December poll are also shown in the new reports.
DeleteDom
undermedia,
DeleteJust wanted to say, great work on the Wikipedia article re: polling post 2011. Nicely done.
Thanks chimurenga. It's not terribly sophisticated, but I think having the graph and table of all the polls since last election is useful and informative. It's hard to contend with fancy-pants Éric here though, what with his exclusive inside access to unpublished polls, elaborate poll averaging algorithms and all. ;)
DeleteDom
Gee Eric you are killing me. I'm desperately waiting for updates on the Forum and Nanos ON polls. Please!
ReplyDeleteThere's a CROP poll that's been out for a week or so too I think.
DeleteThese polls have been added to the averages, BTW.
DeleteI am but one man (with a book to write!).
Éric,
ReplyDeleteAfter finishing that book, you aren't planning on running for something, are you?
Wow, what a slogan that would be: I am but one man.
LOL. Keep up the good work which we greatly appreciate!
This whole thing is going to become a great example in one of my classes about why people should do exploratory data analysis before anything else.
ReplyDeleteAnd whichever way the poll should be it's still bad for Harper.
ReplyDeletePolls are surprisingly good for Harper considering the bad press over the last few months. Conservative support has stabalised in the high 20's which and it appears Liberal support has peaked in the low 30's. In a competitive race Harper with the power of incumbency has the edge.
DeleteAs for the reversed poll numbers much better for the Tories if Mulcair is leading Trudeau. M<=ulcair and the NDP are much less of a threat toward Conservative seats in Ontario.
And as time goes on Dan that incumbency edge just gets worse and worse. Harper is on a losing slide.
DeleteTell that to the BC NDP Peter. Election's not won yet.
DeleteNo Ryan two more years should cement his removal if he doesn't go first himself !
DeletePete,
DeleteHahaha.
Only a fool would underestimate Harper.
We are a long way from the election and in the latest polls the Conservatives appear to have arrested their slide and are enjoying a small rebound.
DeleteAlso the power of incumbency I think is very strong. In BC, Alberta and Ontario the incumbents (McGuinty, Redford and Clark) won re-election even though they had been in office a long time and were not particularly popular amongst voters.
Here is another inconsistency in a poll:
ReplyDeletehttp://www.nationalnewswatch.com/2014/02/02/poll-third-of-canadians-say-pms-trip-to-israel-a-success-most-have-no-opinion/#.Uu7FyvlSYlR
In this poll it says that 50% of Canadians paid attention (or knew about) Harper's trip to Israel.
That is believable.
Out of these "Fifty-nine per cent of those polled believe Harper’s visit will have a positive impact on Canada’s ties with Israel."
What were the other 31% Thinking??? What was their educationally background? Where they stoned or just drunk?
It is arguable whether a closer best friend relationship with Israel is good for Canada but saying that this trip is not good for Canada Israel relationship is like saying that recording "Satisfaction" was not good for the Rolling Stones career.
Polling in Canada. You have to evaluate the overall information of any poll that gets on such an obvious question wrong.
They need to add a trick question on each poll to weight it... suggestions:
Is President Obama an African-American?
Maybe the other 31% thinks it won't have any effect. Israel and Likud are different entities.
DeleteBCVoR,
DeleteIt depends on what you include in the concept of "Canada’s ties with Israel". It may seem a neutral term, but it's actually value-laden. If one is referring to business and political ties between the two countries, then by that narrow definition a positive impact is likely. But considering the concept more broadly, then it's likely to be negative. The issues of Israel's treatment of Palestinians and its aggressive policies in the region have an undermining effect on relations between Canada and Israel. However much Harper favours those policies, most Canadians do not. But more significantly, such policies have the demonstrated effect of increasing insecurity in the Middle East, leading to escalations of violence, with repercussions that are likely to come back to haunt us in the future, as long as the Canadian government supports Israel in this way.
Is President Obama an African-American?
DeleteYes and no. He is African by race and American by citizenship but, he is not an African-American ethnically at least so far as these things can be attributed or in so far as African-American is a separate ethnographic group. Therefore the question: Is President Obama an African-American can only be answered by Mr. Obama himself. If he self-identifies with African-American culture he would probably answer yes. If he does not identify with that culture he would probably answer no.
Ha Ha Chimurenga..
DeleteSo how do you answer the question do you love your Spouse and Family? Do you evaluate the impact of this choice on the world ? Is the world better off because of your choice.
You can argue, as I have previously stated, as the the value of relationship but there can be no argument that the current visit strengthen the mutual support and relationship between the countries.
Bede
It is pretty clear that Mr. Obama clearly identifies as African-American. It has been widely broadcast that he is the first African-American president. He has had ample opportunity to reject this classification and has chosen not to.
Anyone answering that Obama is not African-American.... whatever their rationalization or thinking... will not provide value to a survey representative of the universe being studied.
Maybe The question should be list the last 3 PMs of Canada. If you can't do that you are not a representative sample of the universe being sampled.
"Maybe The question should be list the last 3 PMs of Canada. If you can't do that you are not a representative sample of the universe being sampled."
DeleteA lot of people couldn't answer that question though, and they still vote.
You can poll people on whether or not gravity exists, but the answers don't change if that's true or not. We do polling because we want to know people's opinions and views. If you're looking for something else from polls you're not going to get it.
@Ryan
DeleteThe idea of polling is trying to get the sample that reflect the universe in question.
The universe in political polls is all Canadian's of voting age that will vote in the next election.
Chances of people who do not know the last 3 PMs in Canada are extremely unlikely to actually get out and vote. They have no interest in the political process.
There are going to be some confused people who would not understand the question on Canada-Israel relationship being stronger by Harper's support and visit to Israel. But people who do not understand basic concepts are not going to actually make the effort to vote.
From your posts I clearly understand that you will not be voting for Harper or any Conservative in the next few elections... But I would be shocked if you did not see Harper making a Stronger relationship with Israel.
You might not agree with it and think it is bad for Canada and/or the middle east but surely you would see Harper strengthening the relationship. Wouldn't you?
@Ryan Here is another conntrary strawman
Delete"I never voted before and likely won't vote ever but next time I will vote for Harper because he sends me my a GST refund check.
A few people like thus in the sample after they get normalized to represent the under 25 Canadian demographics would produce poll showing Harper sweeping into another majority based on youth support.
You can see that these non-informed non-voters have to be filtered out of the survey.
No they don't. You can screen by likely hood to vote, but not solely on how informed they are or their motives or intelligence. It's not a scientific survey otherwise.
DeleteWhat if we ask "how old is the Earth" in a survey in the United States? A huge number of voters would get that question wrong, but they still vote in droves.