Thursday, January 30, 2014

Mulcair favoured as PM due to likely coding error (updated)

When a poll result is counter-intuitive, and not marginally so, you should probably take a look to make sure you did everything correctly. Last week, The Hill Times reported on a new poll by Forum Research with a headline that read: "Libs gain at expense of Conservatives, but Mulcair favoured as leader who would make best Prime Minister".

That didn't make much sense. The Liberals were ahead of the New Democrats in the poll by 12 points, yet Justin Trudeau trailed Thomas Mulcair on the PM question by 10 points? The reason it didn't make much sense is because it was probably wrong, and a coding or transcription error on the part of Forum Research is to blame.

UPDATE: As appeared to be the case, Forum Research has confirmed there was an error in its two reports that listed the numbers that should have been Trudeau's as Mulcair's and the numbers that should have been Mulcair's as Trudeau's. So, the error crept in when Forum took the data from their polling and transcribed it into their reports, both of which contained analysis as to why Mulcair was polling ahead of Trudeau on this question. 

This means that for these two polls, Trudeau was ahead on who would make the best Prime Minister with 29%, with Mulcair at 19% on Jan. 16-17 and 18% on Jan. 23-24. Forum says it has corrected the issue.

I've obtained the detailed reports of Forum's most recent polls, and the firm has some numbers out for a Jan. 23-24 poll that shows generally the same numbers that they recorded in the Jan. 16-17 poll reported by The Hill Times. Here is the table showing how the numbers have evolved on this question:

Forum report for Jan. 23-24 poll

What happened between Dec. 12-13 and Jan. 17 to change things so dramatically? Voting intentions did not shift to any similar degree between that time. Polling by Nanos Research concerning who would make the best prime minister has consistently tracked very closely to what Forum has been recording - until these last two polls, that is.

Forum releases a lot of data tables in their reports, so it makes it possible to see how the various demographic and regional breakdowns look. For the Best PM question, the numbers don't make much sense. The Liberals leading with 58% support in Atlantic Canada to 23% for the NDP, yet Mulcair gets 47% to 12% for Trudeau on the Best PM question in the region?

But then you get to the table of Best PM by voting preference. You can see why this is a problem pretty quickly:

Forum report for Jan. 16-17 poll

Apparently, Mulcair is the overwhelming choice of Liberal voters and Trudeau is the overwhelming choice of NDP voters. Time for a leader swap!

If you run the numbers using this table exactly as presented, you get pretty much the exact same national results as reported by Forum. But let's assume that this is the table that is wrong, rather than the overall numbers. That would mean that 53% of 609 Liberal voters prefer Trudeau, and 65% of 381 NDP voters prefer Mulcair. Perhaps the columns were just misplaced on this chart. If you do that, however, you come to a national tally of 24% on this question for Trudeau and 23% for Mulcair.

So, it seems that somewhere in Forum's coding or transcription process, the numbers were swapped between Trudeau and Mulcair, or Trudeau and Mulcair's names were put in the wrong spot on the chart (meaning Trudeau would be the one at 29%, and Mulcair at 19%). Either that, or their detailed report is rife with typos. Both the Jan. 16-17 and Jan. 23-24 polls show the same error, while previous polls have everything in the right place.

It isn't entirely The Hill Times's fault for the misinterpretation, the article was written by a great, but very busy, Parliament Hill reporter who can be forgiven for the omission as the analysis by Forum in the report itself talks about how Mulcair is ahead of Trudeau on the Best Prime Minister question. But there is a lesson in here, though. If a number doesn't look right, it probably isn't right, and both pollsters and journalists should be damn sure to double check to make sure everything is as it should be.