Thursday, April 14, 2011

Pick a poll, any poll

You might as well! The polls that were added to the projection this morning were, in some ways, contradictory. But in other ways they were not, and when we look at the results of the four of them with a closer eye as to why they are different and why they are the same, the polls don't seem as unusual as they might appear at first glance.

No matter, though. Many people will believe the poll they want to believe. And more power to them! There's no way to prove that one poll or another is wrong, especially with more than two weeks to go into the campaign. But I will always default to assuming every pollster takes a professional approach to their work, just as I do with mine. In other words, completely unsubstantiated claims that pollsters cook their numbers will never get any airing here.

So now that that's out of the way, let's get to it. The four polls today are from COMPAS conducted for the Toronto Sun, Innovative Research Group for Maclean's Magazine (and a myriad of radio stations), EKOS Research for iPolitics, and Nanos Research for, of course, CTV and The Globe and Mail. In accordance with the guidelines of Elections Canada (those well-meaning keeners), I should point out that all of these polls sampled eligible Canadian voters, and that Innovative Research uses an online panel.
First off, let's get to the 'controversial' COMPAS poll for the Toronto Sun. As I said yesterday, the biggest problem with this poll is that we have no previous poll from COMPAS for comparison. If we did, and there was a huge swing or no swing it all, we would be better able to judge the validity of their results. This is especially important as the poll is at odds with what others have found. But as far as I can tell, there is nothing in the methodology that easily explains this variation.

Or is there? Although it isn't exactly explained clearly (and my emails to the firm yesterday bounced), it appears that COMPAS's reported results are those of "decided" voters only, excluding "leaners". This explains the huge 35% undecided rate. Virtually every pollster reports on the combined decided and leaning results, and I'm not quite sure why COMPAS would do any different. There doesn't seem to be any compelling evidence that shows that casting aside the leaners brings us to a closer result - a decided voter is not, necessarily, a Canadian who is sure to vote.

Polling firms in the United States have done more work on this "likely voters" issue. We don't really have much to go on up here, and our voting systems are very different (i.e., registered Republican, Democratic, and Independent voters).

COMPAS reports that among decided voters, the Conservatives lead the Liberals by a margin of almost 2:1. This would seem to correspond with the 45% to 24% they report as Canadians' voting intentions. They then go on to say that among leaners the lead is only about 4:3. Well, that's pretty important, isn't it?

So let's try something. Assuming that 65% are decided and 20% are leaners (leaving about 15% undecided, which is what Nanos is reporting), and that the 4:3 lead is the sort of 40% to 30% lead Nanos has found, we narrow down the gap from 21 points to 19, or about 44% to 25%. With rounding (and the fact that I am making a few assumptions), we could even say that the gap is 43% to 26%, or 17 points.

That doesn't look so strange anymore. It is generally where Ipsos Reid has placed the two parties, and is on the outside edges of the margins of error of other recent polls. And when you consider that COMPAS's decided numbers are almost identifical to Ipsos Reid's "absolutely certain to vote" numbers, this poll doesn't look so strange at all.

But the method of reporting the results, and the omission of what questions were asked, is certainly unusual. I doubt that this method of reporting results would meet the MRIA's standards, though more than a few pollsters are truant on that score.

Note that, in the graph above, the regional margins of error have been calculated based on my best estimates of this poll's sample size, as regional breakdowns were not included.

COMPAS's results in Ontario are especially stark, with the Conservatives leading by 49% to only 32% for the Liberals. That would give the Tories 69 seats, with only 27 going to the Liberals and 10 to the New Democrats. But in the Nanos poll, which shows the Liberals leading by 4.4 points, the seat breakdown would be 46 for the Conservatives, 44 for the Liberals, and 16 for the NDP. These two results would create radically different parliaments.

Innovative Research's poll doesn't have anything particularly remarkable in it, and in fact it closely matches my own projection. It does put Michael Ignatieff ahead of Jack Layton on who would make the best Prime Minister, which is new. We last heard from Innovative in a report on Ontario's voting intentions only, taken between April 1 and 3. Since then, the Tories are up three points in the province, which is statistically insignificant.

EKOS's poll, however, is more interesting. It was taken over two days, and those two days will be included in EKOS's weekly report tomorrow for iPolitics. When that poll is released, it will replace this two-day poll in the projection model.

This poll shows a much smaller gap than even Nanos shows: five points. Compared to EKOS's last two-day sneak peek taken between April 4 and 5, the Conservatives have dropped 2.2 points and the Liberals have gained one point. That isn't significant, but the three point gain of the NDP certainly is interesting. Also, note that EKOS has the Liberals in front in the Toronto CMA.

Finally, Nanos. They show a narrowing gap between the Conservatives and Liberals that now stands at 7.8 points, less than it did in 2008. Since their last complete three-day poll, conducted between April 8 and 10, the Conservatives have dropped 2.3 points while the Liberals have gained 0.7. Like EKOS, this isn't significant movement but again the NDP is up, with a 3.1 point gain in three days.

Much of that has come in British Columbia (+9.4) and Ontario (+4.8). In the latter province, the Liberals have gained 5.3 points while the Tories have lost 6.1 points since that last three day poll. Movement definitely seems to be taking place in the battleground province.

In terms of trends, in Atlantic Canada the Conservatives have gained for three consecutive days, as have the Liberals in Ontario and the New Democrats in British Columbia. The Conservatives, however, have lost support for three consecutive days in those two important provinces.

The New Democrats are between 16% and 21% in Quebec in all four of these polls, providing further confirmation of their newfound strength in the province. However, they are also below 20% in Atlantic Canada across the board. That isn't great news for the NDP, but their four MPs in the region are relatively ensconced.

The questions asked by each of the pollsters are as follows: "For those parties you would consider voting federally, could you please rank your top two current local preferences?" (Nanos). "If a federal election were held today, which party would you vote for?" (Innovative). "If a federal election were held tomorrow, which party would you vote for?" (EKOS). The question asked by COMPAS is not available.


  1. Hello,

    Thanks for compiling all of this information. As a reminder, would you be able to state who was "closest" to the final 2008 result?

    Also, are you aware of any changes to the polling fequency between now and May 2nd


  2. Angus-Reid was the closest pollster in 2008. I'll have an in-depth report on how the pollsters fare in this election after it is finished.

    I've been told be a few firms that their polling frequency will be increasing as we reach the end of the campaign. Let's hope that will be the case!

  3. You forgot to mention that EKOS asks questions on direction of the country, leadership and issues BEFORE the vote intention question.

    That's a big difference.

  4. Those EKOS numbers in BC are a joke. Does anyone seriously believe the NDP are ahead of the Tories? Having said that the COMPAS poll seems a little way off too. Ill stick with whatever Eric's poll says as it seems far more balanced.

  5. I wouldn't give to the "who was the closest in a specific election" factor much weight - this is all statistics, after all. If I remember correctly, in 2006 Nanos was the closest (suspiciously so - much less than their MOE). The better approach would be to find a pollster who is _on_average_ (averaged over a few elections) the closest to the true answer. But as elections happen rarely, and pollsters continuously change their methodology, this is probably not a good solution either. I think what Eric is doing (poll modelling, with all biases removed, and tested against past elections) is far better than trying to find the best pollster.

  6. If the NDP support continues to rise in Ontario/Quebec, and the CPC/LPC stay where they are, we could see a couple surprise ridings go blue. Stay tuned.

  7. So Stephan Harper's leadership index score just went up by 27.9 points on the Nanos three day rolling poll. I don't even know how that is possible, because it means his score was 90 points higher than the day that was dropped. it might even be a typo it's so high.

    I'm pretty sure that means he won the English language debate.

  8. Maybe the leadership numbers aren't a rolling poll. Their sample is only 400.

    A jump of 28.7 points over three days isn't possible.

    Do you know Eric?

  9. @Anon

    The leadership numbers are not rolling. From the releases:

    "The Leadership Index score will be tracked daily with updated results from the previous night of polling. Unlike the other measures tracked by Nanos, the leadership index score will not be based on a three-day rolling sample. It will be reported from results from the evening before. This will allow Nanos to measure the performance of the federal party leaders and the key factors driving performance on a daily basis."

  10. **Note: M# denotes the day the poll was completed – for example M29 is the 3-day rolling average for the poll completed on March 29

    That's the fine print at the bottom of that poll. So Stephan Harper's total number did increase by 83.7 over the day dropped from the rolling average, which was Sunday.

    The number is out of a total of 300, because three measures are added and respondents are asked to choose one of the leaders for that quality; trust competence or vision.

    So his average percentage increased by 27.9% over the day dropped. That's a huge jump.

  11. It isn't explained well as there are contradictory things on the PDF, but BT appears to be correct. The methodological explanation on the second page of the PDF explains the leadership index is a nightly tracker, not a rolling average.

  12. Shouldn't we take isssue with the EKOS poll since a disproportionate amount of the people polled had a University education at approx. 43%?

    This doesn't appear to be random and is in no way representative of the actually population.

  13. The results would have been weighed correctly to have the sample reflect Canada's demographics.

  14. Where's Angus Reid been at all this time?

    The leadership numbers look good for Harper... but am I wrong in thinking that there's been no corresponding jump in support in the same polling days? I mean, the gap actually narrowed as per above.

  15. Does anyone know how many people Nanos polls in Atlantic Canada each day?

    I ask because I have noticed some really wide swings and have read that he polls 33 people each time, which might explain the variation

  16. "The results would have been weighed correctly to have the sample reflect Canada's demographics. "

    Samples are weighted for age and gender. I'm not sure they are weighted for other socio-economic or racial factors.

    If it says 43% of the sample had University educations that would skew the results. I'm not sure where it says that and wouldn't mind the link.

    Kevin, no the there wasn't a corresponding jump in vote intention for the Conservatives, but it has to do with the day dropped, as well as vote intention tends to lag behind the leadership number. The leadership number may also, given it is so high for Harper and considerably lower for Ignatieff drive voter turnout numbers on election day.

  17. Do we know who made the decision to quote only the decided numbers in the COMPAS poll - COMPAS or the Toronto Sun?

    It certainly makes sense that the decided vote would skew the Conservatives higher, since there are lots of anti-Conservative voters who aren't sure where to park their anti-Conservative vote. Interestingly, a poll like this might well induce more strategic voting amongst those voters.

  18. Wow! the EKOs poll has the tories @31.9 and the libs@29% it seems the tories are falling like a stone, but voters are parking with the greens.

  19. In the 2008 election the week before the election was called there were 4 polls

    The simple non weighted average for the week before the election was:

    cpc 39 lib 27 ndp 18 bloc 8 and green 8.

    2008 Week 1 - 16 polls,
    week 2 - 16 polls
    week 3 – 21 polls
    week 4 - 21 polls and
    week 5 - 28 polls

    In 2008 the parties averages in weekly polls were:

    CPC 38, 38, 37, 36, 34 election 37.8
    Lib 27, 28, 26, 25, 25, 27 election 26.1
    NDP 16, 17, 19, 19, 19 election 18.2
    Bloc 8, 8, 9, 9, 10 election 10.0
    Green 10, 10, 10, 10, 10 election 6.8

    In the 2011 election the week before the election was called there were 5 polls

    The simple non weighted average for the week before the election was:

    cpc 38 lib 26 ndp 17 bloc 10 and green 8.

    Really close to the start of the last election

    2011 Week 1 – 5 polls,
    week 2 – 10 polls,
    week 3 – 10 polls

    so far the parties averaged the following week:

    CPC 38, 38, 39
    Lib 26, 27, 28
    NDP 19, 18, 17
    Bloc 9, 8, 8
    Green 7, 7, 7

    Note the week 3 simple averages are almost exactly what Eric has as his weighted averages for the 3 main parties. He has the Bloc higher by 1 and the Green lower by 1.

  20. sorry about the 31.9 vs 29% folks,
    I didnt look at the poll year and the headline was Guergised! I assumed it was April 15 (Friday) because she was on TV talking about how she was kicked out of caucus etc. BUT It was april 15 2010!!


COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.