Wednesday, October 7, 2009

What's in a Question?

With only seven national pollsters who are infrequently active, it is difficult to compare them and their methodologies. It doesn't mean we can't try, though.

There are two general ways to ask a survey question concerning vote intentions. The first is to ask something along the lines of:

"If an election were held today, would you vote for the Conservative candidate in your area, the Liberal candidate in your area, the NDP candidate in your area, the Green candidate in your area, or the Bloc Quebecois candidate in your area?"

The pollsters always rotate the order of the parties so that none is given an advantage. This question gives people options from which to choose. It is more likely to get a response from undecided or inattentive voters.

The second question is something along the lines of:

"If an election were held today, which party would you vote for?"

This question does not name the parties and so forces the respondent to give an answer without knowing which parties are his or her options. It is more likely to not get a response from undecided or inattentive voters but is, in my estimation, a more accurate way to judge current public opinion.

Judging from their detailed reports, Strategic Counsel, Angus-Reid, Léger Marketing, and Ipsos-Reid ask the first type of question. EKOS asks the second, while Nanos - in their latest poll at least - asks respondents to give their first and second preference, but does not name the options. While it isn't exactly the same question as asked by EKOS, for the sake of argument I'm going to lump them together.

Harris-Decima does not put the question in their detailed reports. I've sent them an email and hopefully will get a response.

Taking the average result from each of the polling firms' latest poll, we get the following totals for each of the parties when they are listed in the question:

Conservatives - 38.3%
Liberals - 28.8%
New Democrats - 15.0%
Bloc Quebecois - 9.3%
Greens - 7.8%

When respondents are not given a list of the parties, the average result is:

Conservatives - 36.8%
Liberals - 31.6%
New Democrats - 14.4%
Bloc Quebecois - 9.8%
Greens - 7.6%

Looking at it this way, we see that the NDP, Bloc, and Greens don't really benefit either way. But the Liberals certainly tend to do better when respondents aren't prompted with the list of parties.

Another factor that can change the results of polling is whether "leaners" are included. Several polling firms, when given the "I don't know" or "I haven't decided" response to the first question, ask a second. That second question asks what party the respondent is leaning towards voting for.

From what I can tell, Strategic Counsel, Léger Marketing, and Ipsos-Reid include leaners in their results. Angus-Reid, EKOS, and Nanos only use decided voters.

This is the average result for the "leaners included" pool:

Conservatives - 38.7%
Liberals - 29.3%
New Democrats - 14.3%
Bloc Quebecois - 8.7%
Greens - 8.3%

And the decided results:

Conservatives - 36.8%
Liberals - 30.0%
New Democrats - 15.2%
Bloc Quebecois - 10.2%
Greens - 7.0%

This indicates that Liberal, NDP, and Bloc voters are more committed than Conservative or Green voters.

Of course, all of this should be taken with a grain of salt. There isn't a large enough sample size to really determine how much influence the question has on the response.


  1. Mark McLaughlin07 October, 2009 12:12

    I agree with you that a question without options is a better way to get an informed opinion rather than a "What party's name did I hear most recently in the news" response.

    The reality though is that the option question better recreates the ballot box experience. When putting that 'X' down we are given the options.

    It's hard to say what polling method is most accurate. With a poor participation rate it's entirely possible that the option method encourages the uninformed, and less likely to vote, to give an answer. There's also uninformed voters that go anyway simply out of civic duty.

    I guess the pollsters just hope that it all comes out in the wash. It's tough to control for all those factors.

    What is fairly reliable though is that by nature of demographic support and corrosponding participation rates, the Cons usually outperform polls.

    Even the Liberal pollsters know this, so when they see 38% and trust it, they are really seeing a likely 40-41%.

    That's what's getting them so nervous right now.

  2. I'm not sure what makes you think that Conservatives tend to outperform polls. They UNDER performed the polls in the 2004 and 2006 elections and they got about exactly what the polls predicted in 2008. There is also no evidence of any Tory "over-performance" in polls at the provincial level. They did just as badly as predicted in Nova Scotia, Ontario and also the BC Liberals (who are really Tories) were supposed to beat the NDP by double digits according to the polls - that evaporated into a low single digit lead when the votes were counted.

  3. DL,

    Let's put aside the provincial comparisons because they're different leaders and different campaign teams (BC Liberals brought in a Carbon Tax, they are NOT Tories).

    I think what Mark means about over-performing the polls is that Conservative supports GROWS over the course of an election campaign.

    So if you compare all the pre-writ polling to the final election result they tend to gain a few points.

  4. To follow up on Jesse's point, Barry Kay (I think) has put together a list contrasting the last pre-election poll with actual election outcomes from 1962 to 2000. Generally the Tories (or their combined PC and Reform predecessors) have done better in the election than pre-election polls would indicate. Moreover, with only 2 exceptions in that period (1974 and 1993) the Liberals have done worse than pre-election polling would suggest.

    Anyhow, here's the link:

  5. I think that another exception would have to be 1979 where the Tories started out with a big lead and ended up in a minority and 45 behind the Libs in the national popular vote.

    Kay has done a good study, but i think that what he is not controlling for is the fact that in the vast majority of cases, incumbents parties of any stripe tend to lose ground during election campaigns and as we all know during the entire period of 1962-2000 the Liberals were the incumbent party in almost every election.

  6. DL,

    Well then let's look at campaigns where the Tories were the incumbent to test your statement that:

    "incumbents parties of any stripe tend to lose ground during election campaigns"

    1962 - -1% for the Tories, -8% for the Liberals, +5% NDP

    1963 - +1% for the Tories, -5% for the Liberals, +2% NDP

    1980 - +5% for the Tories, -3% for the Liberals, -3% NDP

    1988 - NC Tories, -1% for the Liberals, -2% NDP

    1993 - -20% Tories, +8% Reform, +8% Liberals, -1% NDP, +4% BQ

    2008 - NC Tories, -2% Liberals, -1% NDP, NC BQ

    I don't really see that tendency here.

    There would have to be more serious analysis but my feeling is that:

    A) Campaigns lower Liberal support, campaigns increase Conservative support.


    B) Pre-election polling over-estimates Liberal support, underestimates Conservative support

    OR some mix of A and B.

  7. There is some truth to the notion that Liberal support between elections in canada tends to higher 9and Conservative and NDP support is often lower). I think this is because the Liberals are such a "national institution" and have so much more "brand identity" than the other parties that a lot of people will naturally say they will vote Liberal when they don't have to think about it too hard. Its like someone saying "what's your favourite food?" and the first thing that pops into your mind before you have a chance to think is "chicken".

  8. But the point "Mark Mclaughlin" was making was that Cons tend to outperform POLLS, not that Cons tend to gain ground over the course of an election campaign - and i see no evidence of that.

  9. The conclusion I draw when I see the numbers:

    Conservative - 36.8%
    Liberal - 30.0%

    is that it is the Conservatives who have the most commited voters, not the Liberals, NDP, or Bloc.

  10. DL,

    I think its important to distinguish between pre-writ polls and polls taken during an election.

    How I took his point is that when we see the polls taken now, if an election were to be called, you could probably tack on a point or two to get an idea of how well the Conservatives would do.

    Whether this has something to do with the polls themselves, with the quality of the campaign, or as you said with Liberals tending to be the default setting I don't know.

    When the actual election is called the polls taken toward the end of the election are probably more accurate of how things will turn out. So on the last week of an election nobody should tack on a point or two.

    It's just about strategizing for election timing and whether the parties should go or not.

  11. Hey AJR79,

    not more committed supporters, just more supporters in general.

    The word "committed" is a bit vague because it sort of confuses the concept of having soft support (leaners, independents, swing votes) and the intensity and mood of your supporters.

    Another poll already showed that the Conservatives have the largest number of die hard intense partisans who will show up no matter what, probably give money and volunteer. That sort of enthusiasm is important.

    So don't take it to mean that Conservative supporters are somehow less enthusiastic about their party then the average Liberal supporter.

    It just means Harper has won over a lot of converts recently and he needs to be very careful not to do anything that could screw that up because they're the type of people who could switch over to other parties.

  12. DL,

    I think the point Mark was making was that they outperform pre-writ polls (i.e., the polls we're seeing right now), which is of some relevance

    I wouldn't expect any party to consistently outperform (or underperform) relative to polls at the end of an election campaign since, if pollsters actually know what they're doing, on average their estimates should mirror election results (meaning to the extent they're not spot-on, they're not consistently over or under). For what it's worth, that appears to be the conclusion to draw from the last three elections, where the last polls have, respectively over-, accurately, and under- stated actual Tory results (and vice versa for the Grits)

  13. I disagree. I don't think that comparing pre-writ polls to actual results tells us anything at all. There is a little detail called a federal election campaign that comes between them where each party spends about $20 million dollars on ads there are debates etc... every campaign has a character of its own. In 2003, the Tories in Ontario under Eves started out even with the Ontario Liberals, then they ran a poor campaign and they wound up losing in a landslide.

    I think that it is fair to speculate about whether final polls tend to systematically under- or over-estimate support for any particular party. For example it has been a pretty consistent pattern until very recently in Quebec that the Liberals at either the federal or provincial level would get a "ballot box bonus" compared to the final polls due to the "shy federalist phenomenon". There has also been some speculation that since turnout is higher among older wealthier people - parties that tend to do better among those demographic groups will tend to over-perform compared to the final polls - but then that doesn't explain why the federal Liberals did so much better than the final polls predicted in 2004 and 2006 or why the BC NDP did so much better than expected in the BC provincial election this year.

    With regard to firmness of vote. Its not at all surprising that the Tory vote is the most "solid". face it, if you are a small "c" conservative, there is only one game in town and that is the Conservative party - unless you are one of the few remaining "Blue Grits". But if you think of yourself as left of centre and you don't like Harper - well maybe you'll vote Liberal, but then again maybe you'll vote NDP or maybe once in a blue moon you might even vote Green and if you live in Quebec you might vote BQ as well - so you have a smorgasbord of choices while for rightwingers there is only one choice.

  14. DL,

    the Liberals have traditionally had the most money during a campaign. And as i've said before, looking at provincial politics just confuses the issue.

    Now we've controlled for the variable of incumbency.

    And it seems pretty far fetched to suggest that Liberals always run bad campaigns and Conservatives always run really good ones.

    So the variable of the campaign cannot be responsible alone.

    And we're left with a relationship between the pre-writ polling and the actual election result that seems significant.

    So why would we simply ignore it ?? It clearly IS telling us something.

    As a general rule of thumb I think its a good idea to take what the polls are saying before a campaign and assume the Liberals will do poorer on election day and the Conservatives will do better.

  15. Mark McLaughlin08 October, 2009 09:41

    Some clarification. Some posters hypothosized correctly. My point was that the Cons generally outperform pre-writ polls. I should have made that clear.

    Whether it is due to better campaigns or due to DL's Liberal brand recognition is unclear.

    The Libs haven't run a really good campaign in the last 3 elections. I don't think many are of the opinion that Iggy has the team to change that trend. Too many knives aimed at his back.


COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.