Saturday, October 10, 2009

Polls vs. Votes

A comment from a reader got me thinking about the difference between what people say in polls and what they do at the ballot box. So I decided to take a look.

For the 2006 and 2008 elections, I took the last poll from each of the active polling firms. In 2008 there were six who polled in the last few days (EKOS, Angus-Reid, Nanos, Harris-Decima, Strategic Counsel, and Ipsos-Reid). In 2006 there were four (Strategic Counsel, SES Research (now Nanos), Ipsos-Reid, and EKOS).

This chart shows the differences between the average polling result of these polling firms, and the actual vote totals each party received.It's difficult to draw conclusions from this with only two sets of data. The 2004 election doesn't seem to have been as frequently polled. Perhaps only after the next election will we be able to get a real picture of how parties do in polls vs. elections.

Let's take a look at the numbers. For the last days of 2008, the Conservatives averaged 34.5% in the polls. The Liberals were at 27.0%, the NDP at 19.3%, the Bloc at 9.7%, and the Greens at 8.8%. The actual election results were 37.7%, 26.3%, 18.2%, 10.0%, and 6.8%, respectively.

That means the pollsters under-estimated the Conservative vote total by 3.2 points and the Bloc's by 0.2 points. They over-estimated the vote totals of the Liberals by 0.7 points, the NDP by 1.1 points, and the Greens by 2 points.

But what does that really mean? Turnout was low, and the Liberals took the biggest hit in actual votes. If turnout had been what it had been in 2006, would the Conservatives have gotten the predicted 34.5%? It's impossible to say.

Looking at 2006, the Conservatives averaged 37.1% going into election day. The Liberals were at 27.8%, the NDP at 18.7%, the Bloc at 11.3%, and the Greens at 5.1%. Their actual vote totals were 36.3%, 30.2%, 17.5%, 10.5%, and 4.5%, respectively.

So this time the Conservatives were over-estimated by 1.2 points, as were the NDP (1.3 points), the Bloc (0.8 points), and the Greens (0.6 points). The Liberals were under-estimated this time, by 2.4 points.

As you can see, it is really impossible to say whether one party does better or worse in polls. Both the Liberals and the Conservatives saw relatively significant over- and under-estimations of their vote haul. Only the NDP, the Greens, and (to a lesser extent considering national totals for the Bloc are difficult to quantify) the Bloc were over-estimated in both election campaigns. The NDP's over-estimation is relatively high, at over an entire point in both elections. But the Greens were only 0.6 points higher in 2006, which is actually a pretty close estimate.

This really does show how the MOE is an effective way of safe-guarding polling results. We can reasonably assume that every pollster is within three-points at the national level. But with things being as close as they are, that is a huge margin. And for parties like the NDP, it can mean a historic best or a disastrous campaign.

The commenter in question was wondering if I should take this sort of "ballot box" effect into consideration when making my projections. Looking at these numbers, I don't think I can.

Have a great Thanksgiving weekend everyone! I'll post if a poll comes out during the weekend, but otherwise I'll see you Tuesday.


  1. Hey Eric,

    just to clarify the discussion hinged around "pre-writ" polling versus actual election results. It seems like the Liberal party tends to drop when you compare the last polls taken before an election is called and the ballot box results. There's a couple of things that could be at work here:

    A) The campaign. This is obvious but could the Liberals really be running a lousy campaign each cycle? Wouldn't they adapt and do better or hire better people.

    B) Liberals are the "default" option. During a campaign second, third, and fourth parties are displayed and some people move towards them.

    C) Candidate selection and strategic voting in places like BC where elections become two way races between the NDP and the Cons. Liberal support splits to the other two parties.

    And who knows what else.

  2. Why this matters is because we're in a pre-writ period with pre-writ polling. So if the Liberals are at 28% in the polls they're probably going to end up with a Dion result of 26%.

    Obviously this rule of thumb doesn't work with wave or historic elections or even expertly run campaigns.

    But 40 years worth of polling seemed to show that where the Liberals are now in the polls they will probably be lower on election day.

  3. Yes, I remember the conversation. I looked at this aspect of it, though, since my projection will be affected by the actual election when the time comes.

  4. Hey interesting topic. I've spent a lot of time looking at GPC shortfalls. Reasons, and shortfall in actual vote can vary with polling company, as i assume that different pollsters use different methodology to 'correct' data. As usual, corrections always backwards looking, so suffer from same problem as economists. Always measuring the past, not enough data to meningfully project forward.
    For the Green Party vote, which is most extreme example, I hypothesise two reasons. First, pollsters call representative sample at their homes, (or nowadays cell phones as well). Lot's easier to get voting intention than it is to get out to vote. Greens have no GOTV apparatus, so woefully underperform other Party's on e-day. That will change suddenly one election soon, as Greens getting better organised daily, reducing 1/2 of the error.
    Other reason is same reason. GPC supporters least likely to vote, whether GOTV or no. Easy to answer phone and state voting intention, not so easy to get off butt and go to vote. That won't change much, until GPC support extends beyond kids, and the disinterested. You shouldn't forget that the polls you (and I) are so interested in are intended to approximate the real world. They have a small impact on behaviour, but aside from that are backwards looking measures, not really leading indicators.

  5. How many people lie to the pollster.

  6. There's no reason to think that a lot of people lie to the pollsters. The amount who purposefully lie is probably very low. But people change their minds. I've vote for three different parties over the last three elections. Over the last year, I've had two different answers for vote intentions in the Angus-Reid polls I answer (anyone can sign up).

    A lot of people don't know who they'll vote for, and most of the pollsters ask for who they're leaning towards.

  7. Eric,

    really? I thought I saw somewhere on the site that you're BQ. Or are you just strategic voting or disliking of your local candidate?

  8. I did not live in Quebec for all of those elections.


COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.