Sunday, February 13, 2011

Should we be wary of polls?

In response to a very good, but also somewhat dismissive, article by Joan Bryden of the Canadian Press on polls, pollsters, and methods, I decided to take a look at how the pollsters had performed in the last two federal elections.

I've looked at this before - it is how I came to rank the pollsters according to how they have performed in recent federal and provincial elections. I use this ranking to determine one of the three factors I use to weight the polls included in my model. The ranking takes into account the accuracy of the pollsters in calling the Alberta, Quebec, and Canadian elections of 2008, the Nova Scotia and British Columbian elections of 2009, and the 2010 New Brunswick election.

But this article by Joan Bryden touched a bit of a nerve. After all, I write about polls on a daily basis. Are they all such unreliable twaddle?

If they are, it is remarkable how all of the different pollsters are able to generally provide the same inaccurate results using different polling methods. There's a reason we don't see the Tories at 50% in one poll and 10% in another, and it has something to do with accuracy.

The pollsters rarely disagree with one another in a very strong way, and though we have no election to check their accuracy against we do have the poll results of their peers. All of the pollsters have the Conservatives between 34% and 37%, and the Liberals between 25% and 29%. That is pretty consistent, and since they all agree with one another we have a good reason to believe they are all accurate within their margins of error. We don't have one poll putting the NDP ahead in Alberta and the Conservatives ahead in Quebec - though the regional results vary the usually tell the same general story.

The article does make a very good point about margins of error, however. We should all take more care in reporting on results in the context of the MOE. I try to do that as often as I can, but perhaps I should take more care in the future.

But we shouldn't dismiss polls because they use imperfect methods. As the article points out, no method is without its flaws. But that is why a site like ThreeHundredEight is useful. By aggregating the polls and weighting them, we can cancel out some of the problems of each individual poll.

The results of the last two elections show why polls certainly aren't unreliable, and why aggregating the results can be more accurate. We'll start with the 2006 federal election. In the chart, E = Election Day, so E-1 means the poll was completed one day before the election. All of this data is from the Wikipedia pages for the 2006 and 2008 elections.In 2006, the Conservatives, NDP, and Bloc performed slightly better in polls taken within five days of the vote. The Liberals fared worse. Nanos was the closest, with an average margin of error of 0.3 points. EKOS was next best, with an average MOE of 1.44 points, while Strategic Counsel and Ipsos-Reid were not far behind.

The average MOE of these four polls was 1.23 points, but when we average out the polling results we get an MOE of only 1.16, bettered only by Nanos. This is an indication of why aggregating poll results tend to give a more accurate result than most individual polls. Nate Silver on FiveThirtyEight has also written about this.

In 2008, the pollsters were not as close. For whatever reason the Conservative vote was hard to pin down. Either the polls were somewhat inaccurate, or voting intentions shifted in the last few days. The Dion interview, released at the very end of the campaign, may have had an effect.Nevertheless, the pollsters were not very far off. The best result was Angus-Reid's, off only by an average of 0.88 points. EKOS, again, was off by 1.44 points while Ipsos-Reid, Nanos, and Harris-Decima averaged an MOE of about 1.8 points. The worst performers were the Strategic Counsel and Segma, but both had an average MOE of less than the usual +/- 3.1.

The average MOE was 1.75 points, but the MOE on the average poll result was only 1.38 points, better than all pollsters except Angus-Reid.

Note that Angus-Reid uses an online polling methodology, while Nanos (the best result in 2006) uses the traditional telephone survey.

Unfortunately, ThreeHundredEight wasn't active during the 2008 federal election, so I don't know if I would have done better than a simple average. However, I was active during the 2008 Quebec election.Averaging out the results of the last three polls conducted for that provincial election (by CROP, Léger, and Angus-Reid, the only pollsters active in the last week) gives a MOE of 1.9 points, still quite good. My projection, however, was only off by an average of 1.1 points.

UPDATE: A comment in response to this post has gotten me thinking about the value of pre-writ polls. Are we just gauging the horse race for our own score keeping? Political parties run their own internal polls in pre-writ periods to get a handle on what Canadians are thinking. If this is information they find valuable enough to pay for, and are partly basing their own decisions on the results of these polls, shouldn't we have access to the same kind of information? If the NDP decides to support the next budget, will it be because they truly believe it to be a positive budget, or will it be because they fear an election? Does that matter? Well, that's a question for individuals to answer for themselves.

Polling faces a lot of challenges today: how to reach the cell phone users, how to handle online polls, how to deal with low response rates. But that doesn't mean they are useless or unreliable. On the contrary, they are both useful and reliable - within context, of course. And the media does have a responsibility to provide that context. But let's not throw the baby out with the bathwater.

19 comments:

  1. but the main point in the article is that the polls have been getting progressively worse, due to several factors (doff. methods, smaller sample sizes, increased competition, decreased or even nil payment for them, & far less random participation by the public), so how does dialing the way back machine to how they did 2 or 3 or 5ish years ago do ANYTHING to ameliorate those concerns?

    (granted, I didn't bother to actually READ your apologia, so maybe you do address that, but it sorta smacks of missing the point).

    ReplyDelete
  2. Those are the final polls before each election, though. If you look November 2005 or August 2008, though, which is more similar to the current situation, the polls looked nothing like the final results turned out. In that sense, it's absolutely correct to say that polls mean pretty much nothing and that they give pretty much no information as to the eventual result of an election.

    They're probably somewhat accurately conveying public opinion as it stands now, but that's pretty much irrelevant, I think.

    ReplyDelete
  3. Anonymous,

    I don't believe that the situation has changed radically since 2006 to make the results of those two elections irrelevant.

    Joffré,

    But those polls are never described as being a picture of how people WILL vote. They are always how people intend to vote NOW, which is very relevant.

    The Conservatives will enter the next campaign already having the support of 35%, while the Liberals will be at 29% or so. It will take a lot of work for the Liberals to over take the Conservatives, and tells that they have to approach the campaign as the underdogs, rather than the front runners.

    If the polls were showing current intentions as split 33% to 33%, that would change things entirely.

    ReplyDelete
  4. Several issues with respect to polling.

    Pollsters base their surveys on the societal norm and not on who actually votes. These values are quite divergent. We also have no formal measure of which part of the population votes and which does not. We assume we know that older people vote and younger people do not, but we do not actually have any solid data to prove this.

    So how do we weight the polls for age groups, education, and income when we are not certain who is voting or not but have some strong impressions of what is going on? should seniors be weighted up and youth weighted down?

    The pollsters also get opinions from many people that will not be voting in the election. Counting these people as having the same weight as the people that actually vote is a serious methodological problem.

    About 40% of the population is not voting federally, but the pollsters get 85-90% of the people having opinions. This means a lot of people that will not be voting in the election are counted as having an opinion. Pollsters simply are assuming they are the same representation as people that vote. This is a statistically invalid assumption unless you can empirically show that non-voters do in fact voters.

    It should be very easy to get an accurate reflection of voter turn out and furthermore weight the polling based on who voted in 2008 and who did not. Then asking how certain people are that they will vote in the next election.

    If doing this they get more than 60% of the respondents saying they voted in 2008, there is a huge problem with the survey. Furthermore, the intent to vote in the next election should be no more than 55% given that the trend is downwards. If the pollsters are more than 1% off on either of these numbers, one has to assume there is a serious flaw in their assumptions.

    When we look at the stats math, every polling company should be very close to each other and very close to the results on election day. This does not happen. A polling company should only every any of its results outside of the margin of error once every 20 elections for their poll closest to election day.

    Getting a result on your last election poll that is outside of the margin of error should be almost unheard of.

    The stated margin of error on polls is only the statistical measure or probability and does not measure the much larger margin of error that comes for flaws in the methodology.

    As far as I able to estimate, the margin of error from the methodology is about +- 10% of the stated value of a party - this means a value of 37% is in fact somewhere in the range of 33.5% - 40.5%. You then add the statistical margin of error on top of that and you get a very large margin of error.

    When the number of people voting was the same as the number people expressing an opinion in a poll, the methodology margin of error was likely much smaller.

    At this point in Canada, dead reckoning and Ouija boards are about as useful as polling.

    ReplyDelete
  5. Bernard von Schulmann I believe we do somewhat know the demographics of past voters - elections Canada has a pretty comprehensive survey they conduct.


    Frank Graves echoed your concerns.

    He said its possible his polls UNDER-STATE (his word) the true level of CPC support because they have such a big lead with older voters and do so poorly amongst younger voters.


    So a lot really does depend on who turns out.

    We may already have a CPC majority and just not know it.

    ReplyDelete
  6. "So a lot really does depend on who turns out.

    We may already have a CPC majority and just not know it."

    Yes but also following your logic you can have a much reduced minority. Depending on current polling looks to me to be such a crap shoot for any party I wonder why they bother.

    The link I just posted talks about the Liberal in-house pollster using a sample size of 5,000. No commercial pollster comes close to that. But the bigger the sample as we all know the more accurate the result.

    ReplyDelete
  7. Peter, the link you posted is the same as the one in my article, so I didn't post it.

    A sample size of 5,000 people would have a MOE of about 1.4 points. Nationally, the increase in accuracy would not be significant but large sample sizes would be useful at the regional level.

    ReplyDelete
  8. "A sample size of 5,000 people would have a MOE of about 1.4 points. Nationally, the increase in accuracy would not be significant but large sample sizes would be useful at the regional level.
    "

    Vs a national of 3.0 or larger Eric. Sorry on that we will disagree. A 50% improvement is substantial by any measure.

    Link was to CTV's version of the piece, didn't look to see if there was any variance.

    ReplyDelete
  9. It's the same Canadian Press piece. When you see an article that was written for the Canadian Press, it is usually the same everywhere.

    It certainly is an improvement, but of course still subject to the "19 times out of 20". The point is that it costs a lot more for little return, and a poll limited by its survey methods will be limited whether they survey 1000 or 10,000 people.

    ReplyDelete
  10. "It's the same Canadian Press piece. When you see an article that was written for the Canadian Press, it is usually the same everywhere.

    It certainly is an improvement, but of course still subject to the "19 times out of 20". The point is that it costs a lot more for little return, and a poll limited by its survey methods will be limited whether they survey 1000 or 10,000 people. "

    Yeah re CP. Didn't notice on the CTV site that it was a CP piece. Won't be any change.

    " and a poll limited by its survey methods " Exactly, and that in essence is what that CP piece is talking about. I do question the "land-line" telephone stuff as actually bilge. There hasn't been anywhere near the swing away the pollsters claim. Lot's of people have gone with cell phones as secondary but darn few I think have actually dropped their normal phone service.

    I think the pollsters, based on what they say in that CP piece, actually don't want to pay the big charges involved in a country wide telephone survey so are "weaseling" !!

    ReplyDelete
  11. Eric,

    "They are always how people intend to vote NOW, which is very relevant."

    That claim seems at least debateable. How one would vote now is not terribly relevant since there is no federal election occurring right now.

    I wouldn't go so far as to say pre-writ polls are entirely meaningless (if I did, I'm not sure why I would follow this site!)

    But as others have noted, polls taken months before an actual vote have not shown themselves to be particularly good predictors of an election outcome.

    ReplyDelete
  12. I wonder to what extent polls are self-fulfilling prophecies - not that people want ot back the horse with the best chance of winning so much as people who see polls suggesting their party is going to lose make it so by staying home. More importantly I wonder if this effect is increasing with time. I don't know if this is actually true but I get the sense that polls are released more frequently and receive more press than they once would have.

    If this is true, we need to be wary of polls less for their reliability than for their ability not only to measure public opinion but indeed to influence it.

    ReplyDelete
  13. "But those polls are never described as being a picture of how people WILL vote. They are always how people intend to vote NOW, which is very relevant."

    No they aren't. People that respond to polls are not an accurate sample of the voting public in Canada. Intentions always change when elections are called. For instance, here are the Liberal and Tory poll results from polls taken in August 2008 - just before the election was called.

    HD Aug 7 (L: 33 C: 32)
    IR Aug 12 (L: 30 C: 36)
    HD Aug 21 (L: 34 C: 33)
    Nanos Aug 27 (L: 35 C: 33)
    AR Aug 27 (L: 28 C: 36)
    IR Aug 26 (L: 31 C: 33)
    SC Aug 29 (L: 29 C: 37)

    Predictions of the Liberal vote were off the actual result by an average of 5.4 points (only 2/7 were within 3 points of the actual result). They did better with the Tories, only getting it wrong by an average of about 3.7 points.

    This was less than a month before the election was called. What changed between August and September (in fact during roughly the week the election campaign began) that can explain the surge in Tory fortunes to majority territory (before the financial crisis drove them down once again)?
    And by the way, you can see this story time and time again if you go back. Poll numbers change enormously whenever elections are called. Your final prediction will only look good if it largely discards the data from between elections.

    ReplyDelete
  14. "If this is true, we need to be wary of polls less for their reliability than for their ability not only to measure public opinion but indeed to influence it. "

    Very interesting thought. Just watch some on here sway in the winds of polls. You get exactly that idea.

    ReplyDelete
  15. HtH,

    Again, my projections are a picture of the situation NOW, not a year from now.

    When the next election comes, my projection will reflect the most up-to-date picture of the situation. The projection will, of course, be heavily (if not completely) influenced by polls taken during the campaign.

    However, Nate Silver has written in the past about how, with a lack of polling data, using pre-campaign polls is better than using only a few new polls.

    ReplyDelete
  16. Hey Eric,

    I sometimes see American pundits discuss how their horserace polls can be tallied two ways: counting all eligible voters, or the subset of only likely voters. Some say the "likely voter" numbers are better predictors.

    Which of these numbers do Canadian pollsters report?

    ReplyDelete
  17. Brian,

    Generally, polls in Canada only make a distinction between decided and undecided voters during a campaign.

    Polls that are released outside of a campaign usually include decided voters and "leaners".

    I have heard that distinction for likely voters may be a part of the next campaign.

    ReplyDelete
  18. This morning, Feb 15/2011, Reuters came out with CPC, strong lead. But.
    The pollster was Ipsos Reid, in my opinion the least reliable of them all. So, who are we to believe or trust ?
    The compilation at 308 becomes the most reliable.

    Allen Graham, aged political hack

    ReplyDelete
  19. Eric

    Global's poll seems to back up the Ekos one though I do question the apparent swing in Ont given Hudak is going nowhere ?

    ReplyDelete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.