Wednesday, September 5, 2012

Quebec election aftermath

Last night's election results in Quebec were surprising, but the event that took place in Montreal after the results were known overshadowed to a great degree what happened in the voting booths. The attack perpetrated by an undoubtedly deranged individual was an assault not only on his victims, but on all Quebecers - and all Canadians - and upon our democratic values. My thoughts are with the families of the two victims. Tragically one has passed away.

This sort of heinous act puts a lot of things in perspective, but it is important not to let the crime of a single man carry more weight than the act that millions of Quebecers took part in yesterday, an act that hundreds of millions of people worldwide are forbidden from doing and for which they have given their lives.

I will have some analyses of last night's results on the websites of The Globe and Mail and The Huffington Post Canada later today and I will update this post with links. UPDATE: The Globe article can be found here. The article for The Huffington Post Canada can be found here.

It will take a little while to pore over how the projection performed, especially as I will have to calculate the regional results of last night's vote in order to see how the model would have done with the correct numbers. That was a big problem last night. No poll was very accurate, especially in terms of the Liberals. The highest number they had in a late-campaign poll was 29%, and that was in Forum's election-eve survey that also had the PQ at 36%. The poll that had the best result for the Liberals also had the worst result for the PQ. In other words, it was not very helpful.

The projection did make a few good bets, forecasting that the polls would under-estimate the Liberals and over-estimate the support of the CAQ (edit: at least in terms of Léger and CROP), Québec Solidaire, Option Nationale, and the Greens. It was wrong to believe that the polls would also under-estimate the Parti Québécois and that was the difference between projecting a bare-minimum majority government and the minority government the party actually won.

A preliminary analysis suggests that the projection made the right call in about 82% of ridings (with the projected ranges, that is bumped up to 86%), a decent performance considering the faulty inputs of the polls. There were a few surprises, and like in the Alberta election the resilience of the Liberal incumbents was extraordinary.

Once I plug the actual regional results of the election into the model, I'm very confident that the projection will provide a close result. But that just proves once again that the challenge is being able to forecast results with the polling data that is available. More and more, that data is looking less and less useful. The two polls conducted on the eve of the vote were both fatally flawed (one suggested a clear PQ majority, the other a Liberal catastrophe). It is unfortunate that Léger and CROP were not in the field later in the campaign, but it is impossible to know if they would have performed any better.

In the end, ThreeHundredEight's vote projection turned out to be closer than any of the polls, with a total error of 8.1 points (compared to 8.4 for Léger and 10.4 for Forum/CROP). That is not quite good enough, and more work will have to be done to determine how to best forecast the difference between voting intentions and voting behaviour. Wider confidence intervals and the recognition that there is only so much that polls can tell us about what will happen tomorrow may be in order.

34 comments:

  1. Polls in Canada are getting worse and worse. The federal election, the AB election and now this.
    It's terrible.
    Compare to the US where polls are generally very accurate.

    ReplyDelete
    Replies
    1. If you read http://fivethirtyeight.blogs.nytimes.com/
      The conclusion is that they generally tend not to be until the end. I'm no expert but I think it has to do with the the fact that we have more swing voters/undecideds than our neighbors to the south.
      No poll can compensate for people changing their mind at the last minute or simply being dishonest.

      Delete
  2. Actually in the CAQ did NOT get over-estimated by the polls. They wound up with 27% of the popular vote which is almost exactly the average of the final polls - in fact if you throw in those last IVR polls by Forum and Ekos, CAQ actually did BETTER than the average of the last polls indicated.

    ReplyDelete
    Replies
    1. I'm not sure if it is worthwhile including EKOS, but you're right if you include the Forum poll.

      Delete
  3. Your low bar for the PQ was only 3 off the mark and CAQ's was one off. Overall, I think your model was pretty accurate.

    The final count of PQ 32%, PLQ 31% and CAQ 28% was within the margin of error most polls.

    For pollsters, maybe this is just a matter of distributing the undecideds *heavily* towards the Liberals?

    Thanks, Éric.

    ReplyDelete
  4. This poll of Léger of the 8th of August correctly predicted the result of last night.

    http://www.threehundredeight.com/2012/08/caq-continues-to-gain-ground.html

    ReplyDelete
    Replies
    1. Yes, I noticed that. But throw enough darts at the board and one will hit the target.

      Delete
  5. I believe that Crop and Léger had fairly accurate final numbers, consistently highlighting a tight race and a large number of undecided. However, during the last few days of the campaign, CAQ stalled, as did the PQ. Only the Liberals were on the upswing with a positive time in Québec City. I think that the undecided, from the outset, were a large number of liberals leaning CAQ, but in the end were uninspired and returned to the Liberal fold.

    ReplyDelete
  6. Leger's poll on August 8:

    PQ 32%, PLQ 31%, CAQ 27%, QS 6%, ON 2%, PVQ 2%

    They should have stopped right there, as it turned out.

    It wasn't that the pollsters were off on the popular vote, it was the seat distrubtion based on those votes.

    ReplyDelete
  7. Regardless of the wiggling by all here the results did NOT mirror the polls any more than the Alberta results did.

    The methodology or the "corrections" or maybe people have got to the point they just tell the pollster what he wants to hear ??

    Regardless the accuracy is diminishing. Eric thanks for you hard work and let's hope the data you get to use in the future is more accurate !!

    ReplyDelete
  8. I wonder if an argument can be made that similar forces at were at work here in Quebec as in Alberta, where large numbers of CAQ-Liberal (in Alberta Tory-Liberal-NDP) undecideds were dragging down both parties'numbers, and breaking heavily for whichever of the parties they felt was strongest in their riding on election day.

    Just a hypothesis, but I wonder how detectable a situation like that would be with the current practices of Canadian pollsters.

    Out of curiosity Eric, how close would you have been with the correct poll numbers?

    ReplyDelete
    Replies
    1. As mentioned, I'll have to run the numbers to get the right regional totals first. That could take a while (need to add up votes!).

      Delete
  9. I'm wondering if the automated calls verses live callers reveal any accuracy differences in results.

    I think the by-election in Kitchener-Waterloo may be a good test given these two polls with very different results and very different methods:

    http://www.forumresearch.com/PollsDownload.asp?c=ar95eryrn5r5&f=MFEHM_in7tun0_Xv6pur0r4-jn6r4y11_OB-Ryrp6v105_c1yy_(S147z_er5rn4pu).2qs

    http://www.oraclepoll.com/uploads/Kitchener_Waterloo_Riding_Report_Aug31.pdf

    (I'm also looking forward to seeing how Eric will update the by-election forecasts once he gets a chance to incorporate these polls)

    ReplyDelete
    Replies
    1. It is updated. http://www.threehundredeight.com/p/by-election-barometer.html

      Delete
  10. Frankly, as someone who is from Quebec originally, was in the media during the first PQ win and first independence referendum, there is one immutable fact that the journalists and pollsters fail to adjust for: the Liberal/Federalist vote is always under-estimated because a lot of older people in particular are hesitant to declare themselves to pollsters. A lot of them fall into the category of undecided or would not respond. In Quebec, PQ supporters/separatists tend to be overbearing on the question of their party or their option. It gets uncomfortable to declare oneself for the federalist status quo. I worked in such an environment in 1980 - young federalists just went without badges and kept their mouths shut while PQ/independence supports - the "Oui" - wore their badges and talked openly about the victory to come.

    This dichotomy was true of English/French people in my office, but also among francophones of differing opinions.

    The University of Montreal's Claire Durand captured and predicted a much higher Liberal vote in his pre-election interviews. While she came out slightly high for the Liberals, he assessment probably holds more water than anything I've seen from pollsters in this campaign.

    It's a factor Eric doesn't reflect in his model, but probably should.

    ReplyDelete
    Replies
    1. I agree. The backlash and peer pressure you would get if you said you were liberal in this pass election was harsh (e.g you are willing to vote for a corrupt government, how dare you support an elitist party, etc...). So you said you were undecided. This is exactly why voting is done anonymously.

      Delete
    2. Which fits in with a theory I've sort of developed over the past couple of years.

      Many people are fed up polling/telemarketing. Viewing both as essentially the same invasion of privacy. Tell the pollster what you think he wants to hear and he goes away. Much like the telemarketer being told NO ! He moves on because he can't waste time on non-buyers !!

      Delete
  11. It's too easy to calculate the real regional swing and then apply it. With that logic, calculate the riding swing in every riding and you get 0 mistakes! I understand the logic but the fact is that you could NEVER have all the correct provincial swings. Polls at best provide us with the regional breakdown for Mtl, the big Mtl, Quebec and the rest of Quebec.. And even there, polls are not accurate enough to provide that.

    At the end of the day, I think you should show the projections if you had the correct, global, provincial intentions. Otherwise it's more like cheating. At least provide both.

    Also, being happy to have made the correct calls while being off with the vote intentions is kinda weird to me. It's like you make a mistake with the vote, then doesn't transpose the % correctly and somehow the two mistakes cancel each other. And you weren't that close with the vote projections. I mean, yes the Liberals were higher, but not the PQ and the CAQ was just a little bit below. Nothing like what you had. And it's normal, polls were clearly wrong again.

    ReplyDelete
    Replies
    1. Eric's ultimately not the one making the call on voting intentions though. That's primarily coming from the pollsters.

      Delete
    2. Actually Ryan it comes from the voters. If they want to mislead the pollsters that's their prerogative!!

      Nope this is more about a failed system/methodology than anything else. Eric does a great job with the info he gets but he can't correct the kind of nonsense we've seen recently.

      Delete
  12. Les résultats s'expliquent par la division du vote souverainiste, aussi simple que cela. S'il n'y avait pas eu cette division (entre le PQ, QS et ON), c'est 20 comptés qui seraient allé aux souverainistes (pour le PQ en presque totalité). Vos prédictions étaient exactes, si l'on ne prend pas en compte la division.

    ReplyDelete
  13. Liberal voters tend to be more publicly discreet with their choice because it is frowned upon nationalists to openly support the Liberals.

    As usual, the Liberal vote was underestimated. Which explains the big chunk of undecideds.

    ReplyDelete
  14. Interesting piece in today's Toronto Star by Chantal Hebert on "whence Charest" ?

    ReplyDelete
  15. The polls seem to consistently over-estimate the support of the most left wing party in every election. I noticed the pattern several elections back and it has been holding firm. You would be wise to take that into account next time.

    ReplyDelete
  16. Except that in the last BC election most polls gave the NDP 37 or 38% of the vote and they got 42% on election day. There is also no evidence that polls overestimated the NDP in the last Manitoba or Nova Scotia elections.

    ReplyDelete
    Replies
    1. I don't think there is any systematic way to predict how the polls will be wrong. An adjustment like the one I used will probably be right most of the time for most of the parties, but not all of the time for all of the parties.

      I'm not sure where to go from here. Base the projection on the polls alone, but base the confidence intervals on how the polls have been wrong in the past? That might be the simplest thing to do.

      Delete
  17. Really interesting map here which shows clearly the place of the PQ and the Liberals in the Quebec countryside

    http://www.cbc.ca/news/canada/quebecvotes2012/

    ReplyDelete
  18. Wow Eric - even with the right %, you get the PQ about right but the PLQ and CAQ downright wrong. Is this due to using the ADQ as a baseline, when the CAQ is a different party with somewhat different supporters?

    ReplyDelete
    Replies
    1. That is part of it - the CAQ's support turned out to be much more uniform than the ADQ's, which meant they finished second in a lot of ridings.

      As to it being 'downright wrong', the ranges would have been 41-65 PQ, 34-55 PLQ, 18-38 CAQ, and 1-2 QS, so the sort of scenario that played out Tuesday night would have been envisioned.

      Delete
    2. In the many elections you've followed now are you seeing any difference between the IVR method verses live callers? (that might be an interesting post in future).

      Delete
  19. "It will take a little while to pour over how the projection performed..."

    It's *pore, not *pour.

    Easy enough to fix that, eh?

    ReplyDelete
  20. Monsieur Éric,

    j'apprécie votre candeur lorsque vous discutez des limites méthodologiques de votre modèle.

    C'est tout à votre honneur,

    Salutations,

    CA Ramsay

    ReplyDelete
  21. I'd like to cc you on a Letter to the Globe re "How history would be different if Canada used the U.S. electoral system" Sept 10

    It's great analysis, the "Canada elects no Chief Executive" is one of my pet projects.
    pls see my perspective on this
    "Reversing the Governor General's Vasectomy"

    http://robertede.blogspot.ca/2012_01_30_archive.html


    robertede@gmail.com
    416-819-7333

    ReplyDelete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.