Tuesday, April 8, 2014

Quebec election post-mortem - victory for the polls

Last night's provincial election results in Quebec were most certainly positive for pollsters. They called the race correctly, judging the Liberals to be in a majority position, the Parti Québécois to be dropping significantly, and the Coalition Avenir Québec to be making gains. The final polls were off by no more than a handful of points in any direction, while the aggregate performance was within two points for each of the major parties.

Because the polls did well, the vote projection model did well. In fact, ThreeHundredEight.com's vote projection had a smaller cumulative error than any one pollster. The total error of the vote projection was 3.5 points across the four major parties. Next lowest was Forum Research, which had a total error of 5.6 points. The performance of the polls by Angus Reid and EKOS Research were also quite strong, with a cumulative error of 6.5 and 6.6 points, respectively. Léger's final poll had a total error of 8.6 points, while Ipsos Reid's poll (out of the field six days before the vote) had a total error of 13.6 points among likely voters.

The projected vote results for the Liberals, CAQ, Québec Solidaire, Option Nationale, Greens, and other parties/independents all fell within the likely ranges. Only the PQ's result of 25.4% fell outside even the 95% confidence interval, indicating just how poorly the party did.

The seat projection model performed well for the Liberals, giving them 69 seats (or between 60 and 78) when the party actually won 70. For the PQ, CAQ, and QS, however, the 95% confidence interval ranges were required to capture the eventual results. The PQ was projected to take as few as 29 seats, and they won 30. The CAQ was projected to win as many as 23 seats, and they won 22. These errors occurred because of the CAQ's performance around Montreal at the expense of the PQ. Of the 10 riding-level errors where the ranges did not identify the correct winner, nine of them were seats expected to go to the PQ that actually went to the CAQ. And seven of them were in the greater Montreal area.

This wasn't the fault of the polls, exactly. The regional numbers, as an aggregate, were actually quite good. Rather, it was that the CAQ won these Montreal-area seats by the skin of their teeth. A projection model can falter if a party manages to win a lot of close races.

Overall, the right call was made in 82.4% of ridings, or 103 of 125. In those 22 errors, the confidence in the call averaged just 57.6% - so most of them were genuine toss-ups. Nine of them, actually, were called with 50% or 52% confidence, and more than half of them with 59% confidence or less. The worst call was in Masson, which was projected to go PQ with a 73% chance. That was the only case of the confidence being over 69% and being wrong.

The likely winners through the projection ranges were identified in 92% of ridings, or 115 of 125. Considering the scope for error at the local level, I think that is a very positive result.

The chart above compiles the regional results of last night's election, compared to the projection. As you can see, the aggregate of the polls was quite close across the board.

The Liberals only greatly out-performed the polls on the island of Montreal, where it won them just one more seat than expected. They took 58.5% of the vote here, compared to the 52.1% that the polls suggested. The PQ took 17% instead of 19.7%, while QS took 12% (tying their 2012 performance) instead of 12.8%.

The vote projection in the suburbs around Montreal was very, very close. The Liberals took 35.5% of the vote here, compared to the projected 34.3%. The PQ took 28.8%, compared to the expected 30.5%. And the CAQ took 26.5%, rather than 26.3%.

Quebec City was also quite close, with all parties projected to within 1.1 points. The Liberals won the region with 37.9% of the vote to 35.3% for the CAQ.

And in the rest of the province, the Liberals, PQ, and QS were projected to within 1.2 points, and the CAQ to within 2.1 points. The Liberals won here as well, with 36.9% to 29.7% for the PQ.

More specifically in the 'regions', the PQ won in eastern Quebec with 39.1% of the vote to 37.3% for the Liberals and just 14.5% for the CAQ. In central Quebec, the Liberals had 38.2% support to 30.4% for the CAQ and 23.6% for the PQ. And in western Quebec, the Liberals had 35.4% support to 30.5% for the PQ and 25.5% for the CAQ.

The regional breakdowns show that only the PQ was over-estimated across the board: by 2.7 points on the island of Montreal, by 1.7 points in the suburbs, by 1.1 points in Quebec City, and by 1.2 points in the regions. This explains their over-estimation in the seat projection. The Liberals, meanwhile, were over-estimated in Quebec City and the regions but under-estimated around Montreal. The CAQ was over-estimated in Montreal, but under-estimated in Quebec City and the regions. Even so, we're only talking about a handful of points - a spectacular performance for the polls.

As an aggregate, of course. Individually, there were mixed performances. Forum did very well in the Quebec City and Rest of Quebec regions, averaging 1.4 points of error per party or less. It averaged 2.9 points per party of error in the Montreal area, but that is still quite good considering the margin of error for smaller sample sizes.

Léger was within 1.6 points per party in Quebec City and the regions as well, but was 3.8 points off per party around Montreal.

There was no breakdown of likely voters by region in the Ipsos poll, so it is hard to gauge. But the poll of eligibile voters did well in the regions of Quebec (2.9 points per party), but less well in the rest of the province (between 5.4 and 6.8 points per party). EKOS and Angus Reid both repeated the errors in the Montreal region that other pollsters did. GOTV operations may have been a factor here.

With all the tight races, how would the model have done if the polls for the four regions defined by the model had been even more accurate?

Quite well: 89.6% of ridings (112 of 125) would have been called accurately and in 95.2% of ridings (119 of 125) the winner would have been correctly identified in the likely ranges.

The Liberals would have still been called to within one seat, and the third seat for QS would have been within the likely ranges. But again the PQ and CAQ results would have fallen just outside the likely ranges by one seat for the PQ and two seats for the CAQ.

This should not be shocking - there was a 20.5% chance (or about one in five) of the CAQ having its result fall above the high projection band. But there was only a 7.5% chance that the PQ's result would fall below the low projection. That an incumbent government under-performed to such a great extent says more about how motivated the PQ's supporters were than it does about the projection model. It also shows how well the CAQ did - under normal circumstances, they should not have been able to win all those close races.

The chart to the left shows the projected results at the riding level with the actual regional results plugged into the model. You'll see that in many cases the numbers are quite close.

The six ridings in which the winner would not have been identified by the likely ranges were Borduas, Chambly, Iberville, Masson, Mirabel, and Saint-Hyacinthe. It is no coincidence that these were all CAQ victories in which the PQ was favoured. It is also no coincidence that they were all actual pick-ups by the CAQ in the same region of the province. There was a serious and disproportionate CAQ gain in these regions that went over-and-above what should normally be expected to happen.

This marks the second consecutive election in which the polls do a good job of tracking a race, making all the hard work put into this site worthwhile. The seat projection model performed decently with the polls, and very well with the actual results. In my view, the difference between the results and the model tells us something about what happened last night, rather than about the errors of the model itself. If humans behaved in a way that was uniform and easy to predict, models like mine would be right in 99% of cases. It is where it is wrong that we see real politics at play.


  1. Charlevoix resulting in Marois defeat in her own riding was the surprise of the night that resulted in PLQ going up to 70 from pollster's predicted 69

  2. I'd suggest (as I did before the election) that there was a considerable amount of anti-PQ strategic voting going on. I don't think that's something that you can predict from polls (maybe there'd be a hint on the second preference numbers). I think the best you can do is widen your min-max ranges to make sure that cases like the PQ here are captured by your model. I'd suggest that for the popular vote too, since the PQ fell just outside your minimum popular vote value too.

    Otherwise it seems like a pretty good result for you. Thanks for the great coverage of the race and all the hard work!

    1. The ranges will be updated with the results of this election, so a lower floor for the incumbent party will be the result.

    2. Great. This is really the first example of the governing party under performing in your projections isn't it? It wouldn't surprise me if over the long term further elections slowly push that min number downward for the governing party too.

      I can't help but shake my head at CAQ's results though lol. First they under perform on seats in 2012, now they over perform.

    3. Ryan I would recommend looking at the Ekos latest poll for evidence of strategic voting for the Liberals. This poll asked a number of insightful questions, including a) likelihood of vote changing b) party voted for in previous election c) party of choice federally d) most important issue to voter.

      The Liberal vote came mostly from previous Liberal voters, was the least likely to change, voted Liberal federally, and was disproportionately concerned with the economy (shorthand for referendum). The new Liberal vote came from the CAQ, the QS, but mostly from voters who did not vote in the last election. These new votes shifted early in the campaign (with Peladeau's sovereignty speech) and held until election day.

      If you ask the right questions, you could get every indication that this was a strategic hardcore federalist vote that the PQ reinforced with every speech. Even Couillard's so-called gaffe on language (supporting bilingualism) actually reassured this core vote.

  3. I don't know about the rest of you but to me there are two significant things here.

    First of course is the swing to the Liberals.

    But possibly of even more importance is the shift from the PQ to the CAQ. A major shift ?

    1. I would love it if the CAQ supplanted the PQ as the primary opposition in the province. I've always thought the soft separatism of the ADQ and CAQ made a lot more sense than the PQ's approach, and the CAQ generally has much more sensible economic policies.

    2. I don't think the swing to the Liberals is really significant, it just means people who don't want a referendum went to the safe option. It was an anti-PQ vote, not a for-PLQ vote. Had the campaign gone on one or two more weeks, I strongly believe the PLQ vote would have fallen too, to the profit of the CAQ. I believe the CAQ swing is much more significant and will dramatically change the Québec political landscape for the next few years.

      On the other hand, the PQ's tumble is quite important; they have a tough 4 yeas ahead of them. Where will they position themselves? The right doesn't seem to favour them, will they return to centre-left or will they keep moving into right territory with the PLQ and CAQ? Will they push for a sovereignty option or will they push that back until further notice? Who becomes their new leader?

    3. This comment has been removed by the author.

    4. The PQ lost 15 points among Francophones in 4 weeks two more of campaigning and the PQ would have found themselves with support in the teens. The PLQ may have suffered in a longer campaign but, I think the momentum was against the PQ. A longer campaign would have placed the PQ in third place.

      The PQ needs to win back at least some CAQ and QS supporters if they hope to return to the government benches. I think the pressure for the PQ to turn right will be strong. Fiscal and economic issues are becoming more important especially as interest rates rise and debt servicing begins to eat into an already tight budget. The PQ also needs to regain its credibility on the economy. Most ballot box issues are pocket book issues.

      How the PQ goes through the leadership process will be interesting. I hope they take a more democratic approach than the BCNDP.

  4. Yes Eric an excellent job, thanks.

  5. Hi Eric,

    I'm looking for raw data showing the number of votes per candidate, per riding. Is that publically available anywhere ? I could re-type it in from your site, or from any news outlet, but it there was an official source to download it from, it would be way easier.

    P.S.: Good job on the prediction !

    1. The Elections Quebec site has full results.

  6. I think there is fairly strong evidence that the ethnic vote turned out this time:

    Fatima Houda-Pepin lost to Gaetan Barrette in La Piniere
    PQ Diane De Courcy (Language Minister) lost in Cremazie (Muslim Pop 6150)
    PQ Leo Bureau-Blouin lost in Laval-des-Rapides (Muslim Pop 5130)

    Also, the riding of Ungava in northern Quebec has been a PQ stronghold. It has a population of over 6000 First Nations people, mainly in the Autonomous Nunavik region. The district only has 26,000 electors. It went Liberal for the first time since the founding of the PQ in 1970.

  7. I'm hoping someone will release exit polls. It will be interesting to see some of the various ways in which the votes fractured. We don't even know what the linguistic breakdown is, though I'd imagine we can postulate based on the overall accuracy of the polls.

  8. Anyone who wants to get into a more political as distinct from numbers discussion this link will help.


  9. The projections for my simulator, using the poll average from the site, gave me:

    67 PLQ
    39 PQ
    16 CAQ
    3 QS

    However, I had cheated and removed the trend from Marois's riding, allowing her to win in Charlevoix-Côte-de-Beaupré. If I hadn't done that and let my projector do its job, I would have predicted her losing to the PLQ, giving:

    68 PLQ
    38 PQ
    16 CAQ
    3 QS

    Those results are pretty close to the actual results, only 9 seats would need to switch to a different party. I had underestimated the CAQ, but that was mostly due to the PQ being overestimated by 1,5%, and in that zone, it was the difference maker in a few ridings, especially against the CAQ. In total, I would have been wrong in 16 ridings and right in 109, for an efficiency of 87,2%.

    Using the results of the election, I get:

    73 PLQ
    31 PQ
    18 CAQ
    3 QS

    On the seat total, only 4 seats would need to switch over to the CAQ to have the actual results (3 from the PLQ and 1 from the PQ). I would have correctly predicted 111 ridings and been wrong in 14, for an efficiency of 88,8%. I find this results pretty good for my simulator's first run, especially since it's something I did in my spare time for fun. However, I find that the 88,8% with the actual results is too close to the 87,2% of the polls.

    By making a slight modification to how my riding algorithms are calculated, using the poll average, I get:

    70 PLQ
    38 PQ
    14 CAQ
    3 QS

    The PLQ and QS's seat numbers are correctly projected, but the PQ and the CAQ are still quite off with 8 seats needing to switch between them. I would have correctly predicted the outcome of 111 seats out of 125, for an efficiency of 88,8%, the same as my previous version's actual results efficiency.

    Using the actual vote results however, I get:

    72 PLQ
    34 PQ
    16 CAQ
    3 QS

    Only QS is correctly projected, but overall, only 6 seats need to switch to the CAQ (2 from PLQ and 4 from the PQ), which would be 2 more than my previous version. Riding per riding on the other hand, it becomes more accurate, with only 10 mistakes out of 125, so 115 correct projections, for an efficiency of 92%. I will therefore keep this version.

  10. Eric,

    Clearly you were caught out (to the extent you underestimated the CAQ seat count) by the large number of toss-up PQ/CAQ seats. You calculate the probability of victory in each seat for the party in the lead. Would it be practical to assign (for example) 0.57 seats to the PQ if its probability of victory in a given seat were 57 per cent, and report an expected seat count using the sum of the fractional seat counts? (For reporting purposes, of course, you might have to round up or down to the nearest integer.)

    1. That is an interesting idea. But the probabilities are calculated only for the leading parties. In other words, when a seat is projected with 57% confidence, it means there is a 43% chance another party will win it - not necessarily one party in particular.

      If I used the extreme ranges, and when a seat was not seen to be at play in those ranges put the confidence at 100%, and distribute the remainder to the parties seen at being in play in other ridings, I'd get the following:

      66.4 PLQ
      44.5 PQ
      11.5 CAQ
      2.4 QS
      0.2 IND

      So, actually worse (error of 29.4 seats instead of 28).

      If I applied the percentages to only the first two parties in any riding, then I would have gotten:

      PLQ 63.5
      PQ 42.1
      CAQ 16.5
      QS 2.8
      IND 0.2

      The error would have been 24.5 seats, which is better. Arguably, though, it is better to have projected a strong PLQ majority and over-estimate the PQ than to suggest it could have been a minority and be a little better for the CAQ.

  11. Eric,

    Nice little tribute to Flaherty.



COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.

Note: Only a member of this blog may post a comment.