Tuesday, November 1, 2011

Ontario: Projection vs. Results

The 2011 Ontario election was a tricky one. Throughout the campaign, the polls showed a neck-and-neck race between the Liberals and the Progressive Conservatives. Some of the polls showed a widening gap between the two parties in the final days, but the end result wound up being just as close as the campaign had promised it would be.

The Liberals under Dalton McGuinty won their third consecutive election, but were reduced to a minority government by a margin of one seat. Tim Hudak's Progressive Conservatives formed the Official Opposition while Andrea Horwath's New Democrats remained in third place. Both opposition parties, however, improved their standings in the legislature and all three leaders stayed on.
ThreeHundredEight projected a Liberal majority of 58 seats, with the Progressive Conservatives taking 29 and the NDP 20. The end result was 53 seats for the Liberals, 37 for the Progressive Conservatives, and 17 for the New Democrats.

This was a total error of 16 seats, or 5.3 per party.

Using the actual provincial vote result improved things only slightly, to an error of 4.7 seats per party. The Liberals were still erroneously projected to win a majority.

Whether using the projected vote or the actual vote, the riding level accuracy was 85.0%, with 91 of 107 ridings being accurately called.
The seat ranges, however, envisioned a Liberal minority as it stretched from between 51 and 62 seats. But for the PCs, the upper seat range was 35, or two seats less than the party actually won. For the New Democrats, the lower seat range was 19, or two seats more than they actually won. But the ranges did give a clearer picture of what was possible.
The vote projection under-estimated the Tories by 2.1 points and over-estimated the NDP by two points. But generally speaking, an error level of 1.6% per party was not bad at all.

The chart below shows a few quick facts about the projection at the riding level.
The average margin of victory in the ridings that were incorrectly called was 8.9%, so a relatively significant amount.

The New Democrats and Progressive Conservatives were generally well called, as between 70% and 75% of their riding level projections were within 5%. This dropped to about 59% for the Liberals, and overall 45% of ridings, or 48 out of 107, had every party at or within 5% of their actual result.

On average, Liberal riding projections were 0.4 points higher than their actual result, while NDP projections were 0.8 points higher. For the Tories, the average riding-level projection was 2.5 points lower than their actual result.

This was the real problem. The Progressive Conservatives were under-estimated in 61 of 107 ridings, and only over-estimated in 36 ridings. Ten were called exactly.

This contrasts greatly with the Liberals and NDP, who generally split evenly. The New Democrats were under-estimated in 50 ridings and over-estimated in 48, with nine being called exactly, while the Liberals were over-estimated in 53 ridings and under-estimated in 46, with eight being called exactly.

In all, the margin of error at the riding level was +/- 3.3% per party.

These are the top three ridings in terms of accuracy:
For Niagara West-Glanbrook, the Liberals and NDP were only off by one point apiece, while in Oak Ridges-Markham the projection was off by one point for the Tories, NDP, and minor parties. In Thornhill, the projection was off by one point for the NDP, the Greens, and the minor parties.

But why did the projection get 16 ridings wrong? It wasn't just the polls under-estimating the Progressive Conservatives, as the model would have still gotten 16 ridings wrong with the actual province-wide numbers.

The problem is that province-wide numbers are difficult to use in such a regionally diverse province. If we look at the results more closely, we see where things failed regionally. The bulk of the incorrect ridings were in southwestern Ontario, where the Tories were under-estimated in 17 out of 21 ridings. They were also under-estimated in every riding in northern Ontario.

For the three major parties, the riding level projections had an average error of +/- 4% or less in central Ontario, the Hamilton/Niagara region, and the GTA. But that error was more than 5% in Toronto, southwestern Ontario, and the north, where more than two-thirds of the errors took place. This demonstrates why a regional model is necessary, despite the lack of uniformity in how the pollsters divvy up the province.

This was a big reason for a lot of the errors, but there were also some specific factors at play. For example, Rocco Rossi was not a "star candidate" in Eglinton-Lawrence, or at least that bonus made that riding an inaccurate call.

Overall, however, it was not a bad projection. The 1.6% popular vote error was acceptable, as was the +/- 3.3% error at the riding level. Being off by 5.3 seats per party is less acceptable, but had the Liberals eked out their majority it would not have been as consequential an error as it turned out to be.


  1. Have you thought about looking at the ridings that were furthest off trend to consider what other factors might be at play that aren't now in your model?

  2. I would start by identifying the ridings that were most off trend, considering what they might have in common, and whether what they have in common is quantifiable. Just looking at some of the off trend ones (PEH, SDSG, GPR, NQW, BGM, Oshawa, SRR, Welland, and a number of ridings in Southwestern Ontario) the variances seem to have a correlation with federal results in those ridings.

  3. But there are a lot of ridings that had very different results provincially than they did federally. I try to look for things that can be applied uniformly. It may have been a regional phenomenon, i.e. people in SW were more influenced by the federal election than people in the GTA.

  4. It would be really, really, nice if the pollsters would standardize their regions. How hard can it possibly be to come up with cohesive definitions for Hamilton/Niagara and eastern, central and southwestern Ontario? Northern Ontario, the 905, and the 416 are at least consistent across all pollsters, so that could form a partial base from which to work.

  5. I think there are two reasons the seat projections were off. First, the underprojection for the Tories didn't take into account the big lead for the tories among older voters who vote in much bigger numbers than average and obviously swung 7 or 8 ridings. The other side of the coin is the over estimation of the NDP which polls higher among younger voters who vote in much lesser numbers(33%) and so several ridings went Liberal and one Tory which were projected for them. The other factor is the early voting,which took place when the tories were still riding high in the polls, so likely were skewed more towards them than would have been the case on election day as the Liberals rose and the tories sank in the opinion polls. I think in future the early polls and age demographic of support should be taken into account

  6. I'm not convinced people always vote regionally. In this election, the results seemed based on a rural/urban-suburban split, with the exception of north of Parry Sound.

    In a race where rural seemed to go to the Tories and suburban/urban to the Liberals and the NDP, any region with a larger % of rural ridings is going to look more off. But, in the next election, we may not see either that type of split, or a regional one.

  7. You had Rocco Rossi as a star candidate?

  8. What does the projection look like if you put the final provincewide results into the federal model (but valuing provincial and not federal incumbency)? What about a mixture of the two?


COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.