Saturday, October 15, 2011

Manitoba: Projection vs. Results

If you'll allow me a little immodesty, the projection for the 2011 Manitoba provincial election was ThreeHundredEight's Mona Lisa. Tongue firmly in cheek, of course, but of the 57 ridings in the province, 56 were called correctly and the 57th was so close that it is currently under judicial review.

The Manitoba election might seem trivial to people who aren't one of the 1.3 million Manitobans in the province, but this was an election that was thought to be very close. It was an election that demonstrated perfectly the usefulness of seat projections, as though the polls pointed to a tight race my seat projection model indicated that the New Democrats could win a comfortable majority with an uncomfortable margin in the popular vote. And, as I will show, this projection was no mere accident or stroke of luck. Even at the individual riding level the projection model performed very well.

In the end, Greg Selinger's New Democrats won an even larger majority than was projected, defeating Hugh McFadyen's Progressive Conservatives. Jon Gerrard, leader of the Manitoba Liberals, was the only candidate elected for that party.
ThreeHundredEight projected 36 New Democrats would be elected, 20 Progressive Conservatives, and one Liberal.

The result was 37 New Democrats, 19 Progressive Conservatives, and one Liberal. Using the actual provincial vote tally did not change the projection.

In all, the model called 56 out of 57 ridings correctly for an accuracy rating of 98.2%.
The projected seat ranges anticipated this result, as the New Democrats were expected to win between 35 and 38 seats, while the Tories were expected to win between 18 and 21 seats. The result was within the expected seat range. The Liberals were expected to win one seat and one seat only, and so they did.

As the polls by Probe Research and Angus-Reid at the end of the campaign were excellent, the vote projection was excellent as well. It under-estimated Progressive Conservative support by one point, but was within 0.6 points for the New Democrats, Liberals, and Greens.

The chart below shows a few quick facts about the projection at the riding level.

The margin of victory in St. Norbert, the one riding incorrectly called, was only one per cent.

Generally speaking, the Progressive Conservatives were under-estimated at the riding level. In 16 ridings they were over-estimated, in four ridings the projection was accurate, and in the remaining 37 they were under-estimated.

For the other parties the projection was generally solid, with the New Democrats over-estimated in 35 ridings, correctly called in five, and under-estimated in the remaining 17. The Liberals were over-estimated in 31 ridings, correctly projected in eight, and under-estimated in 18.

In all, 79% of individual riding projections for the New Democrats, or 45 out of 57, were called to within 5% of the result. That increased to 81% for the Tories and in almost every case the riding projections were within 5% for the Liberals and Greens. In 40 ridings, or 70%, every party was pegged to within 5% of their actual result. That is no mere coincidence or accident.

On average, the riding projections were 1.9 points higher for the NDP than the result and 2.2 points lower for the Tories. The Green and Liberal estimations were even closer, putting the margin of error for each party's riding projections at +/- 2.7%. That is, I believe, quite extraordinary.

To compare to the Prince Edward Island election, less than 30% of those ridings had every party called to within 5% and the average margin of error was +/- 4.6%.

Many ridings were called very well, but here are the top three:

In Dawson Trail and Selkirk, the projection was only off by one point for the three parties running. This is a good time to remind readers that the boundaries in Manitoba changed since the 2007 election, compounding the difficulty in making accurate projections.

Rossmere was off by a total of three points across four parties.

Now, what of St. Norbert, the one riding that got away? As mentioned, a judicial recount will take place and the results revealed on October 24. So, it could be that the result will be overturned and the projection will end up being 100% accurate. Note also that the vote in Kirkfield Park, which was correctly called, will also be reviewed. Hopefully that one won't be overturned.

If the result is not overturned in St. Norbert, why did the projection get it wrong? St. Norbert only bucked provincial trends by the tiniest amount, and it appears that the lack of an NDP incumbent did not hurt the party at all. But removing that factor still would have given the riding to the Tories.

So, it would appear that local factors were at play. Perhaps a good performance by Dave Gaudreau, the NDP candidate, or a less than good performance by Karen Velthuys, the Tory candidate, is to blame. Handing Gaudreau the star candidate bonus would have given him the riding, but he doesn't fit the profile of a star candidate and Velthuys does not seem to have been especially problematic.

The wrong call in this one riding needs to be attributed to the fact that the model cannot precisely predict how thousands of individuals will behave. St. Norbert being the one wrong call is, I think you'll agree, perfectly acceptable.

It should come as no surprise that the 2011 Manitoba provincial election projection is the best of the seven that ThreeHundredEight has called. And that is by a wide margin - the per party error on the vote projection was almost half the error of the next best called (2008 Quebec election) and the per party seat error was almost one-third of that in the 2008 Quebec election. I do not consider it likely that the projection model will ever provide a better result than it did in Manitoba.

4 comments:

  1. You did a great job!

    asdf

    ReplyDelete
  2. I think I may be able to shed some more light on this projection.

    When transcribing the votes from the old boundaries to the new boundaries the biggest source of error were absent and early ballots. These ballots were lumped together, so did not correspond to the exact voting area of the voters. This means if one voting area got moved to another riding the absent and early votes in that area may not have been moved.

    You'll notice the 3 top projections were all rural ridings, while the only mistake was a Winnipeg riding. For whatever reason there tended to be fewer absent and early voters in rural ridings. In addition, early voting in rural ridings was often divided by town, so these could be moved if that town was moved to another riding.

    If there was a large number of early and absent voters in Winnipeg ridings voters were moved based on the overall proportion of voters moved in the riding. I took a look at the St. Norbert riding, and there were two of these adjustments made for this riding. One moved votes away, and the other added votes. This indicates there were a large number of absent and early votes in this riding and the ridings that were added to it which may have impacted the results.

    This is just a theory, there's no real way to be sure how much this impacted the result. But it would have been interesting to see how well the projection would have done without boundary changes.

    KV

    ReplyDelete
  3. "It was an election that demonstrated perfectly the usefulness of seat projections..."

    It may have demonstrated the potential _accuracy_ of seat projections. But what "usefulness" do you claim for your seat projections and by what metric do you claim it was achieved?

    ReplyDelete
  4. The usefulness is in showing what polls do not always show.

    As mentioned in the above post, most analysts considered the race to be very close, almost too close to call. A seat projection model, however, showed that a close race was all that the NDP needed to win, and that the Tories needed a much larger margin in order to form government.

    ReplyDelete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.