Tuesday, November 26, 2013

By-election post-mortem

The four federal by-elections last night played out as ThreeHundredEight forecast, with the Liberals retaining Bourassa and Toronto Centre, the Conservatives retaining Provencher, and Brandon-Souris being a toss-up riding that leaned slightly towards the incumbent Tories. So slightly, in fact, that the Liberals were less than 400 votes short of winning it.

By-Election Barometer Record
This makes 22 by-elections without a wrong call by ThreeHundredEight, going back to June 2012. The Barometer will be put to the test again tonight in Carbonear-Harbour Grace in Newfoundland and Labrador. The model suggests the riding is leaning Liberal, though the PCs do have a slim chance of holding on to it. However, the forecast is for the Liberals under new leader Dwight Ball to take it by a narrow margin.

The results in Brandon-Souris, which were the most at-odds with the polls by Forum Research, showed the mettle of the By-Election Barometer. The forecast was that it was a toss-up between the Conservatives and Liberals, with the Tories considered to have a 52% chance of winning to 48% for the Liberals. In the end, the Tories won it by a single percentage point.

This was similar to the by-election this summer in Ottawa South, where the polls suggested the Tories were well-placed to steal former premier Dalton McGuinty's riding away from the Liberals. The model still considered it a toss-up that leaned slightly towards the Liberals, which turned out to be the case. It demonstrates the need to take into account the fundamentals in a riding, particularly when riding-specific polls are showing numbers that are wildly out of step with historical performances.

I assessed the performance of the polls for The Globe and Mail, and the results are not great for Forum Research. They missed Brandon-Souris by a wide margin and Provencher was off by a considerable degree as well. The polls were better in Toronto Centre and Bourassa, however. This record matches their performance in the Ontario by-elections this summer, where I gave them a score of 2.5 out of 5. Last night, they got 2.5 out of 4, for picking the right winner in Provencher but missing the respective levels of support for the Tories and Liberals.

Because of this very mixed record, I employed confidence intervals far wider than the reported margins of error of these polls. This turned out to be a good idea, but only captured some of the error. Of the 20 calls Forum Research made (one for each party + others in the four ridings), 60% fell within the 67% confidence interval and 85% fell within the 95% confidence interval. Clearly, these wide ranges will need to be stretched even further, but that is the usefulness of these extra data points.
The performance of the confidence intervals, based solely on Forum's polling, compared to the results can be seen in the chart above.

The misses at the 95% confidence interval were primarily in Brandon-Souris. The results for the Liberals and Conservatives were so off that a +/- 10-point confidence interval still did not capture the result. The Greens in Provencher also fell outside of the 95% confidence interval.

At the 67% confidence interval, only the Others in Toronto Centre and the Conservatives in Bourassa were wide of the mark. In Brandon-Souris, the Liberals and Conservatives fell outside the interval, while all parties fell outside of it in Provencher. In short, even compared to Forum's past errors in by-elections the ones in Brandon-Souris and Provencher were extraordinary.

Note: I inquired with Forum about their methods for Brandon-Souris in relation to the reports of multiple calls being received by some residents. I was told by Lorne Bozinoff that they used the same random dialing in all ridings, and that no panel of phone numbers was used to dial the same people over and over again. I was also told that if a number was called in consecutive nights, that person's results were dropped from the sample and that all three of their final polls (Forum also polled on Nov. 23, but did not publish the results which were similar to the poll of Nov. 24) were independent samples.

21 comments:

  1. I guess we will see if the Libs can build on this result to further Provencher and win Brandon-Souris?

    ReplyDelete
  2. Good night for you Eric. Crappy night for polling in Canada, as usual.

    ReplyDelete
  3. I think these by-elections clearly demonstrate the huge importance of the parties' get-out-the-vote ground game and how pollsters will be very hard-pressed to accurately predict results when the parties aren't equally competitive on this front.

    Clearly the CPC focused all of its efforts on the MB ridings, particularly Brandon-Souris (as evidence by the highest voter turnout among the 4 ridings) which they were scared to death of losing. The Liberals had a strong showing in both ridings but clearly didn't come close to matching the CPC's ground game, hence the former's support was severely overestimated and the latter's severely underestimated in the polls.

    The second highest turnout was in Toronto Centre, where the polls came decently close, probably because both the Liberals and NDP invested equally strong efforts in their ground game there.

    Finally, the polls were also fairly close in Bourassa, where again I suspect that the Liberals and NDP invested roughly equal efforts, though less overall than Toronto Centre (as evidenced by the dismal turnout) because it was probably deemed by both parties as a safer Liberal riding than Toronto.

    Also note that the CPC underperformed in relation to the polls in Toronto and Montreal, probably as a result of them investing little to no effort in these ridings.

    Dom

    ReplyDelete
    Replies
    1. That's consistent with the CPC's behaviour since they were formed. They don't expend effort in losing causes.

      Delete
  4. The results are in...

    Provencher: The Conservative has won by 28.2 points (Forum predicted in its last poll they would only win by 11 points over the Liberals)

    Bourassa: The Liberal won by 16.5 points (Forum predicted in its last poll they would win by only 12 points over the NDP)

    Toronto Centre: The Liberal won by 13.2 points (Forum predicted they would win by 8 points only over the NDP)

    And Brandon-Souris: The CONSERVATIVE HAS WON by 1.4 points (Which Forum unwisely predicted they would have lost to the Liberals by 29 points)

    Gee Forum... You guys must feel like crap right now... GOOD!

    ReplyDelete
  5. I just can concieve of a situation where I'd hand over a cheque to Forum for poll work. Might as well throw darts.

    ReplyDelete
    Replies
    1. Forum seems to be really hit-or-miss.

      On the one hand, I believe they were the very closest to predicting the result of 2011 Ontario election; they were the closest to predicting the 2012 Alberta election and the only pollster to register any hint of a last-minute closing gap between the PC and Wildrose; and they were the closest to predicting the result of the 2013 BC election and the only pollster to register the Liberals ≥40%.

      But on the other hand, they mistakenly registered a surge in PQ support putting them in majority territory at the end of the 2012 Quebec election campaign, and their cumulative record in all of these by-elections is quite sketchy at best.

      Their federal polls also seem to exhibit more volatility and wider ranges in party support than other pollsters.

      Dom

      Delete
    2. Yes. I wonder how much of this has to do with the frequency of polling. They are active in virtually every provincial and by-election campaign, so they come out good in some and bad in others. Other firms are active in only a few campaigns, so it is difficult to know if they would do better or worse if they were just as active. What other firm was actively polling in NS, BC, QC, AB, and ON? None.

      Harris-Decima, for example, is an excellent firm. But they haven't been active in a campaign since the federal one in 2011. How would they have performed in BC?

      Delete
    3. "Throwing darts" is a little extreme. Forum did register the surge of Liberal support in all the ridings, and did a decent job of predicting the results in several of them. We shouldn't neglect to accounts for the vagaries of by-elections and the importance of political organization, which the Liberals are only beginning to reconstruct out West. It's certainly paltry relative to the Blue Machine.

      Plus, a few darts is better than no darts at all. None of the other polling firms dared to jump in publicly as far as I know. Who knows - maybe they would have done worse (or similarly). Personally I had hoped that Ipsos-Reid would have done something here. Their federal results, which are widely publicized, have been consistently and considerably different from what virtually all the other polling companies have been showing for a while now.

      Delete
    4. A pox on all their houses. A pox on Forum for being outside their margin of error by a lot, and doubly so on the other pollsters for not putting their skin in the game.

      Delete
  6. I have heard some kerfuffle about people being called multiple times by polling firms. I do not know that much about polling, but I know a lot about trying to reach people by phone. When one small riding is being polled so many times over a short period of time, randomized calling will produce a lot of duplicate calls. You typically have to dial 4 times to reach a live person. You might get as many as 10%-20% of those who answer their phone who are willing to answer your questions. So every poll that wants to hit a minimum of say 400 respondants has to dial between 8,000 and 16,000 numbers, (depending upon 'conversion rates' being 10% or 20%). You can see how if only a single company was polling, and they polled the riding 3 or 4 times, pretty well every telephone in the riding will ring at least twice. (assuming approximately 30,000 households)

    ReplyDelete
  7. So the moral of the story is that if you set the bar low enough you will succeed.

    I predict that in 2015 the pollsters will show that the Cons will be ousted by an unholy alliance of Green, NDP and Liberals.

    Using this predetermined information I predict that due to the polls being wrong by +/- 30% there will be 338 toss up and the incumbents will be narrowly favored in the 308 riding they currently hold and the Cons will be mildly favored in the 30 new riding's.

    As I have clearly stated that I am not confident in any of the polling results ...... and have clearly predicted 338 toss-ups that someone will win I am mathematically certain that I will be 100% correct

    ReplyDelete
    Replies
    1. The polls will be off by 30% and the Greens will romp to their first majority. Remember you heard it here first BVOR :)

      Delete
    2. Yeah Ryan given the kind of results the polling is giving lately anyways who can believe ?

      Maybe it's time to change the way polling is done ??

      Delete
  8. I think Brandon has to be the most interesting of all four. To win by only 391 votes out of 27,681 says it was a crap shoot !! It also says the Libs really, really upped their game !!

    ReplyDelete
    Replies
    1. I'm confused, I thought it only mattered who won the seat?

      Delete
    2. Eric it matters who wins Yes. But the manner of the win sometimes has real importance !! Or for that matter the loss as well.

      Delete
    3. I think the wording was : "No matter how you spin, bafflegab or fulminate the only criteria that really matters is number of seats." That doesn't leave a lot of room for "the manner of the win". :~)

      Delete
    4. There’s no spinning it — the Conservatives were spanked in Monday’s byelections

      Andrew Coyne, today's National Post. Sometimes the
      manner"of winning is as important as the win. In these cases the Tories got hammered, even in seats they held. Plus big falls in the other two seats. "Manner" of win or loss matters !!

      Delete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.