Friday, February 14, 2014

How did the polls do in the Ontario by-elections?

Two by-elections were held in the Ontario ridings of Niagara Falls and Thornhill last night. I wrote about how the parties themselves did for The Huffington Post Canada, but let's take a look at how the polls did here.

As is usually the case, Forum Research was the only pollster in the field. The firm was a little less active than it was in the Ontario and federal by-elections of 2013, and conducted its final poll on February 11, rather than on the eve of the vote as Forum has usually done.

Forum and the Toronto Star did seem to be aware of the skepticism that had grown with Forum's by-election polls after the misses in Ottawa South and Brandon-Souris. In their reports, the Star pointed out that Forum uses a proprietary weighting formula that was shown to the Star, and that the raw data of the polls had been deposited at the University of Toronto (if you're a political science student there, take a trek down to the library and take a look at that raw data, please!).

There was no Ottawa South or Brandon-Souris type error last night in the polls. As forecasted, Wayne Gates of the NDP won in Niagara Falls and Gila Martow of the PCs won in Thornhill. The final poll in Thornhill was quite close, with all parties' results falling within the reported margin of error, while the final poll in Niagara Falls over-estimated the support of the NDP. Otherwise, the poll did fine.

A few notes on the charts below. The "With reported MOE" row shows the margin of error ranges for each of the parties when applying the margin of error that was reported (Forum rounds the margin of error off, and calculates it for the entire sample, not just decided voters). The "Actual MOE" row shows what the margin of error for each party actually was, taking into account their level of support (a party with 50% support has a higher margin of error than a party with 5%) and the number of decided voters.

In Niagara Falls, Forum over-estimated the support of the New Democrats by at least five points, and as much as twice of that. So that was a miss, but at least the winner was correctly identified.

The PCs, Liberals, and Greens all did slightly better than the poll expected, though the results were within both the reported and actual margins of error.

The poll conducted by Forum on February 5 would have been much closer. That survey had the NDP at 38% to 36% for the Tories and 19% for the Liberals.

The poll in Thornhill was much better. The results were within three points for the PCs and within 1.8 points for the Liberals, NDP, and Greens. All results fell within the reported and actual margins of error, and comfortably within them (unlike the PCs in Niagara Falls, who fell just within the edge of the margin of error).

This is not the first time Forum has had a good performance in the Toronto area. Its Toronto Centre poll in the November by-elections was the closest of the four, while the polls in Etobicoke-Lakeshore and Scarborough-Guildwood were the best of the five Ontario by-elections of August 2013. Forum has had much less success outside of the GTA. I cannot speculate why that might be.

UPDATE: A reader made an interesting suggestion of why that might be. Forum's by-election record has been generally good in races where voting patterns did not change much (i.e., the GTA) but has been very poor in ridings where those patterns changed significantly. If Forum is making assumptions to model the voting population in its polls, those assumptions may be incorrect when the electorate is shifting so greatly.

This site's By-Election Barometer continues to have a flawless record. Thornhill had been forecast to be a Likely PC win, while Niagara Falls was a Toss-Up that favoured the NDP. This stretches the barometer's record to 29 by-elections without an error.


  1. I'm a U of T student. If you tell me where to look, I can probably get it for you.

    1. It is apparently at the Political Science Data Library. I'm just curious what was actually deposited there.

  2. U of T has a general Data Library, but none for Political Science itself.

  3. Interesting how in both seats the Liberals did better than the polls ??

    1. In a way the by-elections gave all three parties a reason to want a Spring election.

  4. Eric

    It is interesting to see you defending the pollsters. This is yet again a complete mathematical failure.

    To be off by 8 points basically double the the MOE combined with Forum's past track record is a clear almost overwhelming sign that some thing is wrong with their methodology.

    It is not the outlier that happens 1 time in 20 polls ... that outlier would be the NDP getting 43.5 when the poll says they should get 48. That should happen only 1 times in 20.

    they are off by 4 standard deviations rather than the 1.96 . To be off even 3 standard deviations is a quite an accomplishment. This should happen in 1 sample out of 1000

    The Forum polls need to have a rider added to them that these are just for fun and have no basis on statically accurate projection of the out comes of the actual election.

    1. That does not answer how Forum could randomly get one number so wrong but the other nine correct. If the odds are 1000:1 that they'd get that NDP result with a good random sample, what are the odds that they'd get all the others right with a completely unreliable sample? Isn't it more likely that some other factors were also at play and perhaps responsible for more of the error?

      In this I mean turnout and vote switching. Forum could not have done anything more than try to guess at what the voting population would look like. At that stage, we're not talking about polling so much anymore and the theories don't apply as strictly.

    2. I should point out that we have better reasons to be concerned with their methodology due to the raw samples they compile, rather than a miss on vote results. I have written about that several times on this site. Here's one such post:

    3. The big issue in Canada with polling these days is how do you get an accurate random sample of the voting public?

      In the past when 80% of people voted and almost everyone could reached by a phone number in the phone book and the one phone in a home allowed you to ask for youngest female voter or oldest male voter etc, the process of using a phone book was a decent representation of a random sample of the voting public.

      Getting anything close to a true random sample is almost impossible now which means the margin of error calculation is misleading on any poll.

  5. I read your article of May 27.

    You provide a pure motive examination of the problems that the pollsters have been having with their limited resources and inability to accurately establish a representative sample.

    In Forum's case you have to get the the critical antenna up immediately on hearing that they are normalizing to the Census rather than to any demographics of actual voters.

    To assume that under 35 year olds are going to turn out in the same proportion as 55 and older is just a plain bad assumption.

    The other thing that is extremely worrisome is the pollsters motivation....... From what I can gather is that political polling is a loss leader that is used to get their name out in the market-place to have recognition for money making work.

    Does it hurt Forum to have Rob Ford with a higher approval rating than Obama? I think that is just off the wall crazy but Forum was getting its brand mentioned by all the major US News shows.

    The Talking Heads on the news-casts have no idea that Forum has a major reputation of being wrong.

    Did having the Wild Rose party on the verge of kicking out the dynasty Cons in Alberta generate more or less discussion than having the Cons winning again?

    In BC the NDP Victory was a far more impact laden story than the Liberals/Con alliance continuing to govern Canada.

    In Canada the pollsters having the leader of the tiny third party Trudeau as the next PM with the office being his to lose is certainly more news-worthy than any other permutations of results that might have been delivered.

    The incredible thing is that these polls do have an impact..... E.May got a seat at the debate because the polls had incorrectly put the Green support at 10%.

    Without this incorrect/fudged poll it and her national exposure at the debate it would have been even more difficult for Ms. May to get elected as a MP.

    1. May got a seat at the debate because she had an MP BCVOR.

    2. That was one of the arguments but pre-debate Harris-Decima had the Green's consistently polling at 13% -- 4-5 point ahead of the BQ and within 2-3 points of the NDP.

      If that were true there she deserved to be at the debate.... Trouble was it wasn't close.

      bad polling or using polling to push an agenda or make news?

      Or will you present the hypothesis that Ms. May did so bad at the debate that she dropped Green support more than 5 pts?

    3. Ryan,

      She sort of had a M.P., Blair Wislson resigned from the Liberal whip then sat as an independent. In August of 2008 he announced he had joined the Greens but, before he could take up his seat as the country's first Green M.P. the House was dissolved. To my knowledge he was not recognised by the Speaker as a Green M.P. and so I think it questionable whether he was in fact a Green M.P. All academic of course as the media cabal, whoops consortium, thought he was and allowed May's participation.

    4. Yah, a bit debatable.

      BVOR - Are you sure those polls were wrong? I think that that was probably the Greens' genuine support, and the more dishonest/strategic number was on election day.

      In any event, 2008 was the first time in Canadian history that a party took more than 5% of the vote without winning a seat. The Greens' "true" support was definitely in the range where you would expect them to take seats.

    5. Many people say FPTP is exclusionary and doesn't allow for smaller parties but, you're absolutely right with the exception of 2008 every party that has won 5% of the vote or better has elected a M.P. This is no different than Germany where they impose a 5% threshold to enter Parliament.

    6. Difference is under the German system you'd have 15 seats with that 5%, not 1. :/

    7. Yes, if a party receives 5% in the national vote they would have about 15 seats. From what I can find the threshold is 3 constituency M.P.s or 5% of the national vote to get into Parliament. Presumably once you had 3 M.P.s and 5% you would get the additional MMP seats. I think the caveat though is if you elect two constituency M.P.s but, don't receive 5% they can not sit in Parliament.

    8. It should be mentioned that in the German system 5% of the national vote only equals about 2.5% of the seats since half the seats are allotted through constituency voting.

  6. Guesses on today's Canada -Finland game. Price to start in nets ?

    1. After a really tough game we won in overtime.

      Hate the Olympics point system !! Lose a point because you win in overtime ??? Hello ????

    2. Makes more sense to me than the NHL system, with some games worth 3 points, and others worth 2....

    3. Olympics you win in regular time = 2 points

      Win in overtime = 1 point

      Makes no sense to me, a win is a win Eric ?

    4. No, when you win in regular time in the Olympics you get 3 points. If you win in overtime, you get 2 points and the other team gets 1. It is a better system, since in every game there are just 3 points at stake.

      In the NHL, you get 2 points in regular time and 2 points in overtime, but the other team gets 1. That makes it a three point game rather than the usual two point game.

    5. Thanks Eric, the person that explained the system to me had it wrong then. Appreciate the correction.

  7. Ipsos has a new Ontario poll: PC 34%, Lib 31%, NDP 31%.

    Some interesting numbers when it comes to committed voters PC 37%, Lib 27%, NDP 33%.

    With these numbers it is difficult to see how Wynne will survive the budget.

    1. Probably a dice roll for any of them. 10% is hardly much of a lead when you have Tim Hudak as leader.

    2. With the committed voter numbers the Tories would win about 50 seats with the Grits and Dippers winning roughly half that. So if the numbers accurate Hudak is on the cusp of a majority.

      The Regular numbers would be very interesting, probably would see a Conservative plurality of a couple seats but, all parties may be within 1-4 seats of one another. Possible tie even.

      I think the NDP and Tories have to go unless the next poll alters the numbers. For Hudak this may be as good as it gets and for Horvath she has a chance to displace the Liberals as Official Opposition maybe even secure a government. It is once in a blue moon the NDP federally or provincially polls in the 30's in Ontario delaying an election will only give Wynne time to recover or counter attack.

  8. "Forum has had much less success outside of the GTA. I cannot speculate why that might be."

    I can think of a few possible reasons.

    1. There is more data for the GTA than other regions because there are more people. More past data = better sampling and adjustment procedures.
    2. GTA ridings are more uniform and it is easier to get a representative sample. A riding like Niagra, for example, is both small town, suburban and rural. You have to be much more careful here to avoid oversampling an area of the riding.
    3. My guess is that Forum polling is organized and designed by people in the GTA. As such, the poll designers understand the local nuances of sampling better than they would for other areas.

  9. Eric when will we see seat projections for Ontario based on polls.

  10. Forum seems really good at spotting new trends but not in pinning down the numbers.

  11. It is interesting that Nanos has basically taken themselves out of the election results polling.

    They do not poll anything that actually can be measured against actual results.

    Their polls have no practical relevance to the next election rather than generating headlines.

    Who are your 2 choices for PM?

    So the left of Centre get to pick Mulcair and Trudeau. The right of Centre get to pick Harper and in my case it would be Trudeau or maybe any Green except May.

    This has been designed to make Trudeau look so much more popular as a leader.

    Before expanding to the 2 votes per person fantasy Harper was always almost twice as popular a leader in the Nano poll. This was reflected in the last elections results.

    Nanos might be correct.... If every voter is given 2 votes the Liberals under Trudeau might win.

    Given a choice of 1 leader Harper would once again dominate the Field and taking away the people like me that are solid Cons but could appreciate Paul Martin Trudeau would likely fall behind Mulcair and May would fall back down to single digits.

    That would take away a headline where Trudeau is the considered the best leader of some make-believe version of Canada.

    If Nanos wanted to make bigger headlines (and lose more credibility) they could ask who are your 3 favorite leaders and Ms. May might be more popular than Harper.

    1. Nanos only reports the first choice, so your concerns are baseless.

  12. Trudeau 54 Harper and Mulcair 50 each and May 32 adds up to 186 %

    Who ever is the BQ leader gets 25 of Quebec.

    If they can fool you Eric then it makes my concern far greater.

    1. Please fully read the methodology. Those are the index scores, meant to each be out of 100.

      You were referring to the PM question. Only the first choices are reported: Trudeau 30, Harper 28, Mulcair 19, May 4, Bellavance 1.

  13. Did you see the Canada-Latvia game Eric ?? Wow !!


COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.

Note: Only a member of this blog may post a comment.