Friday, January 30, 2015

What are the polls really showing in Sudbury?

As by-elections go, the one in the provincial riding of Sudbury in Ontario has been pretty dramatic. Accusations of skulduggery, a Liberal-turned-independent candidate mixing it up, and a floor-crosser that managed to ford both the ideological gap between two parties as well as the federal/provincial line.

Some of the polls have been showing a close race between Liberal candidate Glenn Thibeault and the NDP's Suzanne Shawbonquit, with independent Andrew Olivier earning a big chunk of the vote. Other polls have shown Thibeault with a relatively comfortable lead. What gives?

At first glance, it is very puzzling that the polls by Forum Research and Oraclepoll Research can differ so much. Both firms have been in the field twice at exactly the same time. Oraclepoll was in for three to four days, with the Forum polls right in the middle of those sampling periods. So timing should have nothing to do with it.

Forum's first poll on January 13 put the gap at just two points, with Shawbonquit narrowly ahead. Olivier scored just 1%. Then on January 21, the gap was three points for Thibeault as he dropped seven points and Shawbonquit fell 12. Olivier made a miraculous 21-point gain, which Forum called a surge.

Meanwhile, Oraclepoll gave Thibeault a lead of 13 points on January 12-15 at 39% to 26% for Shawbonquit, with Olivier at 19%. On January 20-22, the gap was 16 points (42% to 26%), with Olivier at 20%. In other words, while Forum was showing a major shift, Oraclepoll was showing stability.

So who's right?

There are many reasons to trust Oraclepoll's numbers more than Forum's. Here's why.

Firstly, Forum is based out of Toronto and Oraclepoll is based out of Sudbury. Right out of the gate, the local firm has an advantage both in terms of local knowledge and a local area code showing up on the caller ID.

Secondly, and most importantly, Oraclepoll is conducting a superior survey. Forum is doing its polls via IVR, as it always does. It jumps into the field for a few hours on one evening, gathers its 500 to 800 responses, and reports the numbers a day or two later (response rates being what they are for Forum, it probably needs to call 25,000 to 80,000 households to get even those small samples).

Though it doesn't say explicitly, we can assume that Forum is not calling cell phones. This is because, unlike land lines, it is not possible to know for certain where a respondent on a cell phone is living. This is a problem in by-elections, as ridings do not entirely occupy an area code. This is not an issue in provincial or national polling. Cell phones can be included there without issue, and usually are.

Update: Commenter DL (see below) points out that targeting cell phones in a billing area is straight-forward, so Forum may indeed be calling cell phones. The inability to call cell phones pinpointed to a specific riding would be more of a problem in a city like Toronto where there are multiple ridings within one municipality.

Oraclepoll, on the other hand, is doing its polls with live-callers and it explicitly says that it is including cell phones in its sample (perhaps it has a list of local cell phones to call from). It is also in the field for several days, and according to its press release is calling back numbers where respondents did not answer up to five times before giving up. That is how a poll is supposed to be done.

Forum is getting responses from whoever is home between, say, 7 PM and 9 PM on January 13 and January 21 and is willing to answer a survey. Oraclepoll is doing everything that is reasonably possible to reach everyone it is calling. The potential for a biased sample is far lower.

Lastly, Forum's 1% result for Olivier on January 13 defies logic. At the time the poll was released, people were very surprised that Olivier, who had a relatively high profile, was scoring just 1%. That his number jumped from 1% to 22% in eight days makes little sense, particularly when Oraclepoll was showing his support levels to be steady at between 19% and 20% over the same period. While it is conceivable that the discrepancies in the results for the Liberals and NDP can be explained away by sampling issues and statistical probability, the discrepancy for Olivier's numbers are virtually impossible.

I don't know what happened in Forum's January 13th poll. But it simply doesn't make sense.

In that survey, Forum inquired as to whether Sudburians (Sudburites? Sudburers?) approved of the candidates on offer. The responses there just don't line up with the voting intentions numbers it recorded.

At the time, the poll found that 95% of respondents were aware or had an opinion of Thibeault, compared to 75% for PC candidate Paula Peroni and just 61% for Shawbonquit. But Olivier, who had just 1% support in the same poll, had 88% recognition - better than either the NDP or PC candidates.

That poll also suggested that 48% of all respondents (including undecideds and those who did not know him) approved of Thibeault, indicating that he was converting about 80% of sympathizers into supporters. Shawbonquit, with 41% approval, was converting an incredible 99% of sympathizers.

Peroni's approval was just 28%, as she was unknown to many respondents, but she was converting just 45% of sympathizers into supporters.

Olivier, though, had an approval rating of 60% of all of those sampled - and this is including those who did not have an opinion or did not know him. That was higher than any other candidate. According to the poll, we are thus supposed to believe that Olivier was converting just 2% of his sympathizers into supporters. If you can believe that, I have a bridge that spans the width of Lake Huron to sell you.

I think we can reasonably conclude that the January 13 poll by Forum can be tossed aside and discarded. Forum's January 21 poll has no similar problems, and support levels for Olivier and Peroni are similar to those recorded by Oraclepoll. Considering the sample sizes, Shawbonquit's support is also within the margin of error of these two polls, and Thibeault's is only slightly outside of it.

Due to the advantages that Oraclepoll's survey has over Forum's (not to mention the poor record Forum has in by-elections outside of Toronto), the benefit of the doubt should probably go to Oraclepoll. And that's even with the smaller sample size (the margin of error of decided voters would be just under +/- 6%).

If we use that margin of error to estimate support ranges, we'd get Thibeault at between 36% and 47%, Shawbonquit at between 21% and 31%, Olivier between 16% and 25%, and Peroni between 5% and 11%. That is probably as close as we can get to the truth at this stage of the campaign, which will come to a merciful end on Thursday.

33 comments:

  1. Odd, Forum is usually produces polls that are the most favorable to the Liberals.

    ReplyDelete
    Replies
    1. Forum byelection polls tend to exaggerate Liberal momentum or performance in Tory-Liberal races. In NDP-Liberal races, they have been pretty accurate - If anything, they have favoured the NDP slightly.

      Delete
    2. Well, yes & no. Forum is somewhat erratic & unpredictable. Sometimes they have high Liberal numbers, but sometimes they may end up hurting the Liberals by producing inaccurate polling numbers like they did in Brandon-Souris (& Fort McMurray).

      As for this by-election, one would assume Oracle knows the Northern territory since they are based there whereas Forum seems to be more familiar with Toronto, as Eric says.

      Delete
  2. Great analysis, Eric. I'm with one of the campaigns here in Sudbury - not Andrew Olivier's - and I can tell you that we were in complete disbelief regarding where Forum put him in their first poll. However, the "surge" that Forum insisted happened to Olivier has certainly played well for his campaign narrative.

    We Sudburians have become fairly politically engaged in the last several years, but speaking with voters this time out, it's clear that the constant negativity is turning a lot of people off, and even those who are aware that there's a by-election going on and may be somewhat engaged in it are saying they'll be staying at home rather than voting for anyone. And that's really too bad. I had hoped that my community might show up the rest of Canada for turnout in a by-election, but now, I'm not so sure. There's also been a very active "refuse to vote" campaign underway on social media here.

    It's certainly been a very interesting, if unusual, by-election. And we get to do it all over again now in a few months, federally!

    ReplyDelete
  3. a peculiar Government of Canada page... demonyms for you... (sadly, it raise but doesn't answer the question of folks from Sudbury, but from other sources Sudburian is correct)

    https://www.noslangues-ourlanguages.gc.ca/quiz/jeux-quiz-gentiles8-demonyms8-eng.php

    ReplyDelete
    Replies
    1. The Government of Saskatchewan would dispute that page. Their documents consistently refer to the residents of Saskatchewan as "Saskatchewan People".

      Delete
    2. They should be Saskatchweigians in English.

      Delete
  4. Wynne really wants to take back Sudbury. Jean Chretien is campaigning for Thibeault today.

    A recent Toronto Star article claimed that Wynne is not fond of Horwath and wants to "put her in her place" hence the Liberals are really going all out for this by-election.

    I believe it is a three-way race between the Liberals, NDP and independent Olivier. However, the Liberals do have the advantage as they can still ride on the post-election honeymoon. Moreover, I don't see Thibeault party switching having a negative impact on him other than with NDP loyalists who may have felt betrayed.

    ReplyDelete
  5. So is Oracle considered a well-respected pollster in Northern Ontario then? That seems to be conclusion one can take from this article. Does it have a good track record in that region in the past?

    ReplyDelete
  6. Glenn Thibeault will win. Federally, this seat will go liberal,its not a toss-up. A former mayor is running for fed liberals. She has name recognition. I still feel that cons won whitby oshawa because a mayor ran for them.

    ReplyDelete
  7. Do you know what language the polls were conducted in? The riding of Sudbury is 1/4 francophone so to get a representative sample, polling should be done in both languages. As this is not the case in most Ontario ridings, Forum (based in Toronto) might not have thought of it and it could account for part of the discrepancy. This is especially the case since traditionally in the riding, francophones voted Liberal in higher proportions than Anglophones.

    ReplyDelete
    Replies
    1. Really good point. I suspect any pollster based out of Sudbruy would conduct interview in both languages. Your point about Sudburian francophones historically voting Liberal is interesting. As you probably know, the surrounding communities have a considerably higher proportion of francophones, and those areas lean NDP.

      Delete
  8. Forum does not say on its web site that it calls cell phones but Lorne Bozinoff claimed the firm did in an interview with the Toronto Star last summer. They ought to be explicit about this in their statement of methodology.

    See: http://www.thestar.com/news/city_hall/toronto2014election/2014/08/09/confused_about_political_polling_here_are_some_answers.html

    ReplyDelete
    Replies
    1. I don't doubt that Forum does call cell phones in some of their polls. But I seem to recall Forum saying they don't call cell phones for by-election polling - though I'm not sure if I am misremembering that.

      Delete
  9. I find it very sad to hear of a 'dont vote' movement. Any of those are just self defeating - if few vote then the established parties are extremely happy as it means that the most important thing is to get your own voters out and they are good at that. Far better to either support the independent candidate or go for one of the parties that were not involved in the twin messes (the NDP whose member quit right after an election, the Liberals who poached a federal NDP member thus forcing another byelection). I'm a Green supporter so I was hoping voters there would say 'screw the big 3, lets send a strong message we are sick of business as usual' but instead it looks like efforts to not vote will result in a Liberal win, which will just support business as usual. Sigh.

    ReplyDelete
    Replies
    1. I wonder how many 'don't voters' understand they're working for the status quo/Conservatives... then again, maybe some of them are being funded...

      Delete
    2. I think most are just fed up with corruption and don't see the point in voting. Problem is, the more you stay away the worse it will get. The scariest thing to a party in power is an engaged electorate.

      Delete
    3. exactly, which is why Harper & Co. have been undermining voting accessibility, etc.

      Delete
  10. Given that new federal poll from Forum, I wonder if they've recently changed their methodology.

    ReplyDelete
    Replies
    1. Why do you say that? Lately, they've had the Conservatives higher than anyone else, so not surprising they have them ahead by one point.

      Delete
    2. The 12 highest Liberal results since the last election have all been Forum polls. Forum tends to rate the Liberals very highly, and on this one they've fallen neatly amidst the cross-pollster average.

      Two polls ago, Forum showed the Liberals at 41.

      Delete
    3. That's true, but they've also been the highest for the Conservatives over the last few months, averaging about 34% compared to 31% for other pollsters. They've averaged 37% for the Liberals, compared to 35% for everyone else.

      They have also been among the most volatile. I think where they stand out is the very high numbers for CPC/LPC combined. They have been the worst for the NDP of late, also the Greens. I wonder if they are applying their turnout model...

      Delete
    4. A question about Forum, why have the last 2 Forum polls had Andre Arthur winning Portneuf-Jaques-Cartier back? Even when the NDP were at their lowest levels in Forum polling they never had Arthur winning, but all of a sudden they do in 2 polls?

      Delete
    5. Pietro,

      I can only speculate as to Forum's motivations. However, the Conservative party is up 2-3% since Beaulieu became BQ leader. Arthur is known as a conservative so perhaps Forum is assuming a rising Cionservative tide lifts all conservative boats.

      Delete
    6. If I had to guess, Forum is doing what I used to do. That is, adjust Arthur's numbers as if he is a Conservative. But it is a little silly for them to make a point of it, since there is absolutely no indication that Arthur is running again. Why not just count that riding as Conservative and avoid making such an odd statement?

      Delete
  11. FYI, its actually quite easy to incorporate cell phone numbers into an IVR (or live operator) survey in a riding that encompasses a billing area - such as Sudbury. So i suspect that Forum does call cell numbers - in any case cell users tend to be much younger and skew NDP - so excluding all cells would if anything make the Liberals look stronger not weaker

    ReplyDelete
    Replies
    1. Ah, thanks for that. I'm not sure that it would help the NDP, though. It might in a downtown riding, but not so sure about a northern Ontario riding with a long NDP history. I don't think northern Ontario votes NDP because they have a lot of young cell phone users...

      Delete
    2. Does the riding encompass the billing area? Isn't the city rather larger than the riding, including a chunk of the Nickel Belt riding as well as the eponymous riding?

      Delete
  12. No but NDP support does skew younger than Liberal support...and there are just as many people with cell phones in Sudbury as in downtown Toronto

    ReplyDelete
  13. Any thoughts on the Mainstreet poll released today?

    http://www.mainstreettechnologies.ca/sudbury-race-closes/

    ReplyDelete
    Replies
    1. Check out my CBC article tomorrow.

      Delete
  14. My take is since a NDP MPP resigned, the NDP is unlikely to win the riding. A portion of voters will want to punish the NDP for causing an unneeded by-election and wasting voters votes at the election. Normally this would be enough to guarantee a Liberal victory. I don't live in the ridings so I have no knowledge as to whether enough local popularity exists for Andrew Oliver to become a M.P.P. from my reading of the tea leaves it doesn't appear Oliver has much of a hope unless he can capture NDP voters.

    ReplyDelete
    Replies
    1. My take is that there are four type of voters that can go for Olivier.

      1. Supporters and sympathizers of Olivier. (A locked vote.)

      2. Liberals who are upset at how the ex-NDP Thibeault was nominated as candidate for their party.

      3. People who traditionally lean towards NDP, but want to punish the NDP for two resignations.

      4. Tories who may support Olivier as the best candidate to beat both the Liberals and NDP.

      Delete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.