Friday, August 2, 2013

Mixed results for polls in Ontario by-elections

Two polling firms were very active during the Ontario by-election campaigns: Campaign Research and Forum Research. Their polls had a very mixed performance, with two races being called well, one having the broad strokes called accurately but not the precise numbers, and two being called poorly. That gives them a grade of 2.5 out of 5, straddling the pass/fail mark.

But before getting to how the polls did, how did ThreeHundredEight do? The By-Election Barometer was designed to forecast outcomes in general terms in by-elections where precise, reliable information is hard to come by. On that score, the Barometer extended its streak without a missed call to 18 in a row. Scarborough-Guildwod was considered Likely Liberal and stayed Liberal. Windsor-Tecumseh was considered Strong New Democrat, and it was won in a landslide. Etobicoke-Lakeshore and Ottawa South were considered PC/OLP toss-ups, and they were both close races won by the PCs and OLP, respectively. London West was considered a three-way toss-up, and the NDP took it.

The Barometer also gives an indication of how the parties met expectations. The New Democrats won both of the ridings they were considered to be at play in (and for awhile looked like they could take a third in Scarborough-Guildwood), the Liberals went 2 for 4, while the Progressive Conservatives went only 1 for 4. In the end, though, all five were Liberal seats - and generally safe ones at that.

Compared to the polling, ThreeHundredEight did well with my weighted averages (which take into account survey date, sample size, and firm track record) having a lower average error than either Forum or Campaign. The simple turnout model, however, performed poorly: it had a worse average error than either Forum or Campaign. It makes me doubt whether it is worth its salt (it didn't have much salt to begin with, though).

Now, to the polls themselves.

Etobicoke-Lakeshore was the race that was called the best last night, at least in general. Forum had a total error of 4.3 points, significantly better than Campaign's 9.9 points. ThreeHundredEight's weighted average, with an error of only 3.5 points, was the best performer. The turnout model did as poorly as Campaign, though unlike Campaign it still had Doug Holyday as the winner. While the results for Campaign were not far from their forecast, having the winner wrong is a problem. Their under-estimation of PC support puts them just on the edge of their own margin of error, and this was their biggest issue.

Turnout was roughly 39% in the riding (going by the total number of eligible voters from the 2011 election, rounding down to take into account population growth since then).

This was a big win for the Progressive Conservatives, since it gives them a seat in Toronto and a big name in caucus. But who won this riding: Doug Holyday or Tim Hudak? Considering the poor performance of the Tories in other ridings, the answer seems pretty clear. The Liberals put up a decent performance, but did lose about nine points here from the 2011 election. Drop them by nine points throughout Toronto and they are in trouble.

London West was not polled very well, as neither Forum not Campaign had the New Democrat candidate Peggy Sattler as the winner. Forum was the closest, however, with a total error of 14 points against 16.9 points for the weighted average, 17 points for the turnout model, and 22.8 points for Campaign.

But the feather in Forum's cap in London West is not a good one. The result for the NDP and PC candidates was outside the margin of error, whereas Campaign only had the New Democrats outside the margin of error out of the major parties. The turnout model did suggest that the NDP had room to grow, but the Tories should not have been boosted. All in all, a miss.

Turnout was roughly 38% in this riding.

This is a big win for the New Democrats, who have won all three of the by-elections that have taken place in southwestern Ontario since the last provincial election. They gained about 20 points, while the Liberals lost about 30. This was a big defeat for them, as Ken Coran was supposed to be a star candidate. Instead, he had the second worst Liberal performance of the night.

Ottawa South was the worst polled riding of the five, with the polls suggesting Matt Young would win easily against John Fraser. Instead, Fraser won by just under four points.

Campaign did better than Forum with a total error of 19.2 points, but they had Young ahead by seven points. The weighted average performed second best, with a total error of 21.9 points, while Forum had a total error of 26.6 points. The turnout model was the worst of all - it had the Tories ahead by 21 points!

This was a complete miss by Forum, as the results for the Liberals, PCs, and NDP were outside the margin of error. Campaign had only the Liberals outside the margin of error of the three major parties, so it was not a complete miss by them. But it was not a good performance by any stretch of the imagination.

Turnout in Ottawa South was roughly 40%, the highest of the night.

The Liberals have to be happy with this win. To many, a loss here would have represented a repudiation of the McGuinty years as well as a refusal to accept that the new government of Kathleen Wynne is a different beast. The Liberals' loss of support was relatively modest. For the Tories, this was a strong performance - but they had hopes of taking it and from that perspective it is a blow. If they can't win McGuinty's old riding when campaigning against McGuinty's memory, they may have to change their messaging.

Scarborough-Guildwood was well served by the polls, with Forum coming the closest with a total error of only 6.4 points. Campaign had a total error of 8.8 points, their best performance of the night. In fact, this was the only riding in which both Forum and Campaign did better than ThreeHundredEight's weighted average.

This was a good call by Forum in that the four major parties were called within the margin of error. Unfortunately for Campaign, however, Adam Giambrone's performance for the NDP was outside the margin of error. The turnout model suggested that the NDP could do better than Forum had envisioned, but it was over-zealous.

Turnout in this riding was roughly 36%.

A win is a win and the Liberals (and certainly Mitzie Hunter) will take it, but they did drop about 13 points in this riding. The New Democrats picked up nine, which is a strong performance for them. With the PCs also taking Etobicoke-Lakeshore, it puts the Liberals on notice that their Toronto seats are far from secure.

Windsor-Tecumseh was not well polled (the total error was actually worse than London West), but unlike the other misses it did not mislead people. The New Democrats won it easily, as the polls suggested. But that doesn't mean the polls were close.

The weighted average was the best performer, off by 17.4 points. Campaign was off by 18.7 points, while Forum was off by a total of 20.7 points. Worse, their results for the NDP and Tories were outside the margin of error. Campaign was outside of the margin of error for the NDP and Liberals. It was a poor performance by the polls, but at least no one in Windsor-Tecumseh was surprised by the results.

For that reason, perhaps, turnout was the lowest here at around 30%.

Because this win was expected by Percy Hatfield and because the riding votes NDP federally, it does not look as impressive as it should. But the Liberals lost 30 points to the NDP, and it seems that with Dwight Duncan gone the Windsor area is firmly in the NDP camp at the provincial and federal level. Unlike, say, London West or Kitchener-Waterloo, this is a riding that will almost certainly remain an NDP fortress going forward.

Overall, ThreeHundredEight's weighted average was the best performer, with a total average error of 13.9 points per riding, or 2.8 points per party (including Others). Forum Research was close behind, with a total average error of 14.4 points, or 2.9 points per party.

Campaign Research had an average error of 15.9 points or 3.2 points per party, while the turnout model was the worst performer. It had an average error of 18.8 points per riding, or 3.8 points per party.

The turnout model had improved every Forum election-eve poll they had done in the recent past, but it made them all worse here. Similarly, turnout went against the grain. It averaged 38% in the ridings retained by the government, whereas it averaged 36% in those won by the opposition. My research for yesterday's Globe article showed that turnout was, on average, quite a bit lower in by-elections won by the incumbent. It was a topsy-turvy night.

It is hard to spin these results as a good performance for the polls. It was not. While Etobicoke-Lakeshore and Scarborough-Guildwood were well-polled, 2 for 5 is a poor performance (especially in light of the 19 times out of 20 caveats that go along with margins of error). Forum did well in Etobicoke-Lakeshore and Scarborough-Guildwood, but had poor performances in London West and Windsor-Tecumseh and a very bad performance in Ottawa South. Campaign had a generally bad night, as in addition to the misses in London West, Ottawa South, and Windsor-Tecumseh, they had the Liberals winning Etobicoke-Lakeshore and were outside of the margin of error for the NDP in Scarborough-Guildwood.

But by-elections are not easy things to poll, so this shouldn't be taken as another indictment of the industry. The sample sizes were very small, and the response rates very low. Both firms used IVR technology, polling in one evening over a couple of hours. In Campaign Research's reports they mentioned how they needed to call well over 20,000 households to get a few hundred responses, a response rate of under 2%. Undoubtedly, Forum had a similar rate of response. Regional variations in behaviour and availability at the national level probably even things out more, but in a by-election those 2% might be a very different set of people than the rest of the riding. But this is a problem for the pollsters to figure out - they shouldn't put out numbers if they know them to be potentially faulty.

For my part, I'm pleased that the By-Election Barometer continues to give a good, broad-strokes overview of by-elections and that it does not rely too heavily on individual polls (for instance, it still had the Liberals favoured in Ottawa South). It will continue to operate, and hopefully its success will continue in Toronto Centre, Bourassa, and Provencher. But the turnout model may be turfed.

34 comments:

  1. I think turnout is a much harder beast to predict in by-elections than it is in the general. Larger populations tend to have a smoothing effect on odd swings, don't they?

    ReplyDelete
  2. I appreciate the care that you've taken here to be transparent with what went right and what went wrong with the forecast. This is clearly another example of bad polling preventing your forecast model from being accurate.

    Have you considered doing for by-elections what you do for general elections, ie. reporting the vote forecast as a band of possible outcomes while highlighting the outcome that is regarded as most probable? That might be a good way to show the uncertainty introduced by small sample sizes and the very low rates of response.

    I would agree with you that the turnout model may need to be jettisoned. It may be a case of over-fitting the model to the data.

    ReplyDelete
    Replies
    1. It isn't over-fitting, because I actually came up with the (extremely simple) formula and then applied it. In other words, I didn't use the results to come up with the formula - it just worked.

      I think I'd still like to see how it would do in a general election, but I'll definitely give it little prominence after the lessons of last night.

      I'd disagree, though, that my forecasts weren't accurate. The polls were off in a few ridings so the polling average was off, but the forecasts themselves (i.e., who would likely win, which is about as much as you can do in a by-election) were all good.

      Delete
    2. That's a fair point about accuracy. Maybe the better way to put it is that the bad polling kept your forecast from being precise. Of course the most important thing is to pick the correct winner. Precision is gravy.

      Delete
    3. I had hoped with all the polls that precision could be possible, which is why I posted the weighted averages. But, in the end, I was just reminded why I set up the Barometer to be the way it is.

      Delete
    4. I disagree that "the most important thing is to pick the correct winner". You are trying to predict how many people will vote for each candidate - in a close election, the accuracy required to turn that into a predicted winner might be unreachable, but if you predict that candidate A and B will each have 40 +/- 1% of the vote, you've done just as fantastic a job if it's A:41%/B:39% as if it's B:41%/A:39%.

      That said, low turnout dramatically increases the uncertainties, both because of the statistical error (i.e. with a by-election-like 37.5% turnout, the random sampling increases the uncertainty by about sqrt(2) for parties that could win compared to a good-national-election 75% turnout) and because it introduces systematic errors (those 50% of national-election-voters who vote in by-elections are a biased sample). So I'm not particularly surprised by the size of the errors.

      Delete
  3. With regards to Ottawa South it is obvious the polling firms and the PC's simply did not appreciate the strength of the Liberal machine, built up over decades, it could and did triumph over the Mcguinty problems !

    ReplyDelete
  4. Another point being missed on here and it may be the most important? The Left-Right balance.

    Based on yesterdays results the Left won big time and the Right got kicked !!

    NDP gets two, obviously they are very Left

    Libs get two, they are essentially Centre-Left

    PC's get one and they are way over on the Right.

    Good on the Left IMO

    ReplyDelete
    Replies
    1. Actually, if one follows your logic, Left lost one seat to Right yesterday (after all, all 5 seats were Left before yesterday, according to your definition), shifting the balance to the Right.

      But personally I dislike very much the Left/Right simplification - sounds too much like American model.

      Delete
    2. No First the Left retained four out of five seats.

      As to the American model that is very much what the CPC and their PC adherents are trying to do in Canada.

      That it backfired on them yesterday is very obvious !

      Delete
    3. These were all very safe Liberal seats in the general election. That any of them were even in play is a bad thing.

      Delete
    4. First off Peter,

      I have finally come around to your long held view that Hudak is incapable of winning the next election. The Tories should have won London West instead they lost by a significant margin. Hudak should do the honourable thing and resign the party's leadership much as IDS did in the UK.

      Secondly, the Liberal party is hardly a left wing party. Their links to Bay St. are long and nebulous. The Liberals are a centre-centre-right party that very occasionally throw a bone to the left or "progressives" in order to differentiate themselves from the Tories. One need only look to Ornge to see that the Liberals have followed a Thatcherite policy in terms of the provision of services. A left or "progressive" party would have kept ambulance services within the public service.

      Delete
  5. Eric, your regional swing for the final forum Ontario poll got it right every time. It seems like that is a better forecaster than the individual riding polls. It gives a general indication of how things will go on a normal night, and then, using local issues and candidates strength and gut feeling a relatively safe call can be made.

    For example, in London, it had the NDP up 5%. Then, taking a look at Ken Coran's controversies, and the strength of the NDP candidate, that swing can be strengthened.

    I kept that in mind when I made a call last night and got 4/5. The only one I missed was Etobicoke, but I underestimated Doug Holyday's appeal.

    I think a general Ontario poll is more useful than a by-election poll any day. Also, Forum is pretty good. They have done the best in most of the last few elections. Taking into consideration their results and the gut feeling about a candidate or party in a certain riding, I think a relatively safe call can be made.

    ReplyDelete
  6. Sorry but it is not fair to say that the model has "extended its streak of correct calls to 18 in a row." Five of those "calls" have been "toss-up" which is akin to making no call at all. By this logic anyone could have had a perfect 18/18 record simply by predicting that every byelection would be a toss-up.

    Even the toss-up calls were not always correct. London West was called a toss-up between OLP/PC/NDP; the result was 16%/33%/41%, hardly a tight three-party race. By contrast, Scarborough-Guildwood was called outright for the Liberals when the result was 36%/31%/28% OLP/PC/NDP - a much tighter spread.

    The contrast between the "PC/OLP" Etobicoke-Lakeshore call and the "OLP/PC" Ottawa South call suggests that the "toss-up" predictions are still supposed to indicate a more favoured party. Assuming the first-listed party is the actual call, the model has a 15/18 record. 83% success is good, and you can be proud of that, but it is not perfect.

    ReplyDelete
    Replies
    1. We must recognize what we don't know, and that is what the toss-up classification is about. I'm not here to make bets.

      And that classification is not some arbitrary call I make, it is based on my calculations. When I calculate the changes of a party winning the riding to be less than 60%, I consider it a toss-up because it is, at least according to the forecasting model. That one party might be slightly favoured over another is something I include for completeness, but I don't consider it to constitute a 'call'.

      But I do consider the listed parties as being part of the call. For example, in Argenteuil the model did not give the CAQ a chance to win, though due to having a star candidate they were considered to have a chance. If the CAQ had won the riding, but I had considered it a PQ/PLQ toss-up, I would not have considered it to be a correct call.

      Delete
    2. I'm inclined to agree with MF. It seems to me that your model essentially has a broader margin of error than the polls. With that kind of uncertainty, I don't think your model is that useful a predictive tool. You can't just sit on the fence and then say you were right after election night.

      Delete
    3. Most by-elections don't have polls. This is what the model is designed to tackle. These did, but in only one of the races (LW) did they give a better picture of the race than the Barometer did, and in OS the Barometer was a better reflection of the race than the polls.

      Delete
    4. I'd also ask that you consider everything I write about a race. Read what I wrote about London West on the Barometer page - no one reading that would have went into the night thinking the OLP had an equal chance of winning. I don't write the accompanying text to make the page look busy. One of the reasons I include the "Wildcard" in the Barometer is because some things can't be modeled.

      Delete
  7. I'm going to start a polling firm called "Chimpanzee Research Inc." My methodology will be to write the names of the top two placing political parties from the last election on two different bananas, and then throw the bananas into the chimpanzee enclosure at the zoo and see which one the chimpanzee eats first.

    I anticipate to on average call 2.5/5 races correctly, putting me on par with Forum and Campaign Research, but at a fraction of the cost. At least until they ban be from the zoo.

    ReplyDelete
    Replies
    1. Don't chimps have better things to do with their time?

      Delete
  8. Éric,

    If you don't consider a toss-up to be a "call" then you could say you have a 13/13 record, although I question the predictive value of a model where any projection of sufficient uncertainty to be called a toss-up is simply excluded from the win/loss ratio.

    I don't understand how you can measure "correct calls" (your term) in a situation where the "call" is a toss-up between two or three parties. It is clearly possible to determine whether the projection of a winner was accurate or not, but by what post-election metric do you evaluate whether a riding was a toss-up or not?

    For instance you stand by your projection having included the Liberals as contenders in London West's toss-up projection. They got 16% of the vote in that riding. By what post-election metric do you assert that that was a "correct call"?

    Likewise, if sub-60% odds of winning a riding mean a toss-up is projected, doesn't that mean that a firm projection of a race like Scarborough Guildwood that turned out to be very close is "wrong" in the sense that the projection should have been of a toss-up?

    It seems that either London West was "wrong" because it called a tight 3-way race when the Liberal was badly beaten, or Scarborough Guildwood was "wrong" because it called a Liberal win in a much tighter race, but they can't both be "right".

    I actually approve of your being more cautious with your projections, and approve of the concept of calling a "toss-up" where there is significant uncertainty about a result. But if you are going to hedge your bets by backing all the favourites in the same horse race, it isn't fair to boast about making the "correct call" when one of them wins.

    ReplyDelete
    Replies
    1. You are focusing more on the vote tally, which is not what the forecast is about. It is about trying to inform readers who to expect should win a riding - whether it be by 80 points or one point. When a riding is considered a 'Strong' one, for example, it doesn't mean that the party is expected to win comfortably, it just means that the party is expected to win. When a riding is toss-up, it doesn't mean that the margins should be close but that, with the evidence available, the riding could go either way.

      The forecast in London West never said that the Liberals could be close and lose, just that they could win. The forecast in Scarborough-Guildwood never said that the race would be easily won, just that the Liberals would win.

      My use of 'correct calls' is, though, perhaps not the best choice of words, as you suggest. That is why I took care to say that the Barometer has not made a wrong call on that particular page. I should have done the same here, and will change that.

      I'll take your point that a forecast where the three major parties are seen as likely winners is not very useful. I agree. But that sort of forecast is the equivalent of saying that the information isn't clear enough to make a clear call. I don't see the harm in that - and it is the model that is making that assessment, not me. If I put in my own subjective judgement, I would have dropped the OLP as a potential winner in London West. But I also might have done so in Ottawa South, which would have been a bad idea.

      By-elections are hard to call, the information is spotty, and forecasts are difficult to make with precision. That is what the Barometer is about.

      Delete
  9. I don't fault the accuracy of your model. In fact, it appears you're more accurate than the polls. The problem is that you're model is imprecise. While I understand your desire to play it safe after getting burned in BC, ultimately your results are less meaningful.

    ReplyDelete
    Replies
    1. With by-elections, I think that is the right approach.

      But in general, I see my role as trying to make sense of the polls and giving people an understanding of what they are saying, and what they aren't saying. My role isn't to make bets.

      Delete
  10. With regard to Ottawa South, the irony for Forum is that I think they DID have a poll in early July that predicted the basic outcome (eg. something like 42% for Fraser and 38% for Young). The only problem was, that poll was a month early! By the time of their final poll for Ottawa South, they had lost track of the race even though they had it correctly at the beginning!

    ReplyDelete
  11. Interesting. So basically the takeaway from everything in Ontario is that since the 2011 election gave the Liberals a minority, missing it by one seat, and have since then dropped from 50 seats to 47. Tories are now currently unchanged, after losing then gaining one. NDP have picked up 3 total.

    So I think this has got to be some encouraging news for Andrea Horwath, given that she is now creating a solid base in Southwestern Ontario, picking up 2 seats in pretty foreign territory (London West having slightly more NDP roots than Kitchener-Waterloo) and reverting a federal NDP seat to the provincial party (Windsor-Tecumseh). However, the other 3 results show still that if an election were held today, the NDP would still end up in third, even if they may end up in first in the SW (and probably the North too). They simply can't extend much past some extra SW gains, although their Scarborough-Guildwood suggested that at best they can match their 2011 Federal performance provicinally (roughly, which would net them a few seats from the Libs) particularly in the city of Toronto.

    What does tonight mean for the Libs and Tories? Well, it means if an election were held today, they'd roughly tie out. NDP would eat a little bit of their lunch in SW and maybe Toronto/North, but it'd either be a Lib minority of a PC minority. So not tremendously exciting for any party to go for a new election, except, maybe, the NDP (but even here it's debatable).

    ReplyDelete
    Replies
    1. You know where else the NDP picked up seats in between elections?

      Delete
  12. A few interesting quotes from Lorne Bozinoff in this Toronto Star story. Doesn't sound like a lot of introspection going on.

    http://www.thestar.com/news/queenspark/2013/08/02/ontario_byelections_pollsters_got_it_wrong_in_two_ridings.html

    ReplyDelete
  13. Virtually spot-on in Windsor-Tecumseh! Come on, Star...

    Yeah, that's disappointing.

    ReplyDelete
  14. Eric do you think NDP has a legitimate shot now taking Windsor West?

    ReplyDelete
  15. Judge bans burqa in London court

    21-year-old woman from Hackney is not allowed to enter a plea in court until she shows her face

    Guardian UK

    ReplyDelete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.