Mixed results for polls in Ontario by-elections

Two polling firms were very active during the Ontario by-election campaigns: Campaign Research and Forum Research. Their polls had a very mixed performance, with two races being called well, one having the broad strokes called accurately but not the precise numbers, and two being called poorly. That gives them a grade of 2.5 out of 5, straddling the pass/fail mark.

But before getting to how the polls did, how did ThreeHundredEight do? The By-Election Barometer was designed to forecast outcomes in general terms in by-elections where precise, reliable information is hard to come by. On that score, the Barometer extended its streak without a missed call to 18 in a row. Scarborough-Guildwod was considered Likely Liberal and stayed Liberal. Windsor-Tecumseh was considered Strong New Democrat, and it was won in a landslide. Etobicoke-Lakeshore and Ottawa South were considered PC/OLP toss-ups, and they were both close races won by the PCs and OLP, respectively. London West was considered a three-way toss-up, and the NDP took it.

The Barometer also gives an indication of how the parties met expectations. The New Democrats won both of the ridings they were considered to be at play in (and for awhile looked like they could take a third in Scarborough-Guildwood), the Liberals went 2 for 4, while the Progressive Conservatives went only 1 for 4. In the end, though, all five were Liberal seats - and generally safe ones at that.

Compared to the polling, ThreeHundredEight did well with my weighted averages (which take into account survey date, sample size, and firm track record) having a lower average error than either Forum or Campaign. The simple turnout model, however, performed poorly: it had a worse average error than either Forum or Campaign. It makes me doubt whether it is worth its salt (it didn't have much salt to begin with, though).

Now, to the polls themselves.

Etobicoke-Lakeshore was the race that was called the best last night, at least in general. Forum had a total error of 4.3 points, significantly better than Campaign's 9.9 points. ThreeHundredEight's weighted average, with an error of only 3.5 points, was the best performer. The turnout model did as poorly as Campaign, though unlike Campaign it still had Doug Holyday as the winner. While the results for Campaign were not far from their forecast, having the winner wrong is a problem. Their under-estimation of PC support puts them just on the edge of their own margin of error, and this was their biggest issue.

Turnout was roughly 39% in the riding (going by the total number of eligible voters from the 2011 election, rounding down to take into account population growth since then).

This was a big win for the Progressive Conservatives, since it gives them a seat in Toronto and a big name in caucus. But who won this riding: Doug Holyday or Tim Hudak? Considering the poor performance of the Tories in other ridings, the answer seems pretty clear. The Liberals put up a decent performance, but did lose about nine points here from the 2011 election. Drop them by nine points throughout Toronto and they are in trouble.

London West was not polled very well, as neither Forum not Campaign had the New Democrat candidate Peggy Sattler as the winner. Forum was the closest, however, with a total error of 14 points against 16.9 points for the weighted average, 17 points for the turnout model, and 22.8 points for Campaign.

But the feather in Forum's cap in London West is not a good one. The result for the NDP and PC candidates was outside the margin of error, whereas Campaign only had the New Democrats outside the margin of error out of the major parties. The turnout model did suggest that the NDP had room to grow, but the Tories should not have been boosted. All in all, a miss.

Turnout was roughly 38% in this riding.

This is a big win for the New Democrats, who have won all three of the by-elections that have taken place in southwestern Ontario since the last provincial election. They gained about 20 points, while the Liberals lost about 30. This was a big defeat for them, as Ken Coran was supposed to be a star candidate. Instead, he had the second worst Liberal performance of the night.

Ottawa South was the worst polled riding of the five, with the polls suggesting Matt Young would win easily against John Fraser. Instead, Fraser won by just under four points.

Campaign did better than Forum with a total error of 19.2 points, but they had Young ahead by seven points. The weighted average performed second best, with a total error of 21.9 points, while Forum had a total error of 26.6 points. The turnout model was the worst of all - it had the Tories ahead by 21 points!

This was a complete miss by Forum, as the results for the Liberals, PCs, and NDP were outside the margin of error. Campaign had only the Liberals outside the margin of error of the three major parties, so it was not a complete miss by them. But it was not a good performance by any stretch of the imagination.

Turnout in Ottawa South was roughly 40%, the highest of the night.

The Liberals have to be happy with this win. To many, a loss here would have represented a repudiation of the McGuinty years as well as a refusal to accept that the new government of Kathleen Wynne is a different beast. The Liberals' loss of support was relatively modest. For the Tories, this was a strong performance - but they had hopes of taking it and from that perspective it is a blow. If they can't win McGuinty's old riding when campaigning against McGuinty's memory, they may have to change their messaging.

Scarborough-Guildwood was well served by the polls, with Forum coming the closest with a total error of only 6.4 points. Campaign had a total error of 8.8 points, their best performance of the night. In fact, this was the only riding in which both Forum and Campaign did better than ThreeHundredEight's weighted average.

This was a good call by Forum in that the four major parties were called within the margin of error. Unfortunately for Campaign, however, Adam Giambrone's performance for the NDP was outside the margin of error. The turnout model suggested that the NDP could do better than Forum had envisioned, but it was over-zealous.

Turnout in this riding was roughly 36%.

A win is a win and the Liberals (and certainly Mitzie Hunter) will take it, but they did drop about 13 points in this riding. The New Democrats picked up nine, which is a strong performance for them. With the PCs also taking Etobicoke-Lakeshore, it puts the Liberals on notice that their Toronto seats are far from secure.

Windsor-Tecumseh was not well polled (the total error was actually worse than London West), but unlike the other misses it did not mislead people. The New Democrats won it easily, as the polls suggested. But that doesn't mean the polls were close.

The weighted average was the best performer, off by 17.4 points. Campaign was off by 18.7 points, while Forum was off by a total of 20.7 points. Worse, their results for the NDP and Tories were outside the margin of error. Campaign was outside of the margin of error for the NDP and Liberals. It was a poor performance by the polls, but at least no one in Windsor-Tecumseh was surprised by the results.

For that reason, perhaps, turnout was the lowest here at around 30%.

Because this win was expected by Percy Hatfield and because the riding votes NDP federally, it does not look as impressive as it should. But the Liberals lost 30 points to the NDP, and it seems that with Dwight Duncan gone the Windsor area is firmly in the NDP camp at the provincial and federal level. Unlike, say, London West or Kitchener-Waterloo, this is a riding that will almost certainly remain an NDP fortress going forward.

Overall, ThreeHundredEight's weighted average was the best performer, with a total average error of 13.9 points per riding, or 2.8 points per party (including Others). Forum Research was close behind, with a total average error of 14.4 points, or 2.9 points per party.

Campaign Research had an average error of 15.9 points or 3.2 points per party, while the turnout model was the worst performer. It had an average error of 18.8 points per riding, or 3.8 points per party.

The turnout model had improved every Forum election-eve poll they had done in the recent past, but it made them all worse here. Similarly, turnout went against the grain. It averaged 38% in the ridings retained by the government, whereas it averaged 36% in those won by the opposition. My research for yesterday's Globe article showed that turnout was, on average, quite a bit lower in by-elections won by the incumbent. It was a topsy-turvy night.

It is hard to spin these results as a good performance for the polls. It was not. While Etobicoke-Lakeshore and Scarborough-Guildwood were well-polled, 2 for 5 is a poor performance (especially in light of the 19 times out of 20 caveats that go along with margins of error). Forum did well in Etobicoke-Lakeshore and Scarborough-Guildwood, but had poor performances in London West and Windsor-Tecumseh and a very bad performance in Ottawa South. Campaign had a generally bad night, as in addition to the misses in London West, Ottawa South, and Windsor-Tecumseh, they had the Liberals winning Etobicoke-Lakeshore and were outside of the margin of error for the NDP in Scarborough-Guildwood.

But by-elections are not easy things to poll, so this shouldn't be taken as another indictment of the industry. The sample sizes were very small, and the response rates very low. Both firms used IVR technology, polling in one evening over a couple of hours. In Campaign Research's reports they mentioned how they needed to call well over 20,000 households to get a few hundred responses, a response rate of under 2%. Undoubtedly, Forum had a similar rate of response. Regional variations in behaviour and availability at the national level probably even things out more, but in a by-election those 2% might be a very different set of people than the rest of the riding. But this is a problem for the pollsters to figure out - they shouldn't put out numbers if they know them to be potentially faulty.

For my part, I'm pleased that the By-Election Barometer continues to give a good, broad-strokes overview of by-elections and that it does not rely too heavily on individual polls (for instance, it still had the Liberals favoured in Ottawa South). It will continue to operate, and hopefully its success will continue in Toronto Centre, Bourassa, and Provencher. But the turnout model may be turfed.

Ontario Liberals under pressure in all five by-elections

By-elections are being held in five ridings across Ontario today, from Lake St. Clair in the southwest to the Ottawa River in the east. All five ridings belong to the Ontario Liberals, and all five are at risk of being won by another party by the time the votes are counted tonight.

The By-Election Barometer considers one of the ridings, Scarborough-Guildwood, most likely to remain a Liberal hold, while Windsor-Tecumseh is almost certain to go with the New Democrats. The ridings of Etobicoke-Lakeshore and Ottawa South could go to either the Liberals or the Progressive Conservatives, while London West is considered a three-way toss-up.

But the By-Election Barometer was designed to give people an idea of what to expect in by-elections where little specific information is available. We rarely see by-election polls, so there is never much to go on. But in these five by-elections, we've been treated to a wealth of polling data (25 polls in all). Considering that these polls represent better information than the Barometer itself is supposed to represent, we must consider them much more convincing evidence.

That being the case, let's look at each of the five ridings at stake tonight and what the polls (and the Barometer, which does take them into account) are saying.

(Campaign Research's final polls can be seen here. The final polls from Forum are currently on the first page of their news archive - if you are looking at this post after the by-elections have taken place they will have moved down the list at Forum's site. These polls were all released and made public yesterday, before the new-poll blackout imposed by Elections Ontario.)

We'll start with the riding most likely to remain in incumbent hands: Scarborough-Guildwood. The By-Election Barometer considers it a Likely Liberal win, estimating that Mitzie Hunter has a 75% chance of taking it, against 25% for Ken Kirupa of the Progressive Conservatives.

The polls appear to agree with this assessment. In the five polls done in the riding since the beginning of July, Hunter has managed between 37% and 40% support - a very tight grouping. Her weighted average (which employs the same methodology this site uses during elections) gives her 37.9% of the vote, compared to 31.4% for Kirupa. His support has also been tightly grouped, at between 30% and 32% in the last four polls.

Adam Giambrone of the New Democrats is averaging 25% support, but the latest Forum poll suggests he could have some momentum. The Greens have averaged 4.4%, while support for other parties and candidates has averaged 1.3%.

Applying ThreeHundredEight's turnout model to the Forum poll, however, paints a very different picture. Support for the PCs, Greens, and others is hardly changed, but Hunter drops to 32% and Giambrone increases to 32%, putting the three parties in a three-way tie. Applying the turnout model has improved all of Forum's election-eve polls, so if that trend continues we should expect Giambrone to perform better than the 27% he was awarded in Forum's last poll. (Note: the turnout model cannot be applied to Campaign Research's polls, as they do not release support breakdowns by age.)

The riding in which the Liberals have the next best chance of retaining is Etobicoke-Lakeshore, which the By-Election Barometer considers an OLP/PC Toss-up. The Liberals are given a 56% chance of holding on to it, against 44% for the Progressive Conservatives. But here again, we must consider the riding polls as more revealing information.

They reveal that the race has tightened in the final days of the campaign. Peter Milczyn of the Liberals had fallen behind in mid-July, with Campaign Research putting the deficit at five points and Forum putting it at seven points. Both of their most recent polls show that Doug Holyday's support has remained unchanged, while Milczyn's has increased by a few points, narrowing the gap to four points in Forum's estimation and giving Milczyn the edge by one according to Campaign. Holyday is still ahead in the weighted average, however, with 45.7% to Milczyn's 42.2% and P.C. Choo's 8% for the NDP. The Greens are averaging 3.6% while support for other parties has averaged 0.5%.

This is one riding where the calls made by Forum and Campaign differ. I will assess how the polls (and this site) did in the by-elections tomorrow, and this is one of the ridings that will put the two firms to the test.

The turnout model suggests Holyday has it, improving his vote haul to 49%. Milczyn's also improves to 44%, while the NDP and Greens drop to 4% and 2%, respectively.

The Ontario Liberals' hope of winning a third riding starts to decrease significantly as we turn to Ottawa South, formerly the riding of Dalton McGuinty. The By-Election Barometer does give the Liberals a 55% chance to the Tories' 45%, but the riding-level polling suggests that this is far too kind to Liberal candidate John Fraser.

Fraser has not led in a poll since the beginning of July, when he was ahead by four points. In every poll since the deficit has increased. Campaign put it at one point in mid-July, but now reckons it to be seven points. Forum put it at 14 points on July 24, but now sees it as a gap of 16 points in favour of Matt Young.

Young is averaging 48.7% in the polls, compared to 35.4% for Fraser, 10.6% for Bronwyn Funiciello, 4.1% for the Greens, and 1.3% for the other candidates. If Fraser manages to win the riding - and it is still generally considered a toss-up, no matter what the polls say - he will have proven the polls wrong. Either Campaign or Forum will miss the call to some extent, however, because the gap between their numbers is so large. This will be the riding that gives either one of these firms the 'win'.

The turnout model suggests Young's advantage is under-stated in the polls, as his support is increased to 55% against 34% for Fraser. In other words, the turnout model believes that if Forum is going to miss the call, it will be because they didn't give Young enough of a lead. That seems hard to swallow.

From here, we move to ridings where the Liberals are not realistically in the running to hold on to their seats. The By-Election Barometer still considers the OLP a factor in London West, however, due to the party's history in the riding and their decent polling numbers in the southwestern part of the province. It gives them a 39% chance of winning, compared to 35% for the Tories and 27% for the NDP, making it an OLP/PC/NDP Toss-up. That seems far too optimistic for the Liberals, who have been tanking in the polls.

Liberal candidate Ken Coran hasn't been a factor in the polls since he took up the Liberal banner. The poll from February giving the OLP 30% was before his candidacy was known. He has been dropping during the race, and has averaged 16.5%.

The real contest is between PC candidate Ali Chahbar and NDP candidate Peggy Sattler. Chahbar's numbers have been pretty steady in the mid-30s, while Sattler's were consistently around 30%. But the last Campaign poll put the gap at three points, and Forum put it at only two points. The weighed average gives Chahbar 36.6% to 33.5% for Sattler. That makes it the closest race of the night. Support for other parties and independents (Al Gretzky, primarily) has averaged 7.1%, while the Greens have been at 6.4%.

The turnout model gives Chahbar a slightly wider lead than the Forum poll suggests, with his support boosted to 40% against 37% for Sattler. But that is still very close. It seems London West will be the race to watch.

The race that will not be the one to watch is Windsor-Tecumseh, which everyone concedes will be won handily by the New Democrats. The By-Election Barometer considers the riding the only Strong one on offer tonight, giving the NDP a 95% chance of winning (the OLP gets the remaining 5%, based on their history and nothing else).

Percy Hatfield of the NDP has had majority support in every poll since his candidacy was made official, and he's averaging 52.7% going into tonight's vote. The PCs have placed second in the last three polls, and look set to take 24.5% of the vote. The Liberals are well behind, averaging 13.9%. The Greens and others are averaging 5.8% and 3.2%, respectively.

The turnout model suggests both Hatfield and Robert de Verteuil could take a higher proportion of the vote, boosting Hatfield to 54% and de Verteuil to 30%. But it seems that the NDP should have no trouble taking the riding.

So the Liberals will be under a lot of strain tonight. They have a good shot (but by no means a certain one) of retaining one of the ridings, and are in the running in at least one more, and possibly two more. It means they are expected to win one riding, but could win three or none. The Progressive Conservatives are set to be tonight's big winners as they are well positioned to win as many as three ridings, but none of these are slam dunks. They could potentially be shutout if the Liberals have a good night. The New Democrats will win Windsor-Tecumseh and put up good numbers in London West and Scarborough-Guildwood, so it seems they will be under the least strain as they watch the votes come in, since they can safely assume they will have a decent night.

But will it be a good night for Forum Research and Campaign Research? They have put their necks out on this one, with polls in the final days of the campaign. They should be able to call the race correctly, or at least ballpark it considering the unpredictable turnout levels in a mid-summer campaign. Tomorrow, I'll take a good look at how they performed.