Wednesday, July 17, 2013

Forum's by-election track record

Five by-elections are scheduled for August 1 in Ontario, and all five of them are setting up to be interesting races. At least four of them could be considered toss-ups, while the fifth could mean defeat for the incumbent. As by-elections go, these are not bad for a hot summer.

Because of these by-elections, we have heard (and will continue to hear) much from Forum Research. The firm is - by far - the most publicly active political polling firm in the country, and it never misses an opportunity to poll for a by-election (it even put out three for the foregone conclusion by-election in Westside-Kelowna). It has already put out some new numbers for these Ontario races, so it might be a good idea to take a look at the firm's track record in by-elections.

Forum releases a good amount of data with their polls, showing unweighted sample sizes. As I've discussed before, this gives us a good idea of how representative the raw sample is, but tells us nothing about what Forum does with that sample as weighted numbers are not provided. In some cases, it seems that a lot of tweaking needs to be done in order to get the sample sizes right. Unfortunately, the firm is reluctant to provide details about how they go about building its samples and what it does with them. As this is proprietary information, that is within their right - but hopefully they are telling their media partners about it before the numbers are published.

Instead, when asked Forum relies on its own track record which, admittedly, is pretty good. Here is how Forum has performed in the last year of by-elections (the 'Days' column refers to the number of days between the poll and the by-election):

CLICK TO MAGNIFY
As you can see, in these seven by-elections Forum has yet to pick the wrong winner or even get the order of the parties mixed up, even in difficult by-elections like Kitchener-Waterloo or Calgary Centre. But if we take a very critical eye to the firm's performance, we see that only their call in Vaughan for the top four parties had every number within the margin of error.

For those by-elections where the final poll was taken a week or two before the vote, that is more than forgivable. And even then Forum did rather well, showing the Greens to have real strength in Victoria, calling the top three parties within 2.7 points in Calgary Centre, and pegging the NDP as second and the Liberals as third in Durham.

However, the record is not completely clean. Even with a small sample of 177 decided voters, Forum was outside of the margin of error in Westside-Kelowna for the B.C. Liberals, under-estimating them by 10.7 points. The 6.3-point miss on the NDP's vote was within the margin of error, but is still rather substantial. In Labrador, the NDP was over-estimated by almost five points and in Kitchener-Waterloo the Progressive Conservatives were under-estimated by almost six.

Overall, however, the record is rather good for the difficult task of calling by-elections. With the exception of Westside-Kelowna, Forum's polls taken within a few days of the vote have done well. On average, Forum's by-election polls have had a margin of error of +/- 3.5 points for each party (and +/- 3.2 points in polls taken a few days before the election).

What does this tell us about the five by-elections in Ontario?

It certainly suggests that four of them are complete toss-ups. If we apply the +/- 3.5-point margin of error to their latest set of polls, we see that the Tories could potentially win four of the five (or none), the Liberals could retain three (or none), and the NDP is in the running for two (and at least one).

Windsor-Tecumseh is the only one that does not seem to be in question, as the New Democrats range between 48.5% and 55.5% of the vote, compared to 18.5% to 25.5% for the Progressive Conservatives. 

The next 'easiest' call would have to be London West, where the Tories would have between 32.5% and 39.5% of the vote, compared to a band of 25.5% to 32.5% for the NDP and 20.5% to 27.5% for the Liberals.

Despite Doug Holyday's candidacy, Etobicoke-Lakeshore actually appears to be the best shot the Liberals have at retaining a seat, with a range of between 41.5% to 48.5% of the vote to 35.5% to 42.5% for Holyday. In Scarborough-Guildwood, the +/- 3.5-point margin gives the Liberals 35.5% to 42.5%, compared to 30.5% to 37.5% for the Tories.

By this measure, Ottawa South is the closest riding, with the Liberals at 38.5% to 45.5% of the vote against 34.5% to 41.5% for the Progressive Conservatives. However, the Forum poll suggests its sample might be skewed towards the Tories, as 39% of respondents said they voted for the party in 2011, while only 40% said they voted for the Liberals. In fact, the Liberals took 49% of the vote to 33% for the PCs.

Undoubtedly, we will hear more from Forum before the voting begins on August 1. In Windsor-Tecumseh, the question is whether the New Democrats can hold their lead. In the rest of the by-elections, the results will be closely watched as none of them have an obvious winner.

2 comments:

  1. You need to factor in the rounding error - Forum rounds to the nearest whole number which adds a rounding error of +-0.5pp

    Meanwhile, as to the scale of their error, they are working with a bell curve and the assumption that they are getting a representative sample of the public, but when you look at the details of their work you can see that they almost always over sample for the older people and under sample for younger people based on census demographics. They weight their results and doing see decreases the effective sample size and raises the margin of error for a result within the 95% curve on the graph.

    My best estimate is that their weighting cuts their effective sample size by about half. This means that what should be reported as their true margin of error with rounding included would be much higher. It also means that their results tend be only slightly better than dead reckoning. When I work through it for the last poll for Westside-Kelowna I get an error of +-11.5pp

    ReplyDelete
  2. I'm not an NDPer, but I thought I'd just say that the NDP candidate for Etobicoke-Lakeshore (PC Choo) is great candidate and an extremely capable individual. He would be a great MPP, and I'd encourage everyone in the riding left or right or in between to give your consideration.

    ReplyDelete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. Please keep discussion on topic.