Forecasting and Projection Methodology

The following is a detailed explanation of the forecasting and projection methodology for the upcoming provincial election in Ontario. The fundamentals of the model were also employed in all federal and provincial elections held since 2011. Improvements, updates, and new features were added after each election season.

This methodological explanation is for projections and forecasts for an upcoming election. The methodology of seat projections for individual polls and for poll aggregations for Canada outside of an election campaign are slightly different, and is explained here and here.

Poll aggregation

The projection model starts with the aggregation of all publicly available opinion polls. Polls are weighted by their age and sample size, as well as by the track record and past performance of the polling firm.

The weight of a poll is reduced by 35% with each passing week outside of an election campaign and each passing day once a campaign has officially begun. The 'date' of the poll is determined by the last day the poll was in the field.

The sample size weighting is determined by the margin of error that would apply to the poll, assuming a completely random sampling of the population. The margin of error for a poll of 1,000 people, for example, is +/- 3.1%. A poll with a sample of 500 people has a margin of error of +/- 4.4%. Rather than giving the poll of 500 people half the weight of the poll of 1,000 people, the smaller poll would be weighted at 70% (3.1/4.4) of the larger poll.

An analysis of a polling firm's past experience in a province or at the federal level has suggested that polling firms that were not active in a jurisdiction's previous election have a total error 1.27 times that of firms that were active in the previous election. Accordingly, polling firms with prior experience in a jurisdiction are weighted more heavily than those that have none.

Polling firms are also weighted by their track record of accuracy over the last 10 years. Their accuracy rating is determined by three factors: 1) the last poll the firm released in an election campaign, 2) their average error for all parties that earned 3% or more of the popular vote, and 3) the amount of time that has passed since the election. In order to take into account changes of methodology or improvements made over time, the performance of a polling firm in a recent election is weighted more heavily than their performance in an older election. The difficulty of each election is also taken into account: elections where the average error was lower are weighted more heavily than elections in which the error was higher. This is meant to take into consideration elections in which there were particular factors contributing to pollster error that were outside of the pollster's control. Conversely, in elections where the consensus was close to the mark a pollster has fewer excuses for higher error levels.

The accuracy rating is determined by comparing the average error, weighted by how recent the election is, of the best performing polling firm to others. For example, if the best performing firm had an average error of 1.5 points per party, a firm with an average error of three points per party would be given half the weight.

All of these ratings are combined to give each poll in the projection model a weight (no poll is ever awarded more than 66.7% of the total weight, unless there have been no other polls done recently). In short, this means that newer polls with larger sample sizes from experienced polling firms with a good accuracy record are weighted more heavily than older and smaller polls from inexperienced firms with a bad track record.

The vote projection

After weighing all the polls to determine the average result and estimating likely support for independents and smaller parties (based on performance in the last election and the number of candidates running this time), the projection model gives the best estimate of support that each party is likely to get in an election.

But rather than suggest that the poll aggregation, after adjustment, reflects the results of an election "held today", the projection is instead a reflection of the result as of the last day of polling in the projection model. For example, if a poll is released on April 14 but the last day the poll was in the field was April 12, the vote projection will be presented as being the best estimate of what the result of an election held on April 12 would have been.

The provincial or national vote projection, however, has little bearing on the seat projection. That is because the seat projections are calculated regionally, using the same methods described above to estimate support in each region of a province or in the country. Polls whose regional definitions do not exactly match the projection model's definitions are adjusted accordingly, with the difference between the election result in a region as it is defined by the model and by the polling firm being used.

The performance of this method

This adjusted and weighted poll aggregation performs better than most individual polls and better than an unweighted and simple averaging of the last polls of a campaign. In 13 federal and provincial elections, ThreeHundredEight.com's vote projection model has outperformed the polls 11 times and has, on average, had an error level of 2.2 points per party compared to 2.9 points per party for the polls.

Recognizing the limitations and vote ranges

But despite performing better than most polls and the average of the polls, the vote projection is still heavily dependent on what the polls show. It can thus fail catastrophically when the polls do, as occurred in the provincial elections in Alberta in 2012 and British Columbia in 2013. A measure of the likely error in the vote projection needs to be made, using the degree of past error polls have had in recent elections.

This is calculated based on a party's position in the legislature at dissolution: the governing party, the Official Opposition, a third party with multiple seats, a third party with a single seat, and parties without a seat in the legislature. The electoral outcome for each of these parties in recent elections is then compared to the polling average.

All cases in which a party in a particular position in the legislature was under-estimated in the polls is then used to calculate the average "High" range. For example, the average under-estimation (when polls under-estimated a party's support) in recent elections for the governing party has been by a factor of 0.96. That means that the weighted polling average is adjusted by a factor of 0.96. The same is done for cases of over-estimation to calculate the "Low" range.

The minimum and maximum projections ("Min." and "Max." on the chart) are calculated to show the range of outcomes likely to occur 95% of the time, or 19 times out of 20. These are calculated based on the standard deviation of over- and under-estimations from the average over- and under-estimations in the past.

This gives readers a full understanding of the potential range of outcomes that are possible, based on past polling performance and what the data is showing. My role is not to make bets, but to try to figure out what the polls are saying and what they aren't saying, and giving people in idea of what to expect. But to narrow it down a little, the projection also calculates the range of most likely outcomes for each party. The chart below spells this out for the Ontario vote:

As the governing party, there is 62% chance that the electoral outcome for the Ontario Liberals will fall within the average-to-high range. If we want to extend that further, we can say there is an 81% chance it will fall between the low and high marks.

For the PCs as the Official Opposition, the range is not so tight. The most likely individual outcome is for the result to fall within the average-to-high range (41%), but it is more likely that it will fall outside of that range (the remaining 59%). To find the smallest range that incorporates the most likely outcome, we have to stretch that to the low-to-high range. There is a 64% chance that the outcome will fall within that range.

For the NDP, a third party with multiple seats, a slight over-estimation the most likely individual outcome but we have to stretch the range also from minimum-to-high to get to a 74% chance. For the Greens, an over-estimation is almost certain: there is a 54% chance the outcome will fall within the low-to-average projection, and an 86% chance that it will fall between the minimum-to-average.

It is, of course, possible that the outcome will fall outside of even the maximum and minimum projected ranges.

Seat projection methodology

Once the vote projection and likely ranges for each party are determined, the model then makes a seat projection. This seat projection is based on the vote projection: if the first is wrong, the second will be as well. If the vote projection is accurate, the seat projection will also be accurate. With completely accurate polls, the seat projection model would have a margin of error of only three seats per party and make the right call in each riding 85% of the time, and identify the winner via the ranges 90% of the rime.

At its core, the seat projection model uses a simple proportional swing method based on the difference between the results of the last election and current polls. Put simply, if a party managed 20% in a given region in the previous election and is now polling at 40% in that same region, their results in each individual riding would be doubled. The image below shows how this method would have estimated the NDP's support in the riding of Trinity-Spadina in the 2011 federal election.
This swing is applied to every party in each riding. As this will sometimes result in total support of more or less than 100%, the numbers are adjusted upwards or downwards proportionately to equal exactly 100%.

This model is in contrast to the uniform swing method popular in the United Kingdom. With that method, in the example of Trinity-Spadina, the NDP's increase of 7.4 percentage points in Ontario would have simply been added to the NDP's result in 2008 in Trinity-Spadina, estimating that the party would captured 48.3% of the vote instead of 57.7%, as proportional swing would suggest. In this one case, that would put the error of uniform swing at about double the error using the proportional swing method.

The proportional swing method is a better estimation of how support changes between elections, reflecting that a party with a large base of support in a riding is more likely to grow by larger proportions than a party with no real support. It can also perform well when parties make major gains - with the actual provincial results of the 2011 federal election plugged into the model, it would have projected 60 seats for the NDP in Quebec to four for the Bloc Québécois, instead of the actual result of 59 to four.

Taking other factors into account

The swing model alone, however, cannot take into account the individual characteristics of each riding. Other factors need to be taken into account.

Incumbency is the most important factor, as it applies to every riding and can have a significant effect. My own research shows that support for incumbents is far more resilient than for other candidates, and that when parties do not have incumbents on the ballot they suffer a serious loss in support. That drop equals about 10% of what the party managed in the previous election, resulting in a slip of anywhere from four to six points (all else being equal). But the incumbency effect is also determined by a how a party is doing overall. My research shows that an incumbent retains more of their vote when their party's support is dropping in the region. It also shows that incumbents who have been re-elected at least once make lesser gains when a party's support is increasing in a region, while incumbents running for re-election for the first time tend to out-perform their own party's gains.

This would seem to be reflection of the difference between a first-time incumbent and a veteran incumbent. A veteran has a more solid base of support that is harder to move in either direction, whereas a sophomore is now a much safer bet compared to when they first ran for election. They have a record of winning, whereas in the previous election they had none.

Accordingly, when a party is losing support incumbents are given a "bonus" usually worth three to five points, while when a party is gaining sophomores are given a bonus worth about one to two points while veterans are penalized by about that much. When the incumbent is not running for re-election, the party is penalized accordingly.

The effect of having star candidates on the ballot is the largest bonus of the projection model. Star candidates improve their party's performance in the vast majority of cases, though the classification of star candidates is one of the purely subjective aspects of the model, as I have to determine whether a candidate should be considered a "star" or not. This is usually quite obvious, and one of the biggest determinant factors is whether a candidate is widely considered as a star in the media, which has its own effect on how the candidate is perceived by voters. Star candidates are usually former MPs or cabinet ministers, party leaders, or well-known figures from the private sector.

Floor crossing is a difficult factor to take into account, as the amount of support a sitting MP or provincial representative brings with them can vary dramatically. But an analysis of past cases shows that the effect can be very large, with the floor crossing candidate able to increase their new party's support in a riding by about half, while the other party's support drops by about a quarter.

The presence of independents can also be difficult to model. If an independent politician is running for re-election as an independent, their vote is dropped by about one-eighth from the previous election, as has occurred in other cases. The same penalty is applied to popular independent candidates who were never elected. Politicians who left or were forced out of their party caucuses and are running for re-election as independents are treated differently. Based on an analysis of previous cases, these candidates take a proportion of their vote share from the previous election based on the circumstances of their departure from caucus. Those who depart for positive reasons retain much more of their support than those who leave in disgrace. When the circumstances are hard to define, an average proportion is used. Those votes come directly from the party the candidate left.

By-elections are also taken into account. When the result of a by-election was significantly different from the results of the previous general election, the proportional swing is applied to the by-elections results based on how current polling levels differ from where the parties stood in the polls at the time of the by-election.

When available, riding polls are also added to the projection for an individual riding. The weight of the poll is determined by the number of respondents (i.e., 431 respondents would give the riding poll a weight of 43.1%) but is capped at 50%. The riding poll's results are used as a new baseline, from which the numbers are adjusted as regional polling changes. In other words, just as the standard model adjusts party support in a riding by the proportion that the party's support has shifted in the region since the last election, the riding poll is adjusted by the proportion that the party's support has shifted in the region since the date that the riding poll was conducted. When multiple riding polls are released during an election, only the latest one is taken into account.

The particularities of an election

When necessary, the projection model takes into account the individual particularities of an election campaign. One common particularity is the presence of a new party, or a formerly fringe party running a full (or almost full) slate of candidates.

When a party is running candidates where they did not have a name on the ballot in the previous election (whether that be limited to a handful of ridings, as often occurs with smaller parties, or in the bulk of ridings, as occurred in the 2012 election in Alberta for Wildrose), the regional vote projection for the party is applied directly to the riding. For example, if a party is polling at 20% in a region it will be projected to have 20% in each riding in that region. However, that number can be adjusted by any of the factors listed above and is always adjusted when the model makes all of the projections add up to 100%. In this example, in ridings where there is little room for the party to have 20% their vote will be adjusted downwards. When there is a lot more room, the vote will be adjusted upwards. This system performed well when the real results of the 2012 Alberta election were applied: Wildrose would have been projected to win 18 seats (instead of the actual result of 17).

Likely seat ranges

In order to take into account error in the polling and in the seat projection model itself, the vote projection ranges are used to determine likely seat ranges. These are applied directly to each party's projected results in each riding. For example, if the high projected vote for a party in a given region is 5% higher than the most likely projection, then the projected vote for the party in each riding in that region is increased by a factor of 1.05. How these high and low results for each party in each riding compare determines whether a seat is "in play". If the projected high result for a party in a riding is higher than the projected low result for the party expected to win the seat, the seat is then potentially winnable for the trailing party.

This gives the seat projection a confidence interval, based on likely results if the polls over- or under-estimate party support by the same degree as they have historically. The maximum and minimum vote ranges are used in the same fashion to calculate maximum and minimum seat ranges if the degree of error approaches or matches historically poor performances by the polls.

Probability of a correct call

One new feature added to the model in 2013 was the probability that a call made by the seat projection model will be correct. This is based on an analysis of the seat projection model's performance in the nine elections that it has made projections for individual ridings. This probability is determined by the margin the projection model estimates the winner will win by. The following chart tracks how the projection has performed in the past, based on the projected winning margin in each riding.
Each red dot shows the percentage of calls that were correct, while the blue line shows the general trend. As the data is somewhat noisy (but the trend is still clearly visible), the trendline has been used to determine what probability of a correct call should be applied to every riding.

If the riding projection shows that a party leading in a riding by 12 points has a 74% chance of winning, that means that based on past performance the model will be right about 74% of the time when it chooses a winner by a margin of 12 points. It does not mean that there is a 74% chance that the projection for every party in the riding will be correct, or that the trailing party has a 26% chance of winning (a third place party could win as well). It is referring to the odds that the party projected to win will win. This model performed well in its first use during the 2013 B.C. election, as the following chart shows:
Hopefully, this should provide a complete explanation of ThreeHundredEight.com's projection methodology during and in the run-up to election campaigns.