Poll aggregation methodology

The following is a detailed description of how the polling aggregate is calculated. While this method is similar to how the vote projection is calculated in the run-up to and during an election campaign, there are some differences.

Currently, the methodology described here applies to the poll aggregations being maintained for the federal political scene and the provincial situations in Ontario and Quebec.

Poll aggregation

The weighted polling average includes all publicly available opinion polls. Polls are weighted by their age and sample size, as well as by the track record and past performance of the polling firm.

The weight of a poll is reduced by 35% with each passing week. In polls taken over multiple days, the median day of the poll is used for the weight. For example, a poll taken between March 12 and March 14 would be dated for March 13.

The sample size weighting is determined by the margin of error that would apply to the poll assuming a random sample. The margin of error for a poll with a random sample of 1,000 people, for example, is +/- 3.1%. A poll with a sample of 500 people has a margin of error of +/- 4.4%. Rather than giving the poll of 500 people half the weight of the poll of 1,000 people, the smaller poll would be weighted at 70% (3.1/4.4) of the larger poll.

Polling firms are also weighted by their track record of accuracy over the last 10 years. Their accuracy rating is determined by three factors: 1) the last poll the firm released in an election campaign, 2) their average error for all parties that earned 3% or more of the popular vote, and 3) the amount of time that has passed since the election. In order to take into account changes of methodology or improvements made over time, the performance of a polling firm in a recent election is weighted more heavily than their performance in an older election.

The accuracy rating is determined by comparing the average error, weighted by how recent the election is, of the best performing polling firm to others. For example, if the best performing firm had an average error of 1.5 points per party, a firm with an average error of three points per party would be given half the weight.

All of these ratings are combined to give each poll in the aggregation a weight. In sum, this means that newer polls with larger sample sizes from polling firms with a good accuracy record are weighted more heavily than older and smaller polls from firms with a bad track record.

A firm with no prior experience is automatically weighted at half the value of the best firm.

The performance of this method

This adjusted and weighted poll aggregation performs better than most individual polls and better than an unweighted and simple averaging of the last polls of a campaign. An analysis of the seven provincial elections in 2011 and 2012 shows that the model used by ThreeHundredEight.com performed better, on average, than 11 of the 15 polling firms active in at least one of these elections (and two of the four better performers were active in only one campaign) and was about 10% better than a simple average of polls.

Calculating the probability of a win

Along with the poll aggregation, the probability of a party winning the popular vote is also calculated. This is centred on the premise of an election being held today, and is based on an analysis of how often the margin between any two parties has shifted between the average of the last polls of a campaign and the actual result in past elections. This probability incorporates both the degree of polling error that has occurred in the past and the amount of actual change in voting intentions that has occurred between the final polls of a campaign and the vote.

This probability is based solely on winning the popular vote. Due to our first-past-the-post system, the winner of the popular vote does not necessarily win the most seats. But it is generally a good proxy for calculating the probability of one party winning an election.