Thursday, March 3, 2011
Seat projection methodology
When individual polls are reported upon at ThreeHundredEight.com, seat projections are often reported as well. The following is a detailed description of that seat projection methodology. This is the same methodology used during and in the run-up to election campaigns, with some minor differences.
The results of each poll can be plugged into the seat projection model to give likely results, based on the premise that the results of an election would be exactly the same as the results of the poll. Assuming that to be the case, the seat projection model has a margin of error of only 3.2 seats per party and makes the right call in each riding 85% of the time.
At its core, the seat projection model uses a simple proportional swing method based on the difference between the results of the last election and current polls. Put simply, if a party managed 20% in a given region in the previous election and is now polling at 40% in that same region, their results in each individual riding would be doubled. The image below shows how this method would have estimated the NDP's support in the riding of Trinity-Spadina in the 2011 election.
This model is in contrast to the uniform swing method popular in the United Kingdom. With that method, in the example of Trinity-Spadina the NDP's increase of 7.4 percentage points in Ontario would have simply been added to the NDP's result in 2008 in Trinity-Spadina, estimating that the party would captured 48.3% of the vote instead of 57.7%, as proportional swing would suggest. In this one case, that would put the error of uniform swing at about double the error using the proportional swing method.
The proportional swing method is a better estimation of how results change between elections, reflecting that a party with a large base of support in a riding is more likely to grow by larger proportions than a party with no real support. It can also perform well when parties make very large gains - with the actual province-wide results plugged into the model, it would have projected 60 seats for the NDP in Quebec to four for the Bloc Québécois, instead of the actual result of 59 to 4.
Taking other factors into account
The swing model, however, cannot take into account the individual characteristics of each riding. Other factors need to be taken into account.
Incumbency is the most important factor, as it applies to every riding and can have a significant effect. My own research (originally conducted in February 2011) shows that support for incumbents is far more resilient than for other candidates, and that when parties do not have incumbents on the ballot they suffer a serious loss in support. That drop equals about 10% of what the party managed in the previous election, resulting in a slip of anywhere from four to six points (all else being equal). But the incumbency effect is also determined by a how a party is doing overall. My research shows that an incumbent retains more of their vote when their party's support is dropping in the region. It also shows that incumbents who have been re-elected at least once make lesser gains when a party's support is increasing in a region, while incumbents running for re-election for the first time tend to out-perform their own party's gains.
This would seem to be reflection of the difference between a first-time incumbent and a veteran incumbent. A veteran has a more solid base of support that is harder to move in either direction, whereas a sophomore is now a much safer bet compared to when they first ran for election. They have a record of winning, whereas in the previous election they had none.
Accordingly, when a party is down in support in a poll incumbents are given a "bonus" usually worth three to five points. When a party is up sophomores are given a bonus worth about one to two points while veterans are penalized by about that much. When the incumbent is not running for re-election, the party is penalized accordingly.
The effect of having star candidates on the ballot is the largest bonus of the projection model, usually worth about five points. Based on the results of an analysis conducted in November 2010, star candidates improve their party's performance in the vast majority of cases, though the classification of star candidates is difficult to determine. This is one of the purely subjective aspects of the model, as I have to determine whether a candidate should be considered a "star" or not. This is usually quite obvious, and one of the biggest determinant factors is whether a candidate is widely considered as a star in the media (which in and of itself plays a role in how a candidate is seen by voters). Star candidates are usually former MPs or cabinet ministers, party leaders, or well-known figures from the private sector.
Floor crossing is a difficult factor to take into account, as the amount of support an MP or provincial representative brings with them can vary dramatically. But an analysis of past cases shows that the effect can be very large, with the floor crossing candidate able to increase their new party's support in a riding by about half, while the other party's support drops by about a quarter.
The presence of independents can also be difficult to model. If an independent politician is running for re-election as an independent, their vote is dropped by about one-eighth from the previous election, as has occurred in other cases. The same penalty is applied to popular independent candidates who were never elected. Independents who left or were forced out of their party caucuses are treated differently. Based on an analysis of previous cases, these candidates take a proportion of their vote share from the previous election based on the circumstances of their departure from caucus. Those who depart for positive reasons retain much more of their support than those who leave in disgrace. When the circumstances are hard to define, an average proportion is used. Those votes come directly from the party the candidate left.
By-elections are also taken into account. When the result of a by-election was significantly different from the results of the previous general election, the proportional swing is applied to the by-elections results based on how current polling levels differ from where the parties stood in the polls at the time of the by-election.
A post-mortem on the factors described here for the 2011 federal election campaign can be found here.
The particularities of an election
When necessary, the projection model takes into account any individual particularities of a jurisdiction. One common particularity is the presence of a new party, or a formerly fringe party running high in the polls.
When a party is running candidates where they did not have a name on the ballot in the previous election (whether that be limited to a handful of ridings, as often occurs with smaller parties, or in the bulk of ridings, as occurred in the 2012 election in Alberta for Wildrose), the regional poll result for the party is applied directly to the riding. For example, if a party is polling at 20% in a region it will be projected to have 20% in each riding in that region. However, that number can be adjusted by any of the factors listed above and is always adjusted when the model makes all of the projections add up to 100%. In this example, in ridings where there is little room for the party to have 20% their vote will be adjusted downwards. When there is a lot more room, the vote will be adjusted upwards. This system performed well when the real results of the 2012 Alberta election were applied: Wildrose would have been projected to win 18 seats (instead of the actual result of 17).
Precision when there needs to be uncertainty
A seat projection for an individual poll will appear precise: a party is projected to win X seats, no more and no less, with what the poll reports. But just as the poll itself is susceptible to any number of sampling errors, the projection is susceptible to the model's own errors. These are further amplified by the margin of error in the poll.
Seat projections need to be read with a good understanding of what we don't know. An error in the poll by one or two points could change the results in of dozens of seats if a race is close enough, and that would still be within the margin of error of the poll. And an error in the modelling of a half-dozen seats could mean the difference between winning or losing an election. The seat projection for individual polls is a best guess and the most likely outcome - but is no more foolproof than the poll it is based upon. A grain of salt, which needs to be taken with every poll, also needs to be taken with the seat projections.