Wednesday, October 22, 2008

Weighting System

Polling and electoral results are weighted according to the age of the poll, the amount of people polled, and the reliability of the polling firm.

The 2004, 2006, and 2008 federal elections are included because past voting behaviour tends to be a major predictor of future behaviour. Obviously, recent elections are weighted more heavily than older elections.

Older polls are dropped as newer ones are added.

Polling firms are weighted according to their accuracies in national and provincial elections since 2008. In the name of openness, here are the reliability factors of the various polling firms (updated as of the 2010 New Brunswick election) :

Angus-Reid Strategies - 1.11
Corporate Research Associates - 0.98
Léger Marketing - 0.84
Ipsos-Reid - 0.84
EKOS Research Associates - 0.79
Mustel Group - 0.72
Nanos Research - 0.70
Environics - 0.68
Harris-Decima - 0.67
Strategic Counsel - 0.52
Segma Unimarketing - 0.47
CROP - 0.12
Abacus Data - 0.08

12 comments:

  1. Salut Éric,

    I'm wondering if the weighting system you are using - based on accuracy of pre-election polls - is the best method. It would not have been unexpected that the same polling methodology at Angus Reid could have produced a poll that is a little (say 1%) further from actual voting results.

    By letting a single lucky poll stand for Angus Reid, you may be overdoing things a bit. Further, your polling firm leaning page suggests that there is a slight bias to which you may be adding undue weight.

    As well, the poll size should be a factor in the weighing, IMHO. The recent EKOS poll had an exceptional sample size and weighing may want to reflect that.

    Finally, I'm curious about your projection methods. How are you using the polling data to make your projections?

    In any case, great site. I hope it expands in depth and breadth and statistical power. Thanks!

    ReplyDelete
  2. Never mind the EKOS comment. I found your weighings at the bottom of the page. Can I ask, what makes EKOS a weighing of 3.0 versus 3.1 or 2.9?

    ReplyDelete
  3. ok...one last question: What happens for projecting for the two independents in the house?

    ReplyDelete
  4. Donaldstreet,

    You may be right that my weighting system can reflect a lucky poll. I'm not quite sure how else to value a polling firm, however. They can only stand by the work they've done in the past during election campaigns, because there is no other way to measure their success. And while my leanings-rankings gives an indication of (likely unintended) bias, it is not necessarily an indication that the polling firm is wrong. It could well be that one polling firm is correctly measuring support while the other polling firms are under-representing support for a given party.

    My projections are based on past historical results. If a party is projected to have X support, that translates to X seats, based on historical results.

    As to the weightings, I gave EKOS a 3.0 for, well, an arbitrary reason. The poll was so huge (and unusual in its size), that its weight should have been something like 8.0 or 9.0. That would have thrown everything out of whack. So I put a limit of 3.0 for any one poll.

    ReplyDelete
  5. I threw those two independents to the wind. If, in the end, my projection is off by two seats, that would still be extremely good.

    ReplyDelete
  6. I'm curious about the low weighting for Nanos, given that his numbers for two of the last three elections have been extraordinarily accurate. Is it possible to see the numbers on which you have based the accuracy weightings?

    ReplyDelete
  7. I've only used the last election as a measuring post. That may or may not be fair, but after the next election I'll use that result and the one from 2008 to re-value the pollsters.

    As to the numbers I've used, what I have is that Angus-Reid was 4.4 points off, Léger was 8.4, EKOS was 7.6, Ipsos-Reid was 8.8, Harris-Decima was 8.8, Nanos was 10, Strategic Counsel was 10.8, and Segma was 12.3. I also took into account how many days there were between the poll and the voting day, which is why Léger has a better weighting than EKOS despite being further off. For CROP I used their 2008 Quebec provincial prediction, which was 15 points off.

    ReplyDelete
  8. Thanks Eric. When you say "points off", is that an aggregate of the distances of each party from the poll result to the actual result?

    I have another question for you - when allocating seats by province or region, are you doing anything more than uniformly applying the swing in popular vote to the results in each riding?

    A friend and I were having an argument about the wisdom that. I said there was an easy way to check (which I'm too lazy to do but am kind of hoping you might have done it): check the riding-by-riding variance in the swing from the last couple of elections. So for what % of ridings in (say) Ontario does a 5% province-wide shift from Lib to Con mean a 5% shift at the riding level? In how many ridings does it mean a shift of 3% or less? 7% or more? You see where I'm going here.

    Any chance you've done work along those lines?

    ReplyDelete
  9. I'm actually not doing anything like applying the swing in regional votes to individual ridings. I don't look at individual ridings at all.

    What I've done is figure out what percentage of votes equals what number of seats, based on historical performances.

    ReplyDelete
  10. But you must be looking at individual ridings at some level. In your recent best case/worst case post, you even named the two ridings available to the Greens.

    I mean, you obviously have an algorithm that translates votes to seats. If it is not based on actual vote totals in each riding at previous elections, what is it based on?

    I'm not trying to break balls here or anything, I'm genuinely curious.

    ReplyDelete
  11. It's based on previous vote totals and what that equated to for seats. I've done calculations to figure out how many average votes are needed over the last three elections to win X amount of seats.

    When I name individual ridings, like Outremont, Guelph, or Saanich, it is because I know what seats we're talking about. But when the Liberals win a seat in Ontario, I don't know which one it is. When the NDP wins a seat in Quebec, I know its Outremont because that's their only chance. If they win a second one, I know its Gatineau because that was the second best chance.

    This system works pretty well, as if I punch in the results from each of the last three elections into the engine, I get very accurate results.

    ReplyDelete
  12. I'm wondering what logarithm you are using to reduce polls values with respect to time. A year is a political eternity, so a poll nine months old is almost worthless. Also, three elections may be a bit too many to use as a projection. Three elections before the meteoric rise of Reform and the BQ in 1993 (the 1980 Trudeau majority victory), neither party even existed.

    ReplyDelete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.