Sunday, April 25, 2010

New Weights for Polling Firms

Just a short note to inform you that I've tweaked the reliability weighting of each polling firm. I've used the 2008 Canadian and Quebec and 2009 Nova Scotian and British Columbian elections in order to determine the ratings. Polling accuracy, with a small allowance for delay between the poll being completed and voting day, is used to calculate the polling firms' rating.

The 2008 federal election is weighted twice as heavily as each provincial election.

Corporate Research Associates - 1.21
Angus-Reid Strategies - 1.11
Léger Marketing - 0.84
Ipsos-Reid - 0.84
EKOS Research Associates - 0.79
Mustel Group - 0.72
Nanos Research - 0.70
Environics - 0.68
Harris-Decima - 0.67
Strategic Counsel - 0.52
Segma Unimarketing - 0.47
CROP - 0.12


  1. Why is CROP weighted so lightly, Eric? Just out of curiosity.

  2. The results of the election were 42% PLQ, 35% PQ, 16% ADQ, 4% QS, 2% PVQ.

    CROP's last poll results were 45% PLQ, 29% PQ, 15% ADQ, 6% PVQ, 5% QS.

    By far, they were the most inaccurate pollster in that election.

  3. I agree with your ranking of Angus Reid Strategies. Nevertheless, during the last federal election, ARS conducted a federal opinion poll in Saskatchewan (within days of the election):

    CPC - 40% (54% - actual)
    NDP - 35% (26% - actual)
    LIB - 17% (15% - actual)
    GREEN - 7% (6% - actual)

    The 5% ARS spread between the CPC and the NDP was actually 28% on election night. A huge disparity. ARS can be off the mark as well.

    On the other hand, Mustel in BC has always been historically pretty well bang-on with their results.

    Mustel only conducted one federal poll at the beginning of the 2008 campaign while they were in the field 1 - 3 weeks prior to election day with their last BC provincial poll.

    LIB - 47% (46% - actual)
    NDP - 38% (42% - actual)
    GREEN - 12% (8% - actual)
    OTHER - 3% (4% - actual)

    In the last week or so of the campaign it appears that a 4% share of Green voters moved over to the NDP to block the Liberals based upon the NDP plea for same.

    And in all other previous electoral contests Mustel has been bang-on.

    In that vein, I would rank Mustel certainly above Ekos and Ipsos.

    My 2 cents.

  4. Any thoughts on including '04 and '06 accuracy ?

    Wikipedia has a database of polls taken before those elections.

  5. JQ,

    Thanks for your comment. The accuracy ratings will be refined as more elections take place and I can pay closer attention to who is polling what and when. For now, I'm going to use the system I have in place.


    No, I'm drawing the line at the 2008 federal election.

  6. One election seems like a short window. Have the participants changed their methodology so much since 2004 or 2006?

  7. Volkov, I must congratulate you on your milk carton avatar. I used to have an animation of that on my phone until I broke it.

  8. I can see why you might want to draw the line at the 2008 federal election, but that does mean you're limiting yourlsef to a single datapoint and calling it a predictable pattern.

  9. The Quebec, Nova Scotia, and British Columbian elections provide data points as well. Using them along with the 2008 election is a good way of rating a polling firm's accuracy as they are very different situations.

  10. Éric: I've used [federal and provincial] elections in order to determine the ratings. Polling accuracy... is used to calculate the polling firms' rating.

    The goal is admirable but the methodology is questionable.

    Every poll has some error; a hypothetical pollster with zero average error will still be off by a lesser or greater extent with every poll.

    With a large number of samples the standard deviation can be estimated. With a single poll (or a small number of polls) probability matters as much as competence. No, let's not say "probability" in this context; let's call a spade a spade and say "luck".

    That's why dwelling on the 2008 federal election results is not helpful. Nik Nanos may be very good or just lucky--that time. Provincial elections increase the number of sample points, but not by much.

    Shadow is right about looking at the 2004 and 2006 results, although perhaps not for the reason he suggests. The goal isn't to increase accuracy; it's to get a handle on how bad the accuracy is. Even three elections are far too small a sample space for a reasonable margin of error (although that's better than using one election). The interesting analysis would be how pollsters' errors in the 2004, 2006 and 2008 elections compare across each pollster. I've commented briefly on this; my claim is that they don't match and we're seeing more chance than skill.

    Now that I've discharged the firearm into the barrel, what helpful advice can I give? In the absence of better data, I'd assign every pollster an equal weighting. Then for extra entertainment, I'd compare the weighted results to the unweighted results. If all pollsters were equally accurate, assigning weights would increase the standard deviation of the prediction but not change the mean.

    Of course, my undergrad stats course was decades ago, so somebody else may have a better analysis. However--with high confidence--the bottom line will still be, "Not statistically significant".

    And as always, Éric does not claim omniscience. We're all free to apply our own weights to his predictions. I appreciate his willingness to give us a target.

  11. It is by no means perfect and I may go back and add more elections in the future, but I believe that it is a good idea to have reliability ratings.

    If the ratings reflect which of the pollsters have the most "luck", so be it.

  12. I'm curious. Couldn't a polling firm be the most accurate in terms of gathering intentions, but simply be unable to reflect varying success amongst parties in "getting the vote out"? Unless there's a consistency over elections in all party's relative ability to get out the vote and/or some way to factor this into the evaluations of the polling firm numbers, it seems only a equal weighting system is credible unless there are other reasons to question a firm's methodology.


COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.