Monday, October 11, 2010

Polling House Effects Update

What better day to update the house effects chart than a slow and quiet Thanksgiving Monday. I've updated Ipsos-Reid and CROP, incorporating data from September 2010.

Ipsos-Reid is the most favourable pollster for the Conservatives, polling them at an average of 3.5 points higher than other pollsters. They are tied with Environics as the least favourable NDP pollster, polling them at an average of 2.2 points low than other pollsters. In Quebec, however, they are one of the pollsters with the least amount of variation from their competitors.

CROP is the best pollster for the Conservatives in Quebec, polling them at an average of 2.2 points higher than other pollsters. They are also the best for the NDP, putting them at an average of 3.9 points higher. They are the worst for the Bloc Québécois, polling them at an average of 4.2 points lower than others.

We'll use the new house effect numbers to "correct" the latest polls from Ipsos-Reid and CROP, taken at the end of September. First, Ipsos-Reid's national numbers:

Conservatives 35% = 31.5%
Liberals - 29% = 29.7%
New Democrats - 12% = 14.2%
Greens - 12% = 12.7%
Bloc Québécois - 11% = 11%

And now Quebec:

Bloc Québécois - 39% = 37.5%
Liberals - 22% = 22.3%
New Democrats - 16% = 16.4%
Conservatives - 17% = 15.8%
Greens - 6% = 6.1%

While this doesn't change the situation in Quebec very much, it does show a much closer race at the national level.

Now CROP in Quebec:

Bloc Québécois - 32% = 36.2%
Liberals - 23% = 23.8%
Conservatives - 23% = 20.8%
New Democrats - 18% = 14.1%
Greens - 4% = 6.7%

This still has the Conservatives at a much higher level than most other pollsters, but the Bloc numbers look a lot more realistic.

The chart below tracks how each pollster tends to lean when calculating support levels for the various parties, as compared to the average polling results from other pollsters each month. This does not necessarily equate to a deliberate bias, but instead is more reflective of the polling methods used - the "house effects". This is also not a scientific calculation of any kind, but it does give an indication of how each pollster tends to compare to others.

The following chart shows each pollster's average variation from other polling firms. The numbers are the amount of percentage points a particular pollster favours or disfavours that particular party compared to other pollsters over a similar period of time.


  1. Interesting Eric.

    At least on the national scene when the "house effect" is applied we see more consistency pollster to pollster.

  2. I wish you would do the same with actuals versus polling.

    CROP is taken to be by far the most accurate Quebec pollster. (Hearsay evidence only)

    AR was the only pollster that was within the MOE a week before the last election. The rest all had the 1 time out 20 that they were wrong.

    IR had the CPC under polled by 3.6% with 34% as opposed to actual 37.6 in their last pre-election poll and had the Liberals overstated by 2.8 % at 29% rather than the 26.2% they got 5 days later.

    Taking this "bias" into account it would put the

    CPC 35 +3.6 = 38.6 %
    Liberals 29 - 2.8 = 26.2 %

    In other words the CPC up 1% from last election and the Liberals exactly at the same spot.

    I believe that would be a CPC majority using your methodology??

  3. Just a quick note, correcting for house effects may or may not make a set of numbers closer to reality.

    As an example, it may very well be the case that Environics, the least favourable NDP pollster, is right and everyone else is wrong.

    In which case artificially raising the NDP numbers you get from Environics could be making things better than they are for the NDP.

    The assumption being made when determining house effects is that bias in polls should cancel each other out if you have enough pollsters.

    This is likely, but not nessecarily, true.

    A special case might be Greens.

    Pollsters tend to either have high or low numbers for them based on whether they prompt or not.

    If one poll says 4% and another poll shows 12% and we apply house effects to show us more consistent numbers like 6% and 10% Greens should be aware this is likely not helpful.

    Either the 4% or the 12% is the more likely result, with, i'd argue, the 4% being the most likely outcome.

  4. We're all aware of that, Shadow.


    Pollster ratings are based on actual results. Based on those, CROP is the least accurate pollster, while Angus-Reid is the most. Those pollster ratings are taken into account in the projection.

    These house effects are not, they are just a comparison of pollsters.

  5. One reason why Leger and CROP often seem to have Quebec numbers that differ from the Quebec sub-sample numbers from other polling companies is the that (at least in the past), the usual practice in Quebec was always to include the names of the party leaders in the vote question. So, while Harris/Decima (as an example) simply asks "would you vote Liberal, Conservative, NDP, Bloc Quebecois or Green?", Leger and CROP have tended to ask "would you vote for Michael Ignatieff's Liberals, Stephen Harper's Conservatives, Jack Layton's NDP, Gilles Duceppe's Bloc Quebecois or Elizabeth May's Greens?"

    Clearly, the NDP will probably get a boost from Layton's name being mentioned since he is very popular in Quebec, while Ignatieff probably drags the Liberal numbers down - and Harper may be a bit of a plus for the Tories since he personally may be less toxic to Quebecers than the Tory name on its own (maybe). The Green Party needless to say gets crucified the moment you mention Elizabeth May's name. Without her name a lot of people wistfully like the idea of voting for "A green party". But you stick in her name and especially in Quebec, you get cold water thrown in your face as you're reminded of that abrasive woman who speaks French like my ass chews gum!

    Eric, have you ever asked CROP or Leger about whether they still include the leader's names in the vote question?

  6. "We're all aware of that, Shadow."

    I'm certain you are but you shouldn't speak for everyone.

    I see a lot of people trying to explain away bad polling numbers with house effects (on all sides) without inserting the requisite caveat that perhaps the pollster happens to be RIGHT as is.

  7. BCVoR,

    Pollster ratings are based on actual results. Based on those, CROP is the least accurate pollster, while Angus-Reid is the most. Those pollster ratings are taken into account in the projection.

    These house effects are not, they are just a comparison of pollsters.

    I am trying to get my head around this.

    You are saying that in your big overall seat predictions and monthly averages seat predictions you have AR more heavily you know/feel/have statistical analysis showing them to be most accurate.

    You also show AR as more than 3% pts more "bias" in CPC over Liberals relative to other less accurate pollsters. I don't consider this to be bias so much as accuracy relative to the rest.

    Failure to acknowledge this when applying the Polling House Effects is not putting forth the best prediction based on the information available.

  8. I'm not applying the Pollster House Effects to anything, nor is there any need to. The extra weighting given to pollsters who have been more accurate since 2008 already reflects the house effects. It would be redundant to include both.


COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.