Friday, November 26, 2010

Pollster House Effects Update

Time for an update on the house effects chart. This month, I'm updating Harris-Decima and Nanos Research. See below for a full explanation of what these house effects measure.

For Harris-Decima, they did not show any huge disparity at the national level, though their largest was in polling 1.6 points lower than average for the Conservatives. In Quebec, they polled the Conservatives 1.9 points lower than average, while polling the Bloc Québécois and Greens 1.9 points higher than average.

Adding these findings to the chart, Harris-Decima is the worst pollster for the Conservatives at the national level and the New Democrats in Quebec. They are the second worst for the Liberals nationally and the Conservatives in Quebec. They are second best for the Greens nationally, and best for the Bloc in Quebec.

On to Nanos. They did have some large disparities in October. They polled the Liberals 4.3 points higher than the average and the Conservatives 3.3 points higher. This was at the expense of the Greens, who they polled 5.2 points lower than average. Undoubtedly this is due to their method of not prompting party names in their questions.

In Quebec, they polled the Conservatives 6.0 points higher and the Liberals 4.9 points higher, while they polled the Greens 6.3 points lower.

This makes them the worst pollster, both nationally and in Quebec, for the Green Party. They are tied for being the best pollster for the Conservatives in Quebec, and are the second best pollster for the Liberals both nationally and in Quebec. They are the second worst pollster for the Bloc.

So, just for giggles, let's apply these house effects to Harris-Decima's and Nanos' most recent national polls. The first number is the actual poll result, the second is what it becomes with the house effects.


Conservatives - 33.0% = 35.3%
Liberals - 28.0% = 28.7%
New Democrats - 17.0% = 17.2%
Bloc Québécois - 9.0% = 9.0%
Greens - 10.0% = 8.2%


Conservatives - 37.1% = 37.8%
Liberals - 31.6% = 28.6%
New Democrats - 15.4% = 15.0%
Bloc Québécois - 10.8% = 10.8%
Greens - 5.2% = 7.6%

Interestingly, the Liberals and Greens end up at similar results, with the disparity seeming to come between the Conservative and New Democratic results.

Now Quebec:


Bloc Québécois - 39.0% = 36.3%
Liberals - 21.0% = 21.6%
Conservatives - 13.0% = 14.8%
New Democrats - 11.0% = 12.9%
Greens - 11.0% = 11.1%


Bloc Québécois - 42.8% = 46.4%
Liberals - 26.2% = 21.8%
Conservatives - 19.3% = 17.1%
New Democrats - 10.5% = 10.7%
Greens - 1.3% = 4.9%

Much more of a variation here, though again the Liberal numbers are very similar.

The chart below tracks how each pollster tends to lean when calculating support levels for the various parties, as compared to the average polling results from other pollsters each month. This does not necessarily equate to a deliberate bias, but instead is more reflective of the polling methods used - the "house effects". This is also not a scientific calculation of any kind. Methodological differences, field dates, margins of error, and polls released vs. polls unreleased all play a role in these calculations. But it does give a little bit of an indication of how each pollster tends to compare to others - and only to each other.

The following chart shows each pollster's average variation from other polling firms. The numbers are the amount of percentage points a particular pollster favours or disfavours that particular party compared to other pollsters over a similar period of time.


  1. A projected LPC seat in Alberta? HA! Where? This is one of the weaknesses in projections. Its all well and good to generalize, but I'd like someon to explain to me exactly which seat they have a snowball's hope in heck of winning?

  2. More seats would come into play than you'd think, especially if the Liberal vote almost doubles and the Conservative vote drops by a tenth, which is what polls are showing might be the case.

    Edmonton Centre, though, is the obvious choice. Applying uniform swing would give the riding to the Liberals, and with the refusal of the government to fund the 2017 Expo bid their chances are even better.

    But did anyone serious think the NDP would've won a seat in Alberta in 2008? Things shouldn't be taken for granted.

  3. Agreed. But with respect, I'm in Edmonton. Things look different here when it goes from intangible "provincial projections" to reality on the ground. For example, Laurie Hawn won by 10,000 + votes. In Strathcona, the Libs didn't even qualify for their elections Canada deposit refund. They have zero organization, zero money and zero members. It's laughable.

  4. Oh and one other thing... yes, people here did know that the NDP had a chance to win, especially after their strong '06 performance and their candidate's hard work in between the two elections. That, combined with Jaffer's nose diving reputation among his constituency meant E-S was realistically in play. None of those factors are present for the LPC ANYWHERE in Alberta.

  5. Eric I think your adjusted Nanos numbers are off.

    If you're correcting for bias then CPC numbers should go down, not up slightly.

    On the Alberta situation I think a lack of credible opposition candidates and a losing record in the west will sink any Liberal gains.

    Parties throw away ridings all the time by not putting in the resources and recruiting good people.

  6. No, overall Nanos is still rated at under-estimating Conservative support by 0.7 points. They just bucked their usual trend in October.

  7. Traditionally the Nanos effect is just to shift support from the Greens to the Liberals, but now that support appears to be going to the CPC just as much.

  8. Are your "house effect" calculations based on actual election results, or the average results of all polls. If the latter, I think you have a problem.

    Implicitly you assume that the average result of all polls is somewhat close to the true results. I see no reason to believe this to be true. If 5 pollsters use a method that screens out low info voters, while 1 pollster uses a technique that doesn't, it is the latter pollster (ceteris paribus) that is more accurate. Yet they would show up as having the highest bias.

    Real results are the only way to test pollster bias. Canadian pollsters are not likely to hover around the population average either, because they are:
    A. few in number
    B. methodologically diverse

    A second-best approach might be to figure out how differing polling methodologies bias the results in favour of different parties.

  9. To further my point, some numbers...

    2008 projections of Incumbent vote
    A-R: 37%
    Leger: 36%
    Ekos, Segma: 35%
    I-R, H-D, Nanos: 34%
    Strat. Counsel: 33%
    Average: 34.8% (actual: 37.6%)

    2006 projections of Incumbent vote
    SES: 30.1%
    SC, Ekos, Ipsos: 27%
    Average: 27.78 (actual: 30.2%)

    2004 projections of Incumbent vote
    Ipsos: 32%
    Ekos, Environics: 33%
    Compas, SES: 34%
    Average: 33.2% (actual: 36.7%)

    Most Canadian pollsters are terrible. In the past three elections they have been on the margin or outside of a usual 2.5% margin of error in their estimation of support for the incumbent party.

    Looking for consensus from this confederacy of dunces makes little sense.

  10. As explained very clearly in the blogpost, the house effects are a comparison of pollsters to one another, not election results.

  11. Banging away at how bad the Canadian pollsters are:

    The pollsters have consistently indicated that the Liberals will be doing better in Ontario and the CPC worse since the 2008 election.

    Vaughan is in Ontario. The Liberals won this seat by 15%. For perspective Ignatieff won his seat by 11%.

    All indications given by the pollsters is that the margin of Liberal victory in Vaughan should increase.

    If the election result is a tie the pollsters are off by 15-20%.

    If the CPC take the riding by 10,000 votes as is suggested by some Liberal insiders the pollsters are off by 30-35%.

    I would not be surprised if Nanos and Graves release last minute polls (too late to trigger a landslide) that show the CPC in a much stronger position in Ontario. They did that the last GE.

  12. Eric: Edmonton Centre, though, is the obvious choice. Applying uniform swing would give the riding to the Liberals, and with the refusal of the government to fund the 2017 Expo bid their chances are even better.

    Is there any historical evidence of political repercussions on being responsible on large special interest funding?

    I am talking about funding where the people of Edmonton and Alberta (along with the Feds) will be paying off the debt for years..... similar to the Montreal Olympics.

    While the feds have ticked off the Edmonton Worlds fair special interest committee it made a lot of friends with the people who opposed the waste of money that they would end up paying for.

    When the Jets and Nordiques left there was nowhere near the economic fall out predicted. Both these cities would have been in much worse fiscal condition had the Feds backed building new arenas and the cities been on the hook for their portion of the funding.

    The cut to the Arts made an impact as it was minor $ and symbolic. No one was able to ask if the 98% of the population that are not artists was good with raising taxes to cover that special interest bill in stead of health care.

  13. People who oppose government spending were likely to vote Conservative anyway, so I don't think the decision to cancel funding for the Expo bid had gained them any votes. It will most certainly have either no effect or hurt the Conservatives.

    As to your question, I've long believed that the decision of the PQ government not to fund the building of a new arena in Quebec City (which would have kept the Nordiques in town) contributed to the lower-than-expected OUI vote in the capital.

  14. People who oppose government spending were likely to vote Conservative anyway, so I don't think the decision to cancel funding for the Expo bid had gained them any votes. It will most certainly have either no effect or hurt the Conservatives.

    You don't think that the anti-poverty, protect health care, arts councils, teachers unions..groups.. aren't extremely happy that there is a couple of billion more government money that they are entitled to rather than the billions that would be lost to this big party?

    I think this is a big play to the centre. It takes away from the the Left organizing and actually getting motivated to vote.

    In Toronto there will be a huge block of voters very thankful that Toronto did not get the Summer Olympics. That would have set Toronto/Ontario back 20 years similar to the Montreal/Quebec Olympics. Had the Olympics been awarded to Toronto the NDP/Liberals election issue would have been reckless needless spending. The Bloc would have been asking for equivalent federal funding for the Parizeau museum in Trois-Rivières.

    I know there isn't one in the plans but there would be a similar ask in the works to address the fiscal imbalance.

    The CPC learned their lesson with the Vancouver Olympics and the G8/20 summits. The cost of security for international events far exceeds any benefits.

    Security effectively doubles the costs and doesn't generate any benefits.

  15. BC Voice Of Reason:
    Your numbers would be correct if
    1.Every Liberal incumbent in Ontario retired
    2.In each of those ridings the Conservatives recruited a candidate as prominent as Fantino
    3.Voter turnout in the next election would be the same as for a byelection

  16. Éric, since you've been tracking house effects for some time, have you considered plotting these values against time, or simply as bar graphs? That might help us visualize what proportion of house effects are real and continuous, and how much is just jitter.

  17. Hmm. Something to consider.


COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.