Wednesday, May 15, 2013

Polling industry dealt major blow in B.C. election

Last night was a very bad one for Adrian Dix and the New Democrats, who expected victory as much as the pollsters did. And with good reason: a stabilizing, maybe even growing, lead over the B.C. Liberals with hours to go before the polls opened. Instead, British Columbians collectively woke up and changed their minds and swung about 13 points towards Christy Clark. Or, more likely, something disastrously wrong occurred in the polling industry.

I wrote about the implications for the four party leaders for The Huffington Post Canada, and took a look at why the polls went wrong for The Globe and Mail.

Why did they go wrong? I have no explanation this morning. In Alberta, there was the late swing. There was the novelty of the Wildrose Party. There was the relative lack of polling in the final days. There was the inexperience of the pollsters who were active. There was the immensely more well-oiled organization of the Progressive Conservatives.

In British Columbia, there was no indication of a late swing. If anything, there was a sign that Clark's momentum had reversed itself. The New Democrats were not an unknown quantity. There was polling being done as late as Monday. There was the experience of two pollsters with long and successful histories in British Columbia. There was the much-vaunted GOTV organization of the NDP. And yet all the polls said the New Democrats would win, and all the polls were wrong.

(Note: the chart below includes the average standard deviation between the polls from each pollster, meant as an attempt to determine whose numbers were fluctuating the most. It seems like a moot point now.)

Forum Research ended up doing the best, but they should not gloat. As in Alberta, they were the best of a bunch of bad polls. They were in the field six days before the election, when the Liberals appeared to be at their peak in polls by other firms, and in all likelihood were just lucky not to release any new numbers. And it was odd that they didn't, as Forum has released 11th hour polling in Quebec, Alberta, and even Labrador.

My vote projections did second-best, mostly because I had a mechanism for diluting the support of the Greens and Conservatives. On the Liberals and NDP, I was as wrong as anyone else.

The forecasted ranges captured every vote and seat result with the exception of the NDP. Those ranges are designed to account for an Alberta-level event, but even so they were unable to predict that the New Democrats would under-perform in the popular vote to such a great degree. The ranges, implying that the polls should always be considered potentially spectacularly wrong, were apparently a good idea, but if ranges of this size need to be included in every election the usefulness of the forecasting model is virtually zero. In even a modestly close election, they will always span almost the entire spectrum since most ridings come into play at that point.

I have not had the time to input the actual vote results into the seat projection model yet, as I need to calculate the regional vote totals. I will do so as soon as possible. I suspect that the projected results will end up being very close to the actual results, as they have been in almost all the 10 elections I have worked on in the past. I will write a fuller post-mortem in the coming days.

There is no question that seat projection models like mine work. They are an effective way to translate poll results into seats. This is not voodoo magic, it is a rather simple endeavour. The challenge is being the least possible amount of wrong, which is the best that forecasters can hope for. But the models are only as good as the available information.

I have to admit that my confidence in the quality of that information - polling - has been profoundly shaken. Alberta was an aberration, and there was some good reason as to why it occurred (which I now have doubts about). Quebec was only a minor flub, which can be attributed in part to superior Liberal organization (or can it?). But this is a complete disaster. There is no reason why this should have happened, which leads me to believe that the reason it happened is because the pollsters did a bad job.

It might not be their fault exactly. Perhaps it is no longer possible to consistently and repeatedly build a sample that is reflective of the population. Can online panels be reliably effective when they aren't national? Work will have to be done to determine why this is happening and how it can be avoided. I have no doubt that the pollsters will eventually tackle the new challenges that they face. The question is how long it will take and whether it can be done in a country like Canada.

It puts into question the validity of the work I do. I write about polls every day for this site, for The Globe and Mail, for The Huffington Post Canada, and for The Hill Times. I give radio and television interviews about them. It is my full-time job. I've always approached it as a professional and have tried to provide insightful analysis of polling, separately from my role as a forecaster. No one in Canada who doesn't work for a polling firm writes about polls as much as I do.

How can I credibly continue to do so when I myself doubt that the results are reliable? While I was shocked when I saw the results last night, a part of me was not surprised that I was shocked and that they got it wrong all over again. If I go into every election assuming that disaster is more likely than triumph, what is the point?

This site was meant to be a way to cut through the confusion in polling and give a good idea of what, as a whole, the polls are saying. The site can still do that, but if what the polls are saying is not reflective of reality, what use is it?

My projection was wrong because the polls were wrong. Again. I am sorry that it was so. I can blame the pollsters for providing me with unreliable information, but I am nevertheless responsible for what is posted here, for the defense of polling I have mounted for the last few years, and for whatever confidence I expressed when analyzing the numbers in an attempt to inform readers about the state of the race in British Columbia and elsewhere. I apologize for that. Where do we go from here?

102 comments:

  1. Pollsters have bias that they can not overcome with science.

    They are basically political operatives and not able to step away from their roles as political advisors.

    The only people that we the people distrust more than a politician is a pollster.

    You expect a politician to be biased.

    ReplyDelete
    Replies
    1. right... so ALL the pollsters were MASSIVELY biased towards the NDP, and none feared humiliation or the loss of reputation afterwards (pollsters do have to make money to survive after all and their clients rely on, and pay them for, accurate data).

      what is MORE likely is that there are some legitimate methodological and scientific issues to work out, rather than "politics".

      Delete
  2. It is far easier to predict election results by the size of the crowds that the politicians can draw.

    When Ignatieff and the Liberals showed a surge in polling based on his trans Canada getting to know him tour he was drawing pathetic crowds. There was no explaining.

    If Trudeau as leader of the distant 3rd party is expected to win the next election he should be drawing thousands of people to every speaking engagement.

    ReplyDelete
    Replies
    1. This is bullshit conjecture.

      I don't know how many times I heard conservatives on about the superior crowd sizes of Republicans during the last US Presidential race, and yet, the polls were all on the mark for a resounding Obama win.

      Delete
  3. This really shows that polls are not at all reflective of what the true feelings and direction of Canadians are.

    We really have no idea how Canadians feel about Health Care (two-tiered), Capital Punishment, Gun Control, Prisons, Immigration, the environment and the CBC.



    ReplyDelete
  4. No apologies needed. I (we) appreciate that you are taking this on the chin and reporting the failures honestly, however disappointing they may be.

    ReplyDelete
    Replies
    1. Agreed. Please keep doing what you're doing. I can certainly understand the disappointment and frustration, but thus work is definitely still valuable.

      Delete
  5. Here is one possible answer. The reason why polling has such a high profile (lots of articles in main stream media, daily discussion on TV etc.) is because public really wants to have this data - there is high demand for it. That suggests that a whole lot of people actively consume this data, especially at election time. And that can bring very unexpected feedback phenomena. As an example, assume a sizeable fraction of population (say, 25%) learned about the pollster prediction (NDP majority) the day before BC election - because it was widely circulated by the media. And that could have changed the minds of people right before the voting started. And all that soft NDP support evaporated, presumably because those voters didn't realize we are heading towards NDP MAJORITY, so they chickened out last moment.

    So it is a magic circle: the better you guys do polling and polling predictions, the bigger chance is for the "electoral upset", because more people hear your predictions and believe them.

    ReplyDelete
    Replies
    1. But this was always a factor. Polls have been around for decades.

      Delete
    2. I suspect they were not a large part of our daily life, but in this age of total connectivity they probably became much more influential on our decisions.

      Delete
    3. Not sure, people got most of their news from TV and newspapers in the past, where polls were still given a lot of attention.

      Delete
    4. I think what I'm trying to say that polling has been continuously improving over the recent decades, until it became so good (in terms of amount of resources, sophisticated mathematical modelling etc.), that it itself became a major ballot factor. You sort of reached a critical point where the feedback all of a sudden became very strong.

      Delete
  6. A place to start is to demand better regulation of polling and polling firms. Their methods and their results should be explicit and independently verifiable. Internal polls should be forced to be made public. And I'm sure other related measures are necessary. Otherwise, there is indeed, no reason to pay attention to them.

    ReplyDelete
    Replies
    1. Why should internal polls be made public? The respective parties pay for them. The data belongs to them to do what they wish with.

      Delete
    2. In some countries, it's mandatory for even internal polls to be made public, since they are an instance of information-gathering focussed on the public. Otherwise, internal polls are seen as simply a tool for manipulation of the political process with no public scrutiny.

      Delete
    3. It might be an idea to look to see how other countries have approached this though Canada seems a politically quite volatile electorate by comparison.

      British polling took a major confidence dent when it failed to predict the re-election of the Conservatives in 1992, largely because of the so-called "Shy Tory Factor" whereby people were reluctant to admit to voting for a party. The main polling companies formed an umbrella council to ensure standards, such as the public release of the full breakdowns of publically reported polls to allow independent scrutiny of the figures, and there are various methodologies used by different firms to try to handle both "shy" voters and the question of turnout.

      Delete
  7. Eric

    I have empathy for your situation. GIGO

    Try as I might I can find no correlation with polling and actual results. I thought that there was a built in biased of about 5% to the Left in the average Canadian pollster but this election and the Alberta election blow this out of the water.

    ReplyDelete
    Replies
    1. I think the "bias" is not about left or right, but rather towards the incumbent government. Changing a government (especially after >10 years in power, as it was in Alberta and BC), is not something people would easily opt for, without serious reservations. In other words, the swing toward opposition seen by pollsters in such situations should be considered "soft", and has a high probability of evaporating on the election day.

      Delete
    2. The bias is towards rabid partisans. I should know, I am one, and I relish every opportunity to talk to a pollster and make my team look good (even if its only by a tiny fraction of a %).
      If you only get partisans like me talking to pollsters, the bias will be towards the team with the more enthusiastic base. Previously that would be reflected in the turnout, but something has changed so that lots of people who turn out to vote refuse to talk to pollsters... Might be worth examining further, except that the very people you need to talk to about it to get an insight into what they're thinking will refuse to talk to pollsters!

      Delete
    3. Except that it ISN'T always so. Sometimes, people are ready to throw the bastards out. The trick is in knowing when this is the case, and when it isn't. What if this site had been around in 1993 and had a built-in incumbency advantage for Kim Campbell?

      Delete
    4. I agree with Dan F above. I wrote about something like it below before I read his comment. In the QB, AB, and BC elections, the parties that pulled an "upset" were more "moderate" relative to their principle contenders on EITHER the left OR right, and likewise their supporters also seemed "quieter" responding to pollsters.

      There is the question of why this didn't happen in the past. It could be that for whatever reason people are GENERALLY less trustful or interested in talking to pollsters these days for whatever reason, but despite that the "rabid partisans", despite feeling similarly, continue to do so anyways for obvious rabid partisan reasons (i.e. to "move" the numbers for their "team" just that little bit in their rabid partisan imaginations.)

      I have no idea how models might account for this kind of "enthusiasm" though, if it's real, since it would seem to shift from case to case and would seem fairly subjective to measure (even if an objectively real phenomenon). Indeed, while the more extreme parties on the right or left may USUALLY have more "rabid partisan" supporters, this may not be the case in every election (for instance, could the more "centrist" Liberals, but under the "charismatic" Justin Trudeau, facing a tired government, have the most enthusiastic supporters next election?) If enthusiasm per se is hard to account for, maybe one could do the opposite and look at where the "moderate", only mildly-interested in politics voters tend to be heading in any given election, and then give that party a small boost in your models (because this group as a whole is thought to be undersampled). Maybe. If there is available data for this kinda thing for past elections it might be worth checking out, but it will have to be data related to specific elections, and not just "generally speaking", since the enthusiasm dynamic is likely local/idiosyncratic to each election (like what "just a guy" said above).

      Delete
  8. Out of every pundit who covered the BC election, you were the least wrong Eric. There's nothing to apologize for.

    Are you familiar with the "Shy Tory" effect? I wonder if that was at work here. UK pollsters had to refine their methods a lot to account for that, and maybe that's a process that Canadian pollsters should be starting too.

    ReplyDelete
  9. I heard that when Keith Baldrey asked both parties what their internals showed over the weekend, both parties said they saw a 4% gap in favour of the NDP. So the parties themselves had bias in their internal polls, though not as much as the mainstream pollsters. Interesting IMHO.

    ReplyDelete
  10. I'm really surprised there was only 1 landline poll in the entire election, and that one a week out. Any idea what the NDP and Liberals were using for their own internal tracking? I would kill to see those number for the last 4 or 5 days.

    ReplyDelete
  11. It could simply be that Canada isn't able to support a highly-accurate high-volume polling industry the way the US is, for structural reasons. National media organisations have, roughly, 10x the money to pay for polls in the US. US parties probably have an even more pronounced advantage over ours. The problem is that polling in Canada isn't any cheaper than it is in the US - getting a sample of 1000 people in Ohio isn't more expensive than getting a sample of 1000 people in BC.

    So the US gets a lot more polls than we do, and pollsters get to compare their polls against a much larger range of electoral events than ours do - they have gubernatorial races, senate races, state-level presidential results, nation-wide presidential results, even congressional races. It's always election season somwhere in the US, so a given polling firm may get to compare its final polling to actual electoral results 20 or 40 times a year.

    In Canada, I figure that we poll for the provinces, federally, and maybe the mayoral elections in the top 5 cities. An polling firm in the field for all of those would get to compare its final poll to actual results four times a year.

    That means US firms have a huge advantage over ours. Turnout models, proper demographic weighing, or how to select your online or phone samples are all things that you learn how to do by iterating, and our polling firms don't get to iterate as much as firms in the US.

    I don't know an easy way to solve that.

    I was really heartened when you started to include "forks" of possible results to account for this uncertainty in polling results, but it's true that "either the NDP or the Liberals could win" isn't a particularly useful story to tell. Even then, it looks like you underestimated just how crappy the polls were, because the NDP result falls outside that range.

    Doing what you're doing is a really hard problem to solve. I don't envy you.

    ReplyDelete
    Replies
    1. Lots of good points here. But let's remember that US polling is not infallible either. I seem to recall Nate Silver being as surprised as anyone at the size of the 2010 surge for congressional Republicans.

      Delete
    2. Good points here, but I wouldn't push them too far either. I follow American polling too, and frankly their pollsters are as all over the map as Canadian ones, especially at the state level, and neither are there necessarily more polls conducted for particular states than particulat provinces here, especially after controlling for size/influence. Many states in the last American election were hardly polled at all.

      In addition, pollsters in the U.S. might just have been LUCKY of late, since the Republicans/Democrats have been at a relative deadlock in terms of their regional/electoral coalitions. But we know from history that every now and then even these seemingly sturdy coalitions fall apart and get re-written, as hard as it may be to imagine from any particular standpoint. When that time comes, American pollsters might not fare much better than Canadian ones, cause all their data is based and calibrated on the relative stasis of existing patterns and habits. When things really start shifting again, they might have equal trouble keeping up.

      The electorate in Canada may also be IN GENERAL more dynamic and volatile than the American one (this is not the same thing as having more parties, which a lot of people have mentioned, cause you can still have a volatile electorate with two main parties, and multiparty systems can have long steady periods). You just have to look at some of the graphics on this site to see that volatility; or consider the fact that Canada seems to regularly give birth to new parties, which rarely happens in the U.S. even though they're a much bigger place with much more politics (in terms of number and regularity of elections, number of states/provinces and other jurisdictions, amount of political spending, etc.)

      Delete
  12. Éric,

    Is polling more art than science? Your work has always been consistently excellent. We value your contribution -- we found out last night that you are like the rest of us -- not God! (Not that you ever claimed to be.) But we have not lost the faith and don't plan to.

    ReplyDelete
    Replies
    1. There'S no question it's science... the science of statistics, basically. But it isn't political science... Polling may correlate with political phenomena, but it isn't explanatory. (And humanity is nowhere near being able to formulate a true science of politics.)

      Delete
  13. Is there any explanation as to why the polling in the USA seems to be so much more accurate? I mean I know they have a lot more polls, but I don't think anyone sereiously believes more polls in this campaign would have made a difference, the consensus seemed to be pretty strong.

    ReplyDelete
    Replies
    1. and doesn't a big part of it have to do with the much, much simpler situation of a two-party system?

      Delete
    2. More money and more frequent elections IMHO. And it's a lot simpler with just 2 parties.

      Delete
    3. but inherently less democratic

      Delete
    4. Pollsters in the US make a big deal of counting "likely voters". Our pollsters, not so much.

      In elections where turnout is around 50% that's a pretty important point.

      Delete
    5. re: chimuregna: actually a two party system might actually be MORE democratic than multiple parties under a first-past-the-post electoral system.

      on the other hand i totally recognize that it all depends on what one understands as "democracy". there's a lot of ways in which even a PR system may not be as democratic depending on what one understands as most important/salient to "democracy".

      Delete
    6. Agreed chimurenga. 110%. Wish we had STV here, even if FPTP did benefit my party this time.

      Delete
    7. That would depend on the PR system though JDC. IMHO, with the better systems you can make a pretty clear judgement that they outperform FPTP on just about every measure (except complexity). If we're talking about Israel's crapsack system then sure, there's issues there. No one seriously wants to imitate Israel's system though.

      Delete
    8. The type of system alone is no guarantee, of course. On paper, the Australian model seems much more likely to produce a more representative, democratic government, but because their media is owned by virtually one man, and because of other powerful business controls on politics, the end result is just as skewed (if not more so) towards business interests as politics in Canada or the US (with a difference in scale).

      Delete
  14. I think it would be interesting to see a graph that covers the span of the election that includes the "undecided voter". Perhaps if future polls made a point of displaying the undecided voters, it would be easier to see available swings in support. Also if the decrease in the undecided voters is roughly correlated to the increase in support for the BC Liberal, perhaps it would give us a better idea of what happened.

    ReplyDelete
    Replies
    1. I think what also may be important is to determine the probability that a respondent constituent will actually vote on election day.

      People should be asked the likelihood that they will vote on a scale going from:
      1. I will definitely vote.
      2. I probably will vote.
      3. I might vote.
      4. I won't vote.

      Then their preferences can be weighted against their actual likelihood to show up on election day.

      Delete
  15. I think you gave up on the momentum too soon. While the polls did not report any further gains near the end this is not how public opinion usually moves.

    I am as surprised by Liberal win as anyone else, but my last projection was 44 seats for the NDP, 40 for the liberals based on momentum alone.

    The rest I can only attribute to the opposite of Alberta White Rose effect. The WR party was not a sufficiently known quantity, the NDP in BC was too much of a known quantity.

    ReplyDelete
  16. Probably we need individual riding polls with adequate sample sizes, but the cost is prohibitive. Provincial and regional polls may not be able to consider the vote concentrations that work with/against the single-member plurality electoral system.

    ReplyDelete
  17. Wow, Éric, I really feel for you.

    Just yesterday (before the results came in) I was thinking about the tremendous amount of invaluable data you've generated on this site over the past 4.5 years. The in-depth coverage of all federal and provincial elections, the monthly averages, the rolling averages, the meticulous tracking and cataloguing of hundreds (thousands???) of individual polls, the consistently insightful, objective commentary and analysis, etc, etc.

    I really enjoy your site and admire you for having become in a sense the Nate Silver of Canada. And it seems natural that if it can be done accurately in the U.S., one should be able to pull it off here as well. But alas, it appears not to be so straightforward because your input data, over which you have no control, seems to be fundamentally faulty.

    You've pointed out numerous times that Canadian pollsters tend to lack transparency in their methods and results compared to U.S. counterparts and that Canada's increasingly over-saturated polling industry is causing a race to the bottom of sorts in terms of who can provide the cheapest, most plentiful polls. History indicates that when this sort of phenomenon reaches a breaking point (think of the video game industry in the early 80's), consumers (in this case, the media outlets that commission the polls) lose faith, stop buying and a major crash and realignment ensues, with just a few players managing to emerge from the ashes and "reset" the industry (think Nintendo and Sega).

    After the moderate miss of the federal election, the complete but somewhat excusable miss of the Alberta election and now the complete and inexcusable miss of the BC election, I've really gotta wonder if we're about to reach that breaking point as media consumers (aka the public) will now be near-irreparably skeptical of polls and media outlets will have to be growing awfully wary of publishing them as fast and furious as they have been.

    Much as it may bode ill for your day job, maybe a crash, downsizing and realignment is what the Canadian polling industry needs right now to salvage any credibility it has left. I would hope that from the ashes emerge a smaller, more robust, transparent and viable group of pollsters who can raise the standards going forward. Hell, in my opinion with your insight and experience the pollsters should be making you offers to come help them improve their practices and save themselves! You have plenty of great advice for them.

    All the best to you, Éric. You've been doing outstanding, commendable work.

    Dom

    ReplyDelete
  18. A scientific poll lets someone collect the opinions of persons selected at random to project over the universe of all persons living within a bounded area.

    There are 85 ridings (provincial electoral districts) in B.C.

    To be confident that 95 out of 100 times doing a poll of persons selected at random where those 95 would be within 5% of the answers of any other random polling, 376 persons would need to be surveyed within a short time of each other.

    For a scientific poll of the B.C. election, that means a proper survey would need 31,960 completed surveys (376 x 85) on any given day.

    Two key factors to know: (1) News changes people's opinions. (2) For every demographic split an additional 376 need to get surveyed.

    So say a pollster is legit and has the means to poll a whopping 31,960 eligible BC voters maybe as defined by having voted in the last election, and within a meaningful time frame, say 24 hours, if the pollster wants to know how females want to vote, the completed survey count needs to double. So a survey would need to have 85 * 376 * 2 or a whopping 63,920!

    Political polls are bunk nothing but mere fabrication published by news media of TV and newspapers to gain eyeballs. Pollsters are frauds, unscientific.

    ReplyDelete
    Replies
    1. However, the huge error we are seeing is at the provincial level. The popular vote, province-wide, was very different from what every poll predicted. You only need 1000 voters in the province to get a 3% margin of error with 95% confidence.

      Eric has already noted that his model for converting popular votes to seat projections is quite accurate. So, if we can get more accurate forecasts of the popular vote, we'll have a good predictor of the seat counts (although not necessarily of *which* ridings go each way).

      Delete
    2. There were 85 elections held on election day and not one. That means there needed to be 85 surveys, with at least 376 respondents for a margin of error of ±5% with 95 confidence level. That is statistical reality.

      So, 376 respondents x 85 ridings = 31,960 completed surveys, 376 for each riding to predict the winner in that riding. That too is statistical reality.

      The polling of 1,000 voters province-wide is not going to cut it, not going to let anyone project with accuracy the winner of each riding.

      It is possible to keep polling until discovering how many ridings a party shall win to know who shall seat the premier. In that, not all 85 ridings need to get polled.

      Yet, polling merely 1,000 can't tell you how many ridings shall be won or if enough shall be won to gain control of government.

      All Ph.D.s in statistics shall back up my claims.

      Delete
    3. Something that anybody with a Ph.D. -- heck, even a Bachelor's degree -- in any of the mathematical sciences should be able to tell you, is that if a poll accurately counts a party's vote province-wide, but overestimates its support in some ridings, it must underestimate its support in other ridings. That is why, even if a forecast gets individual ridings wrong, it should not get them all wrong in the same direction, and so should get the overall totals fairly accurately.

      The error that comes from a small sample size is random fluctuation. It is quite different from systemic error that has a consistent direction. That's why Eric's projection models are consistently quite good at converting vote numbers to seat numbers.

      This comment that you have now copied and pasted almost verbatim at least 3 times in this thread that I've noticed -- save it for another day, when there's an election in which everybody is surprised at how much the same party overperformed expectations in some areas and underperformed in others. Because that isn't what happened on Tuesday.

      I've said in other threads on this site that I don't put a lot of trust in *which* ridings a projection model predicts any party will take, but I have a lot of confidence in the overall count. So right now, I'm not baffled by how many ridings were projected incorrectly. I'm puzzled by how badly the overall vote, and consequently the overall seat numbers, were projected.

      Delete
  19. There is a question that needs a real look at. Is it possible for polling to be used in a nefarious way to manipulate the outcome of an election. When there are 4 recent examples of polling going very wrong, BC, Alberta, Quebec, and the recent US presidential election, it is a question that needs some deliberate thought.

    ReplyDelete
    Replies
    1. Actually, the recent US election polling was, when aggregated, pretty much spot on. What we are seeing is that Canadian polling is proving to be more and more inaccurate - there were explanations before, but BC is outright incorrect.

      Unfortunate for Eric, as it compromises his work through no fault of his own.

      Delete
    2. May I suggest that you review the Republican's belief that they were leading the polls, that Mitt was going to be the next President of the US of A. Clearly their campaign was very vocal in their belief that they were winning right up to the time of the count where to their amazement they were on the losing side.

      Delete
  20. I'm interested in the methodology of the polling and of artifact was introduced by language issues. A large proportion of the lower mainland speak English as a second language. Was polling conducted in English? Were immigrant communities engaged in the polling?

    ReplyDelete
  21. First of all Eric, these polling errors are not your responsibility. All you do is analyze results, and do a great job of it. As for the pollsters themselves, well, their lack of accuracy could be fatal to the industry. They are either irrelevant, inaccurate, impotent or all three. I'm also wondering if the move away from phone polls, and therefore the lack of undecideds info, is causing this skew. If say the last polls were showing an undecided of about 30 per cent, that could account for the dramatic shift we saw last night.
    In the end though, we should not blame the messenger for these results. This was Dix's race to lose and he did an absolutley astounding job of doing just that. You could excuse Iggy for blowing the Libs chances in 2011, he has little political experience. But Dix has been in the game a long time. He's got to go.
    Also, the NDP needs to take note of how the Libs won this election. They need to start campaigning almost immediately after electing a new leader, within the year. And no more Mr. Nice Guy. It doesn't work. This is war and you need to bring your A game. The Libs get it, the NDP doesn't, therefore, last night's result.

    ReplyDelete
    Replies
    1. I think the NDP simply needs to confidently offer a real alternative. They won't always win with such a strategy (and no strategy would), but they are doomed if they appear too much like the other parties, and they have a real chance to build on being a genuine alternative.

      Delete
    2. Agree chimurenga, but they still need to play the game, confidently yes. I've watched this left-right dance for decades and I think this was the NDP's weakest approach to an election I can remember. They can play the game with vigor and still not appear at all like the other guy. But they need to play hard and counteract the other guy's moves as they go. That didn't happen much or well this time around.

      Delete
  22. Eric, one thing you should keep in mind too is that a lower percentage of people in BC speak English as their first language than Quebeckers and French. If pollsters only polled in French in Quebec, how useful would those polls be?

    ReplyDelete
  23. Has anyone tried to figure out the probability of this being a coincidence? Clearly, it feels like a series of major screw-ups, but humans are notoriously bad at intuitive statistics. I would be curious to know what the math says. Given the number of elections in the world and the number of polls, is this event really as unusual as it seems?

    ReplyDelete
  24. Didn't some of the Canadian pollsters have some really good results polling in the US elections? if so then there is more to the story than just incompetent Canadian polling companies.

    My own theory is that some of the NDP vote was very soft. Just a general desire for change, really. Whereas the "anti-socialist" coalition stayed very solid.

    The actual NDP number was more like their minimal base of supporters. Also fifty years of history shows that the "anti-socialist" coalition in BC is pretty good at winning elections no matter how incompetent and corrupt voters think they are. And the Liberals have been more competent and less corrupt than

    ReplyDelete
    Replies
    1. That combined with complacency. I know of a bunch of left-leaning individuals in my own circle who simply did not vote - some even citing that their vote would be moot due to the significant NDP lead. It really is too bad.

      Truth be told, things aren't horrible under the Liberals - indeed, they are scandal plagued - but the province isn't in economic ruins. Things could be significantly better, but that, unfortunately, is not motivating voters these days. Like you say the "throw the bums out" mentality simply wasn't strong; to say nothing of Dix's failure to reinforce expand upon those feelings.

      Delete
  25. I don't have any suggestions as to why, but I remember that the last federal election's polls were also widely off the mark. Less of a big deal was made at the time, since the Tories won as expected. But the polls were showing it highly questionable whether they would get a majority or not, and it ended up being nowhere near that close.

    ReplyDelete
    Replies
    1. the Conservatives clearly won that majority (seats-wise) by an extreme micro-targeting of specific ridings though.

      that probably only works in specific elections when nothing "huge" is at stake that will move the electorate generally and decisively in one direction or another (though of course that was what happened in Quebec, if not elsewhere, last election. at the same time i'm sure Harper targeted precisely zero seats in Quebec so that didn't matter much for his strategy of "steady as she goes/don't rock the boat/look at all these little goodies just for you".

      Delete
    2. John-David,

      With Due respect Harper won a very small majority government. If the Tories won 12 less seats it would have resulted in a minority government. We did not know until the BC results were counted whether Harper would get a majority or minority.

      Delete
    3. Funny how the memory works. It had been so long that 12 seemed like a huge majority to me.

      Delete
  26. I think everyone is over thinking this. The NDP had this one in the bag and every one knew it. So why even bother voting. I think a lot of soft NDP vote just stayed home. Clark lost her seat. That is what happened when the NDP vote had a reason to get out and vote. There is a difference between saying you support a party and taking the time to get out and vote for a party. Why would anyone bother everyone had been told this election was just a formality we had to go through so Dix could get on with the job of being premier.

    Something kind of fun to look at. The schools had a mock election and cast 100,000 votes and these results turned out to be very close to what the polls had been saying would happen. This may not mean much as a one time thing but this was not the first mock election by the high schools in B.C. In the past the mock elections had been a fairly good indication of how the actual election would turn out, not so this time why?It is what it is.

    ReplyDelete
    Replies
    1. Some anecdotal support for this. I was talking, after the election, to someone who really wanted the NDP to win. "The NDP called me asking for a donation," she said, "and I told them 'No. You already have this in the bag.'"

      Delete
    2. If everyone thought the NDP had it in the bag, then why would Liberal supporters bother turning up to vote either? I would imagine it's Liberal supporters who should have been demoralized by what everyone was saying. A lot of what people are saying today is anecdotal post-hoc rationalizing. Personally I suspect there are deeper methodological problems at work.

      Delete
  27. Perhaps polls need to include...wait for it...a measure of whether a polled person intends to vote the way their preference was stated ON election day? Huge amount of error not accounted for by this one variable. I'm so disheartened for you and as an analyst myself, I trust good modelling. Don't lose faith in your work. See the opportunity to refine what you do and help the 'polling industry' work on getting better data to model from. Keep your chin up. We need you!

    ReplyDelete
  28. I think that the NDP in B.C. now know how the Toronto Maple leafs fans felt.

    ReplyDelete
  29. Maybe a fellow analyst like Nate Silver would be willing to give an opinion on the situation?

    ReplyDelete
    Replies
    1. Nate Silver's attempts in a multiparty systems in both Canada and the UK have not been impressive. I don't know if the model works well in these environments which are characterised by changing allegiances year-to-year.

      Delete
  30. i just wonder if Joffre doesn't have a serious point.

    We are a real multi-party country unlike the USA with it's two party system .

    So there were four formal parties running in most ridings as well as Independents. Thus this multitude does lead to far more nebulous results. And mistakes !

    ReplyDelete
  31. Have you put the actual numbers in your model so as to view the outcome, Eric? Unless that is wildly off, I don't think you need to take any blame for your predictions.


    As to the polls, geez. I haven't checked the turn out, but I wouldn't be surprised if it was down. Maybe NDP supporters just weren't rallied enough to get out and vote.

    ReplyDelete
  32. Just listened to Andrew Weaver on CBC, and he made a point that I quite agree with. I think, for the most part, the polls were right, just nobody showed up.

    This was one of the lowest turnouts in a BC election, so it wouldn't surprise me if a lot of NDP supporters stayed home. Whether it was because they thought they had a win in the bag or were discouraged because of negative campaigning, who knows.

    And Eric, I understand your disillusion with the polls, but I think people overlook how successful polling has been in the past. For every upset like Alberta 2012 or BC 2013, people ignore successes likeSaskatchewan 2011, Manitoba 2011, etc. Even in Ontario 2011, the polls were pretty good!

    It's an unfortunate day for the reputation of polling firms across Canada, but not for polls themselves in Canada.

    ReplyDelete
    Replies
    1. Interesting. I am very politically engaged, but can echo the sentiment that perhaps people just didn't show up. The negative ads were deeply frustrating and angered me to the point that I almost tuned out. I did end up voting, but imagine that those same feelings could have kept many home (combined with a sense of "safety" as many "safe" NDP ridings also fell last night). I suspect turnout will be low again.

      The Liberals won the election, time will tell if they truly won the support of British Columbians.

      Delete
    2. I just posted a comment indicating that I suspected turnout was down; it actually went up! I am completely mystified...

      Delete
  33. This comment has been removed by the author.

    ReplyDelete
  34. Liberal Ted made the comment about Manitoba and Saskatchewan election polls and how right on the predictions were but considering the provinces I would ask if the corporates had as much at stake there as they do in the other provinces that had such a great disparity in the polls. Gas, oil and pipelines as well as free rain for big corps. Statistically these results seem more then just an anomaly. Bad polling in one place, good polling somewhere else seems odd. Is it possible to manipulate the polls? I would tend to believe it is just as possible as manipulating the stock market, setting lending rates, starting wars with lies, collapsing world economies, and the list goes on in this very insane and corrupt world of greed. Liberal Ted made the comment about Manitoba and Saskatchewan election polls and how right on the predictions were but considering the provinces I would ask

    ReplyDelete
  35. Funny how all the conspiracy theorists are coming out. So in one election (AB), the pollsters UNANIMOUSLY favour the farthest-right party, and in another election (BC), the pollsters UNANIMOUSLY favour the farthest-left party, cause ALL the pollsters are clearly in collusion with one another and know exactly how to "manipulate the public" in each case to get what they want. That's how far the science has come.

    To get real, it seems to me that the more "moderate" (so NEITHER far left or right) parties in each case (QB, AB, BC) underperform in their polling numbers, which may be a sign that the people most eager to take time out to express their opinion to pollsters are (usually?) those at the more extreme ends of the spectrum. Maybe models have to account for "enthusiasm" not just at the polls, but those most "enthusiastic" to talk to pollsters in each case (not easy to account for I know, but we shouldn't let available methodology dictate theory).

    ReplyDelete
    Replies
    1. Interesting jdc that you would be on the conspiracy theorists train. I believe that all the incidents that I stated are historical facts and many were covered by using the term conspiracy until proven otherwise.
      For you to state that the pollster are in collusion with each other overlooks other possibilities of manipulation. Clearly there is an lack of understanding on your part of the ability of people to manipulate data for their own purpose, without collusion. After all Christy Clark's spin machine is always in overtime misrepresenting the truth, and I am being kind using the word misrepresenting instead of outright lying.
      Stop with the conspiracy theory response, educate yourself in math and technology, then consider what was simply questioning the status quo.

      Delete
    2. JDC was clearly being sarcastic and arguing against the conspiracy theorists...

      Delete
  36. There are many possible reasons why this might have gone so badly for pollsters.

    It could be that Chinese-speaking populations were underpolled, and they tend to vote Liberal.

    It could be that young people, who disproportionately vote NDP, had terrible voter turnout.

    This result isn't necessarily catastrophic for pollsters (though it might be).

    Is there any pattern in the error? For example, did Éric accurately predict the interior, but not the GVRD?

    ReplyDelete
    Replies
    1. I don't know about the error, but there's a pattern in the outcomes within the GVRD. The NDP gained seats in the "city" (Vancouver and Burnaby/New West) but lost seats in the suburbs (the Tri-cities, Surrey, and Delta).

      Delete
  37. Here's another caveat: I worked at a voting place where there was a surprising number of absentee voters. (more voters than those at any other poll). This was possibly the result of confusion over where to vote, but it may also be the result of Elections BC ecouraging people to vote wherever they were on Election Day. If the NDP also encouraged its supporters to vote absentee if necessary, then you may see some of its lost numbers returning during the final count in two weeks (or not).

    With several close races, a possibly high absentee-voter turnout (to be counted May 27th-29th), and the parties differing abilities to pull the vote, there may still be some surprises to come.

    ReplyDelete
  38. I have every confidence the polls were accurate, where they failed was translating raw data into likely voters. We know the NDP was heavily favoured among the 18-34 cohort but, my suspicion is very few of them voted. Whereas, Liberals supporters vote on a more frequent basis.

    In short "the piblic" and "voters" are not synonymous with one another.

    ReplyDelete
    Replies
    1. All of that is rather easy to overcome if doing proper scientific voting polling.

      Merely, one need only call at random those who have voted in the last three elections.

      The methodology of the B.C. Pollsters is wrong. That is why their polls are wrong.

      In comment above, I have explained how scientific polling works and how it would need to get done for B.C.

      Delete
  39. Maybe the general public is getting tired of pollsters and is responding randomly and/or dishonestly just to mess with them and their partisan masters? I know that if a pollster called me I'd pretty much tell them the opposite of what I'd really vote just so that the side I didn't want winning got complacent.

    ReplyDelete
    Replies
    1. You won't necessary, or even usually, make your opponents "complacent" though by boosting their poll numbers artificially. (That's just the frame that everyone is going by today, the day after the election, even if nobody REALLY knows what happened yet, beyond what they've heard anecdotally among "people they know" who didn't vote -- clearly the most representative sample of the general public possible :P.) Indeed, you might end up making your opponent look like they have momentum (which can lead to further momentum, a la the NDP in 2011), give them a morale boost, make them look as if they're "winning" on the issues/debates, or do myriad other things that actually aids your opponent in public perception, depending on the situation.

      I doubt enough people do this to even matter, and those that do probably cancel themselves out. Clearly there are some methodological and sampling problems that have to be dealt with, but I highly doubt this is one of them.

      Delete
  40. This is seriously interesting stuff, Éric. I wouldn't give up, but take the opportunity to consider a more dynamic model.

    Discussing the issue with my colleagues at an economic forecasting firm, we make a couple of observations as to why the polls and model have failed. The crux of our thinking is that this is no two-party system a la Nate Silver's fivethirtyeight system, and is hence more complicated, in particular voters' behaviour is driven by the information that they have when they go into the booth (or more specifically, twice, including when they decide to go to a booth at all).

    Historically, in a semi-close election, there is generally insufficient information to determine who would win your riding and the province. This would lead to a certain strategic voting behaviour.

    This time, our hypothesis is there is a well-advertised outcome at the provincial level has changed voting behavour, which could have manifested itself as different turnout and different strategic voting, which could be markedly different that what people declared during poling.

    The implication of this is that the high polling numbers which led to a high modelled and reported confidence in an NDP victory could have paradoxically been a negative when it came to actual voting behaviour.

    I don't really blame the model for this, as the published polls and associated media are likely a greater information source for the standard voter, but the model should be adjusted to incorporate voter behaviour under differing information sets. This would be very difficult to do, but a more complicated poll that asks voters their thoughts about who would win their own riding and the province as a whole, and perhaps their confidence in that outcome, before asking them about their voting intentions (if they would vote, and who they would vote for), might be the ticket.

    I wouldn't be surprised if you have had thoughts along these lines, Éric, but would welcome your opinion.

    ReplyDelete
    Replies
    1. I am limited by the information I have available to me during a campaign. But, yes, I will be re-thinking many things.

      Delete
    2. Eric I'm quite sure you and many others will be doing that.

      But right now nobody knows !!

      So let's drop all the babble until some clear consensus is developed.

      Right now nobody knows. Give the experts time please !!

      Delete
  41. Well, I think in both Alberta and BC, they did not properly compensate for the likelihood of the persons to vote. The Wildrose drew alot of support during polling from the trailer park crowd that didn't want to leave the bar stool on election day. The NDP in BC drew alot of support during polling from students and young people who are notorious for not showing up to vote on election day. I believe that in Alberta, if they restricted the polls to the people who had voted in the last election, they would have accurately called it. There were private polls in Alberta(not big media company polls) that had accurately predicted the numbers to within 1-2% point even a week before the Alberta election by using this criteria.

    ReplyDelete
  42. It puzzles why people want to look for excuses rather than look at reality.

    The pollsters failed because they fail to use the scientific method needed for public opinion surveying.

    There were 85 elections held yesterday and not one. That means there needed to be 85 surveys, with at least 376 respondents chosen at random who were screened to have a likelihood of voting, say by having voted in the last three elections. The 376 respondents per riding would yield a margin of error of 5% at the 95th confidence level (95 times out of 100, any other 376 at random selected respondents would respond the same way).

    So, 376 respondents x 85 ridings = 31,960 completed surveys. That is a huge number to get done in one day.

    Never were surveys done this way. Instead, all pollsters did was take the opinions of maybe 500, maybe 1,000 people and thus could not project to the universe of voters in each riding.

    Ask any Ph.D. with a Ph.D in statistics. They shall agree with me.

    85 separate elections requires 85 separate surveys.

    All B.C. polling has been a massive fraud pulled on the people, perpetuated by news media.

    ReplyDelete
  43. 1. I, and my family, hang up on robo or machine calls. There are a lot of those but, of course, it's the cheapest polling method. It's 'no person - no listen' in our house. What about you? Honestly, do you listen to all those calls? All that drivel, stuff, nonsense and hangups? You do? Step right up, pollsters want your opinion.

    2. There are so many polls and surveys that I'm leery of "just a few minutes of your time" and the like and I sometimes demand payment if they want 15 minutes or more. When pollsters hang up on the likes of me the random sample is eroded and the polling sample no longer represents the population as a whole. Bad statsistics!

    3. Automated phone polls and surveys have become so simple that anyone can do it with a personal computer or even link dozens in a virtual call centre. It's too easy and that makes too many calls. ...and I'm tired of it!

    4. My kids don't have phone numbers listed anywhere although their cell phones are always on. I resent polling and survey calls on my wireless phone because I pay for every minute but also because I have better things to do than answer garbage calls while I'm driving, walking or they possibly catch me in the men's room. I may not be at my best, I might even be rude or I might do worse.

    5. When telephone polling first started not all voters had phones. Now a very large proportion of voters have unlisted numbers. I even have two different numbers on my cell phone. Again, voters aren't the same as home phone numbers without unlisted cell phones. Not any more.

    6. My kids are always on the move - three or more mini jobs will do that. Both faithfully promised to vote but I'm not sure either actually did vote. "Is there an app for that?" they asked but there wasn't, of course.

    7. Predictive calling centres are everywhere but unfortunately they ring a huge number of telephones - and hang up on most of them when an operator can't be found quickly enough. I hate those hang-up calls from 1-800 numbers and I'm predisposed to be rude or hang up when I get one of those calls.

    ReplyDelete
  44. I think some of the other commentors are on the mark. Polls for American polls tend to be pretty good, and they typically always tell you the undecideds. I don't know what if anything Canadian pollsters do in terms of a likely-voter screen (i.e., filtering out respondents who aren't likely to vote), but if they aren't doing it, they should. And it would be helpful if they provided both sets of results so people like you can try to decide if their screens are too harsh or too relaxed. Finally, given the results of the last year (and the federal results weren't stellar either), pollsters should probably be testing softness ("certain to vote for X", "likely...", "somewhat likekly...") and reporting those when they seem unusually high.

    Given that the polls are only recently turning to garbage, it seems more likely that the undecided/internet-polling issue is at fault because nothing has really changed about likely versus unlikely voters.

    What truly shocks me is not that the public polls are bad, but that the private ones seem to have been just as terrible. I haven't seen them, but the reporting (including interviews with Liberal minsters last night) suggests their polls were also telling them they would lose.

    ReplyDelete
  45. Hey Eric,

    I have never posted before but I've been reading this site for years now, and just want to say that you shouldn't get discouraged over this. Your projections have tended to be spot on with the correct data, and you can't control the quality of the polls out there. Hopefully after this and Alberta the major companies will look at and try to improve their polling methods.

    When you have new polls constantly coming out in campaigns, having your site here to try and get an aggregate view is incredibly valuable, and I hope you keep at it. No one else in Canada is doing this sort of work in real time that I can find and not having this here would make the political commentary of Canada far poorer for it.

    Hoping you keep at it,
    Douglas F

    ReplyDelete
  46. As far as I know it is standard practice in virtually all political polls in Canada for there to be a question about how likely people are to vote...so if you think polls didn't screen for likely voters you are barking up the wrong tree. In fact Ekos applied a rigid "likely voter" model in their final BC poll and it EXPANDED the NDP lead.

    ReplyDelete
    Replies
    1. Well, it could be that voters just got cold feet in regards to the NDP. I think that Dix's internet ads might have been a turn off, by making it sound like he would prioritize things like the environment over the economic issues that affect the day-to-day lives of the ppl. I do not live in BC, and can say we were saturated by Dix ads on the internet as well in Alberta, so over-exposure might have left people sick of him.
      In Alberta last election, internal PC polling put the PCS exactly where they ended up. People privy to that polling did not believe it, because all the media polls said otherwise, and it would seem they would be more impartial. Yet, ironically, the PC's internal polling was pretty much dead on, even on the riding projections. And I'm talking a week before the election they were saying their polling showed these results, not a sudden weekend-before surge.

      Delete
  47. You shouldn't me discouraged. I'm a Canadian living in the United States and your website is one of the only ones I check regularly hoping for an update. The polls have been bad, but that is not your fault. I can read polls anywhere, what I find interesting about your work is your seat projections from the polls and those have been spot on for a given vote. As a result the one thing I"d find super interesting for you to add would be along with having seat projections from the poll averages, also having seat projections with different voting percentages perhaps +/- 5 percent in either direction.

    Anyway keep up the great work!

    ReplyDelete
  48. Please hang in there you have an interesting website.

    I do think that people have a little polling fatigue and are now starting to fool with pollsters. Considering all of the robo calls that have occurred during elections (Ontario in 2011 we had federal, provincial and municipal elections) to try to gauge people's support i think we are witnessing the backlash of all of these intrusions on people's privacy.


    ReplyDelete
  49. I think there are three main uncertainties here: (1) did individual respondents change their minds when they voted, (2) did pollsters correctly identify who actually ended up voting, and (3) how reliable are the answers to the questions posed by pollsters?

    Re: (1&2): The only reasonable way to get at these questions is to track and re-poll individual respondents, including after the election. This way, we can learn how stable individual voting commitments are and whether certain individuals actually voted.

    Re: (3): Current polling questions ask for a discrete response - which party would you vote for? I would recommend instead that at least one polling company start asking "On a scale from 1 to 10, how much do you support Party A, Party B, etc." This could show more subtle and gradual changes of support before they show up in actual changes in voting preferences.

    ReplyDelete
  50. I know this wouldn't explain it all, but is it possible that part of the difference could be explained by voters thinking it was a foregone conclusion and deciding to stay home? I know the argument could be made that Liberal voters could have assumed it was a guaranteed loss, but if NDP support was softer than Liberal support, maybe that played a role.

    ReplyDelete
  51. I think we had now 4 elections in which the pollster underestimated the willingness of Canadian voters to stay the course http://drupal7.txwikinger.me.uk/content/what-does-bc-liberals-win-mean-canadian-political-landscape

    I think it has less to do with ideology and more with giving up on politics and contention that currently people's situation is better than they are willing to risk what any opposition can achieve.

    The interesting question would be... do people lie to pollsters.. or they tell the truth but don't vote, or pollsters are not capable to find a representative group of people.

    ReplyDelete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.