Tuesday, May 3, 2011

2011 Federal Projection vs. the Results

Well, at least I got the order of the parties right.

Clearly, the final projection was wrong. It under-estimated both the Conservatives and the New Democrats and over-estimated the Liberals and the Bloc Québécois. While I was not alone in making this error, I humbly recognize that of all the projections mine was among the worst.

Of the 308 ridings in the country, ThreeHundredEight.com correctly called 234 of them. That's an accuracy rate of 76.0%, which is absolutely unacceptable.

However, it was not the seat projection model that failed. The seat projection model actually performed very well - or would have had the popular vote projection model not missed the mark so completely.

In fact, inputting the actual provincial-level vote results into the model, treating them as the results of a poll, churns out a very accurate result.

With the actual popular vote results in the 10 provinces, the projection model projects 161 seats for the Conservatives, 106 for the New Democrats, 37 for the Liberals, and four for the Bloc Québécois.

In other words, the model would have been off by five seats for the Tories, three for the New Democrats, three for the Liberals, and one for the Greens. It would have had the shocking four-seat result for the Bloc correct. In total, it would have correctly called 270 ridings for an accuracy rating of 87.7%. That could be better, but considering the wild changes in support throughout the country that is not a bad result at all.

In short, the seat projection model was able to turn regional results into reliably accurate seat results. Claims that seat projection models cannot capture huge, historic changes are false.

Accurately capturing how people will behave at the polls, however, is a different matter. The vote projection model failed irrevocably. It was partly the fault of its slow progression and partly because of the inaccuracies of the polls.

I should be clear that the polls were not terribly inaccurate. They were actually quite good in pegging the support levels of the Liberals, New Democrats and Bloc Québécois. But they under-estimated the Conservatives, arguing for a strong minority rather than a relatively strong majority government. This was especially problematic in British Columbia, Ontario, and Atlantic Canada. On the other hand, full marks go to the pollsters in unpredictable Quebec. They were all very close to the actual result.

ThreeHundredEight.com projected two things: the popular vote and how that would transform into seats. It failed at the first, which means it failed at the second. Work will be done to get the popular vote projection model up to snuff in time for the autumn provincial elections. However, I am very confident that my seat projection model is an accurate model, and will accurately project elections as long as the right numbers are put into it. Had I simply trusted the latest polls, rather than including weighted polls stretching back to the beginning of the campaign, my projection would have been much closer to the actual result.

I do apologize for the inaccuracy of my projections. But in the aftermath of such a historic election result the projection model has proven that it can be used effectively. I will work on ensuring that in the next election I will use it properly.

88 comments:

  1. No apology necessary. It was a complex set of variables you were trying to project and, hey, perhaps the first election was nothing but a learning experience. Please come back next time and try again.

    In comparative to fivethirtyeight, I think your job was exceedingly more difficult. They dynamics of three party politics and vote splitting, especially with the historic pressures put on the liberals made this a difficult election for anyone to call. Personally I think a lot of long time liberals peeled away in the final 48 hours to other parties in order to stop what they thought was the greater threat - the CPC or NDP. It is hard for any poll to capture that.

    ReplyDelete
  2. You have nothing to apologize for...you did a great job and basically got undercut by inaccurate polling. I respectfully disagree with your comments about polls being mostly accurate. When they continually underestimate the winner by 3-5 %, it makes a huge difference because everything is predicated on what the leading party is doing. I believe pollsters may have to look at who they are sampling. My strong suspicion is that they are going to much to groups that are "connected" to various forms of media. In others words, younger, hipper, savvier votes. People that tend to vote left, or centre-left, leaving a huge swath of voters not represented in the polls (ie: older voters who don't have cell phones, aren't on the internet constantly). To be completely honest, I trust the Licks Burger Poll more than all but Nanos. The only polling company that goes too far the other way is COMPASS

    ReplyDelete
  3. I'd like to say that I've very much enjoyed looking at your projections during the course of this election.

    For future elections, will you have a shorter "half-life" for the effects of a poll? I feel that although this will mean "blips" are initially stronger, they will also disappear more quickly, while truly building momentum will also be visible more quickly.

    Again, good job, and I look forward to seeing your predictions for the Ontario election.

    ReplyDelete
  4. I think you have nailed the flaw, which is that including start-of-campaign voting intentions in the end-of-campaign projections gives a distorted picture. It simply doesn't account for a volatile electorate changing its mind massively during the course of a campaign.

    ReplyDelete
  5. Yes, I admit you all told me so. I felt that it could be the case, but I wanted to stick to the methodology I had settled upon.

    My gut told me that the NDP wouldn't get the vote out and so the flaw would inadvertently turn out to be right, but that was not the case.

    Lessons learned, I'll hopefully re-gain some trust in the provincial elections this fall.

    ReplyDelete
  6. To say that the model works with actual versus projections does not make it valid for what you have been doing for the past five weeks.

    Building projections upon projections is assured to be wildly inaccurate. That in itself is fine. Publishing it with an illusion of validity is, in my opinion, irresponsible.

    ReplyDelete
  7. Couple of comments:
    1) You've done a fantastic job -- with work you will do better.
    2) I suspect the seat projection model actually did exceptionally well this time due to election-specific factors, and probably will not do as well in other contexts (e.g., more uniform swing). Not that I have any suggestions for improvement, just recommend that you look closely riding by riding and consider tweaking.
    3) Could you estimate and post a projection just based on the latest polls, going back a few days to get as many companies as possible? Would be really interesting.

    ReplyDelete
  8. I disagree. The model is able to turn provincial level results into accurate seat projections, something that others claimed was impossible.

    I've demonstrated that the seat projection model is well calibrated, and can be used effectively.

    Had the polls not shifted so late in the campaign, the model would have likely given an accurate result. It failed at capturing the swing in voting intentions. For the first three weeks of the campaign, the projection was almost certainly very close to what would have happened.

    ReplyDelete
  9. I came to your site everyday before I went to the newspapers for their chatter. There are a lot of great things going on here and every model needs to try and fail to work the bugs out. Clearly you are correct in that what happens (polls) at the beginning does not influence the end. I would consider dropping earlier polls off from the average each day as time progresses, and adding additional weight to the final polls. Keep up the great work. Looking forward to provincial analysis.
    Jason

    ReplyDelete
  10. I think that there may have been a significant transfer of votes from the Liberals to the Conservatives over the last two days of the campaign, when talk of a "surge" of the NDP made a lot of right-leaning liberal voters nervous. It would have been difficult to capture something like that in the polls.
    The huge uncertainty with which we have to live each time there is an election, of course, has to do with the particular electoral system. If there were a proportional representation system, we would have known a week ago what the House would have looked like.

    ReplyDelete
  11. Hey Eric: Don't be too hard on yourself.
    What happened last night had nothing to do with the logic of statistics, mathmatics and polls. Pretty hard to predict or even record it accurately. I certainly was wildly off. We all got caught off guard by the sudden wild wave of NDP support and the reaction to it, from right-leaning Canadians.
    One of the mistakes we make on this and other sites is to analyse each poll as if it actually reflects what's going to happen. In reality polls only show general trends and as such work well. However, even that fails somewhat when we get a sweep like we did last night.
    It was an amazing experience. We'll see how it all shakes out.

    ReplyDelete
  12. Please don't apologize. This site was a fabulous site with an insane amount of work that was obviously put into it. I thoroughly enjoyed following it during the campaign!

    I have to say that you had me very nervous for my own candidate in South Shore-St. Margaret's, so I was pleasantly surprised last evening when his numbers were so much higher than expected....

    Thanks again for taking so much time to do this. It was a very enjoyable process!

    ReplyDelete
  13. Very interesting to see where the problems lay. I seem to recall making a prediction early in the election that pre-election volatility in the polls would predict the volatility in the polls during the election. Clearly that theory has been shot completely out of the water! ;-)

    So polls need to age faster, that much is clear. How much faster? I think the factor that people are going to miss is to try to age them fast enough that you get 2011 correct without looking back and seeing how that would have affected what you would have predicted in previous elections. I hope you won't end up overfitting your method in this way.

    ReplyDelete
  14. FIRST - thank you for all of your efforts and hard work - very impressive. But just as impressive is your willingness to learn from mistakes. Clearly the problem with the model is not the methodology, but the numbers that went into it. Aside from shortening the time old polls stay in the system, what else do you foresee to improve the program?

    ReplyDelete
  15. Excellent job and hard work you put into the project. It's my theory that the projection problem was complicated by the undecided vote, which I think was 10 to 12% right to the end. I was in that camp right up to Sunday night, torn between the NDP and the Tories, and in the end decided for the Tories. I found it unusually difficult to make my decision.

    ReplyDelete
  16. I've enjoyed reading your blog through the election, regardless of the outcome. But, you're going to have a hard time fixing the polling side. They were not that accurate across all provinces; unless they only focused on that province. Sure, you could pick one or two that might have come close, but your model doesn't just use one or two polls. National polls that take a few people from each province are not going to help you much at all; except for the national numbers.

    Oddly, I think if you run the numbers with the Conservative and NDP ceilings (adjusted where they overlap) your seat projections might have come up with some interesting results. The decision to do that though would have had to been a choice, rather than the model understanding that the Liberals and Bloq were in tailspin mode. May picking off a cabinet minister was very likely to be a random event anyways.

    Last night, vote splitting caused a lot of ridings to go Con. They had less than what could be considered 'threat' support in some of these ridings, but managed to sneak some in. It'll be interesting to see how many ridings ended up this way across the country.

    There was also strategic voting all across the country. I voted NDP, and got others to do it, in Laurier-Sainte-Marie for the sole purpose of doing a happy dance on ending Duceppes federal political career. Some of us would have been considered Liberal or Conservative in the polling. We just wanted him gone, so made the right choice. (for now. Next election we go back to fighting it out)

    On top of all of that... Never trust the statistics. 100 people polled in BC as part of a larger national poll will tell you almost nothing about the BC numbers. It's much harder for your polling projections to work well regionally if the sum total of people polled is so low there. Likely, the only way you could have gotten good regional polls is if there was a total sample size above 2000 for each province over the final few days of the campaign.

    The targeted riding polls likely could have given you better results for the provinces if you inverted the model and adjusted for incumbency + regional history. Ie, a high poll in a Conservative stronghold in Quebec for the Conservatives won't change the model much, but if the NDP shift up 10% there, it should affect their numbers. Similarly, a riding poll in Alberta that shows a conservative below 65% (vs. the usual 70%+) might mean something nationally.

    ... and finally: We all know that each polling group is biased; whether they admit it or not. Their methods are biased to specific types of groups; whether they admit it or not. Asking leading questions and trimming the poll group to ensure that 'most likely to vote' are in there bias the responses. Not to mention the calling at dinner time.

    ReplyDelete
  17. Does the model account for historical differences between polls and actual results?

    Seems to me the common element is that for the last few elections pollsters have underestimated the Conservative popular vote (whether due to bias in the questions, who they're polling, how likely to vote, whatever).

    So, take the popular vote from polls, adjust by how far off they were the last time (by party) and your model should do a better job.

    For me, among friends and family I was alone in predicting a Conservative majority, but my methodology was simply to add 3% to the Conservative popular vote numbers from the Sunday polls.

    ReplyDelete
  18. I'm not sure that any model could have picked the results last night. It was one of those elections with too many variables. An historic election. My gut feeling going into last night was that the CPC would get a majority based mainly on the polls in Ontario and the fact that the party's supporters have shown in past elections they are more committed. However, I would not have been shocked with a similar result to last election either.

    An interesting exercise will be to review the pollsters and their success or lack thereof. It appears that Ekos (and some others) took a big hit to its reputation last night and that Nanos did a much better job of capturing what actually happened. The post mortem of this election will be fascinating.

    ReplyDelete
  19. Eric

    Ekos final polling sample 2011 cpc 33.9 MOE 1.8

    They had CPC under by 5.7% points. This is the one poll that is wrong 1 time out of 20??

    The top end for the CPC from ekos was 35.7

    HD off by 3.6 MOE 3.1 (1 poll out of 20)

    Nanos off by 2.5 MOE 3.0 ( within Moe)

    If you ran your model without EKOS and HD you would be a lot closer and gave less weight to Nanos.

    Forum, Abacus, AR and IR and Nanos are the pols that were within the MOE.

    The Media stars polls Gregg and Graves are for entertainment purposes only.

    ReplyDelete
  20. It is not at all irresponsible to publish the results based on what he thinks is the best methodology. Eric is fully open about all aspects of the model and its methods, and if anyone is ever skeptical they can check up on that. This gives it as much validity as any other projection. There's nothing harmful about having incorrect predictions so long as the methods are transparent and the author is accountable.

    Eric, it is great to see the proportional swing model works so well, there is no reason it shouldn't so far as I can see. I would actually say you should be more hesitant than you seem to completely abandon the conservative assumptions in your model as this election looks more like an outlier than the norm in terms of polling accuracy. Is their enough past data to try and assess some kind of normal behavior for polling changes throughout the campaign translating into popular vote?

    ReplyDelete
  21. Keep up the good work! Your site was a great help in comparing the early results versus projections, and enabled me, in my own mind, to call the Conservative majority at least 45 minutes before CBC did!

    ReplyDelete
  22. In my opinion, not enough credit was given to Nanos's final-day poll (May 1st polling only, 750 ppl) showing a 38.5-30.7% margin for the Cons/NDP - almost identical to the final results. Why did nobody discuss this?

    ReplyDelete
  23. No apologies necessary. I have enjoyed reading following your projections throughout the campaign. The crazy changes in seats for all of the parties made predicting this election particularly difficult.

    I even trusted you enough to use your projection as the starting point for the projection I used in an election pool. I tweaked the numbers based on some of my gut instincts and personal opinions (dropped the conservatives, increased the NDP etc), but it was similar to yours for the most part. I managed to get third place, so you helped save me from losing $10. And for that, I thank you sir.

    ReplyDelete
  24. Je n'ai pas pas la compétence nécessaire pour évaluer votre modèle, mais le refus des faux-fuyants, l'intégrité et la modestie qui caractérisent vos propos, ça, je peux en juger. Ce sont des qualités très rares, qui sont à votre honneur. Merci de votre travail et... à la prochaine. D.Goulet -Montréal

    ReplyDelete
  25. Thanks for the great work - I really enjoyed this site - don't be too hard on yourself!

    ReplyDelete
  26. Hey, no need to apologize. You did your best, you'll learn from it, and in the meantime I (and I'm sure many others) had a very enjoyable and interesting time on your blog. Thanks so much for all your hard work.

    ReplyDelete
  27. Vote splitting won it for the Cons in Ontario. No way you can model that accurately.

    The Con vote did not change at all all, give or take a percent or two, across the country throughout the campaign. In fact, nationally it is only 2% higher than 2008 and 3% higher than 2006, when they won 143 and 122 seats. it's been established that about 38%+-1% of the population will vote Con come hell or high water, over the past 6 years or so.

    so with 3% higher vote, they go from 122 to 167 seats. under 40% of the vote gets them either a weak minority or a solid majority. We truly have a f***ed up electoral system.

    hard to model a screwed up electoral system.

    ReplyDelete
  28. Few ideas:

    1. If EKOs and Nanos polls are the most frequently used, recognize they have bias and counter them. The Canadian election Polls site I think shows all the polls as well as past performance. It shows that EKOs has underestimated CPC by 3-4% (this time by 6% I think).

    2. The Canadian Federal Election Site got 76% right I think - way down from last time. By reviewing that and maybe setting up something like this where you can solicit comments in almost a delphi technique you get valuable feedback. Then you can avoid Peter MacKay type mistakes- he won by his biggest majority ever I think.

    3. Don't alter your weighting too much away from being able to reflect long term trends- you would have underestimated CPC even more given that their 5 week levels were pretty solid and when if you had adjusted them to recognize polling firms bias would have been right on for CPC.

    4. There is no way to reflect the NDP surge in your model except to confine it to the region it is occuring in. Margins of error are broad in the regions and only in Quebec I think where they outside them.

    5. For the frequent daily polls, you may want to adopt some rules for changing polling levels in the regions given their large margins of errors. In manufacturing they have rules I think that say such things if you have x number of results that are to one side of the average but not above the tolerances you can assume change, anything above or below tolerances is change etc. Otherwise no change. I have used it in interpreting marketing signals. I hope you get what I am saying.

    5. I noted that Nanos Sunday Poll did catch the Tory upswing their and outside tolerance levels. Since the last weekend the last day is critical you may want to reflect that in your weighting.

    I assume this is all built on some swing analysis and works like a mechanical watch - the large pinwheel (regional polls) swings all the small pinwheels (ridings)

    It is critical to get that large pinwheel right. There is where your work is needed and if you can drive some from the bottom up from qualitative riding commentary used at the end after your model generates its final results to manual alter some riding predictions go for it.

    Be proud Eric you are in the news while I am just typing poorly into an anonymous comment.

    Congrats guy
    Gar Y

    ReplyDelete
  29. Eric, your model seems fine enough, but I can't help shake the feeling that the cynics are right about the value of projecting.

    The latter half of the campaign was overtaken by the horserace issues where the first half was more about polticial issues. Worse, people ended up making their decisions based on information from pollsters that was incomplete. Through this lens, the failure of the pollsters is all the more appalling.

    I suppose they can only report on what they get, just as a prediction can only model what they give you; but the end result was a distraction in addition to being wrong. Not that that is your, or a pollsters' responsibility of course; Blame for that lies with the newsmedia. Its just that it seems that this endeavor is only of scientific merit and not of civic merit.

    While I don't think I've ever lost my own focus on why I'm voting for anybody, I do think my abundant interest in this kind of news is probably not good for me.

    ReplyDelete
  30. Anshu,

    Inoted this in two elections now the undersetimate fot CPC. Do you think that is Company Polling Bias or people not wanting to say they were going to vote for Harper because he has been so villified.

    ReplyDelete
  31. In fairness to the pollsters, the nature of their business is backwards looking. I'm not sure if the problem is that their polls are inaccurate or if Canadians just insist on changing their minds at the last minute (for what it's worth Nanos' Sunday May 1st numbers were almost right on the actual result) and polling, inherently, can't catch that change (at least not if Elections Canada bars publications of results on E-day).

    In any event, Eric, this is a great project, and you shouldn't be discouraged by the results. I think the value of this project is in the process.

    However, I would be remiss if I didn't make one point. Contrary to your oft-stated opinion, this election shows that you CAN win a majority government without winning seats in Quebec (which will become easier when new seats are added in Ontario, BC and Alberta before the next election). Whilte that's not neccesarily ideal (both because it's easier if you win seats in Quebec, and excluding Quebec from cabinet can't be great for national unity) I'll be curious to see how the dynamics of that plays out in Ottawa (and Quebec) over the next four years. At the very least, I suppose we'll see Maxime Bernier back in Cabinet.

    ReplyDelete
  32. You need to lose previous elections in your weighting and compensate for pollster bias.

    EKOS blatantly underestimates the Conservative vote. 6%??????? Come on, that's not a mistake or random. That's bias in their survey methodology.

    ReplyDelete
  33. You also need to stop doubling a pols weight because it's sample is doubled, when it only reduces the MOE by 1%. You are weighting polls way too much by sample size.

    If you hadn't discounted the Compas poll so much for having only 750 respondents, your model would have been closer.

    ReplyDelete
  34. I calculated the kappa statistic for your model, Too Close to Call and Democratic Spaces riding-by-riding. Here are the results:

    Democratic spaces: 0.73
    Too Close to Call: 0.66
    308: 0.62


    The kappa statistic measures agreement beyond pure chance (i.e. your 76.3% includes both agreement by chance and due to your prediction, and it also doesn't take into account riding-by-riding variation).

    0.62 is still pretty good considering how volatile this election was. Well done!

    ReplyDelete
  35. Desmond Hume03 May, 2011 12:10

    Eric, congrats on the blog and running a very interesting and addictive experiment. While the outcome might have been off the mark your frequent updates and insights were welcome additions to the campaign. Good work and I look forward to seeing your model in action for other polls and provincial elections.

    ReplyDelete
  36. "EKOS blatantly underestimates the Conservative vote. 6%??????? Come on, that's not a mistake or random. That's bias in their survey methodology."

    No, it's called statistical uncertainty. It's called getting the vote out.


    the blatant ignorance about statistical sampling continues to boggle the mind. With all the polls going on, its EXPECTED that somebody gets its off by that much. If it didn't happen, that would be proof positive that the samples weren't random.

    ReplyDelete
  37. I think another factor that hurt your projections was voter turnout. It's higher than 2008, but still historically low (estimated at 61.4%.) Low turnout typically works well for the Conservatives, as they're very good at making sure their supporters get to the polls. I suspect a lot of would-be Liberal voters stayed home last night, whiile there was an increase in Conservative and NDP votes. If people were polled but didn't vote, that will hurt any projection model.

    I don't know if it's possible to include expected voter turnout in a model like yours, but I'd suggest factoring it in if you can.

    ReplyDelete
  38. Eric your site gives us addicts something to feed our addiction. Keep it UP! Ekos was really off the mark, and there was obviously a polarization of the vote in every province Except Ontario where the NDP took enough lib votes to give Harper his majority. Elsewhere than Q the NDP only won 7 more seats but took enough votes to let the tories gain some Liberal seats. Not enough has been said about the provincial factor-BC, Quebec and Ontario having Lib governments worked in Harpers favour. Nobody could have forseen the poor emotional connection Ignatief had, or the NDP Tsunami in Quebec. Mostly Harper ran a masterful campaign, never having stopped since 2004. A tectonic shift.

    ReplyDelete
  39. AverageCanuck03 May, 2011 12:24

    Eric this statement doesn't seem right:

    "In short, the seat projection model was able to turn regional results into reliably accurate seat results."

    You said its 88% accurate.

    I thought 90% was the bare minimum of acceptability.

    By your own standards isn't that a failure ?

    Advice:

    Stop using EKOS and HD or apply some kind of golden rule to their polling (+5 CPC, -5 Greens/Other)

    ReplyDelete
  40. One thing that you SHOULD be doing, but you and every other forecast never does, is plot the uncertainty bands for all the polls and all your predictions. A quoted result, especially based on statistical samples, without an uncertainty is *useless*. No paper in the physical sciences would be accepted without a careful analysis of the uncertainties.

    this is VERY BASIC analysis stuff that we pound into the head of every first year science student. Its endemic that such students scratch their heads wondering why they "didn't get the right answer", when if they did the uncertainty analysis properly, their result was actually consistent.

    For some reason its acceptable in the social sciences, where the systems under study are fraught with all sorts of hard-to-quantify uncertainties, to either completely ignore uncertainty, or to do a half-baked job of it.

    I am quite sure that with a proper accounting of the uncertainties, and making these uncertainties as important as the central value, you will find that your model wasn't all the bad, given its limitations.

    ReplyDelete
  41. Eric, Thanks for all your hard work, even though things didn't work so well. Admire your ability to admit to what went wrong fully then start looking for where immediately. Perhaps more interesting than plugging actual final provincial percentage into seat calculator model would be to *average the provincial level polling numbers from the last week of the campaign, then convert to seats.* Is that terribly time consuming? Or is a quick answer possible?

    Doug Johnson Hatlem

    ReplyDelete
  42. Eric,

    None of the people who are second-guessing you had the guts to actually call the election results. If someone had done a better job, hats off to them.

    ReplyDelete
  43. Anonymous at 11:56, EKOS called the Toronto Mayoralty election right, giving Rob Ford 48% of the vote (he got 47.1) when everyone else had him much lower.

    Doug Johnson Hatlem

    ReplyDelete
  44. Anonymous 11:58,

    Polls were weighted by their MOE, not their sample size (assuming random sample for the online polls).

    AverageCanuck,

    Yes, the riding-by-riding accuracy was still too low, but the overall result was satisfactory. I'm still not pleased with it, but it would have been about the best to be expected in this topsy-turvy campaign.

    ReplyDelete
  45. Hello Eric,

    I enjoyed our site over the election. I will make one suggestion that may have been mentioned already.

    If you look back at the final polls heading into the election for 2008 and 2006 they both show the incumbent party (2008 for the con/ 2006 for the libs) as being about 3 points below what they ended up getting in the final vote.

    Perhaps you need to build into your model a "Power of incumbency" of a percent or 2 in your future projections.

    ReplyDelete
  46. I have a theory as to where some of the polls went wrong which I can't back up so take it for what it's worth, but I think the polls themselves had a strong impact on Liberal voter turnout. I think many would-be-Liberals saw the writing on the wall and on election day simply stayed home.

    ReplyDelete
  47. First, I want to applaud you for creating such an effective seat-projection model. If you had projected 4 Bloc seats, or 106 NDP seats, everyone would have thought you were crazy, but that's what your model would have shown given accurate vote projections. So good job.

    Second, you weren't alone in favouring a conservative vote projection model. I thought your model was really clever - the only problem I saw was how you ignored the historical evidence of the strength of the CPC ground game. I expected a 3-5 point swing to the CPC between the last poll to the ballot box, but I expected your projections for the other parties to be mostly accurate.

    As you've pointed out, simply using new polls only would have significantly improved your accuracy at projecting votes, but beyond that I disagree that the polls weren't accurate. I think the polls were accurate... but the Conservatives were simply better at getting out their vote than the other parties were.

    ReplyDelete
  48. Congrats on an interesting exercise which was/is more helpful than msm noise. And good on you for the mea coupa.

    Correct me if I am wrong or missed your post, but I think you missed the importance of the high voter turnout in the advanced polls and how that translated to increased conversative support. IE.Ottawa West (Baird CON) had the highest advance poll turnout and you can bet that it was all conservative, given that he has a well oiled machine. I think if you look at the other ridings you can make the same argument. The MSM only regurgitated the EC news release with out delving deeper into it's effect/result.

    Polling firms and the MSM though need to have a sit down - right now they have zero cred.

    Great job and look forward to the Ontario provincial anaylsis.

    L

    ReplyDelete
  49. Eric:
    you did a super job.
    your reports tracked the uptick for the NDP and collapse of the BLOC and Liberals.
    its very likely that the bin Laden story affected the undecideds and Blue Liberals at the end, and the PM campaigned well at the end in Toronto where the election was decided
    Think...about 20 Liberal seats turned over in Toronto and the GTA area to conservatives, if they had held, or even if 15 had held, the CPC would have 152 seats and short of a majority.
    I'm wondering if your model needs to look at 9-10 regions instead of 5 or 6.
    Maybe:
    Vancouver
    rest of BC
    Alberta
    Sask/Man/the North
    GTA
    Southern, and Eastern and Northern Ont (maybe 2 regions here)
    Montreal
    rest of Quebec
    Atlantic

    going to 9 - 10 regions would force the pollsters to make some adjustments to sampling, methodology etc.

    we're all looking forward to your future posts.
    job well done!

    ReplyDelete
  50. No need to apologize, thanks for your hard work and looking forward to following your site with the next elections.

    ReplyDelete
  51. Not the end of the world, Eric. When I commented a few days back, I said I thought it was irresponsible to publish widely when it was clear that the historically-anomalous NDP "surge" was clearly taking place, and outrunning the ability of the model to cope. Irresponsible because this was clearly an election in which many people were strategically voting, amongst other factors. So polls and seat projections can themselves influence the race. Saying that your gut told you the NDP turnout might be poor, and thus (your words) "the flaw would inadvertently turn out to be right," means that you knew it was off. Yet the Globe and many others broadcast the projections, without any header saying, "Warning: Due to historic anomalies, polling component of model appears flawed."

    There's nought to be done now, other than improve the model. Though it might be informative if you chose to speak frankly about the dilemma you faced - running an experiment amidst a fast-changing reality, with the draw of media attention, web-based feedback, etc. That in itself, would make an interesting piece.

    Best of Luck and hope you keep on.

    ReplyDelete
  52. I have to second Hawerchuk at 12:38. Obviously you are going to be your own worst critic. We all are (or probably we should all be).

    This site was my daily fix and I enjoyed the ride. I can't imagine too many predicted this election correctly. Those that said it would be a Conservative majority were not basing it on stats but on wishful thinking. Guessing a 160+ Conservative majority is dumb luck, nothing else.

    I'll make sure to tune in for the Ontario election (my old home) and the next Quebec election (my new -- and very orange -- home).

    ReplyDelete
  53. @Eric: In suggesting this might be an exceptionally good election for your seat projection model, I wasn't saying there was anything particularly wrong with it. It may be pretty much the best possible model. All I was saying is that I wouldn't necessarily expect it to do as well all the time (it might also do better sometimes). My vague recollection was that uniform swing models did better than proportional ones in the last UK election. There are no perfect models for mass human behaviour -- too much randomness.

    ReplyDelete
  54. Great job on the site - your excellent work made this site part of mine and countless others' daily routine.

    In terms of your prediction on Quebec, I think you hit the nail on the head in terms of the methodology flaw. You appeared to make the model fit your "gut feel" on the NDP turnout as opposed to letting the stats make the decision for you. History tells us that Quebec voters are very willing to change parties en masse and the polls we had were telling us unequivocally for weeks that this was going to happen again in favour of the NDP.

    Looking forward to coming back in the fall...

    ReplyDelete
  55. I really enjoyed your blog, it helped me decide how to strategically vote. I think 76% accuracy is something to be proud of. We forget, these results also depend on voter turn out. Someone who answered the polls may not have turned out to vote. Nonetheless, a great project, I hope in four years you'll continue on.

    ReplyDelete
  56. As a Québécois and a Canadian I feel you and other pollsters let us down. It is wildly irresponsible to publish these predictions while there is a possibility that they could turn out so wrong.

    The feeling I get from friends and family is that if the pollsters had indicated a total sweep by the NDP and a CPC majority, the actual results would have been different.

    ReplyDelete
  57. AverageCanuck03 May, 2011 14:27

    Eric is there any way to model GOTV ?

    In '04 and '06 the polls seemed to OVER-estimate the CPC because they weren't capturing the strenght of the Big RED Machine.

    In '08 and '11 the polls seemed to UNDER-estimate the CPC because they weren't capturing the strength of the Big BLUE Machine.

    Odds are in the next election the trend of UNDER-estimating the CPC will continue because they'll be in government and still have great fundraising.


    Do you think looking at fundraising numbers and parliamentary resources could give you a hint about the strength of GOTV ??

    ReplyDelete
  58. Dan,

    Though my comment was just below yours, I was actually responding to the one preceding yours. It didn't turn out that way as I was typing my response to Kim when you posted.

    ReplyDelete
  59. You did a lot of good work, and got many of the results right. It's not a science, clearly.

    I hope you'll allow me a victory lap: On your site,I predicted two weeks ago that Harper would win a majority. The following day, I said the CPC had moved beyond a bare majority of seats and was firmly into the 160s.

    It wasn't the polling that told me that, nor do I have any inside information. But Harper acted like a frontrunner from Day One, and never once changed his strategy. That told me that the CPC numbers, of motivated voters in particular, always hovered around 40%.

    Finally, I'd like to say that although I only posted twice, I checked the Comments almost every day. With a few exceptions, the comments here were rarely filled with the hatespeak found on other political sites.

    WildeMan

    ReplyDelete
  60. I enjoyed following your projections throughout the election and look forward to seeing more in future elections. Thanks for doing this.

    ReplyDelete
  61. In 2008, the polls all had the CPC at 33-34%, and in the end they got 37.6%. This time, they were polling at 35-37% and got 39.7%. Is this just a co-incidence or is there something flawed in the methodology of the pollsters? Maybe conservative supporters are less likely to take the time to answer the survey for some reason?

    Or is it that the conservatives suddenly got a boost in the last two days? The Nanos poll results for the final day of polling were quite a bit higher than the previous day (with the usual caveats about higher MoE on a 1-day sample). IIRC this exact pattern happened with the Nanos poll in 2008 as well. Perhaps that's evidence for a last-minute surge.

    frogg

    ReplyDelete
  62. Eric,

    Your blog has been a great reading throughout the campaign. I was going on your site several times per day to look at your prediction.

    Based on your riding-by-riding number, I had correctly called Duceppe's defeat in his own riding, almost a full week before the election.

    Don't give up, your site is gaining credibility. This was your first federal election. You now have 4.5 years to adapt your model.

    I will second what many others said, there is absolutely no need to apologize.

    The mid-campaign surge in NDP votes in Quebec took everyone by surprised. Your model was a weighted average. Perhaps too much towards the beginning of the campaign/older polls versus more recent polls. You already raised that point.

    Keep up the good work.

    ReplyDelete
  63. As a number of others have post, I think you did an excellent job with this site and the analysis. I like you did not think the BQ vote in Quebec would collapse that much and allow the NDP their 100 seats. I was a card carying active Liberal party member for many years and knew that the Conservatives have a great e-day campain and that the NDP have never had a really great one outside of a few ridings. So I was actively agreeing with your assesment of Quebec and as a result your projected seats made a lot of sense to me. It would be awesome if you plan to use the same set up and analysis also for the Ontario election in the fall. I would definetly be back reading every day.

    ReplyDelete
  64. Eric, great job. Thank you for all your hard work. I will continue to follow your blog as I have for the past few months.

    Don't be too hard on yourself. Take the useful from the constructive comments and ignore the negative ones. Obviously the polling was a problem to some extent, and it's hard to account for that. Your model seems to be a good one. You can only do what you can do.

    Thanks again for all your hard work and your excellent blog.

    Geoffrey

    ReplyDelete
  65. One other thing, Eric. Thanks for your suggestion about cropping and printing the projection. I found it a helpful tool as I was following the results, as it showed which party held each seat at dissolution and the way the seat was expected to go. Although some of the percentages were off a bit, I found it generally accurate and very helpful.

    Awesome blog!

    Geoffrey

    ReplyDelete
  66. LOL, Seb. That is wonderfully whiny of you. I suggest you follow fewer prediction blogs and more policy blogs next time

    Frankly, Quebec is the only province that did anything to beat back the Conservatives. It's not as if a chunk of Bloc MPs would have given Quebec more representation against a majority Tory government. Unless you're saying more Conservatives would have been elected?

    Eric, I enjoyed your blog. The dramatic upswing in NDP support in Quebec would have required a major shift in your methodology to capture, and the corresponding vote splitting in Ontario that lead to the Tory majority would have similarly needed the last-minute shift. Which hey, we all could do for ourselves by actually looking at the last polls!

    - H

    ReplyDelete
  67. Éric,

    I imagine that it would be impossible to come up with a popular vote projection that works in every election.

    Having rooted for the Lib Dems, I was very sceptical when the whole NDP thing started. I know that the regional variation, i.e. the essential replacement of the Bloc, ended up being the NDP's biggest boon, but I still think this election will be the exception to the rule when it comes to mercurial third-party surges. Or, given the ADQ, maybe it's Québec that is the exception to the rule :-P

    ReplyDelete
  68. Eric, thanx for your seat projections during the campaign. We've added all 14 2011 model results to the 2004/2006/2008 Scoreboard @ http://www.trendlines.ca/free/elections/Canada/electcanada.htm

    ReplyDelete
  69. Eric, Having followed this blog for the past few years, just had to say thanks so much for all of your hard work! Your work here is an invaluable resource!
    -G

    ReplyDelete
  70. Eric, you did a FANTASTIC job considering what info you had to play with. I dont think anybody really believed those Dipper numbers until they solidified the day before the election! I for one am very glad you got it wrong as the Tories got an even bigger majority than I thought they would get.

    ReplyDelete
  71. Too many 48 hour deciders to get close without Sunday only polls. If the Sunday only Nanos CPC 38.7% had published a regional breakdown you might have had a good projection.

    Polling is all about probabilities. Should be possible to assign probabilities to 1 and 2 party majorities, and individual seats given polling range probability curves. If you accept that the polling range confidence levels are only for time of the poll and assign probabilities of movement, my hunch is that the set of polls you had would have given you something similar to CG's 28% chance of a majority.

    ReplyDelete
  72. Never mind. No need to apologize for being wrong - everyone was. Your site provided lots of thought-provoking fun in the lead-up to the election and I thank you for that. Good luck with the next set of elections!

    ReplyDelete
  73. Thanks for running this site, I found it informative and entertaining to read, and I look forward to seeing how you do with the provincial elections

    ReplyDelete
  74. In fairness, everyone got it wrong.

    One thing that irks me, and which NONE of the news outlets have touched on, is this: the NDP "wave" was almost completely confined to Quebec. The NDP didn't gain all that many seats in English Canada - a half-dozen in the GTA, a couple in BC, one in NS, and that's about it.

    This really is a case of francophone Quebecers simply voting blindly for candidates whose names they didn't even know simply because they thought their neighbours were doing the same - and they were.

    As we all know, popularity in Quebec is often fleeting. It is highly conceivable - if not probable - that the NDP will half to three-quarters of their Quebec seats in the next election as the novelty wanes.

    The question is - who will pick up the slack? Will the Liberals return to glory in franco-Quebec? with the BQ become resurgent? or will the Tories come back - quite likely under a new leader like Bernard Lord?

    But of course, does any of that matter? by 2015 the additional seats will have been added to Ontario, BC, and Alberta, reducing Quebec's seat share in the House of Commons.

    ReplyDelete
  75. Thanks for all your work throughout the campaign. My only comment would be that you should stop calling the Conservative Party of Canada "Tories". The old Conservative Party became the Progressive Conservative Party in the mid 20th century, and then was almost wiped out in 1993, only to merge with the Reform/Alliance crowd in the last decade. It might seem like nitpicking to some, but today's Liberals bear more resemblance to the old Tories than Harper's crew ever will.
    Len in Winnipeg

    ReplyDelete
  76. Amateur Psephologist03 May, 2011 18:37

    Eric

    I add my thanks and admiration for the thought and care that goes into this site.

    It was just your luck to launch your projections in an election like few or no others.

    Maybe there are a couple of things to consider for next time. In this race, there seemed to be three kinds of region with differing patterns. Could the model treat each of them differently?

    Region #1 involves stable or slow-moing party preferences throughout the campaign) - think the Prairies or Alberta, this time around. Here, having the best GOTV machine, combined with the relative likelihod of voting for each party's supporters, really makes the difference and can add 1-2% to one's total.

    Region #2 involves a dramatic swing, week after week, towards one party or away from anotehr party - think the province of Quebec this time around. Here the urge to make a change is so strong that these voters will get out to the polls without the need for a powerful GOTV machine. I would just go with the average of the last few polls before E-day.

    Region #3 is a place where, in multi-say dogfights, one of the parties is seen to have low enthusiasm levels among its supporters - think Ontario and the Liberals. Here, I would go with the low end of the most recent polls for that weakening party, and distribute, say, 2-3 percentage points of their support to the party most on the rise in that period in that region.

    None of this may fit into a statistial model, but I do find it interesting that we seemed to have three different regional types coinciding in this election. No wonder it was tough to forecast!

    Good luck with the next version, Eric

    ReplyDelete
  77. Is it not a reality that polls and projections failed because one can not build an algorithm to correctly determine the results? There are too many random variables that can change the minds of voters right up until they mark their ballot or decide not to go vote after all.

    I believe that political polling and projections are dead now. They were always more of propaganda tool used to swift opinion rather than reveal it. So this might be a good thing.

    Maybe we can start focusing less on predicting outcomes and focusing more attention to the real issues?

    ReplyDelete
  78. did anybody get over 88% in the predicting this election?

    ReplyDelete
  79. Eric, you made this campaign as enjoyable as it could be. I think we all should have paid more attention to the ceilings for the parties because the CPC and NDP crowded their top end projections and the Liberals and Bloc didn't make it out of the crawl space. Thanks for all your work.

    ReplyDelete
  80. Perhaps you should factor in that in 2008 all but Ipsos had the CPC pop vote at least 2% low.

    For some reason the polsters are not contacting a representative number of CPC voters.

    ReplyDelete
  81. Suggestion: Take the weighted polls and run a regression analysis to capture trending (like the surge) instead of just averaging them. Probably want best-fit sigmoid curve rather than linear, since we know there's an upper and lower limit. (Can't have less than zero or more than 100% of the vote.) Might have to fiddle with the weights, probably have to date multiple-day polls by the middle of the range. Rolling polls you can throw math at to isolate day-by-day.

    JWP

    ReplyDelete
  82. With 3- and 4-way races in Ontario and Quebec, the MOE in most polls is a multiple of the very small percent that can tip a close race in any of three or four directions. I am waiting for detailed riding results to see how many were won with just over 30% of the vote. Kudos to the CPC for their mastery in identifying the tight races and putting the resources where they would count -- and telling these voters what they wanted to hear.

    Unfortunately their skill has overwhelmed judgement as a majority that has the support of only 40% of those voting that uses it to lord over the 60% voting against them carries the seeds of its own downfall. Harper in his victory speech stated he would govern on behalf of all voters. We did not see that in the last Parliament. Time will tell whether and how well he delivers on this commitment.

    ReplyDelete
  83. I appreciate all your hard work and analysis. No doubt over future elections, you will have the opportunity to refine the voting model.

    It seems to me that the turnout argument is typically used by the losing party -- a sort of desperate last refuge. People like to vote for a winner.

    ReplyDelete
  84. HC in AB says

    I would submit that the pollsters are getting a representative sample but that GOTV and the demographics relative to likelyhood to vote and the political leanings of the various demographics are making the "actuals" different than the "projections".

    ReplyDelete
  85. I enjoyed this site. My suggestion is, for each poll, please show what the seat projection would be with all 100% weight on that poll. That would give us all more feel for what kinds of seat swings are possible. Thanks. -ST

    ReplyDelete
  86. First of all nice overall work.

    Then I'll admit it's very hard to choose a proper fixed time constant for aging of old pool.

    One thing that can be done is: the more a new pool is precise but far away from your prediction, the less you should trust your prediction.

    That way, the time constant adjust itself depending of the volatility in the pools.

    One correct mathematical way to implement this is the Kalman filter.

    ReplyDelete
  87. Hello Eric!

    Only reactivating this comment thread, hoping that you've perfected your model over the years. This election is trending to be even closer than the last one!! :)

    ReplyDelete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.