Friday, June 13, 2014

Ontario election post-mortem: likely voter models fall flat, eligible tallies good

In the end, the race was not nearly as close as expected, with Kathleen Wynne's Liberals winning a majority government of 58 seats, with the Tories taking 28 and the New Democrats winning 21. The results have cost Tim Hudak his job, as his party took just 31.2% of the vote, with the Liberals improving on 2011 with 38.7% support and the NDP marginally upping their tally to 23.7%. The Greens had a good night, relatively speaking, with 4.8% of the vote.

Note: This post has been updated to reflect the reversed result in Thornhill, originally awarded to the Liberals but reverted to the PCs by Elections Ontario. The projection originally gave Thornhill to the PCs.

How about the polls and the projection model? The polls were not the miss that some commentators have suggested. As I write in The Globe and Mail today, the traditional numbers reported by pollsters actually did quite well, under-estimating the Liberals enough to put a majority in doubt but generally tracking the race accurately. However, the likely voter models employed during the campaign, and favoured by the projection model here, did not do the job at all. Every pollster that used a likely voter model did worse with it than they did with their estimates of eligible voter support.

As a result of this miss by the likely voter models, the projection model was off a fair bit. The projected support levels for the Liberals, New Democrats, and Greens fell within the likely ranges, which is a success. The PC number, however, was below even the 95% confidence interval.

In terms of seats, the Liberals ended up between the high and maximum expected ranges and the PCs between the minimum and low ranges. That is no coincidence, as the vast majority of the projection model's misses were seats projected to go PC, but that actually went Liberal. The NDP result was only one seat off the projection. So, a more mixed record there.

But the riding model itself did extraordinarily well. It only missed the call in 10 ridings, for an accuracy rating of 91%. That is the best performance of the model since the 2011 provincial election in Manitoba, when 56 of 57 ridings were accurately called. Taking into account the likely ranges (which should be the focus of anyone looking at these riding estimates), in only six ridings was the potential winner not identified. That ups the accuracy rating to 94%, or 101 out of 107.

Of those 10 misses, seven of them were seats projected to go PC, but instead went Liberal. One of them was expected to go PC, but instead went NDP, while two of them were expected to go NDP, but went Liberal. Half of the misses were in the 905 area code, where the PCs did unexpectedly poorly.

Nine of the 10 misses were called with 67% confidence or less, with five of them being called with less than 60% confidence. Only Durham, erroneously called for the Tories at 80% confidence, was a serious outlier.

The model would have done slightly better had I ignored the likely voter models. The vote projection would have been 37% for the Liberals, 33% for the PCs, and 24% for the NDP, with 50 seats going to the Liberals (or between 46 and 56 at the most likely range), 35 to the Tories (31-39) and 22 to the NDP (19-24). So, the Liberals and PCs would have still fallen outside the likely range, though less dramatically.

The model would have called 98 of 107 ridings correctly, for an accuracy of 92%, while the accuracy rating when incorporating the likely ranges would have been 102 out of 107, or 95%.

But what about if the polls had been dead-on? The seat projection model needs to be able to turn actual popular vote results into accurate seat projections, otherwise it would be hopeless in turning poll numbers into seats. On this score, the model did quite well.

The actual results for all three parties would have fallen within the likely ranges, with the Liberals on the higher end and the PCs on the lower end. This is one indication of how the Tories really had a poor night.

The overall call, instead of 'likely Liberal, possible PC victory', would have been 'Liberal victory, possible majority'.

The projection model would have called 98 of 107 ridings correctly with actual results, but when including the likely ranges at the riding level the model would have identified the potential winner accurately in 104 of 107 ridings, for an accuracy rating of 97%. The three ridings that bucked the trends? Cambridge, Durham, and Sudbury. The misses would have been called with an average confidence of only 55.4%.

All in all, the seat projection model performed as it should. The vote projection model did more poorly, but it can only be so good as the polls put into it. A look at the regional projections, though, gives us a look at where the polls missed the call - or, perhaps, where the parties over- and under-achieved.

The Liberals out-performed the polls across the board, but nowhere did they do so dramatically. Their actual result fell within the projected likely ranges in every region except eastern Ontario, where their 38.9% result was slightly up on the expected high of 37.2%. Nevertheless, they were up one to three points in each region.

The Tories under-performed in every region of the province. They were below expectations by three points in the north/central and eastern regions and four to five points in Toronto, the 905, and the southwest. Their result fell below the expected ranges in every region. They had a horrible night where turnout is concerned.

The New Democrats aligned quite closely with the aggregate projection, with differences of no more than 0.5 points in every region except the southwest, where they took slightly more of the vote than the higher likely range expected.

The Greens, shockingly, outperformed the polls in every region. That is a very rare occurrence.

But what if the projection had ignored those likely voter models? In most cases, the projection would have been closer. But still, the Liberals would have outperformed the polls in every region and the PCs would have under-achieved across the board. The NDP would have more traditionally under-achieved in most regions as well, as would have the Greens.

So, let's get to grading the pollsters. This should be done with a great deal of caution. Recall that most of these polls have a margin of error (theoretical or otherwise) of about two to four points for each party. And these sweepstakes, decided by a few decimal points, or not nearly as meaningful as many make them out to be. But let's put the results on the record.

In the chart below, I've ranked the pollsters by cumulative error for the four main parties, including both likely and eligible voter tallies. I've also highlighted in yellow every estimate that was within two percentage points of the result. In this regard, kudos needs to go to EKOS Research, the only firm to call two of the three main parties within two percentage points. In terms of the eventual outcome, their poll was probably the most informative, though they had the NDP too low.

Angus Reid Global's poll of all eligible voters of June 8-10 turned out to have the least total error, at just six points. Their numbers suggested a Liberal minority, however, as did Abacus's poll of eligible voters of June 9-11.

ThreeHundredEight.com would have ranked third among pollsters (or fourth among number sets), though if the likely voter models had been ignored the total error would have been just 4.4 points, putting the projection at the top of the list.

Oracle placed narrowly ahead of EKOS's eligible voter numbers, but the portrait of the race they painted (PC lead of one point) was not reflective of the outcome. After that, the errors become more serious, though both Forum Research and Ipsos Reid (eligible only) did have the Liberals in front.

You might be wondering why Nanos Research's poll is not included in the list. With a total error of just 1.5 points, the poll would have been - by far - the most accurate (Nanos had it as 37.7% OLP, 31.2% PC, 23.7% NDP, 5.3% GPO). But the Nanos poll was out of the field on May 26, 17 days before the election. While it is possible that voting intentions remained static during those 17 days, that is not something we can assume. And if we're allowing a 17-day-old poll to be used as a measuring stick, then the Abacus Data poll of May 28-31 (37% OLP, 30% PC, 24% NDP) was almost as good.

So where do we go from here? Clearly, the likely voter models are still in an experimental phase. When employed in Nova Scotia and Quebec, the first time we have seen them used in recent provincial elections, they only marginally improved the estimations, if they did not worsen them. We may come to the conclusion, then, that for the time being Canadian polling is not yet capable of estimating likely turnout with more consistent accuracy than their estimates of support among the entire population.

This is counter-intuitive, however. Likely voter models should improve things, particularly when turnout was only slightly above 50% yesterday. Going forward, ThreeHundredEight.com should perhaps rely solely on those eligible numbers, until the likely voter models consistently prove their worth, and run a lesser, simultaneous model that takes into account likely voter estimates. This may provide the best of both worlds and give readers food for thought and all the information available, though it will not clarify things more fully. The challenges of polling elections in the modern age continues.

51 comments:

  1. Go with the eligible voters in the future until some big pollster comes up with a better way of doing the likely voters thing !

    ReplyDelete
    Replies
    1. For most IVR polls the likely voters question would put the "most like to vote" option as the first option, so I suspect that some "busy" people would just pick that choice without even listen to any other choices for that question. Will it be counted as a bias in the survey???

      Delete
  2. Eric, the registered voter polls in the US presidential elections have generally been more accurate than likely voter polls in the last 2 elections as well. Is it time to dispense with the likely voter screens altogether?

    ReplyDelete
  3. Replies
    1. The most accurate according to some....because all the other firms are filled with liberal bias

      Delete
    2. It was in no way the most accurate this election.

      Delete
  4. I think undecided voters were a big factor here. Anyone who was going to vote PC had already said so and all undecideds were hesitating between Green, NDP and OLP.


    Eric, are your projections more accurate if you send the undecideds 40% to OLP, 40% to NDP and 20% to PC?

    ReplyDelete
  5. Anecdotal data: I was talking yesterday to two likely voters who were 100% they would vote and in the end they didn't. I think one is a lot less likely to run into someone who is 100% likely to vote for party X and in the end votes for party Y.

    ReplyDelete
  6. Aye, aye on relying on eligible voters.

    ReplyDelete
  7. There must be at least a relationship between those that will participate in a public opinion poll and those that will take the time to vote on election day. So if turn out is projected at 50%, a likely voter model should be using a higher number but the problem becomes how do you figure out what percentage makes sense?

    It is interesting that EKOS had the Greens so much higher than all the other pollsters. But they then dropped them too much in the likely voter model.

    ReplyDelete
    Replies
    1. Think you're definitely right about participants in polls being more likely to vote.

      Delete
  8. Eric,

    I have some quibbles with how you're evaluating the success of your riding prediction model that I'm interested to get your response to.

    First of all, one quick point -- I believe you actually missed on 11 ridings, not 10. I think the one you haven't counted is Beaches-East York, which was another NDP-favoured riding that went to the Liberals (you only account for one such riding in your count).

    My broader point, though, is around what you wrote here: "Only Durham, erroneously called for the Tories at 80% confidence, was a serious outlier."

    I would argue that you should expect to be wrong on a call you make with 80% confidence 80% of the time! As such, your model, even though it got a lot right, actually isn't very well calibrated -- and you should have had a bunch more mistakes!

    I looked back at your percentage projections for each riding, and matched them up with whether they got the call right. I then broke them down into 10 percentage point blocks and compared the number of correct calls with the expected number of correct calls of the midpoint of that range. So, for example, I would expect you to be correct on 95% of races you rated 90-99%, 85% of races rated 80-89%, and so on. Here's how it broke down:

    Range Races Correct Expected
    90-99 18 18 17
    80-89 28 27 24
    70-79 27 27 20
    60-69 20 15 13
    50-59 14 9 8

    So across every range, your assigned confidence rating actually understated the probability of the event occurring. This is especially noteworthy in the races in which you had between 70-79% confidence -- it turns out you actually called all of them correctly. While it's great that you're getting it right, I think a miscalibrated model of this degree is actually not all that helpful.

    Do you have any thoughts on this, and any ways you might be able to recalibrate the model to make it more reflective of the actual outcomes?

    ReplyDelete
    Replies
    1. AJS,

      You are right about Beaches-East York, I must have counted it as correct before all the polls had come in. Darn. I'll update the post.

      For the probabilities, you are forgetting that those are still rather small samples. The more races called, the more it will revert to the posted probability. I may have been right in 100% of those 27 races called with 70%-79% confidence, but in the next election I may be wrong in 50% of those calls. In the races called with confidence of between 70% and 79% in elections before this year, the model has been right 73% of the time. We're talking 193 calls, rather than just 27.

      Consider that the probabilities are based on past performance, so they include scenarios like Alberta where only 59% of ridings were correctly called, or Nova Scotia, where only 65% were called. So depending on how the polls do, in some elections the probabilities will be over-confident, and in others they will be under-confident.

      Note, though, that I did not have time to include the performance of the model in the 2014 Quebec election, which would have increased the confidences of every call as the model performed slightly better than average in that campaign.

      Delete
    2. Now that the OLP victory in Thornhill has been overturned, I'm back to having missed 10.

      Delete
    3. Wait, when did that result get overturned? I can't find any mention of that elsewhere... (or of a recount)

      Delete
    4. Apparently Thornhill was due to a "Data Entry" error. Hmm?

      Delete
    5. Thanks Eric, appreciate the explanation.

      One other note, though -- if you had just predicted that every seat would go to the party that held it at dissolution, by my count you would have missed on 14. Obviously, you did better than that by missing 10, but it doesn't seem to me like the model is a substantial improvement over just picking the incumbent in any riding. Obviously incumbency is a huge advantage, so that's not surprising. So are we (you) getting better at devising a method to determine which currently held seats a party is most likely to lose (and the traits that they share)?

      Delete
    6. And now the losing candidate in Thornhill is demanding a recount. Now I suspect the winning margin was so small you would think an automatic recount ??

      Delete
  9. Trying to choose between the likely and eligible voter numbers is a bit of a suckers game IMHO Eric. Which one is better is going to change back and forth depending on the election and what the ballot question is. Maybe include both?

    Either way I think your model performed very well. You correctly factored in the error in the polls and gave people the best information one could hope for prior to the election.

    The only thing to work on IMHO is smoothing out those lurches back and forth that occur as different polls are released.

    Cheers Eric, and thanks for the hard work. :)

    ReplyDelete
    Replies
    1. Gotta wholeheartedly agree with Ryan here Eric.

      A BIG thank you !!

      Delete
  10. Clearly online surveys are very inaccurate because they are essentially opt-in. Ipsos was completely wrong this time around. All the others were fairly correct, though the Liberals did somewhat better than expected. Maybe people changed their minds at the last minute because they were scared of Hudak winning and voted Liberal. This probably explains how poorly the NDP did in Toronto.

    ReplyDelete
  11. I wonder if there was a rather large black swan thing going on in this election (for lack of a better term). Ie: You have public employees, and families of public employees, who will all vote against Hudak. We're talking a couple million people here. A hefty percentage of these people fall outside the "likely voter" criteria, but will definitely vote.

    ReplyDelete
  12. Am I correct in inferring that, though your model called Sudbury correctly given incorrect popular vote numbers for the North, it would have called the riding incorrectly if given the correct popular vote numbers? How weird!

    ReplyDelete
  13. Ipsos Reid, as I predicted some time ago, once again overestimated Conservative support, particularly in its Likely Voters model where it kept claiming that Conservatives would come out in huge numbers and would swamp everyone else by being the most motivated to vote.

    ReplyDelete
  14. Woulda, coulda, shoulda. It's time to put an end to published polls during election periods. When politicians spend more time talking about momentum and strategic voting than they do about policy ideas and promises, we have a problem.

    ReplyDelete
    Replies
    1. Chris: Absolutely wrong. As a strategic voter, I need every bit of information I can get to make an informed decision when I vote. I don't need the government telling me what are legitimate criteria to use when making my decision.

      In fact, I would force political parties to make public ALL their internal polls. The public pays for them through tax credits and government subsidies, so the public should benefit from them.

      Political parties are going to use polling data whether it is secret or not to manipulate public opinion. Making all these polls secret during election campaigns when the public needs them the most would only increase that manipulation.

      Delete
    2. A tax credit is not revenue, tax credits don't pay for anything they reduce the tax payable by the donor.

      If both politicians and people are spending too much time on polls maybe we should simply ban them during election campaigns. I would be interested to find out how much influence polls have on voters. If it is negligible fine but, if it is determinant on whether a voter votes or not that could be a problem.

      Delete
    3. A good polling company would now be in the field to see if anything has changed since the last poll and election. For example, they could ask if people voted in the election and compare them to actual numbers to see how many people lie to pollsters about this.

      Delete
    4. Pete: Polls are indispensible to voters. I for example used polls last election to vote NDP and oust my Bloc MP after years of voting Liberal. It's not the government's business to tell me how to determine my vote.

      Attack ads, patronage appointments, private campaign contributions, and the first-past-the-post system are far more damaging to democracy in Canada. Polls help hold politicians accountable.

      Delete
    5. Sorry to tell you this Guy_Smily but strategic voting is a sham forwarded by the Liberal party. It dose nothing to stop Conservatives from getting elected and it is nothing more then fear mongering used to scare progressives voters from casting a ballot in there own interests (by voting NDP).

      The truth is that strategic voting only helps the Liberals take NDP seats and allow conservative to stay in the seats the have. Case in point the NDP lost 6 seats to Liberals due to "strategic voting". They are:

      * Beaches East-York
      * Trinity-Spadina
      * Davenport
      * Brampton-Springdale
      * York West
      * York South-Weston

      The PC were not even close to being a threat in any of those seats. On the other hand the PC kept 2 seats that should have gone NDP thanks to "strategic" voters.

      * Sarnia-Lampton
      * Chatham-Kent-Essex

      So please tell me what information you as a strategic voter get from polls that don't tell you anything. Because it seems to me, as a candidate, that strategic voters aren't making up there minds based on facts and polls but on fear and lies.

      For the record if strategic voters used facts and not fear when voting election night would have end like this:

      Lib - 52 (Minority)
      NDP - 29 (Official Opposition)
      PC -26 (Third Party)

      Delete
    6. The "strategic voting" card is a tough one to play for Liberals. It did work much for Paul Martin and it defiantly did not work for Michael Ignatieff. But it worked for Kathleen Wynne because she was a genuine progressive who would be at ease within the NDP.

      Voters in downtown Toronto know that the PC are not competitive in their riding. The Liberal strategy there was to portray Horwath as a faux progressive who may prop up a PC minority government. It worked. Wynne understood Toronto, Horwath does not understand Toronto.

      The anti-Horwath strategy did not work much outside of Toronto. Some places such as southwestern Ontario there is a potential for the PC vote to shift to the NDP, or vice versa.

      If the NDP are ever to win ridings such as Sarnia Lampton or Chantam Kent Essex, they need to draw more PC votes.

      Delete
    7. Guy,

      Polls are dispensable. You should vote for the candidate you think best. You say it is not the Government's business to "tell me how to determine my vote". However, you admit in the same breath the polling companies are essentially doing it for you with their polls-they determined your vote last election! Why you would give up your free will to the polling companies is unusual to say the least.

      Polls do not hold politicians accountable. Voters hold politicians accountable through election. If polls held politicians accountable there would be little point in voting. In any case we are not talking about polls writ large but, the publication of polls and their influence on the electorate.

      Delete
  15. Count me as another avid reader and fan of your work. I much appreciate the time and effort you put in. Thanks!

    ReplyDelete
  16. It's clear to me that Abacus was by far the best pollster in this election campaign. They got the numbers right (mostly), they were the most transparent in their methodology, their results (Liberal win) were consistent (the numbers stayed more or less constant) and at odds with the political orientation of their sponsor (Sun News), and they published a slew of useful data and analysis that they made readily available to the public. It's clear that they were on the ball, and interested in getting a picture of the electorate as opposed to punditry. I guess Sun News won't be hiring them back too soon.

    Abacus identified a number of voting blocks and tracked them. Firstly, they showed that PC's had the largest number of base voters. That would explain the PC need to pander to the base. They showed, however, that the largest bloc of swing voters (20%) were NDP/Liberal swing voters and that they leaned slightly Liberal.

    Most interesting was the group that admitted to not voting last time. 40% of those voters supported the Liberals compared to 20% NDP and 20% PC. The 5% increase in participation was probably due to this group. Fear of Hudak seems to have motivated this group to come out and vote. That could explain the difficulties in identifying likely voters. All previous elections would have been characterized by a decline in participation; this one was characterized by an increase.

    ReplyDelete
  17. Eric, thanks for your thoughtful and well researched comments throughout this campaign. They made the election much more interesting!

    ReplyDelete
  18. Eric, I'm a long time reader but posted for my first time on your last piece before the election. Like many others who have posted, I'm very grateful for your thoughtful analysis. I check your site every day during elections. As I indicated in my post on your last piece before the election, I have a few intuitions regarding why I think the polls, and as a result your model (although still fairly accurate), generally under-estimated the Liberals, and over-estimated the PCs. These intuitions also led me to post my election prediction on FB yesterday before polls close. My prediction on FB was: Liberals 54-58, PCs 32-36, NDP 15-20. Clearly, I also over-estimated the PCs and under-estimated the NDP.

    My intuitions are as follows: (1) the polls generally only report decided voter numbers. I think conservative-minded people are more likely to be decided, because, well, they only had one choice. Progressive-minded people, on the other hand, arguably had three choices. As a result, decided voter counts were likely to over-estimate the PCs because progressive-minded voters were more likely to be undecided. This intuition was reflected, as I recall, in some of the polling results that looked at who the undecided voters might vote for. As I recall, there were relatively few undecided voters who would have considered voting PC, but significantly more who would have considered voting Liberal. I think this dynamic was at play in 2011, too, but this time the Liberals entered election day, I think, in a stronger overall position than in 2011.

    (2) the psychology of political orientation. Conservative minded people tend to value loyalty and obedience more than progressive-minded people. As a result, in my view, conservative-minded people are more likely to identify as a "decided" voter (loyalty), and are also more likely to be identified as a "likely" voter (obedience - voting is a duty that a good citizen ought to perform). I think this caused some likely voter models, especially that of Ipsos, to be off. Ipsos, as I recall, asked voters something like whether they were definitely going to vote. I think any likely voter model that relies on voters' subjective perception about their likelihood to vote will overestimate conservative numbers, because conservatives, I believe, are more likely to say they will definitely vote, no matter what. I think the EKOS likely voter model, which as I recall used objective factors like age, education, etc., is and was much more reliable.

    Anyway, I wanted to share with you these intuitions: the Liberals had a lot more upside amongst undecided voters; conservative-minded voters were more likely to be decided because (a) they had only one choice, and (b) they are loyal to that choice; and, the likely voter models that rely on voters' subjective sense of the likelihood that they will vote will, because of the psychology of political orientation, over-estimate conservative results and underestimate progressive results.

    Finally, as many have noted, I think strategic-voting likely played a role in the Liberals' success. I have a hunch that some who vote strategically may, when asked in a poll who they'll vote for, indicate their preferred choice (sometimes Green or NDP), but when push comes to shove and it actually matters on election day, they vote strategically. This might only be a quater percent or even less. But that could be enough to tip a couple of ridings.

    Anyway, as a result of the above intuitions, combined with the aggregated polling results from your website, I went into the election thinking the PCs really were a long-shot to win a minority. I thought the Liberals had at least a 66% chance at a majority, but admittedly, I wouldn't have been surprised by a minority.

    ReplyDelete
  19. Well Eric what's the next big thing ?? Any other elections in sight ??

    ReplyDelete
    Replies
    1. New Brunswick in September, and then Toronto mayoral election in October.

      Delete
    2. I figure that Toronto Mayor one will be a scream !!

      Delete
  20. Well, time for my own post-mortem. I made 18 errors with my predictions using the poll aggregate, which still represents an 83,2% efficiency. If I include the ridings within the margins of error, which represents 10 of my 18 mistakes, it leaves me with only 8 mistakes for an efficiency of 92,5%. It's actually pretty good, but technically speaking, I still only called 83,2% of the ridings right.

    With the election results, the model would have made 14 mistakes, correctly predicting 86,9% of the ridings. Fo those 14 mistakes, 9 would be within the margins of error, for 5 mistakes and an efficiency of 95,3%. I don't really like including the margins of error within my efficiency though, as they are not correctly called in the first place, so 86,9% with correct results is not acceptable to me.

    So after those pretty disappointing results, I went back and included the by-elections. With that new model, with the poll aggregate, I would have projected:

    45 OLP
    40 PC
    22 NDP

    The seat result is quite far from the actual results, but so are the numbers, so mistakes is the best way to look at it. I would have made 15 mistakes, down 3 from the previous version. That gives me an efficiency of 88,8%, a bit more than 5,5% better than the previous version, which I find really good considering how far the numbers were. With the margins of error, I am down to 6 mistakes, for an efficiency of 94,4%, roughly 2% better than the previous version.

    With the actual election results, that version would have given me:

    53 OLP
    31 PC
    23 NDP

    The seat result would be wrong, which is dissapointing. On the seat by seat count though, I would have made only 10 mistakes, for an efficiency of 90,7%. It's a tad low (I would have liked a clear 90%+), but it's not bad either. If I include margins of error though, I get only 2 mistakes, for a total efficiency of 98,1%! That's really good. And looking at my results, I have 4 ridings where the projected winner has less than a 1% lead on its follower, so really just plain toss-ups.

    All in all, even if the seat total would have been wrong, I'll keep using the model with by-election from now on. I'll have to start including them in my other models (Québec is good for the moment since the election just happened, but Canada will have its fair share). Not sure I'll be following the New Brunswick election of September (I don't have a model to begin with anyway...) as I'm expecting a baby boy on August 30. But we all need a little time for ourselves, so I'll see if predicting elections is what I'll do in my spare time!

    ReplyDelete
  21. It looks like the Ekos and Forum results were the closest to accurate. Ekos and Forum both have been projecting Liberal majorities. Ekos with its nightly polling was accurate with the trend - the very last day they said Liberals on the cusp of majority! I would say IVR trumped everyone else. Also, Conservative bias with some of the other polls just devastated them - Ipsos with its self selected panelists saying an NDP surge happening - completely false! You should probablty put more weight on IVR sampling like Ekos and Forum from now on :)

    ReplyDelete
    Replies
    1. Ekos and Forum were both pretty good in capturing the margin for the Liberals over the PCs - but both seriously underestimated NDP support - as did an IVR poll by Campaign Research...why IVR would so consistently low-ball the NDP by 4 to 6 points is an open question.

      Delete
    2. Forum baked a turnout model into their numbers, so we don't know if their unadjusted IVR numbers would have also under-estimated NDP support.

      Delete
  22. I followed the polls very closely and was delighted to read your posts. Great that you are doing this. However, I think in your evaluation you have been too easy on the polls and on 308. True, you and most of the polls were not too far off in the actual percentages for the two main parties. But what most people care about before the vote is who is going to win and whether it will be a minority or majority. On this, you and most of the polls gave the wrong impression. You made it seem as if it were a close race, that perhaps the liberals were slightly more likely to win but the PC also had a good chance, and finally that a majority for either party was unlikely.
    As you said, Ekos had it right, from the beginning; Ipsos had it wrong right to the end. 308 made the mistake of more or less averaging the polls rather than discounting Ipsos which was clearly all on its own. There has to be a system for underweighting polls that differ markedly from all others and that don't even track the changes that other polls are seeing.
    As for the likely voter model, if you had weighted Ekos more heavily, the model would have been dead on. It was on likely that Ipsos was especially bad, which skewed the results of this analysis.
    Still and all, it is great that you are doing what you are doing. Good luck next time.

    ReplyDelete
    Replies
    1. The problem is how can one tell that Ekos is right and Ipsos is wrong? Every so often the outlier is the one that gets it right.... I mean Nat Silver has been an outlier compared to mainstream media, yet he's been far more accurate. For all we knew Ipsos was on to something. I'm sure they themselves thought they were.

      I predict elections from the other side, doing ground work, lots of person to person interviews and trying to detect patterns that can then be projected to large segments of the population. My record is pretty good (only error, I gave a small win to Kerry, instead of a small loss due to Ohio). For this election I couldn't figure out who was going to win. The only thing I completely discounted was a tory majority, but everything else seemed to be in play and could have gone in any direction had some people said/not-said some key things.

      Delete
    2. Silver was not an 'outlier' I don't think.

      The polls were pointing all in the same direction in 2012, assuming one looked at state polls and not the national horserace.

      Delete
    3. I agree that it's hard to tell in advance. But I think sites like 308 have to try. If one pollster is different from the others, that doesn't necessarily mean it's wrong. But if all of the polls show a move toward one side and the outlier doesn't register that, and it it happens more than once, you have to think there is something wrong. In this race, almost all the polls showed the liberals pulling away - not a huge lead but a consistent one. Ipsos didn't see that and, in fact, had a huge lead for the PC among likely voters right at the end. True, they could have been right and everyone else wrong. But that is unlikely. And I don't agree that Silver was such an outlier. Rather, he ut the polls together and came up with a consistent prediction that was similar to what at least some others were predicting.
      You are surely forgiven for getting Ohio wrong, especially when we are not absolutely sure you did.
      Can someone explain how one poll supposedly showed almost a dead heat among all three parties? Maybe pollsters should get graded....

      Delete
    4. It should also be noted that Silver's model does not rely simply on polling numbers. It uses economic numbers and even the ratio of Starbucks to WalMarts in each state.

      Delete
    5. IIRC, those considerations are taken into account when the election is distant. By election day, the model is almost entirely based on polls.

      Delete
  23. Yeah that's right Éric. Though they are given more weight in states where there is little polling. I remember in 08 when Silver had NC switch of from red to blue and it was based on polling of states with similar demographics.

    ReplyDelete
  24. Following the polls is entertaining, but I think they are of limited utility in predicting election outcomes. They're either pretty close, or not very close. Which polling firm had the closest prediction in this election, or in another, is not so important. Who gets it right in one election probably bombs out in another. How the pollsters evaluate, justify, and excuse their own performance is mildly entertaining, especially if one enjoys watching their creative squirming.

    What's more important is what K Wynne will do with her majority, and how the body politic and economic will fare for the next four years under her stewardship.

    John Diefenbaker probably had it right when he said that dogs know best what to do with polls.

    ReplyDelete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.