Tuesday, November 6, 2012

Nate Silver and the trials of a forecaster

Election night update: Congratulations to Nate Silver on a job well done!

Morning after update: After a long night and early morning, there won't be any posting today. But I invite you to read my article for The Globe and Mail in which I assess how the various forecasters did. The polls did quite well last night, though it is worth noting that Silver's forecast was better than that of RealClearPolitics, which does a simple averaging of the polls. It all makes me lament for the polling situation north of the border - how jealous I am of the amount and quality of data that is available in the United States!
-----
There is an election today. Millions of Americans will be heading to the polling booths today - if they haven't cast their ballots already - and will be glued to their televisions as the results start pouring in. Millions of people around the world will also be paying close attention to the election results, including a good number of Canadians.

Polls have taken up a large space in coverage of the election, and in the last week or so that focus has been especially directed at Nate Silver, who runs the FiveThirtyEight blog for The New York Times. As Silver's forecasts a few weeks ago still showed that Barack Obama was favoured to win, despite the President trailing by a narrow margin in most national polls, he was heavily criticized for being a partisan hack or simply wrong. Some of these criticisms were well-argued and thoughtful, but most were innumerate or downright personal. The backlash against these criticisms resulted in some of the best analysis of what forecasters do that I have ever read, and even more attention was thrust upon Silver (though I don't think he welcomes it).

As long-time readers of this site know, and as anyone who can look at the URL in their browser and put two -and-two together will figure out, I am an admirer of Silver's work. I was inspired to launch this site because of his own work during the 2008 presidential election, and many of the methods I employed were inspired by his. The idea of weighing the pollsters by their track record, for instance, comes directly from FiveThirtyEight. But after being in operation for more than four years (almost as long as 538 itself), ThreeHundredEight has developed its own distinct character.

But despite very different methods and backgrounds (not to mention track records, though I suspect Silver would struggle just as much with our tricky electoral system and lack of good and plentiful polling), we work in the same field. As a result, a lot of the criticisms that have been aimed at Silver over the last week have hit close to home. Though I have (thankfully) only rarely received the same kind of treatment in national media, and the scale is greatly different, I have seen some of the same kind of criticisms (and insults) on the Internet, in my inbox, and on my Twitter feed, and know what it is like to have your motivations and competence unfairly called into question.

Though this site and others like FiveThirtyEight, and there are several of them in the United States, are thought of mostly as prediction sites, that only scratches the surface. I have always found Silver's analyses of polls to be far more interesting than his forecasts, which have always been based on what the polls are saying anyway. This site tries to do the same thing - providing analysis of polls that goes beyond what can be found in most media coverage, and providing more than just a guess as to what could happen.

Polls take up a lot of place in media coverage in Canada as well as in the United States, and there is a corresponding need for responsible poll reporting in this country. Readers need to understand what polls are able to tell us and what they are not. And they can only gain so much from reading about a single poll in their favourite newspaper. As the US election has made quite clear, different polls can say very different things.

Ten polls released in a short time span each tell 10 different stories. Some of those stories will be an accurate depiction of what is happening, and some of them will not. Focusing on one poll or another will lead people to miss the forest for the trees.

The usefulness of a site like ThreeHundredEight or FiveThirtyEight is in telling the story of what those 10 polls are saying, and not just in terms of aggregation and prediction. Silver takes a macro look at all of the polls and describes a narrative that the polling data - and only the polling data - backs up. I try to do the same here. Our projections are about what the polls are saying will happen based on the information available now and what has happened in the past, not what we think or hope will happen, and both of us are always quick to include plenty of caveats.

And those caveats are incredibly important. As of writing, Silver's model indicates that Obama has a 91.6% chance of winning. That means Mitt Romney could still win, but the odds are heavily stacked against him. With an eye towards the limitations of public opinion polling, Silver is forecasting that the odds that the polls will be wrong enough to give Romney the win are very low. There is little valid argument to make against such a statement.

Looking at our recent electoral history in Canada, the fact of the matter is that in only 44.5% of cases has the polling error been large enough to erase the 2.6-point margin that Silver current gives Obama in the popular vote. And as those errors have an equal chance of going one way or the other, that would give Obama a 77.7% chance of winning the popular vote, based on how often the polling margin between two parties in Canada has turned out to have been proven wrong on election night. Add Obama's electoral college advantage to that, and you can see why Silver gives Obama such good odds. But that still doesn't mean Romney can't win.

I had my own set of caveats in my final projection for the Alberta election, which was very open to the possibility of the Progressive Conservatives winning a majority government. I was heavily criticized at the time for that openness - until the results started pouring in. (Then I was criticized for not giving the Tories a thumping majority to begin with.) In the recent Quebec election, I forecast that the Parti Québécois would win, but the question was whether it would be a majority or minority government. In future elections I intend to emphasize more of this uncertainty.

These sorts of caveats and confidence intervals might seem like hedging, but it is in fact the responsible thing to do. There always needs to be a recognition of what we do not know - and the potential error in polls is one of those things. But there also has to be a recognition of what is most likely to happen based on the information that is available to us, and that the information is going to be reliable more often than not.

Criticisms of the specific nature of Silver's model might be warranted. It is a very complicated model that will probably perform only slightly better than a simpler model most of the time. But there are a half-dozen well-known forecasting models operating in the United States, and dozens of lesser known versions. Each has their pros and cons, but to limit the appreciation of the work that Silver does to the percentages in his charts is to miss out on the real value of his work: objective, fact-based analysis of polling data.

Forecasters are not fortune-tellers - our work is based on the data that is available to us and is only as good as the information we are provided. Some critics expect the sort of accuracy that cannot be achieved by models that are based on the polls. When the polls are wrong, models based on them can only do so much. I would have been raked over the coals if I had claimed that the polls were under-estimating the Tories by 10 points in the Alberta election. Expecting any statistical model to foresee and emphasize those 19-times-out-of-20 outcomes is unrealistic, and reduces their usefulness (if the confidence intervals are wide enough, a model will never be wrong). There are plenty of other places to find predictions based on intuition and opinion alone - and anyone who has studied those kinds of predictions can tell you how valuable they are.

Most frustratingly from this side of the argument, a lot of the critics that have leveled their guns at Silver are well-placed to do so. It is very easy to criticize from the sidelines when nothing is risked, like a heckler at a comedy club who would die of stage fright if he was forced on stage. As long as a critique is reasonable (and not based on something as ridiculous as the sound of Nate Silver's voice), the critic is in a win-win situation. He or she is just showing a healthy journalistic skepticism (especially if the forecast ends up being good). If Silver's forecast turns out to be wrong (and a Romney victory will be seen as such, even if Silver's forecast remains open to the possibility), he or she will have been a clairvoyant.

If Romney wins, Silver will have been proven foolish for having relied too much on those increasingly inaccurate polls. Those who are still calling the race a toss-up also risk nothing, even if Obama ends up winning by a relatively comfortable two or three points. "The race was never as clear-cut as Silver claimed, the undecideds just broke towards the president because of Hurricane Sandy in the last days of the campaign" - you know that argument is coming (and it is already being made).

This no-lose gamble is worth taking if it means "I told you so" gets to trend on Twitter. Everyone wants to be smarter than the egg-head.  Silver's reputation is riding on tonight's result. The reputation of those Doubting Thomases won't be dented by their dismissal of his work - but they'll take the credit if he ends up being wrong.

On top of this, Silver can be dismissed as having been "lucky" if his model proves successful. Colby Cosh of Maclean's made the implication in the last paragraph of his otherwise mostly fair assessment of Silver's track record. But focusing on those occasions when Silver was wrong (how his baseball forecasting model couldn't accurately predict a few exceptional individuals, for example, or his struggles with the UK election) seems to miss the point, particularly from someone who wrote a piece the day before the Alberta election in which it was argued that I was under-estimating Wildrose. I don't mean to pick on Cosh, but merely to point out that, sooner or later, everyone trips over something like the 2012 Alberta election or Ichiro Suzuki. And while Cosh does a good job of showing that Silver is not infallible, Silver would be the first to admit that himself.

But all of Silver's hard work, his thousands of words of analysis, his painstaking attention to detail - and if he's right? Luck. The same people who level criticisms at him for providing probabilities to a one-off event will claim that, if he is right, it doesn't matter because it could have gone either way. It could, of course, but there is a reason why his forecast says one outcome is more likely than the other.

Silver and others in the forecasting business don't have the luxury of chirping from the cheap seats. We bring this upon ourselves, of course, but the best of us are willing to admit when we were wrong and also accept that we could be wrong. A lot of our critics assign a degree of brash, arrogant certainty that most of us simply don't have.

In the end, Silver's forecast is all about the odds. I hope that the cards fall well for him tonight, and if they don't I will still look to him for the best polling analysis anywhere. For my part, this will be the first election in four years that I will watch as a mere observer, and I'm looking forward to a fun night.

44 comments:

  1. The vitriol that has been thrown towards Silver is disturbing. The complete lack of understanding of simple statistics by most of those casting the insults, or, their unwillingness to learn simple stats says more about them than it does about Silver.

    ReplyDelete
  2. Pew, the US pollster, discloses that only 9% of the voters contacted were willing to give out their voying intention. Pew said this was down from 38% in 2008. I would think this could influence the accuracy of polling. Eric, has this phenomenon been reported in Canada?

    JKennethY

    ReplyDelete
    Replies
    1. JKennethY, do you mean that 9% of people called were willing to answer the survey, or that 9% of people who answered the survey were willing to give their voting intention?

      If it is the former, that sounds about right. Pollsters almost never disclose their response rates, but Campaign Research did during the Alberta election and the RR was about 8% or so, if I recall correctly.

      I am surprised that it has fallen so quickly in just four years in Pew's polling, though. That is worrisome.

      Delete
    2. Did Campaign research report the contact rate? I'm looking for some commercial polls to compare some academic survey results I have (50% contact rate and 13% response rate, including those who did not pick up the phone as non-responses)

      ap

      Delete
    3. Yes, they did. In their final poll, 38% of people called picked up the phone, and 5.6% of the total called completed the survey. Of those who answered the phone, 15% completed the survey.

      Delete
  3. If I may say so, I think that the tendency of media or partisan narratives to regard the work of 308 and 538 as more akin to fortune telling, is part of the general rejection of scientific method and knowledge in contemporary life. In this country, the federal government has done much to reduce our access to good scientific data and statistics, while Conservative partisans have repeatedly suggested that "science" is nothing more than liberal dogma. It is true that we can choose whichever principles we like to govern our decision-making, but some of those principles lead (on average) to predictable outcomes, while others merely reflect an ideological bias.

    In much the same way, you and Nate provide an analysis which is based on proven methodology, and which can offer the most predictable interpretation of all available data. Others can still prefer to interpret data based on ideological preference, but you can easily point to your track record to show that your average is considerably better. But two things: first, it's performance ON AVERAGE, and therefore will always have the occasional outlier, which is not a flaw but a feature; and second, that individuals, political parties, and media organizations will always select the narrative (and supporting data) that they like, because the performance of their model is almost completely unimportant, and the expression of their desired outcome almost all that matters. So, like it or not, you'll always face criticism, even if most of it is unwarranted and uninformed.

    At the same time, what I like best about your and Nate's approach is that you generally shy away from speculative interpolation of the data. Once you start to speculate, you move more into the realm of "punditry", and away from the solid ground of statistical analysis. And as a pundit, you'll have to face the wrath of those who disagree on level ground--when speculating, almost any outcome can be predicted. If it matters, I appreciate the service you render, and I hope you keep it as neutral and non-partisan as possible. And thanks!

    ReplyDelete
    Replies
    1. "Conservative partisans have repeatedly suggested that "science" is nothing more than liberal dogma."


      You are confusing "real science", with "stuff we say often enough people will believe it is true". Hockey stick curve for global warming, (now climate change). etc.

      When tested the crap that is made up often falls apart given to new made up stuff. Much of what you mistakenly call science is just liberal dogma.


      For the next example of liberal dogma: The polling in this election. (And Nate Silver's acceptance of it). Look at the internals of the polling. Most pollsters give a large internal identification advantage to the democrats. One that is even higher than was achieved at the peak of Obamamania 4 years ago.

      Nate could choose to model this out (although admittedly I have know idea how one would come up with a real number to put into the model without the (almost) untainted data from tonight). Unskewed polls tried one method... just backing the polls to the voter id of 33/33/33 repub, dem, indy.


      I don't have a fancy model like Nate or Eric. Only my eyes and ears.

      And what they tell me is that Clinton and Biden are campaigning in Minnesota and Pennsylvania... millions upon millions were spent in the last couple weeks on 2 states many considered out of reach for Romney. They tell me that hundred of thousands of pre election day voters that favored Obama in 08,... didn't get cast in '12.

      The democrats are acting like they are desperate... So when I see a 90% confidence despite all the evidence starting me in the face

      well I guess Mr. Bob, sir. We are going to get a lesson tonight about what real science is. And what is a bunch of made up crap being served to us.

      My (unscientific) prediction: R/R 54, O/B 45. R/R take 298 on the electoral college.

      Delete
    2. I might also point out that I question what exactly is a track record.

      It is often pointed out that Nate only missed 1 state in 08.

      How many actually took really making a prediction? Is there anyone on this site or Nates that couldn't have gone 2 for 2 on Texas and California model or no? In 08 there was a blue wave... Toss-up states where polling was even close were pretty few. 1 in 2 or 1 in 5 actual predictions?? that isn't so good.

      In 2010 through the midterms Nate's model got all the safe ones right. But on the ones that everyone identified as a tossup? not so good. He missed the teaparty turnout effect an continued believing that the partisan id hadn't changed much since 08. (And I believe atleast that he is making the same mistake again).


      In Canada. I thing Eric has a much harder job. with fewer pollsters, fewer poll data points. And when it comes to modeling a much bigger problem with there being many riding's where you can have not just 2 competitive candidates but sometimes 3 or 4.


      When I look at Nate silver's model. I don't think his accuracy is anywhere near what it should be. Not given an A-B choice where 30-40 of the state electoral college races can already be chalked up to one party or the other before the first poll was taken.



      "Atleast he has skin in the game??"

      So do another several million people who made a prediction. Even I made one. He put more work into his, built a model, based more of his reputation on it. So? Can anyone tell me his track record?? without go the the article that Eric mentions? Can anyone tell me the record of intrade? real politics? PPP? Rasmussen?

      Nobody really cares until one of them hits close on a result that they like.... That was in their own head.

      Delete
    3. The "skin in the game" that pollsters and 'professional' forecasters have is not comparable with the millions of people who make predictions in the office pool. Livelihoods can be at stake.

      Delete
    4. As much as I respect your opinion Eric.... That is complete and utter Bull.

      You and I might be able to look at every pollster out there and tell others how close they were in the last 5 elections (well that is bull... I can't hold that much data in my head and still think about football).

      But go ask your neighbor to rank them. better yet. don't give them the list.. see if they can rattle off a few names.


      The only reputation that rally matters is that lucky hit. And only if you were the closest. noone remembers second place. Nano's was within 1/10's of points one election. Silver got all but 1 state in 08.

      Ask people that can tell you that to tell you how he did in 10?? how nano's did in other elections? nope, nada. most of them will have no clue.

      You might get some extra traffic for being unique (and lets face it... with only a dozen or projectors in the US... even that is still pretty unique.

      But win, lose or draw,... their traffic isn't going to change.

      Same with the pollsters. without that magic. "I got the closest,... once" Their traffic isn't going to change... HD, Abacus, PPP etc... they are all still faithfully employed.

      And in 6 months.... noone will remember that Silver's Model was 90% wrong this election cycle. Noone but the heavy politics watchers... partisans... who will still hold him up. or condem him for his prediction. And in 2 years everyone will still rush to see his writings.

      Delete
    5. Maybe you are right, but I think that has more to do with the good work Silver does as a whole. If all I did was put up numbers without commentary, I would not have survived past misses.

      Delete
    6. Barcs
      I'm sure you did not intend to (and I am sure you will disagree), but your posts prove my point exactly. First, you demonstrate that you have little understanding of Nate Silver's methodology, while at the same time complaining that his methodology does not take into account effects which Silver specifically does take into account. Moreover, he very articulately explains that the odds he assigns based on data are just that--odds. Even with a 90% chance assigned to Obama, what this means is that, with the current available data, the model indicates that Obama would win an electoral college victory 90% of the time. That still allows for 10% of the time where Obama would not win--again, based on the very same data.

      So there are really two kinds of arguments which could be made against an Obama win: ones that explain why this election will fall under the 10%; and others that simply dispute Silver's analysis based on other "factors". In the former case, the only reasonable explanation would be a significant amount of systemic bias in a majority of the polls carried out in most of the states. It is certainly possible, but very unlikely, hence the less than 10% chance ascribed to that. In the latter case, it is an argument as to whether we go with statistical analyses, or something that is not based on this method of data collection and aggregation. Any alternative determining principle can be supplied, including one's "gut". The problem with these alternatives is that they are generally not systematic, and mostly rely on highly subjective interpretations, such as believing that "The democrats are acted like they are desperate...", and then using that interpretation for a speculative conclusion (democrats acting as though they are desperate could indicate that they know they are about to lose, AND it could also indicate that they are passionately engaged and therefore much more likely to vote in numbers, thus the same subjective observation can support two diametrically opposed predictions).

      Finally, my comment about Conservatives was based primarily on the actual fact that Conservatives have frequently made statements to the effect that facts and science matter less than what they "know" to be right. And often making policies and decisions based on that. I don't deny that the scientific/empirical model is based on a set of assumptions as well, but as I said, I like the fact that those assumptions create a methodology that delivers predictable outcomes. And that is an advantage I won't easily be persuaded to give up.

      I would also echo Eric in stating that I can't think of anyone who would have more skin in the game than someone whose reputation and livelihood depend entirely on how accurate her analysis is, rather than on who wins. Which is another reason that Eric and Nate will simply have to suffer the criticism of those who have little in the statistics game, but everything in supporting their preferred candidate/party.

      Delete
    7. Science, you keep using that word, but I don't think you know what it means.

      Based on the statements of a couple conservatives you assume that all deride science. This is untrue. Except as it relates to the crap the left dresses up as science. Insite or anti-vax for example? Write one or 2 badly written studies cherry picking data and fudging numbers. Write a couple dozen more reviewing literature without actually doing any new research... and presto we have 25 or 26 studies in support and anyone who disagrees is a evil tory who hates science....

      Global warming is another good example of the crap science.. Climate change you say? Well maybe, but I have been hearing how the science is settled way back when it was only global warming. The chief spoksman to tell me I need to reduce carbon??? has a carbon footprint several thousand times the size of mine. Somehow Canada reducing 1/3 of its emissions will save it.... while in the next breath China, india, and the rest of the developing world can continue adding the equivalent of several canada's every year. And to come up with the solutions we need to reduce carbon... we will hold conferences who emit as much as many small countries every couple years..

      This is science to people like you.... So you can understand when a Conservative says "I think I will go with my gut" He isn't really against "real science" He is only against the junk science...



      Now, when you figure out what real science is.... then come back to me and we can talk more about Nate's methodology. The point I already contested for example. RCP with a straight average of the polls... most of them average about a +6 democrat..... has the 2 camps tied. You tell me that Nate accounts for that in his model. and other variables.... which increase the Democrat camp advantage to 2.5 up from 0.

      Admittedly there is other factors involved in his equations than just a polling bias. But, it would seem to me that in correcting for a democratic bias should swing towards the republicans...... Maybe he is just putting the numbers in backwards

      Delete
    8. Or, perhaps, he was dead on correct and there was no manipulation, no need to 'unskew' polls and no overall bias in the polls.

      Delete
    9. Barcs
      Reflect on what you write here--it is an apologia for liking some science and not other science, which you then deride as "junk science". I never said ALL Conservatives are against ALL science, I only reported (accurately) that Conservatives have repudiated the need for basing decisions on scientific evidence, when it has suited them to ignore that evidence.

      "Science" is really a belief in a methodology, not a specific outcome. The empirical method posits that we can learn more and better information about the world around us by using a specific means to test our hypotheses and verify the results. It also has a track record of providing information which, when acted upon, leads to the expected result far more frequently than the alternatives. One could decide that the daily horoscope was a better source of information for making significant decisions than a combination of logic and reliable sources, but for most this would lead to adverse outcomes. And, while there is only one defensible empirical choice, you need only scan 3 or 4 news media to see that there is a multitude of conflicting astrological information available.

      I am not talking here about global warming. The science in this case can only tell us how today's climate compares with historical observations. There is no scientific consensus that "proves" that it is due to human activity rather than natural cycles or vice versa. It is warmer, but that doesn't mean it's because of fossil fuel use or any other specific human intervention. On the other hand, no successful business enterprise would fail to develop a risk management plan for something that was potentially devastating, regardless of what the "cause" of the devastation. The problem with those wailing about it being "junk science" is that it doesn't matter: what matters is that we are choosing to do nothing rather than something meaningful. People in the path of Katrina simply wanted better levees; they didn't think twice about the cause of the hurricane when they were busy trying to stay alive.

      Doing the same thing time and again and expecting different results is a definition of insanity. Given the results this morning, you might want to consider some of the implications for your approach to counterfactuals which you prefer not to countenance.

      Delete
  4. People have a real hard time wrapping their minds around probability. You can have a high probability of winning by a small margin, small margin doesn't necessarily translate into lower probability.

    Combine it with the fact that a disturbing number of people are take polls about popular voting and making conclusions about electoral colleges. It's like counting only first-downs in football and claiming that the game is too close to call.

    Another reason is that partisans have a real interest in pretended that the election is a tossup to encourage their supporters to come out. A lot of evangelicals haven't warmed up to Romney and Democrats typically have a lower turnout from minorities, unless they can convince them that the election is a tossup without their vote.

    Finally, Silver and you are dangerous to pundits who'd rather be booked and paid because of their "experience and gut", rather than Math that really puts them out of business.

    ReplyDelete
    Replies
    1. True. But a 2.5% vote advantage that Silver gives Obama over Romney... is not a small margin. Neither is 88 EC seats.


      I think you will see a resurgence of experience and gut pundits when it turns out tonight that math is both dependent on good data... and that even hard data can be extrapolated and manipulated in different ways.

      Delete
    2. A related issue that is rarely explored here is that polls do not exist in isolation, that is, apart from the dynamics of society. The moment a poll comes out, the concerned parties (political parties and their respective backers) digest it, then create and implement strategies to improve or combat its results, while voters respond to it in their way(s). Polls can become self-fulfilling prophecies - at least, that is the hope for those who try to manipulate them, or who commission polls of borderline scrupulousness. The polls of the Alberta election in particular give rise to plenty of speculation along these lines. As for the US election, I have strong doubts that Obama will lose, but if Romney were to win, would it prove that recent polls showing him in the lead were correct, or that those polls had simply moved people to vote for him? Polling statistics don't actually tell you what's going on, i.e. why the outcome of a given election is what it is. At best, they provide a snapshot of the horse race at a given moment. Obviously, Éric and Nate are trying to do a sort of Muybridge-like sequence of snapshots to achieve an overall picture of a dynamic phenomenon. They might tell you whether or not all four hooves leave the ground in a gallop (gallup?) but can tell you very little about how or why.

      Delete
  5. I am a huge fan of both sites and the inciteful articles are very interesting. I find, and was referenced in an article on 538, is that most media is trying to entertain and sell. A "close race" is more exciting then a "Obama has map on his side" article. It is amazing the number of times the popular vote has held a headline when that is not how a president is elected. I find sites like this more educating and more like news then most sites. I don't want to be entertained I would like to know the real story. Even if the prediction is wrong or off that does not matter to me as much as getting the analysis of the pills wrong.

    -interested reader

    ReplyDelete
    Replies
    1. That I can absolutely second.

      A close race is definitely much more fun. And the best part about both 538 and 308 (or are you 330 now) is the in-depth analysis and punditry based on it.

      Even if it isn't the exact result. It is fun to see how close one can get and to read on the various issues that are brought up.

      (Big thanks to you Eric, and to all the others ones like RCP, Nate, Unskewed, cook, sabatos, etc etc)

      Delete
    2. Thanks Barcs, but I wouldn't include Unskewed in your list. The guy who runs it wrote a critique of Nate Silver based on his effeminate look and girly voice. It was embarrassing, if his work wasn't enough already.

      Delete
  6. You might also look at the work done on here Eric. Has been quite accurate over the last few elections

    http://www.electoral-vote.com/

    ReplyDelete
    Replies
    1. Yeah he is pretty much the pioneer. He is doing straight up poll averaging (pretty much).

      Delete
  7. Hear hear, Eric. I've been following Nate's work closely and comparing it with others, such as RealClearPolitics and Politico and Charlie Cook, and they all generally agree - but because Silver comes forward with actual odds, he's criticized. Moreso from the right, because they know they're losing the race right now and Nate's continued propogation of proper analysis annoys them.

    I feel you get the same thing, and yes, from conservatives mostly, though my side of the spectrum isn't innocent. Warren Kinsella comes to mind, and I've had to defend pollsters and projections several times across several sitesm including my own. It can get heated at times.

    I'm not sure what it is - people either just don't trust you (obvious in my case because of my partisan leanings, but I do it for a hobby, not as my life's work), or they don't trust "punditry" or "big pollsters" - like there is some giant conspiracy to work against the Conservatives or tarnish to reputation of the Liberals. The media's insistence on putting singular polls out there as the end-all and be-all of public opinion doesn't help either, especially if you end up having an incorrect prediction. Its a damn shame people don't look at it the way these things are meant to.

    But you do good work Eric and I'm glad that someone out there is working hard to ensure we have this kind of site in Canada. Its important to get these kinds of analyses out there. We need to study the electorate and its machinations and fluctuations, no matter where the results lead us.

    ReplyDelete
  8. Just note on the Alberta election. It may be a little off topic but I think it is worth noting and may help make your point.
    Timing can be everything, I think the pols for the Alberta election may have been wrong but not as wrong as they were just 5-7 days behind a very volatile electorate. Given our electorial system, our lack of good polling numbers I think it is amasing you do as well as you do Eric. It must be very hard some days to cut through the spin and just go with what the numbers are telling you at the time.

    ReplyDelete
  9. Charles Harrison06 November, 2012 11:51

    "If Romney wins, Silver will have been proven foolish for having relied too much on those increasingly inaccurate polls."

    Which inaccurate polls?

    ReplyDelete
    Replies
    1. They may be pointing to the concern that the sampling of the latino vote is being done poorly:

      http://2012.talkingpointsmemo.com/2012/11/latino-polling-problems.php

      I am very curious to see how Nevada, Florida and Colorado play out as I think these states should show if the polling firms are accurately sampling the latino population

      ap

      Delete
  10. I just finished voting down here in Indiana on my lunch break... it went very well. We have to show our IDs, obtain a two-sided paper ballot, and mark with a permanent marker our selections by connecting arrows. Although we may not be the most important state Presidentially (unlike four years ago), we do have an important and lively US Senate race, and several down-ballot and statewide races (Governor) that are important.

    I do want to say that I do value your and Silver's commentary tremendously. You both provide invaluable services and great analyses of your respective elections. The only people that will attack the two of you are either uneducated or are desperate to find anything that supports their position/candidates. While polls have recently suffered unfortunate inaccuracies in Canadian elections, I feel comfortable enough with the sheer amount of American polls that an average of the relevant pollsters minus outliers will provide a gauge for the final vote tally tonight. Hopefully Canadian pollsters can learn from their brethren down south and implement new strategies to reflect more accurate polling in future elections.

    ReplyDelete
  11. Great article as always. 538 was was some great reading 4 years ago. I kept thinking how great it would be to have the same analysis and commentary there. And along you came!

    Honestly, the post-morton and results analysis are nice to look at the morning after. But it is the hours and hours of good reading that keeps me coming back to 538 and 308. Keep up the good.

    ReplyDelete
  12. Silver uses Monte Carlo simulations to generate those projections. Taking the polls available, and their confidence and margins of error, he simulates the election 10,000 times (or more - he probably does more) resolving the ambiguity in the data randomly each time.

    When he's done, he measures how often each candidate won in the simulation. It's a clever approach, and not one prone to error.

    If Obama wins 90% of the time, that's a significant edge, but it's not a dominating lead. The 10% outcome is unlikely, but it's not unthinkable. If you generate a random number from 1-10, you'll get 10 from time to time.

    According to Silver, Romney can win this. I think it would be very interesting to read Silver's analysis of the results if Romney does, because, depending how Romney were to win, it might still be entirely consistent with Nate's projection.

    You're right - if Romney wins people are going to tell Silver he was wrong, but that's not necessarily going to be true. It's entierly possible that Romney could win with Nate's projection still being accurate.

    ReplyDelete
    Replies
    1. there is also a 0.9% chance that neither will win.... but only a 0.2% chance that they will tie the electoral college.

      Presumably that means there is a 0.7% chance that someone will come from outside, and independent or libertarian or write-in..



      Everything is obviously possible. but Nate has given odds at 90% that Obama will win by 2.5 points, and 88 college votes.....

      Those are not small margins. nor as jbailin notes farther down is most of his readership (or the electorate at large) likely to shrug their shoulders and say "well there was a 10% chance that he would lose so the model does cover it."

      On Wednesday or possibly later it will be judged as a win or a loss... As any prediction should be. And no matter the outcome the guy with a vested interest will spin and skew and rework as hard as he can to call it a win.

      Delete
  13. Nicely put, Eric. For those of us who want to understand what the polls are (and are not) saying, we turn to you and Nate. The analysis you provide is, really, an important service to the democratic process. And like anything important in a democracy, it will be targeted by all-sides, at one point or another. Especially those who are backing their Truman against your Dewey.

    ReplyDelete
  14. Hear hear! I hope Nate reads this.

    I think that you have a more sympathetic audience here - most people who visit understand the nature of what you do. Nate, by virtue of being attached to a large media organization and having now a fair bit of celebrity, doesn't have that luxury - a relatively larger fraction of his readership doesn't understand uncertainty as a quantitative measure, only a yes/no proposition.

    ReplyDelete
  15. Well, I had more faith in the american electorate than in the pollsters.

    It appears right now that I was wrong on both counts.


    Anyone want to bet on whether Obama can double the debt that existed at the day he took office?? he's already more than 1/2 way there. :/



    I think I will start construction on my bunker tomorrow. In the words of one of my favorite bloggers: "Poor yourself a shot of Rumpie, kick up your feet and enjoy the decline!".

    ReplyDelete
    Replies
    1. I was waiting for your mea culpa on Nate.. I guess this will have to do.

      Delete
    2. Barcs
      I think the results of last night should cause you to question some of your fundamental assumptions about how the world, or at least the American electoral process, works. It also might suggest, therefore, that your feelings about what Obama is doing to the deficit are also based on faulty assumptions. Do you truly believe that if McCain had been President the last four years there would be a smaller deficit? And a better economy?

      If so, your faith seems to exceed your understanding of global economics, electoral gridlock and history. Or perhaps you are simply choosing to go with your gut in the face of an avalanche of evidence against it. I'm only saying this because I don't believe anyone could have done much better--the choice confronting the winner of the 2008 election was massive stimulus (and Obama's was not massive enough according to people like Krugman), or massive recession. Killing the economy was no way to save it.

      Delete
    3. What I believe is what the data shows.

      When you ask could another president have done better? McCain maybe? Well I don't know if he would have. But hard data suggests that every president recovering from a recession in US history has done a better job than Obama. And did so while shoveling out less money than it took him to create jobs at a rate lower than the population growth.

      You talk about gridlock like bush didn't deal with a democrat house and senate in the last 2 years and a split for 2 years at the end of the first term... or Clinton who had to deal with the republican owning both houses for 4 years and a split for 2 more. Bush senior never saw republican control of either house and Regan before him had control of only 1 house for 6 years.

      Obama in contrast to the 4 presidents before him had a supermajority in both houses for 1/2 his 4 year term.... And yet we haven't even seen a full budget passed in any of his 4 years.

      gridlock is not new... only the degree to which this president is unable to work with others... even the democratic senate.

      Your mountain of evidence is just more crap the left made up and called science to cover up for Obama. Real science. Real numbers, show that one of the very very few numbers Obama improved on in his term... was his golf score.

      Enjoy the decline.

      Delete
    4. Scott.. on Nate it depends....

      He did run the table on the president

      But if you look at his senate data he missed 2.

      Now is that 2 misses out of 33? Or, does he really deserve credit for predicting California and texas where there is a 20 point separation??

      Even on his site he claimed that only 7 were competitive. Maybe it is 2 misses out of 7. 62% on the senate isn't that good of a record though. Gotta be a better way to spin that, eh?


      I do think he did pretty good. But not 98% good. Lets keep a little perspective shall we?

      Delete
  16. Seems Silver was right, love what you do here at 308, keep it up.

    ReplyDelete
  17. One should put statistics and facts before personal ideology. The math clearly favoured Obama in crucial swing states.

    ReplyDelete
  18. Silver kept saying that but the R's wouldn't listen and the media just loves a horse race.

    Looks like Silver is vindicated on Obama. How did he do state by state?

    ReplyDelete
  19. Silver is 50 for 50 on the states. His most likely outcomes were 332, followed by 303 EVs. Obama has 303 right now, and when Florida is called, he will have 332. He also called the popular vote pretty closely. In other words, I suspect even Nate Silver is a bit surprised at just how accurate his model turned out to be. He even got the Senate situation bang on. Kudos, Nate.

    ReplyDelete
  20. Just want to show my love and support of the fine work being done by 308. Something that can give the straight numbers and a fair analysis is greatly appreciated. Keep up the great work

    ReplyDelete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.