Wednesday, October 9, 2013

Nova Scotia: how did the polls and the projection do?

After the trauma of the elections in Alberta and British Columbia, it was almost more surprising to see last night's results in Nova Scotia generally align with what the polls were suggesting they would be.

Some pollsters did better than others. My detailed analysis of how the polls did can be found on The Globe and Mail's website. The article is available to Globe Unlimited subscribers only. If you like this kind of analysis and want to see more of it on The Globe and Mail as well as here on ThreeHundredEight.com, it needs your support. If you don't already have an online subscription to The Globe and Mail, please consider signing up as you might find my articles available to subscribers-only more often. You can find details on how to sign-up via the link above.

The projection did moderately well, particular in terms of the Liberals. Their vote haul of 45.5% fell just outside the likely ranges, which bottomed out at 45.8%. But there was a 21% chance that their vote would fall between the minimum and low projected ranges. Their total of 33 seats was projected exactly right.

At 26.9%, the New Democrats were just 0.8 points off from their projected vote totals. Falling between the average and high ranges, as they did, was considered a 68% chance probability. That they only won seven seats instead of the projected range of 12 to 15 was a surprise, but this was largely due to the performance of the Progressive Conservatives. Those seven seats they won did fall between the minimum and low range, though that was projected to be only a 5% likely occurrence.

The Tories ended up just outside the projected maximum level of support with 26.4% instead of 26.3%, and 11 seats instead of nine. That is unfortunate, to say the least. But this is partly due to the large percentage of the projection taken up by the final poll from Forum Research. More on why that was a problem later.

The Greens took 0.9% of the vote, just below the projected low range. Falling between the minimum and low level of support was considered a 35% chance, so nothing untoward there.

But one of the main reasons why the projection did miss on the seats for the Progressive Conservatives was that the polls did not do a particularly good job of gauging regional support, especially in Cape Breton. This is how it broke down last night:
The polling in Halifax and in the rest of the mainland was generally good, though no one got it bang-on. Cape Breton was a little more difficult. The full sample from Abacus, spanning the entire final week, was relatively close but the last set of numbers from them had the PCs doing much better, while the last set of numbers from Forum and CRA had them doing much worse. The samples were generally too small in Cape Breton, though, to get a good bead on the race.
The model did have some trouble translating the regional numbers into good seat totals. With the actual regional numbers plugged into the model, it would have projected a likely outcome of 22-33 seats for the Liberals (28 to be exact), 13-17 seats for the NDP (13), and 7-16 seats for the Progressive Conservatives (10). The Liberals ended up at the high end of that range and the PCs right in the middle of it, but again the New Democrats would have been over-estimated, falling in the minimum-to-low range. This is actually the first time in ThreeHundredEight's history of projecting a dozen elections that the model performed worse with the actual results plugged into it,

This demonstrates two things. The first is the limitations of a seat projection model in smaller provinces - on average, only about 8,100 votes were cast in any riding. That means that local factors can be especially important, and that is shown by the direction in which the model made errors: every which way. Normally, the model makes errors in the same direction as a party is over- or under-estimated. In Nova Scotia, however, that was not the case. For example, in the eight ridings the Liberals won where they were not projected to win, the NDP was projected to take six of them and the Tories two. In the seven ridings the PCs won where they were not projected to win, the Liberals were projected to win five of them and the NDP two.

Secondly, it highlights just how poorly the New Democrats did. They should have won more seats with the amount of votes they captured, particularly in Halifax. They ended up losing too many races that should have been winnable, with the Liberals benefiting.

In an election where just over half of those ridings that were not entirely new changed hands, the projection model was not particularly strong at the riding level. It made 18 errors, calling 33 of 51 ridings correctly for an accuracy rating of 65%. Taking into account the projected likely ranges, the likely winner was identified in 37 of 51 ridings, for an accuracy rating of 73%. That is not very good, but shows the hazards of small elections. The idea that the overall numbers are more important, and that riding-level errors balance each other out, is especially true in smaller elections.

The probability ratings for the riding projections did a decent job, however, with an average confidence of 72% in the ridings incorrectly called compared to 81% in the ridings correctly called. Half of the ridings incorrectly called had a confidence level of 70% or less.

One of the problems with this campaign was the emergence of the Forum Research poll on the eve of the election. Because the model weighs a poll by its median date, it rewards a firm like Forum that does its polling on a single day. This is actually not really the very best practice since it is better to poll over a few days to iron out the data and ensure that it isn't too dependent on the factors that can skew a poll when it is taken on a single day (i.e., are the people available to take a call on a Monday night different from those who can take a call on a Sunday night?). Coupled with Forum's large samples (due to the cost effectiveness of IVR polling), the final Forum poll took up almost three-quarters of the projection.

I don't think it is appropriate to reward this kind of poll, so going forward I will be weighing polls by the final date in the field, particularly during an election campaign. Doing so would have made CRA's poll weighted for Oct. 3 instead of Oct. 2 and Abacus's weighted for Oct. 6 instead of Oct. 5. At the very least, this would have brought the Progressive Conservative result into the maximum range and undoubtedly produced a better projection.

It is gratifying, however, that the polls did a good job in Nova Scotia. That makes the coverage this site provided throughout the campaign useful, since it was an accurate reflection of the ups and downs of the last four weeks. The lessons learned from this campaign will be digested and the new data added to the model's calibrations, and on we go to the next vote, slightly more confident than we were yesterday that we can trust what the polls are telling us.

19 comments:

  1. Time to downgrade Forum and upgrade Abacus in your pollster ratings? The latter almost hit a bullseye with their likely voters numbers. Ironically though, it's a good thing you didn't use those cause you would've underestimated PC support even more: they had PC 27% among likely voters vs. 28% among all committed voters.

    Dom

    ReplyDelete
  2. I have a hypothesis that part of the reason the PCs outperformed their popular vote totals is that the ballot question for many people was a vote for or against Dexter's leadership, which helped strong PC candidates by emphasizing their own strengths, but didn't allow the NDP candidates to do the same.

    ReplyDelete
    Replies
    1. Ryan,

      I must disagree with your hypothesis. The reason why the PCs out performed the polls is twofold. 1. PC base support is higher than the polls suggest. This may be the result of a sampling error (particularly lack of rural polling) or the result of the "shy Tory phenomenom" well documented in the UK- that generally accepts pollsters underestimate the Tory vote by 2-3% because Tories are reluctant to diverge their opinions to pollsters.

      Secondly, Bluenosers voted against the Government and not specifically against Dexter. Consequently, many ridings reverted to their traditional anti-government vote which helped the PCs.

      Finally, regarding the strength of NDP candidates it should not be lost on anyone that some of the most popular and competent MLAs did not re-offer; Bill Estabrooks, Howard Epstein, Michelle Raymond, Graham Steele-all likely would have been re-elected had they run instead their ridings turned red. I would submit that their leaving sent a very poor message to voters: "the best and the brightest don't want to be part of this government".

      Delete
    2. And the NDP performed badly in the election because they performed badly as a government. Once again (will no provincial NDP strategists learn this lesson from Bob Rae, et al.?), when the NDP acts like the Liberals and Conservatives, they get dropped like a hot rock. When non-habitual supporters vote NDP, it'S a leap of faith, an expectation that the NDP will do something different than the two establishment parties. When they do the same thing, there'S no longer any reason to support them... better to vote for the status quo business parties, rather than the johnny-come-lately...

      Delete
    3. Except when they're the Manitoba NDP. They've governed like middle-of-the-road Liberals and they won 4 straight elections.

      Delete
    4. Chirumenga,

      You have put forth a strange hypothesis. It is akin to saying Quebec's economy is systematically under-productive because they are not French enough! Of more concern you have been extremely selective in your examples; as if your assertion "comes from an ideological position" (chirumenga, Oct. 3, 2013).

      While you mention Mr. Rae you fail to include NDP governments in Saskatchewan, Manitoba who are genuinely thought of as fiscal conservatives and who routinely get re-elected. It appears you have data mined and purposefully excluded examples that do not adhere to your ideological viewpoint(s).

      I am also shocked that you who accuse English Canadians of holding an ideological bias toward Quebec part of a "larger English Canadian mainstream political view of Quebec" feel you posses enough knowledge of English Canada to unilaterally conclude; "When non-habitual supporters vote NDP, it'S a leap of faith, an expectation that the NDP will do something different than the two establishment parties".

      I would suggest you get to know English Canada better before making such provocative conclusions.

      Delete
    5. Ryan,

      But the NDP in Manitoba (and in Saskatchewan) is an establishment party - they've been around so long they are the first or second party in a two-party context. My point was centred on situations where the NDP has perennially been a third party and suddenly moves into power: Ontario, Nova Scotia, arguably BC.

      Bede,

      Your digs are noted, but it's important to note that, unlike Bozinoff, I don't make any claim to being objective or without a political orientation.

      Your point about Sask/Man I answered above.

      And you've reawakened the corpse of an earlier discussion. Nowhere did I say that English Canadians have an ideological bias. You didn't read me or you didn't read me carefully. I referred to mainstream English Canadian media and conventional wisdom. I would not presume to claim knowledge of what individuals think about anything, except from their own statements, and I would never presume that all people of one linguistic group think the same way about anything. You'll note, I also didn't make reference to such imaginary phenomena as the "Gallic race" or their possible Anglo counterpart.

      And, you've presumed you know something about me and what I know about English Canada. What I know about the country is more or less like anyone else who has more than a passing interest. We all have partial knowledge, but I do have some particular awareness : I grew up in Calgary, as an adult I lived for a decade in Ontario, my family is from Newfoundland and now dispersed from Vancouver to St. John's, and I've lived in Quebec for 23 years... so I've learned a thing or two along the way. I don't see where I've said anything so provocative that it can't be discussed.

      Delete
    6. Um. The NDP have been either the second or first party in every election in BC for over 50 years.

      Delete
    7. The reason I said that BC was an arguable example is because, apart from the 1990s, the NDP has struggled for mainstream media acceptance in the face of more obviously dominant Social Credit or Liberal or (back when) Conservative parties.

      Delete
  3. I know there's no straight-up way to build this into the projection model, but consider this as a victory lap for having mentioned this a few times during the lead-up.

    Cape Breton incumbents are made of kevlar.
    I repeat:
    Cape Breton incumbents are made of kevlar.

    Yes, Pam Eyking was actually able to edge out Keith Bain by ~300 votes, but
    A) She's the wife of the sitting MP, so she had some measure of incumbency herself.
    B) Keith Bain has lost in CB as an incumbent before, so his armor was already dented.

    The projection model missed 50% of the CB seats because it doesn't understand that CB's seat counts laugh in the face of regional swing.
    Year: Percentages: Seats
    2006: 43 PC, 30 Lib, 26 NDP: 4 PC, 3 Lib, 2 NDP
    2009: 40 NDP, 39 Lib, 28 PC: 4 PC, 3 Lib, 2 NDP
    2013: 48 Lib, 28 PC, 25 NDP: 3 PC, 3 Lib, 2 NDP 1 lost to redistricting.

    Cape Breton incumbents are made of kevlar.


    ReplyDelete
  4. All I see is yet another "who can tell" election.

    About all that can be taken from the polls is that the Liberals would win. Fine but anything else remains speculation.

    ReplyDelete
    Replies
    1. As someone who correctly predicted on Oct. 3 the NDP would end up in third, it was common sense not speculation that influenced my prediction.

      Those in the know would have come to a similar conclusion as me because the NDP only has 2-4 rock solid seats in the Province whereas, the Liberals and Tories have a base between 6-10 seats each. In addition we continually see the Tory or conservative vote in polls below the actual election result. As above this may be due to a "shy Tory factor" or the result that conservative voters vote with greater frequency than NDP supporters and to a lesser degree Liberal voters.

      Not to point fingers but, eric's projection had the NDP win Timberlea-Prospect, this is a very conservative projection as the NDP nearly won 70% of the vote in 2009. What Eric's projection did not pick up was a large number of NDP votes were votes for Estabrooks not the party. As a result a modified swing model was not able to capture the changing dynamics within the constituency.

      Delete
    2. The Shy Tory factor refers to an unpopular but relatively benign governing party, not the third place party.

      Delete
  5. How did Darrell Dexter lose his own seat? It was very close (and may go to a re-count), but it was still a surprise no one was predicting. Both Jean Charest and Christy Clark lost their seats, but it was not a total surprise in either case because they'd had close calls in those ridings in the past. Surprising it happened to Dexter.

    ReplyDelete
    Replies
    1. My personal projection showed Dexter's seat to be a close race, so I don't think it was outside the realm of possibility on anyone's mind.

      Delete
  6. Eric - do you think your underestimate of the PCs' seat and vote totals might have to do with how large the bins you group parties into are? You class the PCs as "other party with multiple seats," but there's still a big difference between a party like the PCs in Nova Scotia and Quebec Solidaire. Perhaps using some other more continuous rather than discrete measure? Like something based of the index for the effective number of parties?

    ReplyDelete
    Replies
    1. I'll be looking into these sorts of things as I digest the data. You might be right.

      Delete
  7. By the way Eric, does this mean you're one of the incentivized writers the Globe was talking about? May have to subscribe if that's the case.

    ReplyDelete
    Replies
    1. I'm not, but it certainly helps the more the Globe sees the value of this sort of analysis.

      Delete

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.