Friday, December 13, 2013

Much ado about nothing in Quebec polling

"Qui dit vrai?" asked one headline. It was a sentiment I saw repeated several times on Twitter, and a whole slew of newspaper articles emerged to highlight or question the 'contradictory' polls that emerged from CROP/La Presse and Léger/Le Devoir over the past week. The fact of the matter is that these polls are not really contradictory at all, and certainly not enough to precipitate such urgent questions.

Pollsters themselves carry some of the blame for the growing cleavage between them and the public on matters of trust, due to the corner-cutting methodologies and unimpressive disclosure standards of some of them. But the media has played a role in shaking that trust, by reporting numbers in a manner that leave readers perplexed, confused, and ready to give up on polls entirely.

Kickstarter Update: Amazing, we've passed the 2/3rds funding mark with 135 backers for the ebook project. We still have 14 days to go! Let's keep the pledges coming in. I'll have some news about the project next week!

After the Léger/Le Devoir poll came out on Saturday, CTV picked up the story with the headline "Liberals lead in new poll; support for sovereignty waning". Their interpretation of the trendline in support for sovereignty was erroneous, as we will see, but it clearly implied that the Parti Québécois was not doing very well. It was an interpretation repeated elsewhere, and the headline was easily shared on social media. Narrative established.

After the CROP/La Presse poll came out on Wednesday, CTV was back on the story with the headline "Poll shows growing support for the Parti Quebecois". They accordingly had a picture of a happy Pauline Marois to counter the sad picture of her they had in their previous story. Narrative contradicted.

Let's deal with the first problem in how these two polls were interpreted, and one that is the most problematic since it was the most easily avoidable.

Léger reported that support for sovereignty stood at 33%, with 51% against and 16% undecided. CROP reported that support for sovereignty stood at 44%, with 56% against.

Notice anything different with those two sets of numbers? No, it is not the 11-point spread between support for sovereignty in the two polls. It is that Léger included the undecideds in their report, and CROP did not. CROP was following standard reporting practice, or at least what has been the standard for years in Quebec. Léger opted not to portion out the undecideds, something which they almost always do. It is odd that they chose not to this time, and probably should have known that as a result they would cause an unnecessary firestorm. One article even went so far as to suggest CROP was doing something unusual in excluding the undecideds.

When CTV reported that support for sovereignty was waning, it was this 33% number that they were focusing on. Yes, if support among decided voters was at 33% that would certainly be a sign that support for independence was on the wane. But if reported normally with the exclusion of undecideds, that 33% would have turned into 39%. That is well within the norm, and even at the higher end of the usual 35% to 40% support that Quebec sovereignty has registered in recent years. Worse, that 39% is one point higher than the last time Léger reported on this issue in May 2013. The headline attached to the poll was not just wrong in its interpretation, it was factually wrong, at least if you're comparing apples to apples.

In addition, the comparison of the 44% in the CROP poll to the 39% in the Léger poll as somehow contradictory is wrong as well.

Both of these polls are online surveys and both firms are correct to say that a probabilistic margin of error cannot be applied. That doesn't mean, though, that the polls don't have a margin of error of some sort. Ipsos-Reid, for example, applies a 'margin of credibility interval' to their online polls, a practice that probably captures correctly how to measure these things but which, it must be said, still raises some eyebrows in some corners of the industry. Nevertheless, these polls do have a margin of error of some kind, and if we apply the standard margin of error that would apply to a probabilistic sample of this size (Ipsos's credibility interval is usually comparable to a probabilistic margin of error), we see that the two latest polls from CROP and Léger overlap at around 41% to 42% support for sovereignty.

(If, as Michel David of Le Devoir contends, the sample was skewed and support should instead have been 37% in the CROP poll, then the overlap is even greater, at between 36% and 40% support.)

Update (Dec. 14): David published a correction notice today, noting that CROP did indeed weigh their decided sample correctly. Also, an editorial in the Montreal Gazette repeats the error of comparing the 33% number in Léger's poll to the 44% in CROP's, and then goes so far as to give their readers a lesson in statistics.

The chart above shows just how much in common the CROP and Léger polls have had on this question, especially when you consider the amount of error that has to apply to these polls. And the idea that support for sovereignty is on the uptick is certainly consistent with these trendlines. The likelihood that this is a blip, and that CROP will report a drop in support next time, is also strong - and the trendlines will still be in agreement with one another. Broadly speaking, things are remaining stable on this front and both Léger and CROP are telling the same general story.

Now to the question of whether these polls are contradictory in terms of voting intentions. Léger reported 37% for the PLQ and 32% for the PQ, a widening gap from their previous poll. CROP reported a 35% tie between the two parties, a tightening gap. How can this be?!

It be. The change in support in these two polls from their previous survey are within any sort of reasonable margin of error (if not a probabilistic one). So for it to be reported that the PQ was dropping in support in the Léger poll or that the PQ has experienced a "considerable change in support" in the CROP poll is not entirely accurate. If the parties were seeing their support change by four or five points, then yes, there would be something there. A wobble within three points is not worth a headline.

Let's first look at CROP's polling, going back to March, and applying a reasonable interval for the margin of error. As you can see in the chart above, support for the Liberals and PQ has not wavered to any considerable degree from one poll to the next for some time.

For the Liberals, support of between 37% and 38% falls within the error band of CROP's polling going all the way back to April. For the PQ, a band of between 32% and 33% falls within the last four polls done by the firm. There is not much here to go on in terms of a trend - the PQ and Liberals have been wobbling back and forth since at least September. The only real trend worth considering is the gain of the PQ between May and October.

For Léger, the bands are even wider over their last four polls, denoting even more stability: 34% to 39% for the Liberals and between 31% and 35% for the PQ. More importantly, those bands overlap with the bands recorded by CROP since the summer. A band of between 35% and 36% has also been consistently recorded for the Liberals going back to the spring in Léger's polling. It is impossible to look at their numbers and see anything but relative stability.

It should be pointed out as well that results that fall on the outside edges of a normal margin of error are not as likely as results that fall closer to the actual reported number. When looking at the bands above, then, the emphasis should not be on the extremes (i.e. 36% to 37% is probably a better range for the Liberals in Léger's polling than the wider 34% to 39%).

To go on about how these two polls are contradictory, how one is showing a wide lead for the Liberals and failing support for the PQ and their option while the other shows the PQ making huge strides and sovereignty just around the corner, is to make it seem like there is a contradiction and thus that the polls cannot be trusted, when that is not the case whatsoever. Léger and CROP did their polling on different days and the two firms use different methodologies (each do their polls online, but they have different panels and recruiting methods), also contributing to some of the minor differences.

That is all that readers really need to know to explain the contradiction, and they can be told with confidence that things remain relatively stable and close between the Liberals and PQ (particularly since the PQ can pull more seats out of fewer votes due to their advantage among francophones) and that support for sovereignty is probably holding at about the same levels that it has wobbled within for years. Léger and CROP are good pollsters that do good work and are transparent about their methods and samples, particularly with their clients. There is no need to portray their findings in this dismissive way. Save the scoffs for polling firms that deserve it.


  1. Hopefully Quebec doesn't separate. TwoSixty just doesn't have the same ring as ThreeHundredEight.

    1. I would indeed like to avoid another round of "Will you be changing the name?" questions...

  2. Jean Lapierre mentioned that the CROP poll oversampled francophones (82% of their sampling) while Léger sampled 79% (which matches demographics) which created the disparity between the two polls

    1. CROP says that is not the case.

    2. 3% isn't much of a difference. Léger oversampling people in the Québec region by 21% (31% compared to 10%) is much more important. Crop is also oversampling a bit (15% comapred to 10%), but it's far more manageable after ponderation. I think Jean Lapierre is letting his personal preferences show and became biased a bit.

    3. Over-sampling in the Quebec City region makes some sense, in order to have a usable sample. It is also turning into a three-way race, which means a larger sample is even more useful.

    4. I can understand oversampling some (like 5% by Crop, maybe even all the way to 10%), but isn't 21% way too much and actually undersamples other regions?

    5. I'm new to this discussion, but if a pollster oversamples any region by X%, surely that's simply to bring numbers into line with overall population distribution. It wouldn't undersample anyone else -- they'd still get their representative share of the poll.

    6. It's fine to oversample in Quebec, no problem there (we will never complaint to have too many observations!). The problem is that by oversampling Quebec city, sicne they keep the 1000 observations in total, they are undersampling the rest of Quebec by a big margin. And this is where it can cause problems. It was fine when the rest of Quebec was very "uniform", but that might not be the case anymore.

      One solution, if they want to keep oversampling Quebec city, would be to then increase the sample size to 1200.

  3. I'm sure you've covered this at length somewhere, but oversampling can increase magnitude of errors in polling, can it not? Because you're counting a smaller subset of votes more?

    1. Yes, indeed that can be a problem as well if they are particularly skewed.

    2. Well you are more precise for the region you are oversampling. But as I was saying before, you are thus less accurate for the other regions. In Quebec, I think the best way to look at the problem is that pollsters don't sample from all of the province. They pretty much do three polls: one from the census region of Montreal, one from the census region of Quebec city and one for the rest of Quebec. If you oversample one region 9and then correct for this by weighting), then you undersample another one (and if this sample is very skewed as Eric said, then multiplying each observations for this sample will just increase the bias and errors. You can't correct for having a skewed poll by weighting, not really).

      This is why I go way beyond the normal margins of error in my model because polls are much less precise than they should be statistically. Eric does the same. Look at his amrgins of error, especially for by-elections. I thought at firsthe was wrong but no, he's basing his calculations on polls' past performances.

  4. Glad to see that you finally changed your mind of whether polls from an internet panel have margins of error (or credibility intervals or whatever). Very glad to see that.


COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.