Interviews with Christian Bourque and David Coletto

To finish off the series of pollster interviews related to my articles on polling methodology for The Globe and Mail, today we talk to Christian Bourque, Executive Vice-President of Léger Marketing, and David Coletto, CEO of Abacus Data.

Previously, I spoke with Don Mills of Corporate Research Associates (who should be very active in the upcoming Nova Scotia election), Frank Graves of EKOS Research, and Darrell Bricker of Ipsos-Reid.

Léger Marketing has been an active political polling firm in Quebec for a very long time, but has moved over to online in the last few years. This is what Christian Bourque had to say about the methodology:

308: Léger Marketing has used online polling for some time now. Why was the decision made to move over to that methodology?

CB: We made the decision based on the fact that we control the sample. It is our panel. We control it from recruitment to data collection to data cleaning. We started the panel in 2004 and felt comfortable using it for political polling after almost three years of comparative polling telephone to Web. Really, we have focused on Web-based electoral polling since 2007.

308: What are the advantages of conducting your polls online instead of over the telephone?

CB: Cost, timing is crucial and no social desirability bias. We can get up and running faster if the demand from the client requires quicker turnaround. Rushing telephone projects can mean “burning” samples to quickly fill quotas. Given that media clients do not have a lot to spend on polling we have been able to produce larger samples at a fair cost compared to telephone. The “honesty” factor also works in favour of the Web. 

308: What are the disadvantages?

CB: Controlling for potential differences between panel members and the general public is always something we need to control for. Our panel is over 70% RDD recruitment coming from our call center, so we are already very confident about the source. Panelists here are profiled at length so we can compare them, not only on socio-demographics but we can do it as well on technology variables, health-related profiling questions and we could even weight on beer brand to conform to market share statistics if we wanted to.

308: How is your online panel recruited and what steps do you take to ensure the sample is representative?

CB: Most of the panel is recruited from our telephone studies and telephone-based recruitment. It has a higher cost compared to online recruiting but generates more reliable and loyal panelists. We profile on over 90 variables over time so get a good grasp on who these panelists are and we stratify samples at the invitation stage to ensure that the output will not require important corrections or weighting. We also have a data cleaning and data quality protocol that gets rid of speedsters, straightliners and potential fraudsters.

308: What challenges do you face in building a representative sample of the population, considering that not everyone has access to the Internet and the potential for opt-in panels to attract a different sort of respondent?

CB: 86% of Canadians went online last week. That’s more that households who have a landline phone. In a market of rapidly decreasing response rates over the telephone, no methodology should feel they can take the high road and look down on the others. 

308: There are debates in the industry about the problems surrounding online polling not being probabilistic, despite some good performances. Why is this, or isn't it, a problem?

CB: Compare the results of probabilistic vs. non-probalistic polls in BC, Albera, Ontario, Quebec and the last few federal elections and you will not find a clear pattern of who has done best or worse. You either feel very strongly about one or other, but it comes down to faith and preferences more than any clear conclusions one can reach from historic data.

308: Léger Marketing has a very long history of polling in Quebec. How has political polling changed over that time?

CB: Like everywhere else, declining participation in elections is making our work more challenging. How do we account for that 30% to almost 50% who simply do not show up on election day.  Should we move to “likely voter” models in Canada? If you cross-tabulate participation by age and voting intent by age, you can explain most of the differences between polls and election results in the recent past (except Alberta). But age is not the only factor. Disengagement and cynicism need to be factored in too, outside of age.

When doing comparative polling or comparing our historical results and those of the competition, differences between phone and web tend to be rather small and not necessarily consistent over time. We found that only weighting by age and sex will tend to produce slightly more left-leaning results on the Web. We have been using a more complex weighting scheme over the past six years to account for that (education, income and household composition are now factored in).

I'd also like to add the following:gGiven the recent critique and, some would say, controversy in the market, I believe we, as an industry, should come to agree to greater disclosure mechanisms to allow the industry to develop a better understanding of the changing landscape out there. This would benefit us all.  

**************

Abacus Data is a relative newcomer to the political polling world, having first appeared in 2010. Abacus uses an online panel for its polling, but has used other methodologies in the past. I thought David Coletto could give us an interesting perspective as he has worked in the industry in an era where online polling was always an option, as opposed to the other major players who cut their teeth in the telephone-only age.

308: Why has Abacus settled on using online panels for political polling, and what are the advantages and disadvantages of the decision?

DC: Abacus Data decided to exclusively use online panels for political polling as we decided that the advantages outweighed the disadvantages.

Advantages

Online panel research provides for a variety of advantages over live telephone or IVR research. Online research allows a large number of respondents to be contacted simultaneously, meaning the study can be completed much faster than with other methods. Also, the nature of online design allows for great flexibility in the visual appearance of the survey and in question design. Such design aspects allow for the creation of scale questions, visual sliders, drag and drop, or even the presentation of audio and video to respondents.  Further, online research allows for broad flexibility in sample design to target groups of respondents along virtually any screening criteria. Finally, online research is considerably more affordable than telephone or IVR, making it attractive for smaller firms and repeat projects.

Disadvantages

As online polling involves drawing sample from a panel of potential respondents, it does not constitute the entire population and therefore cannot be considered a true random sample – this is the primary disadvantage of online polling. Further, with online research it is currently not possible to verify that the respondent is exactly who they claim to be. While this is also true of IVR, it is much easier to verify an identity with a live telephone interview. Online polling can also result in certain coverage bias, especially among lower income and older groups.

308: What are the costs of online polling compared to over the telephone?

DC: The capital costs of setting up and maintaining a call center are high, just as building a panel is expensive. However, licensing costs of online are much more affordable than contracting out to a call center.  Online allows us to be a full service firm in house and still be small, meaning Abacus is able to control the research process from beginning to end.

Actually carrying out the online research requires somewhat less effort than live telephone, as there is no need for a paid bank of phone interviewers. More importantly, online has far more quality control, as all responses can be easily monitored as they arrive. Further, there is no need to observe or control for interviewer bias.

308: What are the challenges faced in building a representative sample?

DC: There are challenges, but not around representativeness of demographics or regional variables, but rather psychographic representation like interest in politics, political participation and engagement in public issues are likely greater for anyone who answers a survey. However, large panel management firms have a vested interest in maintaining quality panels and ensuring that samples are as representative as possible.

308: There are debates in the industry about the problems surrounding online polling not being probabilistic, despite some good performances. Why is this, or isn't it, a problem?

DC: It is a problem, but it is something that all survey firms must face. Probability issues become more significant when respondents are over-surveyed, meaning they change their behaviour or attitudes because they are surveyed often. We have an in-house policy to screen out frequent survey participants. Abacus tries to solve the problem by making the sample as representative as possible, use minimal weighting, weeding out survey frequent takers, and using high quality large panels.

The problem with probability sampling extends to telephone surveys however with large portions of the population refusing to answer surveys whether because of increased call screening or because of the refusal to respond.

The growing use of cell phone or internet based calling will continue to make telephone surveys more difficult, more expensive, and therefore less representative.

308: What role does weighting play in good polling?

DC: Although weighting plays an important role in helping us to make our samples representative of the population, our data is not heavily weighted. 

We use balanced sampling and interlocking quotas, similar to stratified sampling strategy, to ensure that the respondents captured are as representative as possible and heavy weighting is not required.

Weighting is particularly challenging for IVR polling, because certain demographics are more likely to answer the phone. Therefore, IVR surveys bias towards women and older demographics.

308: What are the challenges involved in building a representative sample of voters, rather than just of the entire population?

DC: The number one challenge is trying to predict who will actually vote, as people are less likely to admit that they do not vote, or that they don’t plan on voting. 

Further, we know that those who answer surveys are likely to be more engaged than those who don’t. 

To address these challenges, online research allows us to use varied question types and measure likelihood to vote in different ways. By being transparent with the models and try to forecast what the electorate will look like versus the general population, Abacus attempts to be as clear and accurate as possible.

In Canada, this problem is evolving. As a result of BC and Alberta, we are taking this issue seriously and will test a number of models in the next election we participate in.

308: As a relatively new polling firm, what challenges did you face in getting into the market?

DC: We face a small-c conservative industry that is adverse to change and innovation, and, quite frankly, are a little threatened by a new crop of researchers like us that are testing the established ways.

I think early on we established our credibility by demonstrating that our online research methodology could accurately forecast the 2011 federal election and 2011 Ontario provincial election. 

We, like many other pollsters, failed to really understand what was going on in Alberta, using a methodology we no longer use (IVR). In BC, our only poll was conducted before the leaders debate, so our performance there is difficult to judge.

308: How is the business of polling evolving?

DC: I foresee more smaller players emerging. 

The business of polling will be completely online, within the next 10 years nobody will answer their phone unless they know who is calling, if we are using telephones at all.

So, the industry needs to perfect and refine how we conduct internet surveys. As Google is showing us, there will be new ways to generate sample that are only emerging as alternatives now, many of which lean towards indirectly observing behavior rather than asking direct questions.

308: What has to be done to ensure that online polling can produce good results in the future?

DC: If you mean being able to predict elections, the question is predicting who is going to turn out to vote. I do think online polling is producing good results now for our clients, whether it’s testing new marketing concepts or public support for policy proposals, or the potential for new product success.