In the last of my series of articles for The Globe and Mail on political polling methodology, I look at online polling.
For this article I interviewed, among others, Darrell Bricker, the CEO of Ipsos Public Affairs. The transcript of the interview can be found below. It is a very interesting one, particularly on the topics of the business of polling and the role of the media.
In the past few weeks, I've posted the interviews with Don Mills of Corporate Research Associates and Frank Graves of EKOS. Over the next week, I'll also post the interviews I had with David Coletto of Abacus Data and Christian Bourque of Léger Marketing.
308: Recently, Ipsos-Reid moved from traditional telephone polling to use of an online panel for its political polls. Why was that decision made?
DB: We’ve been considering the move to on-line for some time. That’s because the market research industry, especially in North America, now uses almost exclusively on-line data collection methods for quantitative studies. Phone is becoming a smaller part of the mix and is usually focused on either specific audiences (B2B), or calling lists. So, the investment in research platforms is going into on-line methods, and the “research on research” that’s being done is also focusing on on-line. The clincher for us was the 2012 US Presidential election – we had an opportunity to work extensively in the on-line space for Reuters and saw how strong it was in terms of sample stability and representativeness.
308: In the past, you have criticized the amount of weighting that has to be applied to online polls. What has Ipsos-Reid done to mitigate this problem?
DB: What I've been critical of is not the amount of weighting (although less is always better), it’s been the lack of disclosure about how much weighting is being done and according to which variables. But, this doesn’t just apply to on-line, it applies to all forms of data collection. As for our experience with on-line, we don’t actually do much weighting (usually just some light demographics), and we always disclose both our weighted and unweighted data.
308: What are the advantages of conducting your polls online instead of over the telephone?
DB: The biggest advantage is coverage. Over 80% of the Canadian population is now on-line. Another advantage is that we can control our sample “input” by heavying-up on hard to reach categories – especially with the river sampling portion of our sample frame. Also, we like the fact that we can ask longer questionnaires on-line. As you know, questionnaire length isn’t a big driver of costs for on-line surveys as it is for telephone. Dual-frame telephone (that’s combo landline and cell) is cost prohibitive, and there’s no advantage in terms of sample accuracy, especially when non-responses are taken into account.
308: What are the disadvantages?
DB: The biggest disadvantage is that on-line research in politics is a relatively new. We’re still learning every day about what potential issues might exist. BC is a good example of this – although the miss in BC was more of an issue with predicting differential party turnout than it was about a specific methodology or under-representing a specific group in the sampling. The way to solve these problems, in my view, is to follow good scientific practice – be your own worst critic and disclose your errors (painful as this can sometimes be) to review by your peers and other interested parties.
308: Generally speaking, how does online polling compare to other methodologies in terms of costs and effort?
DB: To do on-line well doesn’t save a lot of money. And, the amount of effort is basically the same as any other quantitative survey method.
308: Though online polls have performed well in some recent elections, for example in the 2012 US presidential vote, the methodology struggled in this year's B.C. election. Was there anything particular to this methodology that contributed to the error?
DB: The evidence shows that this is a bit of a red herring. The issue in BC was predicting which groups of the public would vote. This was a problem for ALL methodologies. The exit poll that we did on election day (which got the results very close) shows that if we had all done a better job of selecting actual voters to interview we all (regardless of methodology) would have come closer. As for on-line excluding parts of the population that don’t have access to the Internet, the truth is that these groups (usually less affluent, more transient, etc) are also among the least likely members of society to vote. For certain types of social and commercial research getting to these more marginal groups is important and using on-line to get them won’t work. But, for political research this isn’t a major issue.
308: What challenges do you face in building a representative sample of the population, considering that not everyone has access to the Internet and the potential for opt-in panels to attract a different sort of respondent?
DB: We don’t just use opt-in panels for our samples – we also use a proprietary form of river sampling that intercepts participants on the Internet regardless if they are part of an opt-in panel or not. All opt in panels have holes. They are impossible to prevent (for all of the obvious reasons). That’s why the world leaders in this space use a combination of their own and other opt in panels, and some form of river sampling. There’s a whitepaper on our website on blended sampling methods that describes what I’m talking about in detail.
308: There are debates in the industry about the problems surrounding online polling not being probabilistic, despite some good performances. Why is this, or isn't it, a problem?
DB: There are almost no probabilistic samples in any area of social science research these days. Even the ones claiming they are “probabilistic” significantly depart from the classic model and rules. In our case, we take a different approach to understanding both probability and sampling error. And, that approach borrows from the Bayesian side of statistical theory. That’s why we report a “credibility interval” instead of a margin of error with our on-line polls. There’s another whitepaper on our website that explains how to calculate a credibility interval in detail.
308: Ipsos has a long history of polling in Canada and worldwide. How has political polling changed over the years in this country?
DB: Susan Delacourt’s new book on political marketing in Canada does a great job of describing the history of political polling in our country. I’d start with that. But, the biggest change I’ve seen is the willingness of the media to publish polling without doing even the most rudimentary investigation of the pollster or their methods. Blame the lack of resources or the pressures of the 24 news cycle, but it’s led to an embarrassing environment in Canada that hurts polling, the media and our democracy. Want to fix it? The media needs to start demanding disclosure from pollsters and refusing to publish those who don’t comply.
308: Are there any differences between polling in Canada and elsewhere, both in terms of how polls are conducted and the challenges of polling in Canada?
DB: The biggest difference I see in polling around the world compared to Canada is the degree to which media in other countries both value polling and are stingy about giving it coverage. For example, major media outlets in the US like Reuters, CNN, Associated Press, and the New York Times all have polling experts on staff that strictly enforce their organization’s quality standards. They are also active players in polling – each has their own proprietary poll that they pay for and release. This used to be the case in Canada. Now, only a couple of media outlets (including our partner, CTV) do this. As a result, some so-called “pollsters” in Canada simply shop their free results around to various media outlets until they get a bite. If the free poll is “juicy” enough (never mind being accurate or conducted according to reasonable standards), it gets published. If the poll is wrong, who is to blame? For the media, they have the convenience of throwing the pollster under the bus. But, by then the media cycle has moved on and the “pollster” is already working on their next free release. It’s shameful, and Canadians deserve better.
By the way, while I’ve used the US as the point of comparison I could have easily used France, the UK, Italy, Spain, Mexico, Australia or New Zealand. Ironically, we did the polling in Nigeria a couple of years ago and even there the amount of disclosure and review we went through with our media client would put most newsrooms in Canada to shame.
308: Do you have an explanation as to why Canadian media treats polling differently from other countries? Newspapers everywhere are going through the same financial issues.
DB: It is a mystery to me. It just seems that Canadian media don't really take polling seriously anymore. I know that's not entirely true, but it does seem that way. A good example is the CBC making a virtue out of not covering polls for awhile. Instead of doing what the standard-setters do in other countries - which is to create a quality poll of record and challenge others to match, they decided to abandon the field all together. The BBC, AP, Reuters, etc all went in the other direction.
308: How has the business of polling in general changed?
DB: The research business is in major transition. It’s funny that we get caught up in conversations about data collection methods like on-line vs off-line, it’s almost a bit quaint. The truth is that the marketplace has already decided much of this – and on-line is winning in all markets where it’s feasible. The people who used to set the standards for what is acceptable in research, mainly governments and academics, are being supplanted by global corporations like P&G, Unilever and Coca Cola. Outside of the US government and the EU, they are the biggest buyers of research in the world, and they are the ones setting the standards. And, the new standard is methodologically ecumenical. It’s increasingly about creating global averages, speed and direction. Whatever gets you a quick, usable answer be it on-line surveys, social listening, qualitative research, ethnography, passive data collection, Census data, that’s what will be used.
Apart from how global packaged goods companies are redefining research, the other major trend is the domination of the research industry by a few global firms. Given the capital requirements necessary to service global clients, the big players in the market (which are mostly European) are now dominating the business and their domination can only grow. Global clients increasingly want global research partners who can deliver a similar level of quality in all markets. To do this, the major global players are acquiring companies in all the markets that matter. Yes, there will always be important boutiques in all markets, but their competitors will increasingly be the global players. And, the global players are smart, well financed and tough.
308: What changes, if any, need to be made to ensure that online polling produces good results in the future?
DB: On-line already produces terrific polling, so we’re not talking about a fundamentally flawed methodology. But, where things are moving is away from on-line surveys conducted via single opt-in panels. Increasingly, we’ll be seeing more blended samples that select people from wherever they can be found on the Internet. That’s where the big players are all headed. But, in all seriousness, if the competition to on-line is IVR (robocalls), I already know how this battle ends. To directly answer your question though, making on-line surveys better is no different from making any other survey better – we need to satisfy the primary rules of validity and reliability.