"Wonky" Calgary election polling under investigation

A self-regulating polling industry group says it will investigate what went wrong in Calgary's mayoral election after several surveys predicted the opposite of what actually happened.

During the campaign, three polls published in the city's largest newspapers indicated that Calgary Mayor Naheed Nenshi was far behind in the race and expected to lose. But when the results poured in on Oct. 16, Nehshi was re-elected with a majority of the votes.

Mainstreet Research released all three surveys that showed Nenshi was trailing in the race. It is doing an internal review of its polling practices to get to the bottom of the mysterious results, and it told National Observer this week that it recruited a high profile Canadian journalist to investigate how it shared results and responded to criticisms throughout the campaign.

Meantime, the industry group, the Market Research and Intelligence Association (MRIA), says it will investigate all of the surveys done during the municipal campaign.

"We will be conducting our own independent review of what went on in the Calgary election," Kara Mitchelmore, the chief executive officer of the industry group told National Observer Wednesday.

"The spread was so wide, admittedly people were putting out things that, if you looked at social media, had a wonky sample. And people were still publishing this. So we, as standard holders ... feel that we need to go in and take a look at all of the pollsters in the area, and what they were collecting and what they were releasing."

Mainstreet's mea culpa

The industry group, MRIA, is a not-for-profit advocacy association for pollsters with about 1,400 members. Mainstreet is no longer part of the association, following an unrelated controversy triggered by its criticism of another polling firm. But MRIA noted that its new investigation is targeting all pollsters and that Mainstreet was invited to be a part of it.

When asked if his firm would participate in the MRIA investigation, Mainstreet Research president and chief executive officer Quito Maggi said Wednesday he wasn't aware of the new probe, but welcomed it. He added that all of his firm's polling data was public, and they are conducting their own internal reviews.

"We've never gotten it wrong like this before, and I don't ever want it to happen again, ever ever ever, not just to me, but to anyone," Maggi said. "I hope that the review actually tells us what occurred, what happened, what was it (that) was so different about Calgary that made such a huge miss possible that didn't happen anywhere else."

Mainstreet's mea culpa contrasts with its position before the election, when it defended its findings and slammed its critics.

One day before the vote, Mainstreet executive vice-president David Valentin told 660 News that the critics were biased and that his firm planned on "singling people out" over their criticism of the polls.

Three Mainstreet Research polls had forecasted a huge win for mayoral candidate Bill Smith, over Nenshi, with one survey showing the challenger had a 17-point lead. Mainstreet's findings were reported by Postmedia newspapers the Calgary Herald and Calgary Sun. These numbers contrasted with survey results from other pollsters. For example, another firm, Asking Canadians, asked voters about both a new transit line in Calgary and the mayoral race, and reported that Nenshi was leading Smith by 15 points. A separate Forum poll had Nenshi ahead by 17 points.

Some analysts, including Mount Royal University political scientist Duane Bratt, questioned how the polls were conducted; on Wednesday, Bratt explained the first of Mainstreet's three polls seemed to have a gender gap and a low number of undecided voters, which didn't jive for him.

"In this case what you had was sustained criticism of the polls throughout the campaign that were ignored, or given lighter (media) coverage," Bratt said. "Two of the five polls, or two of the three companies, got the winner right."

On Oct. 16, Nenshi won the election with 51 per cent of the vote, comfortably ahead of Smith, who had 44 per cent.

Mainstreet hires Justin Ling to review polling fiasco

Maggi issued a statement on Oct. 19 saying he was shocked and embarrassed by the results, and the firm's practices would be reviewed internally. Additionally, Maggi said his firm hired former Vice reporter Justin Ling, to study the entire issue from top-to-bottom.

"Mainstreet has commissioned from him (Ling) an independent 360-degree view of the polling, the coverage, media, social media, what went wrong, what happened. Because half the problem with the polling in Calgary was the polls being wrong, but the other half was all the communications. How the media covered it, how Mainstreet responded to criticism online and in social media. Many many mistakes were made," Maggi said. "So he's going to come up with some concrete recommendations for us and some broader recommendations in terms of how polling should be covered. And we look forward to his report."

Ling, who is based in Toronto, told National Observer on Thursday he brings an outsider's perspective to the audit. He will be conducting interviews and looking at how Mainstreet conducted itself in public, including how it released and marketed its polling data, and how it responded when it became clear the data was wrong.

"It's helpful, both for them and for the media going forward, to have Mainstreet come up with a more formalized and thought-through media strategy," Ling said.

While he is not investigating media practices, he said polling methodology can be made more transparent for the public. "I think it's about talking about those things in a more intelligent way, and making clear that polling is not the voice of God. It's definitely a science, there's a math behind it, it's not pulling numbers out of the air, but it can be wrong, it's not infallible."

Ling expects to make his media recommendations to Mainstreet in November, and for his report to be public.

Postmedia announced on Oct. 21 that it is putting its relationship with Mainstreet on hold, but it was reluctant to comment on the latest developments.

"How we deal with things internally is internal, so that (a review of practices) is something that we obviously consider, and it's something that could be part of it for sure, but again if there's a position or comment on that, that would come via corporate communications," said Lorne Motley, Postmedia's vice-president of editorial for the West, who oversees the Herald and Sun.

Postmedia's vice-president of communications, Phyllise Gelfand, declined further comment.

Maggi said Mainstreet had been polling for Postmedia for nearly three years. Maggi said the newspaper company never determined the order of questions asked in the polls, and Mainstreet had final say on scripting and content. Mainstreet would deliver the poll results to the newspapers the night before so they could do interviews, and the stories would be released at the same time as the firm released all its polling data online.

Calgary-based independent pollster Janet Brown said one of her biggest concerns through the campaign was "a blatantly false narrative." She still wants to know more about Postmedia's practices and the questions they were asking about the poll results.

"We had the two largest newspapers in the city that were very focused on these horse-race polls and were very focused on this false narrative. I just think the voters deserved better," she said.

Political scientist Bratt has also said Postmedia needs to review its practices.

"I thought that they should have looked at themselves a bit more. In particular, if people are raising all of these issues about methodology from the get-go, not after the fact, but throughout the election, maybe you should have looked at that a lot closer," Bratt said, adding more attention was needed to the demographic samples.

The Calgary newspapers have experienced downsizing in recent years. Bratt noted that, to cover the city election, they pulled reporters from all corners of their newsrooms and collaborated with journalism students at Mount Royal. But Postmedia isn't the only news organization with fewer resources, something made obvious by how few polls were actually in the field through the campaign period.

"It wasn't just the polls that were done, it was the polls that weren't being done, and why weren't they being done?" Bratt said. "This is not a Postmedia story. They're the ones who got caught. This is a media story. And a polling story. And (about) the symbiotic relationship between the two."

Investigating the bigger picture

Mitchelmore said the review of Calgary election polling will be independent. While the specifics of the review have not yet been determined, they will include looking at how questions were asked, how people were sampled, whether leading questions were asked. The review panel will include academics and, she said, it will be intensive. The aim of the marketing research association's investigation is not to conduct a "witch hunt," but to share best practices for researchers and for members of the media.

"The media's really an important piece, they need to ask questions before they start reporting and releasing things. They're not asking the right questions," she said. "You should know what size was your sample, how does this look, this looks really odd, why are we doing this. That's not what's happening out there. It's people have relationships with pollsters, and they release whatever it is they put out, and they're not asking the questions about the sample. We feel in the MRIA that that's vitally important because it goes out in the media, and it's what people are making their decisions based off of."

In Calgary, Brown said it's difficult to say how polls affect voters. But, she said, "The purple army showed up in big numbers to support Nenshi. The population looked at these polls and said that doesn't match with what I'm thinking, I'm definitely going to get out there and I'm going to support Nenshi."

"Purple army" refers to Nenshi supporters. A spokesman for Nenshi, Daorcey Le Bray, declined to comment on the matter on the mayor's behalf.

Bratt notes that, while it is difficult to lock down the effect of the polls, it is clear initial results pointed to a two-candidate horse race between Nenshi and Smith, all but taking a third promising candidate out of the running.

Mitchelmore, who is from Calgary, said that the issue is important since polling can sometimes influence the way people vote. For example, she suggested that some people in the U.S. might have stayed home in the 2016 election after seeing polls suggesting that Hillary Clinton would win.

"I was in town for the (Calgary) election and was hearing everything that was going on," Mitchelmore said. "We knew that this was going to be something that we would have to address for the Canadian public. Because I've gotten numerous inquiries from my members who are in polling asking what is going on...

"We don't want the Canadian public to perceive that polling is a wasted exercise. Because it isn't, it is the voice of the people, but it has to be performed properly and it has to be reported properly."