Polling is an integral part of political storytelling. The news of who’s up and who’s down, who’s ahead and who’s gaining ground drives the narrative of much of our news. But what happens when polling is wrong or miscalibrated? If polls aren’t a true reflection of reality, then we’re doing a great deal of damage to public discourse when we make them the focus.
Going into the final stretch of the Conservative leadership race, things seemed to have finally settled down. Maxime Bernier was cruising to certain victory
In the last month of polling Bernier had a clear lead that averaged to nearly 10 percentage points over Andrew Scheer. In the last poll, taken less than a week before the final vote, Bernier’s lead opened up to more than 14 points.
The problem was that when all the votes were tallied through the party’s weighted ranked ballot system, and each of the dozen candidates were one-by-one eliminated, Scheer was the winner, finishing with a hair’s breadth more than 50 per cent of the votes.
As Scheer’s digital director, Stephen Taylor, put it on Twitter, “We should be losing faith in traditional polling methods. They set false narratives and bias (rather than predict) reality.”
We should be losing faith in traditional polling methods. They set false narratives and bias (rather than predict) reality.
— Stephen Taylor (@stephen_taylor) May 30, 2017
Taylor has a point, especially at the top tier of news conversations. News coverage of the race was closely tied to polling. A month-by-month analysis of the final stretch of the campaign showed the polling leaders found themselves getting more coverage.
Kevin O’Leary, for example, led the polling from January when he joined the race until he dropped out in April. From the start of the year until the end of the race, he ended up being mentioned in some 2,200 stories in print and online tracked by media-monitoring site Infomart.
Bernier’s lead wasn’t clear until O’Leary was out of the race—O’Leary would still receive plenty of coverage—but once Bernier became the clear polling front runner, he also became the beneficiary of the highest coverage. But he’d only have about 1,400 stories written mentioning him. Scheer, the eventual winner, would have only 840 written about him. The fourth-place finisher, Brad Trost? Three-hundred sixty.
Which perhaps goes some distance to explaining the Toronto Star front page splash the next day: “Andrew Who?”
(The headline was a throwback to what they ran when Joe Clark won the PC leadership in the 70s, “Joe Who?”)
But beyond that, it was fairly evident opinion polling wasn’t going to be able to adjust itself to the granular level necessary to give a solid representation of where the race was at. Trost, to pick one example, finished in fourth, on the strength of his social-conservative bona fides, well ahead of the 7th-or-so place polling put him in.
The Conservative voting system required members to vote not just for their first choice, but up to as many as 10 candidates, in their order of preference. You could vote for just one candidate, pick just three, or fill out all 10. Any space beyond the first choice was entirely optional.
To add another layer of complexity, the votes were tallied by riding and weighted so that each riding was equal. This meant votes from ridings with only a handful of party members in Quebec were given the same weight as more heavily populated ridings in suburban Toronto, and Alberta. Making things even more difficult, the party membership is privately held, so most poll respondents would have to self-identify as Conservative members.
Even with all of that making polls difficult to trust, they still went a long way to shaping coverage that in retrospect seems largely inaccurate.
But the Conservative leadership race is just one narrow example of how polling shapes public narratives in the news. It goes well beyond even shaping the news, polls can shape events. Look no further than the most recent general election in the United Kingdom.
There, British Prime Minister Theresa May called an election to get a clear mandate to finish off the country’s exit from the European Union. Polls at the time of the election call had her Conservative party on track for a historic majority, with an eye-popping double-digit lead over the opposition Labour Party.
It’s hard to imagine May making the same call if polls were tighter.
Things would not end favourably for May, though. The Conservatives would find themselves several seats short of a bare majority, and have been forced to negotiate an agreement with one of the smaller parties in order to solidify their hold on government.
What that means for the country’s negotiations for the EU exit are anyone’s guess—those talks are still ongoing—but many observers have said it’s put the country in a much weaker position from where they started before the election.
So, it’s worth asking whether the faith we put in certain polls, the time we spend discussing them, and the energy we put into dissecting them is worth it.
Historian and writer Jill Lepore took a stab at this problem a few years ago in the pages of The New Yorker. Lepore lays out the case that polling is hurting democracy because it’s seeding us with false information. What’s presented as representative fact, is based on limited surveys of a few thousand people extrapolated out and weighted just so to give just the right mix of people—black and white, rich and poor, old and young, etc.—to represent the country as a whole.
If you’re, for example, the only suburban Estonian single mother who answers a poll, your views are being extrapolated out to all suburban Estonian single mothers. (It’s a little more complex than this, but indulge us, for the sake of argument.) This skews results in a way you can’t use math to weigh your way out of. And this isn’t even getting into whether the people you’re asking questions of have actually thought about the issues enough to have a strong opinion.
Think back to the Conservative race for a moment. If you’re polling the small pool of Quebec Tories, how do you know you’ve asked the right Quebec Conservatives about their favoured candidate? It would take only a handful of calls to significantly nudge the numbers in one direction or the other. But this is the system we lean on for insight.
The consequences of relying on polling can be far-reaching and have all sorts of unintended consequences, as we’ve seen. The heavy reliance on polls has done serious reputational damage to the news media, an industry already struggling with a serious trust deficit among the public.
In an era where “fake news” has become the sort of catch-all phrase to label false information and discredit the true, it’s foolish to rely on polling to continually set the narrative. It gives the public yet another reason to distrust media at a time when trust is harder than ever to regain.
Robert Hiltz is a producer with TNKR Media.
