Martin Boon: A pollster entering uncharted waters

Dilemmas of a pollster entering uncharted waters looked at by Martin Boon

OUR poll today suggests that an independent Scotland is within touching distance - the Yes Campaign continues to win the hearts and minds of the Scottish electorate (or perhaps, the No Campaign are losing them) and the race could be viewed as neck and neck.

At first glance, we should not be surprised. Since ICM began polling for Scotland on Sunday we have seen the Yes share increase from 32% last September, to 37% in January and February, to 39% in March, to 42% this month. When we strip out the Don’t Knows, this equates to a real Yes share rising from 40% last September to 48% now. On this trend, the arrival of the first poll suggesting a Yes lead should not be too long a wait.

But let’s put to one side the jubilation or despair for a moment and think about a few things from a pollster’s perspective. Firstly, it’s only one poll, a time-honoured remembrance that we trot out when a sizable movement is observed. Of course, other polls have shown the same direction of travel and we will keenly look for the next one to come along to see if it agrees with us, but it is only a snapshot which is subject to the usual vagaries that affect every single poll.

None the less, there’s much more to the production of these polling numbers than meets the eye; many factors which are troubling to pollsters, as they can, and do, have a fundamental impact on the headline figures we produce. And we all have to make subjective decisions on how we deal with them. Chief amongst them is the rather trite observation that there’s never been an independence referendum before, and this prevents pollsters from being able to peg our data to actual, directly comparable outcomes from the past – a technique central to ICM’s success in general election polling, where we have produced the most accurate final prediction in four out of the last five occasions. That said, it didn’t stop us doing rather well in the equally unprecedented AV referendum in 2011, in which we happily stumbled on the bullseye.

Secondly, we do “weight” our data to ensure it reflects a representative sample of Scots, both in terms of demographic profile, and political balance. In the case of the latter, we tie the data to the 2011 Holyrood election result. But the net effect of both these procedures has been to lift the power of SNP voter voices in the poll – and hence it follows, as you might expect, the power of the Yes voters themselves. Allied to this is the fact that people who tend to sign up to online panels in order to take part in surveys disproportionately happen to be voters. In this poll, we found that 77% of Scots recalled voting in the 2011 Holyrood election when only 50% actually did turn out on that occasion. Given that the SNP did well in that election, it follows that having too many voters in the poll artificially inflates the Yes share. For this reason, ICM has included a new weight for turnout, which deals with non-voters in the same way we deal with voters – by weighting everyone back to their recall of their 2011 behaviour, whether they voted for one party or not at all. For our next poll, we also plan on introducing our standard likelihood to vote question as well – something that worked well in UK-wide elections, and which will benefit whichever side is better at getting its vote out.

But there’s one more material point to ponder. In other elections we have seen a so-called “Spiral of Silence” effect, where the supporters of one party don’t admit to that fact if they perceive their party to be generally unpopular. This manifests itself in people refusing to tell pollsters who they will vote for – or saying “don’t know” when asked (although many of them really do know, they’re just not telling us). This contributed to massive Labour leads over the unpopular Tories in the 1990s, and in today’s context, lower poll ratings for the Liberal Democrats. So what if people perceive that it’s now patriotic to vote Yes, making No voters reluctant to admit their views for fear of being somehow labelled unpatriotic or anti-Scotland? Ask yourself, dear potential No voter: would you tell a pollster if they phone you up or knock on your door (or even ask you to check a box on a website) that you intend to vote No? If you feel you would have a problem doing so, then us pollsters really do have a problem in accurately assessing and ultimately predicting the result accurately.

Evidence on this phenomenon is anecdotal thus far rather than measurable, and we plan on doing some of our own research to see if we can get some grip on this phenomenon. But even if we do find something tangible, there remains a chunk of voters who flatly refuse to tell us anything at all on their electoral behaviour – and it’s these (representing about 10% of the total Scottish electorate) who hold the true balance of power in this campaign, although a large chunk of them didn’t vote in 2011 and may not again.

As always there’s counter-evidence to confuse things. It’s hard to reconcile that one of the two largest groups who currently “don’t know” are 2011 SNP voters. Intuitively, one might imagine this would be the last group of people who are shy to admit where they stand on Scottish independence.

Such are the vagaries of indyref polling in Scotland, and while it’s impossible to know whether our good intentions will be rewarded with pinpoint accuracy or not, what we can safely presume is that we should all be cautious about poll results until we get that one poll that politicians’ always tell you matters most – the one in which people actually vote. «

Martin Boon is director of ICM Research