Newspaper Watch: Lies, damned lies and polls
As the budget approaches, the campaign waged by Independent Newspapers against stamp duty intensifies. For the second week running, the Sunday Independent's lead story focused on the issue, although it was enough to glance at the headline – 'Stamp duty changes are not ruled out' – to appreciate how little information it contained. Not only was the headline a quote from an anonymous "informed source", it was formulated in such a way as to be quite meaningless. Many more interesting things can't be ruled out between now and budget day – the implosion of the universe due to collisions in particle accelarators for example.
In the absence of any meaningful information, the story was fleshed out with the results of a "Sunday Independent phone poll", which revealed that 73 per cent of people responded "yes" when asked,"Should the government make some adjustment of stamp duty in next month's budget?"
The first big problem with these polls is that they are rarely accompanied by any indication of the methodology used. If somebody was to phone their friends and ask them a list of questions, they could, without any inaccuracy, publish the findings as the "results of a phone poll", although such results would be entirely useless in estimating public opinion. There are plenty of reasons to suspect that there may be significant sampling biases in the Sunday Independent's polls. For example, following the recent crisis over the constitutionality of statutory-rape legislation, a poll revealed that 45 per cent of people blamed the attorney general, 39 per cent blamed Michael McDowell and 16 per cent blamed the Supreme Court. The fact that the respondants had such a good understanding of constitutional law that they were all confident enough to venture an opinion and that a majority were sophisticated enough to identify the most obscure of the protagonists as the guilty party leads one to suspect the sample may not have been entirely representative, to put it mildly.
On the one recent occasion that poll results were accompanied by an indication of the methodology used, the Sunday Independent announced it had conducted "an expanded telephone poll in which 500 households were contacted at random by professional market researchers using questions prepared by the Sunday Independent editorial team". At first glance this methodology looks reasonable. However, although 500 households were contacted in this "expanded" poll, there is no indication of how many completed responses were received – which could make the actual sample far smaller than 500. Secondly, the fact that these people were contacted "at random" says nothing about how they were selected and how representative they were of the population. The phrase is general enough to cover 500 hand-picked people being contacted in random order. Thirdly, a "professional market researcher" is anybody who is paid to do market research.
Finally, one of the most difficult tasks in the design of surveys is to formulate questions in such a way as to eliminate bias and to adequately capture the complexity of the responses – there is little evidence the Sunday Independent editorial team appreciates these problems. Overall, therefore, such poll results are only useful as pretty illustrations.